Search

Introduction: The aim of this systematic review was to compare the effectiveness of electronic recording with paper-based recording of clinical skills assessments for entry-level health professional students. Methods: A comprehensive database search was undertaken using AMED, CINAHL, CENTRAL, EMBASE, Medline, PsycINFO, Scopus, and Web of Science on 12–16 June 2017, and updated 9 April 2018. Studies investigating electronic and paper-based recordings of clinical skills assessments of students in entry-level health professional programs were eligible for inclusion. Two independent researchers completed screening of studies for inclusion, quality assessments, and data extraction, with discrepancies resolved by consensus. Quality assessment was performed using the Critical Appraisal Skills Program Diagnostic checklist. Results: From 2264 studies identified, five observational cohort studies were included. Published between 2006 and 2016, included studies investigated electronic and paper examinations of clinical skills assessments of students from medical, dentistry, and physical therapy programs. Electronic assessments were reported to be more time efficient than paper-based assessments with an added advantage of no missing data. Quality and quantity of quantitative and qualitative student feedback increased with electronic assessment compared to paper-based assessments. Conclusions: Electronic assessments were used successfully across a range of health professional programs, take significantly less time to complete and provide higher quality feedback to students. Future studies are needed with more robust psychometric testing and cost-effective analysis to inform the increasing uptake of electronic assessment tools in health professional training. [ABSTRACT FROM AUTHOR]

This study investigated the performance of four widely used data-collection designs in detecting test-mode effects (i.e., computer-based versus paper-based testing). The experimental conditions included four data-collection designs, two test-administration modes, and the availability of an anchor assessment. The test-level and item-level results were analyzed with inferential statistics and multidimensional scaling, respectively. The test-level results supported the superiority of the single-group counterbalanced design and the random-groups design over the single-group design without counterbalancing and the anchor test design in the recovery of the actual test-mode effects. Analysis at the item level revealed the presence of a two-dimensional solution, with the data-collection design contributing significant variability over and beyond the test mode. [ABSTRACT FROM AUTHOR]

There are sound educational and examining reasons for the use of coursework assessment and practical assessment of student work by teachers in schools for purposes of reporting examination grades. Coursework and practical work test a range of different curriculum goals to final papers and increase the validity and reliability of the result. However, the use of coursework and practical work in tests and examinations has been a matter of constant political as well as educational debate in England over the last 30 years. The paper reviews these debates and developments and argues that as accountability pressures increase, the evidence base for published results is becoming narrower and less valid as the system moves back to wholly end-of-course testing. [ABSTRACT FROM AUTHOR]

The terminal examination of post-primary education in Ireland, the Leaving Certificate, is often criticised for the reliance on memory recall over higher order thinking skills in the assessment process. In order to examine the evidence base for these critiques, this article presents an empirical investigation of the intellectual skills and knowledge domains implicit in the tasks in the written examination papers of 23 subjects in the Leaving Certificate in Ireland from 2005 to 2010. Data were collected from two sources: examination papers and student interviews. In an in-depth document analysis of the examination papers, 14,910 occurrences of command verbs were coded for the intellectual skill and knowledge domains required by the assessment task. As the same verb can require different intellectual skills in different subjects and in different tasks, each occurrence of every verb was assigned a specific value depending on its context. The article presents the frequencies and distributions of intellectual skills and knowledge domains within and across subjects. In light of key points in the literature search, the findings indicate concern regarding the level of challenge and stimulation for the development of students of the Leaving Certificate. [ABSTRACT FROM AUTHOR]

Construct: In this study, we examine the differences in test performance between the paper-based and the computer-based version of the Berlin formative Progress Test. In this context it is the first study that allows controlling for students’ prior performance.Background: Computer-based tests make possible a more efficient examination procedure for test administration and review. Although university staff will benefit largely from computer-based tests, the question arises if computer-based tests influence students’ test performance.Approach: A total of 266 German students from the 9th and 10th semester of medicine (comparable with the 4th-year North American medical school schedule) participated in the study (paper = 132, computer = 134). The allocation of the test format was conducted as a randomized matched-pair design in which students were first sorted according to their prior test results. The organizational procedure, the examination conditions, the room, and seating arrangements, as well as the order of questions and answers, were identical in both groups.Results: The sociodemographic variables and pretest scores of both groups were comparable. The test results from the paper and computer versions did not differ. The groups remained within the allotted time, but students using the computer version (particularly the high performers) needed significantly less time to complete the test. In addition, we found significant differences in guessing behavior. Low performers using the computer version guess significantly more than low-performing students in the paper-pencil version.Conclusions: Participants in computer-based tests are not at a disadvantage in terms of their test results. The computer-based test required less processing time. The reason for the longer processing time when using the paper-pencil version might be due to the time needed to write the answer down, controlling for transferring the answer correctly. It is still not known why students using the computer version (particularly low-performing students) guess at a higher rate. Further studies are necessary to understand this finding. [ABSTRACT FROM AUTHOR]

It is widely recognised that high-stakes assessment can significantly influence what is taught in the classroom. Many argue that high-stakes assessment results in a narrowed curriculum where students learn by rote rather than developing higher cognitive skills. This paper describes a study investigating the various cognitive objectives present from Bloom’s Taxonomy Educational Objectives on the Leaving Certificate biology examination. The study analysed examination papers from the past and current biology syllabuses. Analysis was also carried out to determine the marks being awarded to the different cognitive objectives. The findings show that the examination predominately includes questions that do not promote higher levels of thinking. The majority of the marks on the paper were allocated to the lower objectives of the taxonomy, suggesting students can rely on rote learning to succeed when undertaking the biology examination. This study strongly highlights how high-stake examinations have a narrow scope in terms of student achievement and shows how current biology examination procedures promote low-level learning. This low level of thinking promotes rote learning and regurgitation of facts, requiring little to no understanding of the topics. To prepare students for the working world, there needs to be a shift from only terminal exams to a mixed approach. [ABSTRACT FROM PUBLISHER]

The relationship of expected and actual difficulty of items on six mathematics question papers designed for 16-year olds in England was investigated through paired comparison using experts and testing with students. A variant of the Rasch model was applied to the comparison data to establish a scale of expected difficulty. In testing, the papers were taken by 2933 students using an equivalent-groups design, allowing the actual difficulty of the items to be placed on the same measurement scale. It was found that the expected difficulty derived using the comparative judgement approach and the actual difficulty derived from the test data was reasonably strongly correlated. This suggests that comparative judgement may be an effective way to investigate the comparability of difficulty of examinations. The approach could potentially be used as a proxy for pretesting high-stakes tests in situations where pretesting is not feasible due to reasons of security or other risks. [ABSTRACT FROM AUTHOR]

The use of computer-based tests (CBTs) has spread rapidly in recent years, as such tests offer real-time scoring and immediate feedback, facilitate the use of individualised testing methods, improve test administration and reduce test expenses. Thus, most previous studies have tended to focus on the technical advantages of CBTs and on implementation issues. However, objections to the use of CBTs have begun to surface, and the primary concern is whether the scores of CBTs and those of paper-based tests (PBTs) are equivalent. The aim of this article is to compare the scores of Korean students on computer-based and paper-based versions of the same test. We focus on the differences between the scores of male and female participants and between scores on tests examining different subject matter. Surprisingly, even though the Korean students who participated in this study had more exposure to advanced information technologies such as computers, the Internet and multimedia than did students in other countries, they did not achieve higher CBT scores than PBT scores. This finding shows that familiarity with information technology and adaptation to CBTs are distinct. We also identified a fundamental reason for low CBT scores. [ABSTRACT FROM PUBLISHER]

As part of GCSE (General Certificate of Secondary Education) reforms in England, requirements for assessing application in science increased. Setting examination questions in context facilitates testing application as students need to apply what they know and understand to a particular situation. This research explored the nature of the contexts used in reformed GCSE combined science examinations and compared contexts used in a specification which specifically emphasises contextualised learning to those in other specifications. Eight combined science specimen examination papers were selected, including four from GCSE Twenty First Century Science (21C). A qualitative coding frame was used to code each contextualised item. Various strategies for testing in context were present. Contextual features that might risk introducing construct-irrelevant variance were infrequent but may suggest areas for attention in setter training. 21C papers included a higher proportion of items with detailed contexts and a higher proportion of items set in science-related adult/professional settings. [ABSTRACT FROM AUTHOR]

Background: Medical workplace learning consists largely of individual activities, since workplace settings do not lend themselves readily to group learning. An electronic Learning Management with System Computer-Supported Collaborative Learning (CSCL) could enable learners at different workplace locations to discuss personal clinical experiences at a distance to enhance learning. Aim: To explore whether CSCL-enabled structured asynchronous discussions on an authentic task has additional value for learning in the medical workplace. Methods: Between January 2008 and June 2010, we conducted an exploratory evaluation study among senior medical students that were engaged in clinical electives. Students wrote a Critical Appraisal of a Topic paper about a clinical problem they had encountered and discussed it in discipline homogeneous subgroups on an asynchronous forum in a CSCL environment. A mixed method design was used to explore students'' perceptions of the CSCL arrangement with respect to their preparation and participation, the design and knowledge gains. We analysed the messages recorded during the discussions to investigate which types of interactions occurred. Results: Students perceived knowledge improvement of their papers. The discussions were mostly task-focused. The students considered an instruction session and a manual necessary to prepare for CSCL. A high amount of sent messages and a high activity in discussion seem to influence scores on perceptions: ''participation'' and ''knowledge gain'' positively. Conclusion: CSCL appears to offer a suitable environment for peers to provide formative feedback on a Critical Appraisal of a Topic paper during workplace learning. The CSCL environment enabled students to collaborate in asynchronous discussions, which positively influenced their learning. [ABSTRACT FROM AUTHOR]

This paper describes a study comparing paper-based marking (PBM) and onscreen marking (OSM) in Hong Kong utilising English language essay scripts drawn from the live 2007 Hong Kong Certificate of Education Examination (HKCEE) Year 11 English Language Writing Paper. In the study, 30 raters from the 2007 HKCEE Writing Paper marked on paper 100 scripts that they had marked on screen 9 months previously. Subsequent to the rating, they completed a questionnaire, from which it was concluded that, technologically, raters had no problems with OSM. Statistical results indicated no bias favouring either form of rating. Attitudinal differences surfaced, however, between new raters who had solely rated on screen as against veteran raters, whose previous rating experiences had only been with paper-based rating. The paper concludes with the prediction that as OSM becomes more prevalent, it will be accepted as the norm, with prejudice about fairness or convenience gradually fading. [ABSTRACT FROM AUTHOR]

There is a lack of empirical research in the use of computers in assessment. A study was conducted to compare student performance in computerized and paper-and-pen multiple-choice tests and assess the extent and effect of 'computer anxiety'. The LXR.TEST 5.1 software was used for the computerized test. A validated rating scale was used to assess computer anxiety. Most students had little computer experience other than in word processing. Only about a third of the students had moderate to mild computer anxiety. Neither computer experience nor anxiety correlated significantly with performance in the computer test. Whereas students performed significantly better in the paper test than in the computer test, there was no statistical difference in ranking in the two assessments formats. This suggests that computer-based assessment could be used with confidence for the purposes of ranking students, but care should be taken in using it for grading. [ABSTRACT FROM AUTHOR]

The author investigated the impact of assessment methods on student performance on accounting tests. Specifically, the author used analysis of variance to determine whether the use of computer-based tests instead of paper-based tests affects students' traditional test scores in accounting examinations. The author included 2 independent variables, student gender and student class, as covariates. The findings indicate that there was no significant difference in the values of the students' performance according to the 2 methods of assessment. The findings also revealed that neither student gender nor class was correlated to the test scores in either form of testing. [ABSTRACT FROM AUTHOR]

This study investigated factors related to score differences on computerized and paper-and-pencil versions of a series of primary K–3 reading tests. Factors studied included item and student characteristics. The results suggest that the score differences were more related to student than item characteristics. These student characteristics include response style variables, especially omitting, and socioeconomic status as measured by free lunch eligibility. In addition, response style and socioeconomic status appear to be relatively independent factors in the score differences. Variables studied but not found to be related to the format score differences included association of items with a reading passage, item difficulty, and teacher versus computer administration of items. However, because this study is the 1st to study the factors behind these score differences below Grade 3, and because a number of states are increasing computer testing at the primary grades, additional studies are needed to verify the importance of these 2 factors. [ABSTRACT FROM AUTHOR]

Examination questions need to be sufficiently novel if they are to be effective as measurement instruments. Novelty, however, presupposes creativity, suggesting that question writing is, or should be, a creative process. To explore the boundaries of creativity in question writing, this study made use of two data sources: two corpora of examination questions, and interviews with professional question writers. The study identified various factors that appear to restrict question writers' freedom, exposing a tension between creativity and constraint in question writing. In addition, it illuminated the architecture of examination questions and the mechanisms via which this architecture may affect the predictability of assessment design. The paper argues that, in the context of question writing, constraints are essential as they help to enhance the quality of the assessment. However, when they exceed a certain threshold, they can undermine the purpose that rendered them necessary in the first place. [ABSTRACT FROM AUTHOR]

Computer assisted assessment is becoming more and more common through further and higher education. There is some debate about how easy it will be to migrate current assessment practice to a computer enhanced format and how items which are currently re-used for formative purposes may be adapted to be presented online. This paper proposes an evaluatory framework to assess and enhance the practicability of large-scale CAA migration for existing items and assessments. The framework can also be used as a tool for exposing compromises between delivery mechanism and validity--exposing the limits of validity of modified paper based assessments and highlighting the crucial areas for transformative assessments. [ABSTRACT FROM AUTHOR]

Reading is critical to success in college. Faculty members often decry students who come to class without reading, and unprepared for the lessons of the day. Yet, relatively little empirical research assesses how to best stimulate collegiate reading and what types of reading assessments provide the best student learning outcomes. This paper assesses two common ways of assessing reading compliance and learning—reading quizzes and Course Preparation Assignments (CPAs)—using a randomized trial in a large introductory political science course. The data show that students are more compliant with completing the reading quizzes vs. the CPAs, and that students prefer completing the reading quizzes to the CPAs. Data from the 2020 Assessing Critical Reading Techniques study demonstrate little substantive difference between the two groups on the measured learning outcomes through either the exams or the papers. These findings provide empirical support that traditional methods of reading checks or assessments can provide value to the student learning process. Implications for instructors across different course formats are discussed. [ABSTRACT FROM AUTHOR]

Describes an experiment in which term projects, a preprinted evaluation form, and generalizability theory were used to judge the reliability of student grading. The results suggest that students can be both consistent and fair in their assessments. Creates a strong case for peer review when evaluating student papers; Peer review assessment; Analysis; Additional issues; Summary and conclusions.

Proposes a grading system for composition papers. Problems faced by teachers with regard to grading composition papers; Advantages of the proposed system over other grading methods; Scales used in the proposed system.

Medical education has historically relied on high stakes knowledge tests sat in examination centres with invigilators monitoring academic malpractice. The COVID-19 pandemic has made such examination formats impossible, and medical educators have explored the use of online assessments as a potential replacement. This shift has in turn led to fears that the change in format or academic malpractice might lead to considerably higher attainment scores on online assessment with no underlying improvement in student competence. Here, we present an analysis of 8092 sittings of the Prescribing Safety Assessment (PSA), an assessment designed to test the prescribing skills of final year medical students in the UK. In-person assessments for the PSA were cancelled partway through the academic year 2020, with 6048 sittings delivered in an offline, traditionally invigilated format, and then 2044 sittings delivered in an online, webcam invigilated format. A comparison (able to detect very small effects) showed no attainment gap between online (M = 0.762, SD = 0.34) and offline (M = 0.761, SD = 0.34) performance. The finding suggests that the transition to online assessment does not affect student performance. The findings should increase confidence in the use of online testing in high-stakes assessment. [ABSTRACT FROM AUTHOR]

Errors that occasionally manifest in examination papers and other educational assessment instruments can threaten reliability and validity. For example, a multiple choice question could have two correct response options, or a geography question containing an inaccurate map could be unanswerable. In this paper we explore this oft-neglected element of fairness. We adopt the theoretical position that assessment instrument errors stem from different types of human failure, which in turn stem from system-level failure. We then explore how the term 'error' is used in conceptually different ways, potentially engendering confusion in academic discourse. We analysed error incident records from an international assessment organisation to develop a taxonomy of assessment instrument error types. Our taxonomy covers errors that manifest in assessments in all school subjects. We used it to develop an approach to process analysis for any system of checks applied during the later stages of instrument construction. This facilitates subsequent evaluation of the psychological causes of failure. The taxonomy and process analysis approach have applications in analysing types and rates of errors in different educational assessment instruments, in systemically comparing the demands and efficacies of alternative construction processes, and as structural and communication tools in qualitative research on error culture. [ABSTRACT FROM AUTHOR]

History has shown that despite favourable changes in policy and perception, undesirable or ineffective educational practices in the Caribbean have remained largely unchanged, or superficially or inadequately changed. It has also shown that high stakes summative assessment wields considerable influence on the content to which students are exposed and the teaching and assessment methodologies used in the classroom through the widespread practice of teaching to the test. It is against this background that this paper purports that the integration of sustainable development competences (SDCs) can be advanced through high stakes assessments by capitalizing on the widespread practice of teaching to the test. By focusing specifically on the school-based assessment (SBAs) component of the summative, high stakes examinations used by the Caribbean Examinations Council, the paper examines how SBAs can be optimized to measure and thereby advance the integration of SDCs. The paper opines that through its allowance for formative assessment, integrated assessment, inquiry-based projects and group work, SBAs are optimal for assessing and furthering SDC development. [ABSTRACT FROM AUTHOR]

The article presents abstracts of medical education, which include the use of games as teaching tools in surgical residency, computer-based testing of the modified essay question and the joint U.S.-German collaboration project in medical education.

The objective of this paper is to comment on the think-aloud methods presented in the three papers included in this special issue. The commentary offered stems from the author's own psychological investigations of unobservable information processes and the conditions under which the most defensible claims can be advanced. The structure of this commentary is as follows: First, the objective of think-alouds in light of test development and validation goals are considered for each of the three papers in the volume. Second, the response processes (psychological constructs) described in the three studies are assessed vis à vis think-aloud methods. Third, the methodological details that are essential to properly evaluate response processing data for educational assessment goals are elaborated. Fourth, the possible impasse of using a psychological technique to collect psychological data about non-psychological content forms the basis of the commentary's conclusion. [ABSTRACT FROM AUTHOR]

Multiple-choice questions have become ubiquitous in educational measurement because the format allows for efficient and accurate scoring. Nonetheless, there remains continued interest in constructed-response formats. This interest has driven efforts to develop computer-based scoring procedures that can accurately and efficiently score these items. Early procedures were typically based on surface features of the responses or simple matching procedures, but recent developments in natural language processing have allowed for much more sophisticated approaches. This paper reports on a state-of-the-art methodology for scoring short answer questions supported by a large language model. Responses were collected in the context of a high-stakes test for medical students. More than 35,000 responses were collected across 71 studied items. Aggregated across all responses the proportion of agreement with human scores ranged from .97 to .99 (depending on specifics such as training sample size). In addition to reporting detailed results, the paper discusses practical issues that require consideration when adopting this type of scoring system. [ABSTRACT FROM AUTHOR]

A recent revamping of a three-credit-hour course, which the library at Morehead State University has offered for more than two decades, prompted the authors to contemplate what it means to modernize information literacy instruction in a credit-bearing format. For the authors, this modernization meant both committing to asynchronous delivery and adopting the ACRL Framework as the course's learning objectives. In this paper, we discuss some concerns with the previous course, the revision process, and the major components of the modified course. Some reflections after first teaching the course and subsequent revisions are likewise discussed. [ABSTRACT FROM AUTHOR]

This paper describes the past, present, and future of medical education in Cambodia. Although doctor training began in 1902, the first medical school was not founded until 1946. Since the colonial era, the curriculum and teaching strategies have been strongly influenced by the French system, dominated by didactic lectures and the apprenticeship model. Three chronic issues have plagued medical education in the country following the Khmer Rouge regime: a shortage of doctors, poor-quality training, and lack of relevance to the current and future population needs. An increasing number of medical schools and yearly student enrollment have addressed the first issue. Today, the fundamental challenges have shifted from quantity to ensuring the quality and relevance of medical education. Competency-based medical education (CBME) has been adopted as a new curricular model to tackle the latter two issues. Active collaboration between government institutions, public universities, and development partners drives this curricular reform at the national and institutional levels. This paper further examines the challenges associated with medical education and proposes recommendations. [ABSTRACT FROM AUTHOR]

International large-scale assessments have gained much attention since the beginning of the twenty-first century, influencing education legislation in many countries. This includes Spain, where they have been used by successive governments to justify education policy change. Unfortunately, there was a problem with the PISA 2018 reading scores for this country, meaning the OECD refused to initially release the results. Therefore, in this paper we attempt to estimate the likely PISA 2018 reading scores for Spain, and for each region within. The figure finally published by the OECD for Spain – in terms of reading scores – was 476.5 points, which is between the lower and upper bound of the interval we find (475 to 483 test points in 2018). Additionally, we report some robustness checks for the OCED countries participating in PISA 2018, which show that the difference between the actual scores and the ones we found with the imputation methods are quite close. [ABSTRACT FROM AUTHOR]

This concluding essay offers a reflection on the set of the papers contained in this special issue of the Educational Assessment journal. In it the author situates formative assessment squarely in the realm of teachers' continuous professional learning and considers the essential nature of formative assessment as centering on three questions that guide the practice for both teachers and students: where are the students going (well-defined learning goals), where are they currently (evidence collected during the course of learning), and how to close the gap between these two (instructional adjustments, feedback, and student involvement)? The author returns at the end to note the importance of professional learning in the context of formative assessment. [ABSTRACT FROM AUTHOR]

Conscientious marketing faculty spend extensive hours grading student essays and projects. As instructors work on grading papers, they may wonder how effective their comments are. The author explored how students in a marketing principles class reported their use of various grading methods. Students generally preferred rubrics with ratings and accompanying comments and indicated that they used comments to understand their grades and also to improve their writing. Students appreciated feedback on what they did well on projects and how they could have improved. [ABSTRACT FROM AUTHOR]

The article focuses on the use of context in post-16 pure mathematics questions set by the Examination Boards for national assessment in Great Britain. The study uses the grounded theory approach with six items from each specification analyzing and re-analyzing for identifying categories and relationships. Base from these questions, a framework analysis with three components is proposed. The accessibility of the context to all students is the first framework component. The second component pertains to its realistic appearance in the modeling experience and in test specific mathematics. The authenticity of the context in mathematical content is the third component. Further research is conducted for the verification of the framework result.

Results of variations on a categorical standard-setting procedure for use with performance assessments with multiple performance standards are reported. Panelists were asked to make independent ordinal categorical assignments of student work. Student work came from the administration of 1 of the 1996 Grade 8 National Assessment of Educational Progress Science Booklets. This study focused on the comparability of cutscores using 2 data analysis strategies, the effects of 2 strategies for making the categorical classifications (sorting vs. direct classification), the efficacy of long and short versions of the classification scale, and the effects of discussion on the basic, proficient, and advanced cutscores. The results indicate that there were minimal differences in cutscores across the 2 data analysis strategies. The direct classification approach was more feasible than the sorting approach, at least with the design as it was implemented in this study. The short version of the classification scale yielded comparable results to the longer version and took less time (about 20%) for the panelists to complete. Discussion produced the expected effect of more agreement among panelists. These results should be interpreted with caution due to the small sample sizes. [ABSTRACT FROM AUTHOR]

Despite the rapid growth of English Medium Instruction (EMI) programmes around the globe, the language goals in such programmes have not been clearly defined, making it difficult to evaluate their success. Accordingly, our paper reports on a study that attempted to identify the language goals, operationalised as the vocabulary expected of learners, by investigating the under-explored area of EMI assessment. We sampled a corpus of 16 sets of examination papers in two subject areas, namely, Biology and Geography, spanning eight years in Hong Kong. The examination papers, analysed for their vocabulary profiles, indicated that students were expected to learn beyond the K3 level (the most frequent 3,000 words in English) and master mid-frequency words (K4−K8) as well as academic words. Our findings suggest that the less frequent words were not often shared by the two subjects. Analysing academic words also enabled us to showcase how some words were common across the two subjects while others were more subject-specific. Pedagogical implications are discussed, highlighting the need for teachers and students to attend to target language goals in EMI, and to consider the subject-specific nature of vocabulary items in EMI. [ABSTRACT FROM AUTHOR]

Looking back, I am disappointed after having served on syllabus and examination reform committees in various states and universities for over two decades, that the weight given to the reading test section, if at all, rarely exceeded about 25% of that given to the prose and poetry section in the model paper for English Language. Tell me a little more about how tests conducted by the UPSC influenced the teaching of English. i The endorsement of the suitability of the objectively marked test type for testing English by the UPSC turned out to be a significant development. It is unfortunate that despite this visible presence of a reading comprehension test in the real world and the awareness of the value of reading skills, English syllabus committees made no move to actively promote reading comprehension as a learning objective and include a reading test in the model papers they prescribed. i The most well-known English test was a component of the selection test for the Indian Administrative Service (IAS), the successor to the Indian Civil Service (ICS) of British India that was called the I steel frame i , conducted by the Union Public Service Commission (UPSC). [Extracted from the article]

This introductory essay provides a framing for and an overview of the papers contained in this special issue of the Educational Assessment journal. In it the author begins with a brief description of the history of classroom observation tools, and then articulates a rationale for observation protocols that target formative assessment. Examining similarities and differences across the protocols, the author orients the reader to the approaches to classroom observation that are the focus of the special issue. [ABSTRACT FROM AUTHOR]

In a re-marking exercise to investigate the reliability of marking in GCSE English examinations, the Principal Examiners’ marks were adopted as ‘true’ marks for comparison with the actual or ‘live’ marks that had been awarded by the appointed examiners. A variety of approaches to investigating mark/re-mark reliability collectively led to the conclusion that marking reliability was particularly strong in the Higher tier of the specification that differentiates by outcome. [ABSTRACT FROM AUTHOR]

In sub-Saharan Africa over 52 million children are living with the death of one or both parents. Drivers of this parental mortality include afflictions at levels endemic to the region, including: HIV; malaria and other parasites; lower respiratory infections; diarrhoeal illnesses; and road accidents, among others. This paper examines the impact of orphanhood on learning outcomes among girls and boys in sub-Saharan Africa, conditional on school enrolment. By analysing test scores for approximately 60,000 pupils in 12 countries, we estimate the effect on student test scores by comparing paternal, maternal, and double orphans to non-orphans in the sample, specifically for the subjects of reading, mathematics, and HIV-AIDS knowledge. No previous study has analysed how orphanhood might influence learning by using student test score data, making this paper's approach unique in the literature. This study employs two estimation techniques: Coarsened exact matching calculates the sample average treatment effect on the treated, while matching on students' family structure, household wealth, school resources, and geographic location; and double lasso (DL) regression applies applying machine-learning for variable selection with high-dimensional controls for regional and school identifiers, school location, and student age. Our results show both CEM and DL consistently report a significant negative impact of orphanhood on test scores among specific countries, especially those which faltered in addressing the HIV-AIDS crisis. [ABSTRACT FROM AUTHOR]

The purpose of this study was to evaluate whether variance due to computer-based presentation was associated with performance on a new constructed-response type—Mathematical Expression (ME)—that requires examinees to enter mathematical expressions. Participants took parallel computer-based and paper-based tests consisting of ME items, plus a test of their skill in editing and entering data using the computer interface. Analysis of variance was used to assess differences in ME performance across delivery modes, with and without controlling for edit/entry performance. Comparisons of reliability, speededness, and relations with external indicators were also conducted. Although no statistical evidence of construct-irrelevant variance was detected, some examinees reported mechanical difficulties in responding and indicated a preference for the paper-and-pencil test. [ABSTRACT FROM AUTHOR]

The OECD's Programme for International Student Assessment (PISA) has become one of the key studies for evidence-based education policymaking across the globe. PISA has however received a lot of methodological criticism, including how the test scores are created. The aim of this paper is to investigate the so-called 'conditioning model', where background variables are used to derive student achievement scores, and the impact it has upon the PISA results. This includes varying the background variables used within the conditioning model and analysing its impact upon countries relatively positions in the PISA rankings. Our key finding is that the exact specification of the conditioning model matters; cross-country comparisons of PISA scores can change depending upon the statistical methodology used. [ABSTRACT FROM AUTHOR]

Peer-led assessment (PLA) has gained increasing prominence within health professions education as an effective means of engaging learners in the process of assessment writing and practice. Involving students in various stages of the assessment lifecycle, including item writing, quality assurance, and feedback, not only facilitates the creation of high-quality item banks with minimal faculty input but also promotes the development of students' assessment literacy and fosters their growth as teachers. The advantages of involving students in the generation of assessments are evident from a pedagogical standpoint, benefiting both students and faculty. However, faculty members may face uncertainty when it comes to implementing such approaches effectively. To address this concern, this paper presents twelve tips that offer guidance on important considerations for the successful implementation of peer-led assessment schemes in the context of health professions education. [ABSTRACT FROM AUTHOR]

In vocational education and training of computer literacy as part of vocational rehabilitation, learners often work on problem-solving exercises as self-study assignments, and check if their answers are correct. Sighted learners can get information on their incorrect answers by comparing their answers with the correct answers. However, learners with visual impairments largely depend on their teachers for getting this feedback. To remove this dependence, we designed a self-checking system for learners with visual impairments to verify the correctness of their answers. In this paper, we report the results of a usability study to evaluate whether learners with visual impairments can self-check spreadsheet problem-solving exercises using our system in a teacherless environment. Usability evaluation experiment was conducted using 2 × 2 crossover design with people with visual impairments (n = 11). The participants checked their answers (detected and corrected errors) after working on problem-solving exercises in two ways: (i) manually; and (ii) using our system. The system usability was evaluated by measuring Detection-And-Correction (DAC) ratio as effectiveness, time taken and the number of steps required for DAC as efficiency, and System Usability Scale score as satisfaction. The results show that all the participants could complete the DAC task by using our system, and the time required for DAC task was significantly reduced by using our system as compared to by checking manually. Our system enables learners with visual impairments to self-check problem-solving exercises answers. However, to increase the user satisfaction, the number of required keystrokes needs to be decreased. Vocational rehabilitation for learners with visual impairments to improve their computer literacy is becoming increasingly important. Learners with visual impairments have the potential to acquire computer literacy in a teacherless environment by using simple assistive software like our self-checking system. Simple assistive software for learners with visual impairments like our self-checking system may have a positive effect not only on learners with visual impairments but also on sighted people. Moreover, our system reduces the teaching load of the teachers so that they can be more effective in helping learners with visual impairments. [ABSTRACT FROM AUTHOR]

This article discusses the particular consequences for equity issues in testing and evaluation with the inclusion of performance assessment components in public examinations. The article outlines the relative contribution that examination components make, especially teacher-assessed coursework, to gender-related differences in performance. Since the introduction of coursework in 1988 into mainstream examining in the UK, via the General Certificate of Secondary Education (GCSE), it has been generally welcomed as an effective motivator of pupils and has enhanced performance. However, since 1994, we have seen a considerable reduction in the amount of coursework available within the GCSE syllabuses. This has had major implications for how this type of assessment is seen within the examination system. There is also a widespread perception that while teacherassessed coursework generally favours pupils, it actually favours girls more than boys in terms of better (GCSE) performance. The research presented in this article investigates this issue further. Two factors are considered: The extent to which examination components in reality operate as intended and whether coursework does contribute disproportionately to girls' final grades. Both these factors have implications for the consequential and differential validity of these examinations. [ABSTRACT FROM AUTHOR]

A simultaneous equating of four new test forms to each other and to one previous form was accomplished through a complex design incorporating seven separate equating links. Each new form was linked to the reference form by four different paths, and each path produced a different score conversion. The procedure used to resolve these inconsistencies was applied separately at each score level. Considering each equating (at a given score level) as a simple additive increment and imposing constraints on those increments led to a system of seven equations in seven unknowns. The solution produced a set of adjusted increments, so that the linking of any new form to the reference form was the same by all four possible paths. [ABSTRACT FROM AUTHOR]

This study investigated the usefulness of the bifactor model in the investigation of score equivalence from computerized and paper-and-pencil formats of the same reading tests. Concerns about the equivalence of the paper-and-pencil and computerized formats were warranted because of the use of reading passages, computer unfamiliarity of primary school students, teacher versus computer administration of the tests, and slightly lower scores on the computerized format than on the paper-and-pencil format across all 4 grades. A confirmatory item factor analysis implemented through the bifactor model in TESTFACT indicated that the best-fitting model had a general factor as well as skill-group factors. This model was more consistent with the data than a model with 2 method factors, paper-and-pencil and computer administration. In addition, the general and skill factor loadings for most of the items were reasonable. Although several instances of negative loadings were found for items on the skill factors, these did not appear to have any practical importance. As a result, the bifactor model proved useful for studying paper-and-pencil and computerized score equivalence because of the reasonable results and delineation of loadings for the method and skill factors at the item level as well as for the general factor. [ABSTRACT FROM AUTHOR]

The principle that 'what you test is what you get' is widely appreciated in education, so conventional examinations--bound by the limits of pen and paper--are often criticised for preventing teachers and students from fully embracing the new opportunities that modern technology provides. In this paper we argue that the need to be fair to students regardless of their schools' technological capabilities, and the requirement to avoid sudden discontinuities so that standards may be compared, may require a transitional period when computer and paper versions of conventional external examinations run in parallel, creating little scope initially for new types of assessment. We outline some of the issues that must be considered and the difficulties that must be overcome before conventional examinations can be computerised, and describe two illustrative cases. We conclude by noting how the computerisation of conventional examinations could lead to substantial changes to the nature of external assessment. [ABSTRACT FROM AUTHOR]

Three of the papers in this issue consider college admissions testing and a fourth high-stakes testing. For a number of reasons, the vast majority of colleges and universities have now halted their use of college admission tests. [Extracted from the article]

In recent years the use of expert judgement to set and maintain examination standards has been increasingly criticised in favour of approaches based on statistical modelling. This paper reviews existing research on this controversy and attempts to unify the evidence within a framework where expertise is utilised in the form of comparative judgement. Initially, the paper introduces a mathematical model for the way in which comparative judgement may operate. Data from existing studies of comparative judgement are then used to estimate suitable parameters for this model. Having derived a working mathematical model for the operation of expert judgement, the paper will then demonstrate that this model provides results that are broadly consistent with existing research, including both studies that are critical and studies that are supportive of the use of expert judgement. The model will then be applied to examine the required scale and design of expert-driven approaches to standard maintaining in order for these to have any chance of being as reliable as the currently favoured, purely statistical approaches. The paper also discusses the minimum conditions required for this approach to designing expert-driven approaches to be appropriate. [ABSTRACT FROM AUTHOR]

The Programme for International Student Assessment (PISA) is an important cross-national study of 15-year-olds’ academic knowledge and skills. Educationalists and public policymakers eagerly await the tri-annual results, with particular interest in whether their country has moved up or slid down the international rankings, as compared to earlier rounds. In 2015 a major change was implemented in PISA, with the introduction of computer-based assessment. This has the potential to reduce comparability of PISA test scores across countries and over time. We investigate this issue using PISA 2015 field trial data for three countries: Germany, Sweden, and Ireland. We show how, if left unaccounted for, the change to computer-based testing could limit the comparability of PISA test scores. We then describe the methodology the study organisers have used to account for such mode effects. Our key conclusion is that although the adjustment made is unlikely to overcome all the potential challenges of switching to computer-based tests, it represents an improvement over the alternative of making no adjustment at all. [ABSTRACT FROM AUTHOR]

Tests, especially high-stakes ones, occupy a prominent role in education and impact on students’ personal and professional trajectory. Therefore, it is crucial that they are well understood. Enhancing understanding of the workings of tests requires a multidisciplinary approach, one that treats tests not only as assessment instruments, but also as sociolinguistic and sociopolitical products. Using data collected by means of non-participant observation, stimulated recall and one-to-one interviews with writers of GCSE and IGCSE tests, this paper attempts to provide a macro-analysis of the process of test writing. It does so by identifying the various actors and discourses that directly or indirectly take part in test writing. It uses Bakhtin’s framework of voice as a tool to facilitate this. The paper concludes by proposing a macro model of test writing, one that explains the macro process of test writing by capturing the main voices involved in it and their interrelationships. [ABSTRACT FROM AUTHOR]