Search

Background: The paper discusses how the design of summative assessments of practical science can be enhanced through the use of robust theoretical models that capture the diversity of scientific methods. The developments in assessment policy in England and the role of practical science in teaching and learning of science are reviewed. Purpose: The paper has two main purposes: (a) to explore how summative assessments of practical science can potentially reflect the breadth of scientific methods; and (b) to investigate how Year 10 students in England engage with and perceive summative assessments designed using Brandon's Matrix, a framework clarifying the diversity of scientific methods. Sample: The paper draws on data from Year 10 (14-15 years old) students with a sample of physics assessment tasks developed using Brandon's Matrix. Design and methods: The study focused on the development of a series of assessment questions based on Brandon's Matrix, a framework by a philosopher of science, who classifies scientific methods in a taxonomy. Students' performance on the assessments as well as their perceptions about practical science are investigated. Conclusion: The results suggest that students held a very narrow understanding of what a scientific method is and often linked it to hypothesis testing, variable manipulation and measurement of several variables. Furthermore, they reported not to enjoy drawing conclusions from data.

In large-scale educational assessments, it is generally required that tests are composed of items that function invariantly across the groups to be compared. Despite efforts to ensure invariance in the item construction phase, for a range of reasons (including the security of items) it is often necessary to account for differential item functioning (DIF) of items "post hoc." This typically requires a choice among retaining an item as it is despite its DIF, deleting the item, or resolving (splitting) an item by creating a distinct item for each group. These options involve a trade-off between model fit and the invariance of item parameters, and each option could be valid depending on whether or not the source of DIF is relevant or irrelevant to the variable being assessed. We argue that making a choice requires a careful analysis of statistical DIF and its substantive source. We illustrate our argument by analyzing PISA 2006 science data of three countries (UK, France and Jordan) using the Rasch model, which was the model used for the analyses of all PISA 2006 data. We identify items with real DIF across countries and examine the implications for model fit, invariance, and the validity of cross-country comparisons when these items are either eliminated, resolved or retained.

High stakes examinations can have profound implications for how science is taught and learned. Limitations of school science such as the 'cookbook problem' can potentially be addressed if high stakes assessments target learning outcomes that are innovative. For example, less mindless procedural engagement and more thoughtful consideration of practical science can potentially improve science learning. In this paper, we investigate how practical work is represented in the assessment frameworks of several countries that demonstrate above average performance in the latest PISA science assessments. The main motivation is the need to understand if there are aspects of high stakes summative assessments in these countries that can provide insight into how best to structure national examinations. Assessment documents from a set of selected countries have been analysed qualitatively guided by questions such as 'what is the construct of practical science' and 'what is the format of assessment?' The examined jurisdictions used different approaches from traditional external pen-and-paper tests to internal teacher assessments that included different formats (e.g. laboratory report). Innovative approaches to the assessment of practical skills (e.g. PISA computer-based tasks) do not seem to be represented in these high-stakes assessments. Implications for innovative assessments for high-stakes purposes are discussed.

International large-scale assessments are on the rise, with the Programme for International Student Assessment (PISA) seen by many as having strategic prominence in education policy debates. The present article reviews PISA-related English-language peer-reviewed articles from the programme's first cycle in 2000 to its most current in 2015. Five literature bases were searched, and results were analysed with SPSS. Results map the frequency of publications according to journal, country, and scientific discipline. They also summarise major themes within three identified categories: secondary analysis, policy impact, and critiques. Findings indicated that studies based on the PISA dataset has led to progress in educational research while simultaneously pointing to the need for caution when using this research to inform educational policy.

Predicting item difficulty is highly important in education for both teachers and item writers. Despite identifying a large number of explanatory variables, predicting item difficulty remains a challenge in educational assessment with empirical attempts rarely exceeding 25% of variance explained. This paper analyses 216 science items of key stage 2 tests which are national sampling assessments administered to 11 year olds in England. Potential predictors (topic, subtopic, concept, question type, nature of stimulus, depth of knowledge and linguistic variables) were considered in the analysis. Coding frameworks employed in similar studies were adapted and employed by two coders to independently rate items. Linguistic demands were gauged using a computational linguistic facility. The stepwise regression models predicted 23% of the variance with extended constructed questions and photos being the main predictors of item difficulty. While a substantial part of unexplained variance could be attributed to the unpredictable interaction of variables, we argue that progress in this area requires improvement in the theories and the methods employed. Future research needs to be centred on improving coding frameworks as well as developing systematic training protocols for coders. These technical advances would pave the way to improved task design and reduced development costs of assessments.

We investigate the extent to which language versions (English, French and Arabic) of the same science test are comparable in terms of item difficulty and demands. We argue that language is an inextricable part of the scientific literacy construct, be it intended or not by the examiner. This argument has considerable implications on methodologies used to address the equivalence of multiple language versions of the same assessment, including in the context of international assessment where cross-cultural fairness is a concern. We also argue that none of the available statistical or qualitative techniques are capable of teasing out the language variable and neutralising its potential effects on item difficulty and demands. Exploring the use of automated text analysis tools at the quality control stage may be successful in addressing some of these challenges.

The Lebanese Brevet Professionnel (BP) is an occupationally-specific vocational qualification at lower secondary level. Despite being on the margins of Lebanese education, the BP has been showing signs of a resurgence over the past few years. This paper discusses the structure and role of the BP in the context of the Lebanese education system and presents the results of a survey of BP students. The BP appears to fulfill important functions for socio-economically identifiable sections of Lebanese society both through the direct connection which it forges with the employment market and through its bridging function to Baccalaureat-level technical-vocational programmes. The study is discussed in the context of the challenges facing MENA (Middle East and North Africa) educational development. (Contains 1 table and 1 note.)

Vocational and technical education (VTE) has traditionally been stigmatised in Arab societies owing to the lowly image of manual work and the perception of VTE as a last resort for weak school students. This paper focuses on the transition to upper secondary VTE from the regular Lebanese school system and on students' aspirations once they have done so. While regular school students' career aspirations largely reflected the conventional academic orientation of the school system and few contemplated moving into the VTE realm even if they failed, students who had made the transition expressed well-informed positive views regarding VTE, mostly concerning its direct linkages with career formation and the employment market. Despite the recognised "job ticket" value of the Baccalaureat Technique, most envisaged pursuing a higher VTE qualification, and the aspirations of those intending to complete the Licence Technique were comparable to those of terminating Year 12 students in the regular academic system who were contemplating university. The paper ends by addressing specific problem areas in Lebanese upper secondary VTE. (Contains 3 tables.)

The purpose of this study was to gauge the profile of evolutionary theory in Lebanese undergraduate biology programs. The research focused mainly on the views of university biology department heads, given that they are the people who exercise the most direct influence over their departments' ethos. An Australasian sample was chosen as a reference group. Results showed that two groups of academics who participated in the study differed markedly with respect to some issues raised but displayed a convergence of views with respect to others. The Lebanese sample was ideologically more heterogeneous than the Australasian, spanning the full range from literalist creationism to the explicit acceptance of materialistic evolution. The role of evolution in biology education was, however, a converging issue. On the whole, the Lebanese participants in the study clearly regarded the current situation with regard to evolution education as being unsatisfactory, and they were responsive to suggestions made in informal postinterview discussions that steps be taken to improve on it. (Contains 1 table.)