Search

Disengaged responding poses a severe threat to the validity of educational large-scale assessments, because item responses from unmotivated test-takers do not reflect their actual ability. Existing identification approaches rely primarily on item response times, which bears the risk of misclassifying fast engaged or slow disengaged responses. Process data with its rich pool of additional information on the test-taking process could thus be used to improve existing identification approaches. In this study, three process data variables—text reread, item revisit, and answer change—were introduced as potential indicators of response engagement for multiple-choice items in a reading comprehension test. An extended latent class item response model for disengaged responding was developed by including the three new indicators as additional predictors of response engagement. In a sample of 1,932 German university students, the extended model indicated a better model fit than the baseline model, which included item response time as only indicator of response engagement. In the extended model, both item response time and text reread were significant predictors of response engagement. However, graphical analyses revealed no systematic differences in the item and person parameter estimation or item response classification between the models. These results suggest only a marginal improvement of the identification of disengaged responding by the new indicators. Implications of these results for future research on disengaged responding with process data are discussed.

In large-scale educational assessments, interviewers should ensure standardized settings for all participants. However, in practice many interviewers do not strictly adhere to standardized field protocols. Therefore, systematic interviewer effects for the measurement of mathematical competence were examined in a representative sample of N = 5,139 German adults. To account for interviewers working in specific geographical regions, interviewer and area effects were disentangled using cross-classified multilevel item response models. These analyses showed that interviewer behavior distorted competence measurements, whereas regional effects were negligible. On a more general note, it is demonstrated how to identify conspicuous interviewer behavior with Bayesian multilevel models.

This study examined learning strategy use in mathematics among East Asian students in East Asian educational systems. By employing latent class analysis on the Programme for International Student A...

This study aims to investigate how test scores from PIAAC (Programme for the International Assessment of Adult Competencies) can be interpreted, by comparing the PIAAC competencies literacy and numeracy to reasoning and perceptual speed. Dimensionality analyses supported, that the PIAAC competencies can be separated into a common factor overlapping with reasoning and perceptual speed, and domain-specific factors. For the common and specific factors, relations to other variables were analyzed. The nested factor for PIAAC literacy was as expected unrelated to age, positively related to learning opportunities during one’s lifetime, and positively related to literacy skill use. The nested factor for PIAAC numeracy was also as expected unrelated to age, against expectation unrelated to learning opportunities during one’s lifetime, and as expected positively related to numeracy skill use. Results support the validity of the intended test score interpretation for PIAAC literacy, while results for PIAAC numeracy were less clear.

Abstract. Response styles can influence item responses in addition to a respondent’s latent trait level. A common concern is that comparisons between individuals based on sum scores may be rendered invalid by response style effects. This paper investigates a multidimensional approach to modeling traits and response styles simultaneously. Models incorporating different response styles as well as personality traits (Big Five facets) were compared regarding model fit. Relationships between traits and response styles were investigated and different approaches to modeling extreme response style (ERS) were compared regarding their effects on trait estimates. All multidimensional models showed a better fit than the unidimensional models, indicating that response styles influenced item responses with ERS showing the largest incremental variance explanation. ERS and midpoint response style were mainly trait-independent whereas acquiescence and disacquiescence were strongly related to several personality traits. Expected a posteriori estimates of participants’ trait levels did not differ substantially between two-dimensional and unidimensional models when a set of heterogeneous items was used to model ERS. A minor adjustment of trait estimates occurred when the same items were used to model ERS and the trait, though the ERS dimension in this approach only reflected scale-specific ERS, rather than a general ERS tendency.

Land use change, natural resource use and climate change are challenging Sustainable Development issues (SDGs 13&ndash, 15). Fostering the competencies to deal with such issues is one core task for current educational endeavors. Among these competencies, decision-making competencies are central. In detail, we investigate how learners evaluate alternative decision-making options to improve existing competence models. We exemplify our competence modelling approach using the designation of a Marine Protected Area. The cross-sectional sample consists of secondary school students and student teachers (N = 760). Partial Credit modelling shows that quantitative modelling of decision-making options is a different competence dimension than perspective taking if contextualized for Sustainable Development. In quantitative modelling, mathematical modelling is used to evaluate and reflect on decision-making options. Perspective taking covers the ability to consider different normative perspectives on Sustainable Development issues. Both dimensions show plausible (latent) correlations with related constructs within the nomological net, i.e., with qualitative arguing, economic literacy, mathematical competencies, reading competencies and analytical problem solving. Furthermore, person-abilities increase with level of education for both dimensions. The identified competence dimensions quantitative modelling and perspective taking were successfully modelled and shown to be distinct, the resulting measuring instrument is reliable and valid.

Das Nationale Bildungspanel (NEPS) untersucht seit 2009 Bildungsverlaufe in Deutschland. In sechs Startkohorten werden bildungsrelevante Kompetenzen und Pradiktoren in regelmasigen Abstanden vom Neugeborenen- bis zum Erwachsenenalter erfasst. Im Rahmen der Schulkohorten (Startkohorte 3 im 5. Jahrgang; Startkohorte 4 im 9. Jahrgang) wurde eine Serie von Machbarkeitsstudien an Forderschulen mit dem Schwerpunkt Lernen implementiert. Diese hatten das Ziel, empirisch zu uberprufen, ob und wenn ja, wie Schuler*innen mit einem sonderpadagogischen Forderbedarf Lernen (SPF-L) angemessen in die langsschnittliche Untersuchung des NEPS einbezogen werden konnen, um valide und aussagekraftige Ergebnisse zu erhalten. Im Rahmen der NEPS-Erhebungen werden neben unterschiedlichen Kompetenztests (z. B. Lesen, Metakognition, Mathematik) auch schriftliche Fragebogen eingesetzt. Fur Erhebungen an Forderschulen wurden die unterschiedlichen Instrumente angepasst und in experimentellen Variationen administriert. Der vorliegende Beitrag skizziert deren Design, liefert einen Uberblick der zentralen Erkenntnisse der bisher veroffentlichten Ergebnisse und diskutiert die Implikationen fur die Einbeziehung von Schuler*innen mit SPF-L in gros angelegte Bildungsstudien.

Jurgen Rost passed away on July 1st, 2017, after a long and serious illness. For those who do not know him well, he was a professor of psychology at the Christian Albrechts University of Kiel, and research director at the institute for science education before his early retirement.Jurgen was a brilliant and passionate psychometrician. He is known internationally as a leading scholar in this area for his groundbreaking contributions to item response theory and latent class modeling. His work on the development of mixture distribution Rasch models had a lasting impact on the field. Jurgen was not only interested in developing the sophisticated theory of his models but he was also passionate about making his work, and more general, advanced psychometric modeling, accessible to applied researchers. He wrote a textbook on test theory that became the essential German language reference on item response theory. This textbook was published in a first edition in 1996, and then republished in 2004 in extended and revised form. It became the go-to reference for applied researchers and has been used by academic teachers for entry-level and advanced classes on modern test theory.In collaboration with his doctoral students he also developed user friendly computer programs, among these WINMIRA and MULTIRA, that allowed researchers to apply complex psychometric models through easy to use graphical interfaces. These software packages are central tools accompanying the textbook, and were included first on CDROM and later as a download, together with exercises and example data sets that were part of the chapters. That way readers could practice what they learned immediately using free versions of these programs, which can be used to estimate a vast majority of the models discussed in Jurgen's textbook.Aside from writing textbooks and developing teaching tools, Jurgen was also a very active book editor, often in conjunction with colleagues, and often based on groundbreaking conferences that brought together scholars from across the world who followed different lines of thought, and who would consider their own approaches different from those of others. A (mis-)conception that was often either softened or resolved after either attending one of these conferences, or after reading the proceedings that came out of these meetings. Among many of the impactful edited volumes, two are of particular and of continued interest to the world of psychometrics: "Latent Trait and Latent Class Models" edited by Rolf Langeheine and Jurgen Rost, published in 1988, and named by Neil Henry one of the most important references on latent class analysis in his 50-year review of research in this domain. This volume is recognized as a classic, and was followed by another emerging classic entitled "Applications of Latent Trait and Latent Class Models in the Social Sciences", published in 1997, and edited by Jurgen Rost, and his long-term friend and colleague Rolf Langeheine as co-editor. These two volumes brought together researchers and results that stirred developments of integrated and general families of psychometric models. …

The selection and measurement of competencies, reflecting educational effects in a lifelong learning perspective, represents a major challenge for the German National Educational Panel Study (NEPS). Data on the development of competencies serves as a central point of reference for all other parts of the study. These competencies have to be relevant not only for a successful and responsible individual life but also for a well-functioning modern democratic society. Hence, the aim is not just to describe the development of such competencies, but also to analyze relevant prerequisites, conditions, and courses of competence acquisition. The lifelong learning perspective will shed light on how different competencies are acquired over the life span, how they interact over time and across educational stages, and in which way they may contribute to individual and group-specific life-course outcomes. This chapter gives an overview on the selection, rationale, and conceptualization of competencies within NEPS.

Bestimmt erinnern Sie sich noch an Situationen in der eigenen Schulzeit, in denen Lernergebnisse beurteilt wurden. Die Mathearbeit wird benotet zuruckgegeben und Emily ist voller Erwartung. Die mundliche Note in Geschichte, die Moritz bekommen hat, war ungerecht; in der Pause wird daruber heis diskutiert. Die Lehrerin verschiebt den geplanten Test in Englisch und schiebt noch eine Ubungs- und Fragestunde ein. Ein Lehrer lobt Annes Wissen und regt an, ob sie sich nicht vorstellen konne, Biologie zu studieren. Luis bekommt in Deutsch eine Ruckmeldung zu seiner Erorterung und versteht nun auf einmal, wie er es hatte besser machen konnen.

Dieser Beitrag zeigt die Analysepotentiale des Nationalen Bildungspanels fur die Bildungsarmutsforschung anhand der Erwachsenenkohorte auf. Wir analysieren, ob die absolute Zertifikatsarmut uber die Kohorten abgenommen hat und ob es soziale Herkunftsunterschiede in der Wahrscheinlichkeit, zertifikatsarm zu sein, gibt. Des Weiteren untersuchen wir Veranderungen in den Mittelwerten der Lesekompetenz uber die Kohorten hinweg sowie wie sich die Lesekompetenzverteilungen in Bezug auf die soziale Herkunft unterscheiden. Abschliesend werden Zertifikats- und Kompetenzarmut zueinander in Beziehung gesetzt.

When questionnaire data with an ordered polytomous response format are analyzed in the framework of item response theory using the partial credit model or the generalized partial credit model, reversed thresholds may occur. This led to the discussion of whether reversed thresholds violate model assumptions and indicate disordering of the response categories. Adams, Wu, and Wilson showed that reversed thresholds are merely a consequence of low frequencies in the categories concerned and that they do not affect the order of the rating scale. This article applies an empirical approach to elucidate the topic of reversed thresholds using data from the Revised NEO Personality Inventory as well as a simulation study. It is shown that categories differentiate between participants with different trait levels despite reversed thresholds and that category disordering can be analyzed independently of the ordering of the thresholds. Furthermore, we show that reversed thresholds often only occur in subgroups of participants. Thus, researchers should think more carefully about collapsing categories due to reversed thresholds.

The occurrence of differential item functioning (DIF) for gender indicates that an instrument may not be functioning equivalently for men and women. Aside from DIF effects, item responses in personality questionnaires can also be influenced by response styles. This study analyzes the German NEO-PI-R regarding its differential item functioning for men and women while taking response styles into account. To this end, mixed Rasch models were estimated first to identify latent classes that differed in their response style. These latent classes were identified as extreme response style (ERS) and nonextreme response style (NERS). Then, DIF analyses were conducted separately for the different response styles and compared with DIF results for the complete sample. Several items especially on Neuroticism, Agreeableness, and Conscientiousness facets showed gender-DIF and thus function differentially between men and women. DIF results differed mainly in size between the complete sample and the response style subsamples, though DIF classification was overall consistent between ERS, NERS, and the complete sample.

The consistency of extreme response style (ERS) and non-extreme response style (NERS) across the latent variables assessed in an instrument is investigated. Analyses were conducted on several PISA 2006 attitude scales and the German NEO-PI-R. First, a mixed partial credit model (PCM) and a constrained mixed PCM were compared regarding model fit. If the constrained mixed PCM fit better, latent classes differed only in their response styles but not in the latent variable. For scales where this was the case, participants’ membership to NERS or ERS on each scale was entered into a latent class analysis (LCA). For both instruments, this second order LCA revealed that the response style was consistent for the majority of the participants across latent variables.

In Germany, the respondents who had participated in the 2012 survey of the Programme for the International Assessment of Adult Competencies (PIAAC) were re-approached for the panel study PIAAC-L. PIAAC-L aims at investigating the longitudinal effects of skill outcomes over the life course and the development of the key skills assessed in PIAAC. Moreover, additional and alternative background information was collected and analyzed within PIAAC-L. PIAAC-L consists of three follow-up waves to the initial PIAAC 2012 survey. The present paper describes the rationale for PIAAC-L and the benefits of conducting a longitudinal PIAAC follow-up study in Germany. In addition, we outline the general design of PIAAC-L and the specific design of the three waves of data collection. Finally, we address the analytic potential of PIAAC-L data set and its availability to the scientific community.

Naturwissenschaften sind ein Bildungsbereich des Kindergartens, gleichzeitig ist nur wenig daruber bekannt, wie das naturwissenschaftliche Lernen von Kindergartenkindern gefordert werden kann. In diesem Beitrag wird uber eine quasi-experimentelle Studie mit 245 Kindern berichtet, in der die Lerngelegenheiten „Experimente“ und „Gesprache uber naturwissenschaftlich relevante Alltagssituationen“ verglichen werden. Dazu wurden drei Experimentalgruppen, in denen die genannten Lerngelegenheiten systematisch variiert wurden, mit einer Kontroll- und einer Baselinegruppe verglichen. Kinder aus dem letzten Kindergartenjahr nahmen an drei jeweils 90-minutigen Lernangeboten zum Thema „Schmelzen und Gefrieren von Eis/Wasser“, „Verdampfen und Kondensieren von Wasser/Dampf“ und „Losen und Nicht-Losen in Wasser“ teil. In einem Pra-Post-Test-Design wurde der Lernzuwachs im Wissen ermittelt, die erhobenen Daten wurden Rasch-skaliert. Kognitive Fahigkeiten und verschiedene familiare Hintergrundvariablen wurden kontrolliert. Es lasst sich nur bei der Kombination der beiden Lerngelegenheiten ein signifikanter Vorteil im Vergleich zu der Kontrollgruppe und der Baselinegruppe nachweisen.

Fruhkindlichen Bildungsprozessen wird seit einiger Zeit grose Bedeutung beigemessen. Dabei wird auch das naturwissenschaftliche Lernen in den Blick genommen. Fur das Vorschulalter liegen bislang jedoch kaum Untersuchungen zur Naturwissenschaftlichen Kompetenz und ihrer Forderung vor. Im vorliegenden Beitrag wird die Entwicklung und Validierung eines Testinstruments zur Erfassung von Naturwissenschaftlicher Kompetenz bei funfjahrigen Kindern vorgestellt. Thematisch konzentriert sich der Test auf Wasser in unterschiedlichen Aggregatzustanden (fest – flussig – gasformig), die Ubergange zwischen den Aggregatzustanden (schmelzen/gefrieren, verdunsten/kondensieren) und das Losen von Stoffen in Wasser. Der Test wird als strukturiertes Interview administriert; es lassen sich eine hohe Reliabilitat und hohe interne Validitat feststellen. Zwischen den naturwissenschaftlichen Kompetenzen der untersuchten Kinder lassen sich deutliche Unterschiede erkennen, die vier unterschiedlichen Kompetenzstufen zugeordnet werden konnen.

The selection and measurement of competencies, reflecting educational effects in a lifelong learning perspective, represents a major challenge for the German National Educational Panel Study. Data on the development of competencies serves as a central point of reference for all other parts of the Panel Study. These competencies have to be relevant not only for a successful and responsible individual life but also for a well-functioning modern democratic society. Hence, the aim is not just to describe the development of such competencies, but also to analyze relevant prerequisites, conditions, and courses of competence acquisition. The lifelong learning perspective will shed light on how different competencies are acquired over the life span, how they interact over time and across educational stages, and in which way they may contribute to individual and group-specific life-course outcomes. This chapter gives an overview on the selection, rational, and conceptualization of competencies within the National Educational Panel Study.

The aim of the research presented here is the use of extensions of longitudinal item response theory (IRT) models in the analysis and comparison of group-specific growth in large-scale assessments of educational outcomes. A general discrete latent variable model was used to specify and compare two types of multidimensional item-response-theory (MIRT) models for longitudinal data: (a) a model that handles repeated measurements as multiple, correlated variables over time and (b) a model that assumes one common variable over time and additional variables that quantify the change. Using extensions of these MIRT models, we approach the issue of modeling and comparing group-specific growth in observed and unobserved subpopulations. The analyses presented in this paper aim at answering the question whether academic growth is homogeneous across types of schools defined by academic demands and curricular differences. In order to facilitate answering this research question, (a) a model with a single two-dimensional ability distribution was compared to (b) a model assuming multiple populations with potentially different two-dimensional ability distributions based on type of school and to (c) a model that assumes that the observations are sampled from a discrete mixture of (unobserved) populations, allowing for differences across schools with respect to mixing proportions. For this purpose, we specified a hierarchical-mixture distribution variant of the two MIRT models. The latter model, (c), is a growth-mixture MIRT model that allows for variation of the mixing proportions across clusters in a hierarchically organized sample. We applied the proposed models to the PISA-I-Plus data for assessing learning and change across multiple subpopulations. The results of this study support the hypothesis of differential growth.

The study analyzes the effects of grade repetition on science and mathematics achievement and on self-concept in mathematics using longitudinal data from a representative sample of 9th graders in Germany. Same-age comparisons were applied between three groups: (a) the retained students, (b) a matched group of promoted students, and (c) the entire group of promoted students. The results show that the retained students have a higher self-concept in mathematics after one year of retention than before. Summarizing these empirical findings concerning achievement, we conclude that retention in the 9th grade does not reduce the gap in academic achievement to the regular promoted students. The study analyzes the effects of grade repetition on science and mathematics achievement and on self-concept in mathematics using longitudinal data from a representative sample of 9th graders in Germany. Same-age comparisons were applied between three groups: (a) the retained students, (b) a matched group of promoted students, and (c) the entire group of promoted students. The results show that the retained students have a higher self-concept in mathematics after one year of retention than before. Summarizing these empirical findings concerning achievement, we conclude that retention in the 9th grade does not reduce the gap in academic achievement to the regular promoted students.

This study analyses the effects which repeating a class has on ninth grade students’ development of mathematical competency. The following research questions were addressed: How many students repeat grades in the different types of schools? How do students who repeat a grade differ from those who do not in their performance and background characteristics? How much extra mathematics do students repeating a grade learn in one school year? What are the differences between various types of school? Can students with poor mathematics grades in particular profit from repeating a grade? The sample is a sub-sample of the PISA-I-Plus study and comprises N = 360 ninth grade students. The total sample of PISA-I-Plus is representative for all ninth/tenth grade students from the different school types in Germany. The data survey was carried out in the ninth grade and then repeated after the students had repeated a year. The results document differences in the amount of grade repeat quotas between types of school. Furthermore, compared to students not repeating, those repeating a grade had lower mathematics (d = 1.02) and german (d = 1.14) grades, a lower level of mathematical literacy (d = 0.51), and lower test results with regard to basic cognitive abilities (d = 0.32). In terms of the development of mathematical literacy, the students repeating a grade could improve by an average of 23 points (d = 0.27) on the PISA mathematics scale. However, the results identify 38 percent of students repeating a grade who do not make any significant improvement in mathematics or even get worse. A differentiation according to school types shows that students repeating a grade in integrated comprehensive secondary schools and in schools with several educational levels in particular do not, on average, show any noteworthy improvement in their mathematical literacy. The analysis of the school grades received in mathematics shows that students whose mathematics grades are unsatisfactory do not benefit more from repeating a grade than students whose mathematics performance has been rated as being “satisfactory” or better. The article concludes with a discussion of the possible consequences of changing the way in which repetitions of grades are dealt with.

In order to precisely assess the cognitive achievement and abilities of students, different types of items are often used in competence tests. In the National Educational Panel Study (NEPS), test instruments also consist of items with different response formats, mainly simple multiple choice (MC) items in which one answer out of four is correct and complex multiple choice (CMC) items comprising several dichotomous “yes/no” subtasks. The different subtasks of CMC items are usually aggregated to a polytomous variable and analyzed via a partial credit model. When developing an appropriate scaling model for the NEPS competence tests, different questions arose concerning the response formats in the partial credit model. Two relevant issues were how the response categories of polytomous CMC variables should be scored in the scaling model and how the different item formats should be weighted. In order to examine which aggregation of item response categories and which item format weighting best models the two response formats of CMC and MC items, different procedures of aggregating response categories and weighting item formats were analyzed in the NEPS, and the appropriateness of these procedures to model the data was evaluated using certain item fit and test fit indices. Results suggest that a differentiated scoring without an aggregation of categories of CMC items best discriminates between persons. Additionally, for the NEPS competence data, an item format weighting of one point for MC items and half a point for each subtask of CMC items yields the best item fit for both MC and CMC items. In this paper, we summarize important results of the research on the implementation of different response formats conducted in the NEPS.

Assessing competencies of students with special educational needs in learning (SEN-L) poses a challenge for large-scale assessments (LSAs). For students with SEN-L, the available competence tests may fail to yield test scores of high psychometric quality, which are—at the same time—measurement invariant to test scores of general education students. We investigated whether we can identify a subgroup of students with SEN-L, for which measurement invariant competence measures of adequate psychometric quality may be obtained with tests available in LSAs. We furthermore investigated whether differences in test-taking behavior may explain dissatisfying psychometric properties and measurement non-invariance of test scores within LSAs. We relied on person fit indices and mixture distribution models to identify students with SEN-L for whom test scores with satisfactory psychometric properties and measurement invariance may be obtained. We also captured differences in test-taking behavior related to guessing and missing responses. As a result we identified a subgroup of students with SEN-L for whom competence scores of adequate psychometric quality that are measurement invariant to those of general education students were obtained. Concerning test taking behavior, there was a small number of students who unsystematically picked response options. Removing these students from the sample slightly improved item fit. Furthermore, two different patterns of missing responses were identified that explain to some extent problems in the assessments of students with SEN-L.

In Reaktion auf die Ergebnisse der TIMS-Studie wurde das BLK-Modellversuchsprogramm SINUS konzipiert und 1998 mit 180 Schulen uber funf Jahre betrieben. Im vorliegenden Beitrag berichten wir uber zentrale Ergebnisse der summativen Evaluation des Programms. Ziel des Programms war es, (1) die Professionalisierung der Lehrkrafte zu unterstutzen, (2) die Qualitat des mathematisch-naturwissenschaftlichen Unterrichts zu verbessern und (3) die Lernprozesse und Lernergebnisse der Schulerinnen und Schuler zu fordern. Zur Uberprufung der Ausgangslage und der Wirkungen des Programms diente ein Vergleich mit einer reprasentativen Schulstichprobe. Dazu wurden an den SINUS-Schulen Erhebungen mit nationalen PISA-Instrumenten durchgefuhrt. Die Ergebnisse der Abschlusserhebung im Jahr 2003 zeigen, dass SINUS im Verlauf der Programmzeit auf allen untersuchten Ebenen Wirkungen entfaltet hat. Dies betrifft die erfolgreiche Umsetzung der Projektinhalte auf Seiten der Lehrkrafte, die positive Wahrnehmung des Unterrichts auf Seiten der Schulerschaft sowie die Interessen, Haltungen und Kompetenzen der Schulerinnen und Schuler an SINUS-Schulen. Die schulartspezifischen Analysen zeigen jedoch, dass SINUS nicht in allen Schularten die gleiche Wirksamkeit erzielt hat. In erster Linie scheinen die Hauptschulen, die Schulen mit mehreren Bildungsgangen und die Integrierten Gesamtschulen profitiert zu haben.

The National Educational Panel Study (NEPS) aims at investigating the development of competencies across the whole lifespan. Competencies are assessed via tests and competence scores are estimated based on models of Item Response Theory (IRT). IRT allows a comparison of test scores—and, thus, the investigation of change across time and differences between cohorts—even when the respective competence is measured with different items. As in NEPS for most of the competencies retest effects are assumed, linking is done via additional link studies in which the tests for two age groups are administered to a separate sample of participants. However, in order to be able to link the test results of two different measurement occasions, certain assumptions, such as, that the measures are invariant across samples and that the tests measure the same construct, need to hold. These are challenging assumptions regarding the linking of competencies across the whole lifespan. Before linking reading tests in NEPS for different age cohorts in secondary school as well as in adulthood, we, thus, investigated unidimensionality of the items for different cohorts as well as measurement invariance across samples. Our results show that the tests for different age groups do measure a unidimensional construct within the same sample. However, measurement invariance of the same test across different samples does not hold for all age groups. Thus, the same test exhibits a different measurement model in different samples. Based on our results, linking may well be justified within secondary school, while linking test scores in secondary school with those in adult age is threatened by differences in the measurement model. Possible reasons for these results are discussed and implications for the design of longitudinal studies as well as for possible analyses strategies are drawn.

A multidimensional Rasch model is presented here, which is a multidimensional extension of item component models. Relations to existing multidimensional item response theory models are discussed. Apart from other applications, it is also suitable for analyzing tests and questionnaires, which are designed according to two or more facets. An application to a 77-item questionnaire on students’ interest in physics with a two-facet structure demonstrates that the model parameters can even be estimated when 17 latent dimensions are to be measured simultaneously (by means of joint maximum likelihood methods).

Zusammenfassung. Es werden die Originaldaten der Entwicklung der deutschen Übersetzung des NEO-FFI-Inventars mit Modellen der Item-Response-Theorie reanalysiert. Dabei wird der Frage nachgegangen, inwieweit die 5 Stufen des Antwortformates eine Ordinalskala bilden, die mit dem gemessenen Trait korrespondiert. Es wird geprüft, inwieweit das eindimensionale ordinale Rasch-Modell für jede der 5 Skalen angemessen ist, und es wird mit dem mixed Rasch-Modell Personenheterogenität im Sinne einer zwei-Klassen-Struktur nachgewiesen, die jedoch allein auf unterschiedliche response sets der befragten Personen zurückzuführen ist. Lediglich die Extraversionsskala erweist sich als substantiell heterogen. Die Verzerrung der Personenmeßwerte infolge unterschiedlicher response sets wird durch die Anwendung des mixed Rasch-Modells gleichsam automatisch korrigiert.

When competence tests are administered, subjects frequently omit items. These missing responses pose a threat to correctly estimating the proficiency level. Newer model-based approaches aim to take nonignorable missing data processes into account by incorporating a latent missing propensity into the measurement model. Two assumptions are typically made when using these models: (1) The missing propensity is unidimensional and (2) the missing propensity and the ability are bivariate normally distributed. These assumptions may, however, be violated in real data sets and could, thus, pose a threat to the validity of this approach. The present study focuses on modeling competencies in various domains, using data from a school sample ( N = 15,396) and an adult sample ( N = 7,256) from the National Educational Panel Study. Our interest was to investigate whether violations of unidimensionality and the normal distribution assumption severely affect the performance of the model-based approach in terms of differences in ability estimates. We propose a model with a competence dimension, a unidimensional missing propensity and a distributional assumption more flexible than a multivariate normal. Using this model for ability estimation results in different ability estimates compared with a model ignoring missing responses. Implications for ability estimation in large-scale assessments are discussed.

Standard unidimensional Rasch models assume that persons with the same ability parameters are comparable. That is, the same interpretation applies to persons with identical ability estimates as regards the underlying mental processes triggered by the test. However, research in cognitive psychology shows that persons at the same trait level may employ different strategies to arrive at the solutions. This is a major threat to the construct validity of a test since the construct representation of the test changes for different classes of respondents. In this study a reading comprehension test composed of 20 multiple-choice items is analysed with mixed Rasch model. Findings show that a two-class model fits the data best. After investigating class specific item profiles the implications of the study for test validation along with the contribution of the research to our understanding of reading processes are discussed. Accessed 7,812 times on https://pareonline.net from March 05, 2013 to December 31, 2019. For downloads from January 1, 2020 forward, please click on the PlumX Metrics link to the right.

Since the publication of the PISA 2006 study results the question of reporting trends over the PISA cycles has received a lot of interest. This chapter discusses the possibilities and limitations of trend analyses based on data from this international comparative study and using complex test designs. The chapter succeeds trend analyses which were carried out with the German data from the first three PISA studies in 2000, 2003 and 2006 (Carstensen CH, Prenzel M, Baumert J, Trendanalysen in PISA: Wie haben sich die Kompetenzen in Deutschland zwischen PISA 2000 und PISA 2006 entwickelt? [Trend analyses in PISA: how did competencies in Germany develop between PISA 2000 and PISA 2006?] Zeitschrift fur Erziehungswissenschaften, Sonderheft 10:11–34, 2008; Prenzel M, Artelt C, Baumert J, Blum W, Hammann M, Klieme E et al (eds), PISA 2006. Die Ergebnisse der dritten internationalen Vergleichsstudie [PISA 2006. Results of the third international comparison]. Waxmann, Munster, 2007).

International audience; This paper focuses interest in science as one of the attitudinal aspects of scientific literacy. Large-scale data from the Programme for International Student Assessment (PISA) 2006 are analysed in order to describe student interest more precisely. So far the analyses have provided a general indicator of interest, aggregated over all contexts and contents in the science test (Organisation for Economic Co-operation and Development [OECD], 2007a; Prenzel, Schütte, & Walter, 2007).With its innovative approach PISA embeds interest items within the cognitive test unit and its contents and contexts. The main difference from conventional interest measures is that in most questionnaires, a relatively small number of interest items cover broad fields of contents and contexts. The science units represent a number of systematically differentiated scientific contexts and contents. The units' stimulus texts allow for concrete descriptions of relevant content aspects, applications and contexts. In the analyses, multidimensional item response models are applied in order to disentangle student interest. The results indicate that multidimensional models fit the data. A two-dimensional model separating interest into two different knowledge of science dimensions described in the PISA science framework is further analysed with respect to gender, performance differences and country. The findings give a comprehensive description of students' interest in science. The paper deals with methodological problems and describes requirements of the test construction for further assessments. The results are discussed with regard to their significance for science education.

Die Veroffentlichungen von Ergebnissen der letzten PISA-Erhebung geben Anlass, Moglichkeiten und Grenzen von Trendanalysen in internationalen Vergleichsstudien mit komplexen Testdesigns zu untersuchen. Der Beitrag schliest an Trendanalysen an, die in Deutschland bei PISA 2006 durchgefuhrt wurden. In einem ersten Schritt werden Voraussetzungen fur Trendanalysen bei Studien wie PISA benannt, um dann anhand der Daten aus den drei Erhebungen in Deutschland zu prufen, ob sie erfullt sind. Im Zentrum des Beitrags steht die Diskussion eines Modells fur Trendanalysen innerhalb eines Staates, das auch dann angewendet werden kann, wenn bestimmte Voraussetzungen fur Trendanalysen mit den internationalen Kompetenzindikatoren nicht gegeben sind. Die Ergebnisse einer Anwendung dieses Modells auf die deutschen Daten fur alle drei Kompetenzbereiche (Lesen, Mathematik, Naturwissenschaften) werden vorgestellt und interpretiert. Dabei werden die Einschrankungen bezuglich einer international vergleichenden Interpretation hervorgehoben. Im Ausblick werden weiterfuhrende Moglichkeiten diskutiert, in einer Studie wie PISA mit denselben Kompetenzindikatoren sowohl querschnittliche Vergleiche als auch tragfahige Trendanalysen durchfuhren zu konnen.

In this paper, we use data from the German PISA 2003 sample to study the effects of central exit examinations on student performance, student attitudes, and teacher behavior. Unlike earlier studies we use (i) a value-added measure to pin down the effect of central exit exams on learning in the last year before the exam and (ii) separate test scores for mathematical literacy and curriculum-based knowledge. The findings indicate that central exit exams only improve curriculum-based knowledge but do not affect mathematical literacy. Moreover, teachers in German states with central exit examinations are more active and tend to be more performance oriented. Students, although showing a better performance, are less motivated in school.

The Berkeley Evaluation and Assessment Research (BEAR) Center has for the last several years been involved in the development of an assessment system, which we call the BEAR Assessment System. The system consists of four principles, each associated with a practical “building block” [Wilson 2005] as well as an activity that helps integrate the four parts together (see the section starting on p. 325). Its original deployment was as a curriculum-embedded system in science [Wilson et al. 2000], but it has clear and logical extensions to other contexts such as in higher education [Wilson and Scalise 2006], in largescale assessment [Wilson 2005]; and in disciplinary areas, such as chemistry [Claesgens et al. 2002], and the focus of this chapter, mathematics. In this paper, the four principles of the BEAR Assessment System are discussed, and their application to large-scale assessment is described using an example based on a German assessment of mathematical literacy used in conjunction with the Program for the International Student Assessment [PISA 2005a]; see also Chapter 7, this volume). The BEAR Assessment System is based on a conception of a tight inter-relationship between classroom-level and large-scale assessment [Wilson 2004a; Wilson and Draney 2004]. Hence, in the process of discussing this large-scale application, some arguments and examples will be directed towards classroom-level applications, or, more accurately, towards the common framework that binds the two together [Wilson 2004b].

In large-scale educational assessments, such as the Programme for International Student Assessment (PISA) and the Trends in Mathematics and Science Study (TIMSS), a primary concern is with the estimation of the populationlevel characteristics of a number of latent variables and the relationships between latent variables and other variables. Typically these studies are undertaken in contexts in which there are constraints on sample size and individual student response time, yet there are high expectations with regard to the breadth of content coverage. These demands and constraints have resulted in such studies using rotated-booklet designs, with each student responding to a limited number of items on each of a number of scales. This paper describes the techniques that have been employed in such studies to enable the reliable estimation of population characteristics when there is considerable unreliability at the student level. It also discusses the methodology that is used to make the data sets produced in such studies amenable for use by data analysts undertaking secondary analyses using standard analytic tools.

The present volume is a collection of chapters on research and development work on extensions of the Rasch model (RM; Rasch, 1960) that have focused on relaxing some fundamental constraints of the original RM, while preserving many of the unique features of the model. More specifically, the volume presents extensions of the RM in which certain homogeneity assumptions on the item level and the population level have been relaxed. With these two types of assumption intact, the original RM decomposes the probability of item responses in two independent components: an item-specific difficulty parameter that is constant across all examinees in the population, and one ability parameter for each examinee that is the same across all items in a given assessment.