Search

With the increasing demand for precise test feedback, cognitive diagnosis models (CDMs) have attracted more and more attention for fine classification of students with regard to their ability to master given skills. The aim of this paper is to use a highly effective Gibbs algorithm based on auxiliary variables (GAAV) to estimate the deterministic input noisy "and" gate (DINA) model that is widely used for cognitive diagnosis. The applicability of the algorithm to other CDMs is also discussed. Unlike the Metropolis–Hastings algorithm, this new algorithm does not require repeated adjustment of the turning parameters to achieve an appropriate acceptance probability, and it also overcomes the dependence of the traditional Gibbs sampling algorithm on the conjugate prior distribution. Four simulation studies are conducted, and a detailed analysis of fraction subtraction test data is carried out to further illustrate the proposed methodology. [ABSTRACT FROM AUTHOR]

Feedback is a powerful instructional tool for motivating learning. But effective feedback, requires that instructors have accurate information about their students' current knowledge status and their learning progress. In modern educational measurement, two major theoretical perspectives on student ability and proficiency can be distinguished. Latent trait models identify ability as a continuous uni- or multi-dimensional construct, with unidimensional item response theoretic (IRT) models presumably the most popular type of latent trait models. They report a single ability score that allows for locating examinees relative to their peers on the latent ability dimension targeted by the test. Latent trait models have been criticized for lacking diagnostic information on students' specific skills, their strengths and weaknesses in a knowledge domain. Cognitive diagnosis (CD) models, in contrast, describe ability as a combination of discrete skills (called "attributes") that constitute (partially) ordered latent classes of proficiency. The focus of CD is on collecting information about the learning progress for immediate feedback to students in terms of skills they have mastered and those needing study. CD has been underused in education; performance assessment still mostly relies on latent-trait-based methods. The motivation for the study reported here arose from the desire to conduct a side-by-side evaluation of the two seemingly disparate psychometric frameworks, CD and IRT. Data from a biochemistry end-of-term exam were used for illustration. They were fitted with multiple CD and IRT models, among them also HO-GDINA models that permit for a close approximation to several unidimensional IRT models. [ABSTRACT FROM AUTHOR]

With the continuous development of education, personalized learning has attracted great attention. How to evaluate students' learning effects has become increasingly important. In information technology courses, the traditional academic evaluation focuses on the student's learning outcomes, such as "scores" or "right/wrong," which seldom reflects the development of students' cognitive level and lacks effective diagnostic information. This article proposes a non-parametric multi-level scoring cognitive diagnosis method based on the KNN and the characteristics of information technology courses named the EW-KNN (E-weight K-Nearest Neighbor). Compared with the KNN, the EW-KNN improved two key points. One is that it takes the number of IRP (Ideal Response Pattern) as the K value to adapt to different types of tests. The other is that the nearest neighbor distance is introduced to solve the problem of misjudgment of the categories. The Monte Carlo simulation method is used to test its performance. The results indicate that the EW-KNN has a higher accuracy rate and is suitable for information technology courses. Furthermore, the method is applied in information technology course to make a cognitive diagnosis of 120 students of high school in Shanghai. Results demonstrate that the EW-KNN can accurately diagnose each student's cognition levels and knowledge structure accurately. [ABSTRACT FROM AUTHOR]

To achieve adaptive learning, a dynamic assessment system equipped with a cognitive diagnosis was developed for this study, which adopts a three-stage model of diagnosis-intervention-assessment. To examine how this system influenced spatial geometry learning, the study used a quasi-experimental method to investigate student learning outcomes between different groups. One hundred sixteen junior high school students participating in the experiment were divided into experimental and control groups. Experimental group engaged in adaptive learning as they corrected their misconceptions in the learning intervention stage by using learning content provided according to their cognitive diagnosis results. Control group engaged in nonadaptive learning; they corrected misconceptions using self-selected learning content and thus engaged in learning activities that were not focused on their misconceptions. The results revealed that the adaptive learning students exhibited learning performance and misconception correction ratios superior to those in the control group. Through analysis, we discovered that nearly half of the nonadaptive learning students failed to select appropriate learning content for correcting their misconceptions. In addition, the time spent on learning by the students in the two groups was significantly different; the adaptive learning students spent significantly less time on learning than the nonadaptive students, thus exhibiting higher learning efficiency. [ABSTRACT FROM AUTHOR]

Researchers continue to develop and advance models for diagnostic research in the social and behavioral sciences. These diagnostic models (DMs) provide researchers with a framework for providing a fine-grained classification of respondents into substantively meaningful latent classes as defined by a multivariate collection of binary attributes. A central concern for DMs is advancing exploratory methods for uncovering the latent structure, which corresponds with the relationship between unobserved binary attributes and observed polytomous items with two or more response options. Multivariate behavioral polytomous data are often collected within a higher-order design where general factors underlying first-order latent variables. This study advances existing exploratory DMs for polytomous data by proposing a new method for inferring the latent structure underlying polytomous response data using a higher-order model to describe dependence among the discrete latent attributes. We report a novel Bayesian formulation that uses variable selection techniques for inferring the latent structure along with a higher-order factor model for attributes. We report evidence of accurate parameter recovery in a Monte Carlo simulation study and present results from an application to the 2012 Programme for International Student Assessment (PISA) problem-solving vignettes to demonstrate the method. [ABSTRACT FROM AUTHOR]

Most existing cognitive diagnosis models (CDMs) assume attributes are binary latent variables, which may be oversimplified in practice. This article introduces a higher-order CDM with ordinal attributes for dichotomous response data. The proposed model can either incorporate domain experts' knowledge or learn from the data empirically by regularizing model parameters. A sequential item response model was employed for joint attribute distribution to accommodate the sequential mastery mechanism. The expectation-maximization algorithm was employed for model estimation, and a simulation study was conducted to assess the recovery of model parameters. A set of real data was also analyzed to assess the viability of the proposed model in practice. [ABSTRACT FROM AUTHOR]

Learning path and learning progression have received extensive attention from broad disciplines. The existing research In the field of learning path is rarely applied in curriculum learning and teaching. Learning progression is usually constructed through observations, interviews but not quantitative analyses. With 726 Grade 8 students' mathematical knowledge in TIMSS-2015 as the research object, this research adopted a newly generated assessment theory – cognitive diagnosis assessment as the research tool and exploited methods such as K-means clustering analysis to construct learning path by combing the relationships among the attributes. We obtained the students' ability θs for each classified group through the 3PL model in the Item Response Theory (IRT) and constructed the learning progressions based on the θs and the attribute relationships. From a data-driven approach, this method has provided a new perspective as well as the data support for the construction of the learning paths and the learning progressions. [ABSTRACT FROM AUTHOR]

In the typical application of a cognitive diagnosis assessment, the Q matrix, which illustrates the relationship between skills and the items, is assumed to be known. However, the Q matrix is usually determined by subject matter experts and test developers, and so there may be misspecification of its elements. This paper proposes a data-driven approach to jointly estimate the Q matrix, model parameters, and the examinees' skill profiles. The key component is the likelihood ratio statistic that relates the Q matrix, responses, and the cognitive diagnosis model. Simulation studies show that the Q matrix validation based on the likelihood ratio statistic has a promising performance. [ABSTRACT FROM AUTHOR]

Cognitive diagnosis models (CDMs) have gained increasing popularity recently because of their potential to provide diagnostic inferences that can inform learning and teaching. However, the development of cognitive diagnostic assessments (CDAs) is lagging behind the development of the associated psychometric models. At present, it is not clear whether and how much extra information can be obtained by analyzing CDA data with a CDM rather than simply using unidimensional item response theory models. This study fits CDMs and three-parameter logistic (3PL) model to the data from a CDA. Results showed that the CDMs fitted the data well, but the 3PL model did not. Compared with the 3PL model, the CDMs can highlight student differences from a multidimensional perspective, thus offering finer-grained information that unidimensional IRT models are not capable of providing. [ABSTRACT FROM AUTHOR]

This article focuses on a family of restricted latent structure models with wide applications in psychological and educational assessment, where the model parameters are restricted via a latent structure matrix to reflect prespecified assumptions on the latent attributes. Such a latent matrix is often provided by experts and assumed to be correct upon construction, yet it may be subjective and misspecified. Recognizing this problem, researchers have been developing methods to estimate the matrix from data. However, the fundamental issue of the identifiability of the latent structure matrix has not been addressed until now. The first goal of this article is to establish identifiability conditions that ensure the estimability of the structure matrix. With the theoretical development, the second part of the article proposes a likelihood-based method to estimate the latent structure from the data. Simulation studies show that the proposed method outperforms the existing approaches. We further illustrate the method through a dataset in educational assessment. Supplementary materials for this article are available online. [ABSTRACT FROM AUTHOR]

A nonparametric technique based on the Hamming distance is proposed in this research by recognizing that once the attribute vector is known, or correctly estimated with high probability, one can determine the item-by-attribute vectors for new items undergoing calibration. We consider the setting whereQis known for a large item bank, and theq-vectors of additional items are estimated. The method is studied in simulation under a wide variety of conditions, and is illustrated with the Tatsuoka fraction subtraction data. A consistency theorem is developed giving conditions under which nonparametricQcalibration can be expected to work. [ABSTRACT FROM AUTHOR]

Problem solving is considered a fundamental human skill. However, large-scale assessment of problem solving in K-12 education remains a challenging task. Researchers have argued for the development of an enhanced assessment approach through joint effort from multiple disciplines. In this study, a three-stage approach based on an evidence-centered design framework is proposed to analyze problem-solving behavior, abilities, and performance. The approach is applied to assess the online problem solving of 554 students in a Shanghai primary school. The study reveals four clusters with distinctive problem-solving behavior, abilities, and performance. The findings of this approach also corroborate the results of the Programme for International Student Assessment of Shanghai students’ problem-solving performance. The implications and limitations of this study are also discussed. [ABSTRACT FROM PUBLISHER]

The Reduced Reparameterized Unified Model (Reduced RUM) is a diagnostic classification model for educational assessment that has received considerable attention among psychometricians. However, the computational options for researchers and practitioners who wish to use the Reduced RUM in their work, but do not feel comfortable writing their own code, are still rather limited. One option is to use a commercial software package that offers an implementation of the expectation maximization (EM) algorithm for fitting (constrained) latent class models like Latent GOLD or Mplus. But using a latent class analysis routine as a vehicle for fitting the Reduced RUM requires that it be re-expressed as a logit model, with constraints imposed on the parameters of the logistic function. This tutorial demonstrates how to implement marginal maximum likelihood estimation using the EM algorithm in Mplus for fitting the Reduced RUM. [ABSTRACT FROM PUBLISHER]

Traditionally, teachers evaluate students’ abilities via their total test scores. Recently, cognitive diagnostic models (CDMs) have begun to provide information about the presence or absence of students’ skills or misconceptions. Nevertheless, CDMs are typically applied to tests with multiple-choice (MC) items, which provide less diagnostic information than constructed-response (CR) items. This paper introduces new CDMs for tests with both MC and CR items, and illustrates how to use them to analyse MC and CR data, and thus, identify students’ skills and misconceptions in a mathematics domain. Analyses of real data, the responses of 497 sixth-grade students randomly selected from four Taiwanese primary schools to eight direct proportion items, were conducted to demonstrate the application of the new models. The results show that the new models can better determine students’ skills and misconceptions, in that they have higher inter-rater agreement rates than traditional CDMs. [ABSTRACT FROM AUTHOR]

By analysing the test data of 1029 British secondary school students’ performance on 20 Programme for International Student Assessment English reading items through the generalised deterministic input, noisy ‘and’ gate (G-DINA) model, the study conducted two investigations on exploring the relationships among the five reading comprehension skills defined by six English language experts. Skill groups, skill types and conflicting relationship were discovered in the investigation about the relationship among reading comprehension skills as part of the cognitive structure of the examinees, which provided insights into the sequence of reading skill training. The investigation about the relationship among reading comprehension skills in the problem-solving process, conducted at the item level, conceptualised skill relationship patterns as strategies for solving items. The study also demonstrated that saturated G-DINA model catered to the characteristics of reading comprehension skills and could be applied to tests involving highly interactive and hierarchical skills. [ABSTRACT FROM AUTHOR]

In item response theory (IRT), the invariance property states that item parameter estimates are independent of the examinee sample, and examinee ability estimates are independent of the test items. While this property has long been established and understood by the measurement community for IRT models, the same cannot be said for diagnostic classification models (DCMs). DCMs are a newer class of psychometric models that are designed to classify examinees according to levels of categorical latent traits. We examined the invariance property for general DCMs using the log-linear cognitive diagnosis model (LCDM) framework. We conducted a simulation study to examine the degree to which theoretical invariance of LCDM classifications and item parameter estimates can be observed under various sample and test characteristics. Results illustrated that LCDM classifications and item parameter estimates show clear invariance when adequate model data fit is present. To demonstrate the implications of this important property, we conducted additional analyses to show that using pre-calibrated tests to classify examinees provided consistent classifications across calibration samples with varying mastery profile distributions and across tests with varying difficulties. [ABSTRACT FROM PUBLISHER]

Objectives: Verbal memory assessment metrics are an essential component of cognitive screening tools. We compared the relative utilities of word list and story recall metrics in predicting cognitive functioning in nondemented and demented nursing home patients. We selected memory metrics from the Brief Cognitive Assessment Tool (BCAT) and the Brief Interview for Mental Status (BIMS). The BCAT incorporates both word lists and story recall metrics, while the BIMS only has a word list feature. Method: Two hundred and thirty-nine individuals residing in a Maryland skilled nursing facility were referred for neurocognitive evaluation over a one-year period. These residents met inclusion criteria for retrospective data analysis by completing the BCAT and BIMS and were aged 60 or older. Results: For the entire sample and for demented individuals, all four verbal memory metrics significantly predicted cognitive diagnosis. For nondemented individuals, only the BCAT delayed word list significantly predicted cognitive diagnosis. There appears to be enhanced utility in using both verbal memory metric types, as the inclusion of word list and story recall was a stronger predictor of cognitive diagnosis than any individual verbal memory metric. Conclusion: This study highlights the importance of using cognitive screening tools that contain both story recall and word list metrics. This is particularly true in long-term care settings where the base rate of cognitive impairment is high. [ABSTRACT FROM AUTHOR]