Search

Over several years, the evaluation of polytomous attributes in small‐sample settings has posed a challenge to the application of cognitive diagnosis models. To enhance classification precision, the support vector machine (SVM) was introduced for estimating polytomous attribution, given its proven feasibility for dichotomous cases. Two simulation studies and an empirical study assessed the impact of various factors on SVM classification performance, including training sample size, attribute structures, guessing/slipping levels, number of attributes, number of attribute levels, and number of items. The results indicated that SVM outperformed the pG‐DINA model in classification accuracy under dependent attribute structures and small sample sizes. SVM performance improved with an increased number of items but declined with higher guessing/slipping levels, more attributes, and more attribute levels. Empirical data further validated the application and advantages of SVMs. [ABSTRACT FROM AUTHOR]

Computerized adaptive testing for cognitive diagnosis (CD‐CAT) achieves remarkable estimation efficiency and accuracy by adaptively selecting and then administering items tailored to each examinee. The process of item selection stands as a pivotal component of a CD‐CAT algorithm, with various methods having been developed for binary responses. However, multiple‐choice (MC) items, an important item type that allows for the extraction of richer diagnostic information from incorrect answers, have been underemphasized. Currently, the Jensen–Shannon divergence (JSD) index introduced by Yigit et al. (Applied Psychological Measurement, 2019, 43, 388) is the only item selection method exclusively designed for MC items. However, the JSD index requires a large sample to calibrate item parameters, which may be infeasible when there is only a small or no calibration sample. To bridge this gap, the study first proposes a nonparametric item selection method for MC items (MC‐NPS) by implementing novel discrimination power that measures an item's ability to effectively distinguish among different attribute profiles. A Q‐optimal procedure for MC items is also developed to improve the classification during the initial phase of a CD‐CAT algorithm. The effectiveness and efficiency of the two proposed algorithms were confirmed by simulation studies. [ABSTRACT FROM AUTHOR]

The Q‐matrix is a crucial component of cognitive diagnostic theory and an important basis for the research and practical application of cognitive diagnosis. In practice, the Q‐matrix is typically developed by domain experts and may contain some misspecifications, so it needs to be refined using Q‐matrix validation methods. Based on signal detection theory, this paper puts forward a new Q‐matrix validation method (i.e., β$$ \beta $$ method) and then conducts a simulation study to compare the new method with existing methods. The results show that when the model is DINA (deterministic inputs, noisy ‘and’ gate), the β$$ \beta $$ method outperforms the existing methods under all conditions; under the generalized DINA (G‐DINA) model, the method still has the highest validation rate when the sample size is small, and the item quality is high or the rate of Q‐matrix misspecification is ≥.4. Finally, a sub‐dataset of the PISA 2000 reading assessment is analysed to evaluate the reliability of the β$$ \beta $$ method. [ABSTRACT FROM AUTHOR]

Concept mapping is widely used as a tool for assessing students' understanding of science. To fully realize the diagnostic potential of concept mapping, a scoring method that not only provides an objective and accurate assessment of students' drawn concept maps but also provides a detailed understanding of students' proficiency and deficiencies in knowledge is necessary. However, few of the existing scoring methods focus on the latent constructs (e.g., knowledge, skills, and cognitive processes) that guide the creation of concept maps. Instead, they focus on the completeness of the concept map by assigning a composite score, which makes it difficult to generate targeted diagnostic feedback information for advancing students' learning. To apply the diagnostic classification model to the quantitative analysis of concept maps, this study introduced the novel application of the item expansion‐based diagnostic classification analysis (IE‐DCA) for this purpose. The IE‐DCA can not only assess students' concept mapping abilities along a continuum but also classify students according to their concept mapping attributes when constructing the concept maps. The application and benefits of this approach were illustrated using a physics concept‐mapping item related to particle and rigid body. Results showed that the estimated attribute profiles via the IE‐DCA provided more detailed information about students' latent constructs than the composite score. Overall, this study illustrates the feasibility and potential of applying IE‐DCA to analyze concept maps. Future applications of IE‐DCS in other assessments in science education are discussed. [ABSTRACT FROM AUTHOR]

In learning diagnostic assessments, the attribute hierarchy specifies a sequential network of interrelated attribute mastery processes, which makes a test blueprint consistent with the cognitive theory. One of the most important functions of attribute hierarchy is to guide or limit the developmental direction of students and then form a hierarchical learning trajectory. To address the issue that the existing longitudinal learning diagnosis models cannot track the development of hierarchical attributes, this study proposes a new hierarchical longitudinal learning diagnostic modeling approach and two sample models. Compared to the longitudinal learning diagnosis models that do not consider the attribute hierarchy, the proposed models, by taking the sequential mastery tree into account, can accommodate various attribute hierarchies and simultaneously track an individual's learning developmental trajectory. An empirical study was conducted to illustrate the advantages of the proposed model. The results mainly indicated that the proposed model can properly diagnose the development of hierarchical attributes in longitudinal assessments. A simulation study was further conducted to explore the model parameter recovery of the proposed models. [ABSTRACT FROM AUTHOR]

This paper addresses the absence of an international diagnostic taxonomy for cognitive disorders in patients with epilepsy. Initiated through the 2020 Memorandum of Understanding between the International League Against Epilepsy and the International Neuropsychological Society, neuropsychological representatives from both organizations met to address the problem and consequences of the absence of an international diagnostic taxonomy for cognitive disorders in epilepsy, overview potential solutions, and propose specific solutions going forward. The group concluded that a classification of cognitive disorders in epilepsy, including an overall taxonomy and associated operational criteria, was clearly lacking and sorely needed. This paper reviews the advantages and shortcomings of four existing cognitive diagnostic approaches, including taxonomies derived from the US National Neuropsychology Network, DSM‐V Neurocognitive Disorders, the Mild Cognitive Impairment classification from the aging/preclinical dementia literature, and the Research Domain Criteria Initiative. We propose a framework to develop a consensus‐based classification system for cognitive disorders in epilepsy that will be international in scope and be applicable for clinical practice and research globally and introduce the International Classification of Cognitive Disorders in Epilepsy (IC‐CODE) project. [ABSTRACT FROM AUTHOR]

As a core component of most cognitive diagnosis models, the Q‐matrix, or item and attribute association matrix, is typically developed by domain experts, and tends to be subjective. It is critical to validate the Q‐matrix empirically because a misspecified Q‐matrix could result in erroneous attribute estimation. Most existing Q‐matrix validation procedures are developed for dichotomous responses. However, in this paper, we propose a method to empirically detect and correct the misspecifications in the Q‐matrix for graded response data based on the sequential generalized deterministic inputs, noisy 'and' gate (G‐DINA) model. The proposed Q‐matrix validation procedure is implemented in a stepwise manner based on the Wald test and an effect size measure. The feasibility of the proposed method is examined using simulation studies. Also, a set of data from the Trends in International Mathematics and Science Study (TIMSS) 2011 mathematics assessment is analysed for illustration. [ABSTRACT FROM AUTHOR]

Cognitive diagnosis models (CDMs) have been used as psychometric tools in educational assessments to estimate students' proficiency profiles. However, most CDMs assume that all students adopt the same strategy when approaching problems in an assessment, which may not be the case in practice. This study develops a generalized multiple‐strategy CDM for dichotomous response data. The proposed model provides a unified framework to accommodate various condensation rules (e.g., conjunctive, disjunctive, and additive) and different strategy selection approaches (i.e., probability‐matching, over‐matching, and maximizing). Model parameters are estimated using the marginal maximum likelihood estimation via expectation‐maximization algorithm. Simulation studies showed that the parameters of the proposed model can be adequately recovered and that the proposed model was relatively robust to some types of model misspecifications. A set of real data was analysed as well to illustrate the use of the proposed model in practice. [ABSTRACT FROM AUTHOR]

Constructed‐response items have been shown to be appropriate for cognitively diagnostic assessments because students' problem‐solving procedures can be observed, providing direct evidence for making inferences about their proficiency. However, multiple strategies used by students make item scoring and psychometric analyses challenging. This study introduces the so‐called two‐digit scoring scheme into diagnostic assessments to record both students' partial credits and their strategies. This study also proposes a diagnostic tree model (DTM) by integrating the cognitive diagnosis models with the tree model to analyse the items scored using the two‐digit rubrics. Both convergent and divergent tree structures are considered to accommodate various scoring rules. The MMLE/EM algorithm is used for item parameter estimation of the DTM, and has been shown to provide good parameter recovery under varied conditions in a simulation study. A set of data from TIMSS 2007 mathematics assessment is analysed to illustrate the use of the two‐digit scoring scheme and the DTM. [ABSTRACT FROM AUTHOR]

The maximum likelihood classification rule is a standard method to classify examinee attribute profiles in cognitive diagnosis models (CDMs). Its asymptotic behaviour is well understood when the model is assumed to be correct, but has not been explored in the case of misspecified latent class models. This paper investigates the asymptotic behaviour of a two‐stage maximum likelihood classifier under a misspecified CDM. The analysis is conducted in a general restricted latent class model framework addressing all types of CDMs. Sufficient conditions are proposed under which a consistent classification can be obtained by using a misspecified model. Discussions are also provided on the inconsistency of classification under certain model misspecification scenarios. Simulation studies and a real data application are conducted to illustrate these results. Our findings can provide some guidelines as to when a misspecified simple model or a general model can be used to provide a good classification result. [ABSTRACT FROM AUTHOR]

To provide more refined diagnostic feedback with collateral information in item response times (RTs), this study proposed joint modelling of attributes and response speed using item responses and RTs simultaneously for cognitive diagnosis. For illustration, an extended deterministic input, noisy ‘and’ gate (DINA) model was proposed for joint modelling of responses and RTs. Model parameter estimation was explored using the Bayesian Markov chain Monte Carlo (MCMC) method. The PISA 2012 computer‐based mathematics data were analysed first. These real data estimates were treated as true values in a subsequent simulation study. A follow‐up simulation study with ideal testing conditions was conducted as well to further evaluate model parameter recovery. The results indicated that model parameters could be well recovered using the MCMC approach. Further, incorporating RTs into the DINA model would improve attribute and profile correct classification rates and result in more accurate and precise estimation of the model parameters. [ABSTRACT FROM AUTHOR]

Abstract: The deterministic inputs, noisy, “and” gate (DINA) model is a popular cognitive diagnosis model (CDM) in psychology and psychometrics used to identify test takers' profiles with respect to a set of latent attributes or skills. In this work, we propose an estimation method for the DINA model with the No‐U‐Turn Sampler (NUTS) algorithm, an extension to Hamiltonian Monte Carlo (HMC) method. We conduct a simulation study in order to evaluate the parameter recovery and efficiency of this new Markov chain Monte Carlo method and to compare it with two other Bayesian methods, the Metropolis Hastings and Gibbs sampling algorithms, and with a frequentist method, using the Expectation–Maximization (EM) algorithm. The results indicated that NUTS algorithm employed in the DINA model properly recovers all parameters and is accurate for all simulated scenarios. We apply this methodology in the mental health area in order to develop a new method of classification for respondents to the Beck Depression Inventory. The implementation of this method for the DINA model applied to other psychological tests has the potential to improve the medical diagnostic process. [ABSTRACT FROM AUTHOR]

This paper proposes a general polytomous cognitive diagnosis model for a special type of graded responses, where item categories are attained in a sequential manner, and associated with some attributes explicitly. To relate categories to attributes, a category-level Q-matrix is used. When the attribute and category association is specified a priori, the proposed model has the flexibility to allow different cognitive processes (e.g., conjunctive, disjunctive) to be modelled at different categories within a single item. This model can be extended for items where categories cannot be explicitly linked to attributes, and for items with unordered categories. The feasibility of the proposed model is examined using simulated data. The proposed model is illustrated using the data from the Trends in International Mathematics and Science Study 2007 assessment. [ABSTRACT FROM AUTHOR]

There has recently been much interest in computerized adaptive testing ( CAT) for cognitive diagnosis. While there exist various item selection criteria and different asymptotically optimal designs, these are mostly constructed based on the asymptotic theory assuming the test length goes to infinity. In practice, with limited test lengths, the desired asymptotic optimality may not always apply, and there are few studies in the literature concerning the optimal design of finite items. Related questions, such as how many items we need in order to be able to identify the attribute pattern of an examinee and what types of initial items provide the optimal classification results, are still open. This paper aims to answer these questions by providing non-asymptotic theory of the optimal selection of initial items in cognitive diagnostic CAT. In particular, for the optimal design, we provide necessary and sufficient conditions for the Q-matrix structure of the initial items. The theoretical development is suitable for a general family of cognitive diagnostic models. The results not only provide a guideline for the design of optimal item selection procedures, but also may be applied to guide item bank construction. [ABSTRACT FROM AUTHOR]

In order to look more closely at the many particular skills examinees utilize to answer items, cognitive diagnosis models have received much attention, and perhaps are preferable to item response models that ordinarily involve just one or a few broadly defined skills, when the objective is to hasten learning. If these fine-grained skills can be identified, a sharpened focus on learning and remediation can be achieved. The focus here is on how to detect when learning has taken place for a particular attribute and efficiently guide a student through a sequence of items to ultimately attain mastery of all attributes while administering as few items as possible. This can be seen as a problem in sequential change-point detection for which there is a long history and a well-developed literature. Though some ad hoc rules for determining learning may be used, such as stopping after M consecutive items have been successfully answered, more efficient methods that are optimal under various conditions are available. The CUSUM, Shiryaev-Roberts and Shiryaev procedures can dramatically reduce the time required to detect learning while maintaining rigorous Type I error control, and they are studied in this context through simulation. Future directions for modelling and detection of learning are discussed. [ABSTRACT FROM AUTHOR]

The Asymptotic Classification Theory of Cognitive Diagnosis (Chiu et al., 2009, Psychometrika, 74, 633-665) determined the conditions that cognitive diagnosis models must satisfy so that the correct assignment of examinees to proficiency classes is guaranteed when non-parametric classification methods are used. These conditions have only been proven for the Deterministic Input Noisy Output AND gate model. For other cognitive diagnosis models, no theoretical legitimization exists for using non-parametric classification techniques for assigning examinees to proficiency classes. The specific statistical properties of different cognitive diagnosis models require tailored proofs of the conditions of the Asymptotic Classification Theory of Cognitive Diagnosis for each individual model - a tedious undertaking in light of the numerous models presented in the literature. In this paper a different way is presented to address this task. The unified mathematical framework of general cognitive diagnosis models is used as a theoretical basis for a general proof that under mild regularity conditions any cognitive diagnosis model is covered by the Asymptotic Classification Theory of Cognitive Diagnosis. [ABSTRACT FROM AUTHOR]

Cognitive diagnosis models of educational test performance rely on a binary Q-matrix that specifies the associations between individual test items and the cognitive attributes (skills) required to answer those items correctly. Current methods for fitting cognitive diagnosis models to educational test data and assigning examinees to proficiency classes are based on parametric estimation methods such as expectation maximization (EM) and Markov chain Monte Carlo (MCMC) that frequently encounter difficulties in practical applications. In response to these difficulties, non-parametric classification techniques (cluster analysis) have been proposed as heuristic alternatives to parametric procedures. These non-parametric classification techniques first aggregate each examinee's test item scores into a profile of attribute sum scores, which then serve as the basis for clustering examinees into proficiency classes. Like the parametric procedures, the non-parametric classification techniques require that the Q-matrix underlying a given test be known. Unfortunately, in practice, the Q-matrix for most tests is not known and must be estimated to specify the associations between items and attributes, risking a misspecified Q-matrix that may then result in the incorrect classification of examinees. This paper demonstrates that clustering examinees into proficiency classes based on their item scores rather than on their attribute sum-score profiles does not require knowledge of the Q-matrix, and results in a more accurate classification of examinees. [ABSTRACT FROM AUTHOR]

Diagnostic classification models (aka cognitive or skills diagnosis models) have shown great promise for evaluating mastery on a multidimensional profile of skills as assessed through examinee responses, but continued development and application of these models has been hindered by a lack of readily available software. In this article we demonstrate how diagnostic classification models may be estimated as confirmatory latent class models using Mplus, thus bridging the gap between the technical presentation of these models and their practical use for assessment in research and applied settings. Using a sample English test of three grammatical skills, we describe how diagnostic classification models can be phrased as latent class models within Mplus and how to obtain the syntax and output needed for estimation and interpretation of the model parameters. We also have written a freely available SAS program that can be used to automatically generate the Mplus syntax. We hope this work will ultimately result in greater access to diagnostic classification models throughout the testing community, from researchers to practitioners. [ABSTRACT FROM AUTHOR]