Search

The linear composite direction represents, theoretically, where the unidimensional scale would lie within a multidimensional latent space. Using compensatory multidimensional IRT, the linear composite can be derived from the structure of the items and the latent distribution. The purpose of this study was to evaluate the validity of the linear composite conjecture and examine how well a fitted unidimensional IRT model approximates the linear composite direction in a multidimensional latent space. Simulation experiment results overall show that the fitted unidimensional IRT model sufficiently approximates linear composite direction when correlation between bivariate latent variables is positive. When the correlation between bivariate latent variables is negative, instability occurs when the fitted unidimensional IRT model is used to approximate linear composite direction. A real data experiment was also conducted using 20 items from a multiple-choice mathematics test from American College Testing.

A computerized adaptive testing (CAT) solution for tests with multidimensional pairwise-comparison (MPC) items, aiming to measure career interest, value, and personality, is rare. This paper proposes new item selection and exposure control methods for CAT with dichotomous and polytomous MPC items and present simulation study results. The results show that the procedures are effective in selecting items and controlling within-person statement exposure with no loss of efficiency. Implications are discussed in two applications of the proposed CAT procedures: a work attitude test with dichotomous MPC items and a career interest assessment with polytomous MPC items.

The present study aims to examine gender differential item functioning (DIF) in the reading comprehension section of a high stakes test using cognitive diagnosis models. Based on the multiple-group...

Cognitive diagnosis models (CDMs) are used in educational, clinical, or personnel selection settings to classify respondents with respect to discrete attributes, identifying strengths and needs, and thus allowing to provide tailored training/treatment. As in any assessment, an accurate reliability estimation is crucial for valid score interpretations. In this sense, most CDM reliability indices are based on the posterior probabilities of the estimated attribute profiles. These posteriors are traditionally computed using point estimates for the model parameters as approximations to their populational values. If the uncertainty around these parameters is unaccounted for, the posteriors may be overly peaked, deriving into overestimated reliabilities. This article presents a multiple imputation (MI) procedure to integrate out the model parameters in the estimation of the posterior distributions, thus correcting the reliability estimation. A simulation study was conducted to compare the MI procedure with the traditional reliability estimation. Five factors were manipulated: the attribute structure, the CDM model (DINA and G-DINA), test length, sample size, and item quality. Additionally, an illustration using the Examination for the Certificate of Proficiency in English data was analyzed. The effect of sample size was studied by sampling subsets of subjects from the complete data. In both studies, the traditional reliability estimation systematically provided overestimated reliabilities, whereas the MI procedure offered more accurate results. Accordingly, practitioners in small educational or clinical settings should be aware that the reliability estimation using model parameter point estimates may be positively biased. R codes for the MI procedure are made available, Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. This work has been funded by the Community of Madrid through the Pluriannual Agreement with the Universidad de Universidad Autónoma de Madrid in its Programa de Estímulo a la Investigación de Jóvenes Doctores (Reference SI3/ PJI/2021-00258), and by the Spanish Ministry of Science and Innovation (FPI BES-2016-077814)

In this study, person parameter recoveries are investigated by retrofitting polytomous attribute cognitive diagnosis and multidimensional item response theory (MIRT) models. The data are generated using two cognitive diagnosis models (i.e., pG-DINA: the polytomous generalized deterministic inputs, noisy “and” gate and fA-M: the fully-additive model) and one MIRT model (i.e., the compensatory two-parameter logistic model). Twenty-five replications are used for each of the 54 conditions resulting from varying the item discrimination index, ratio of simple to complex items, test length, and correlations between skills. The findings are obtained by comparing the person parameter estimates of all three models to the actual parameters used in the data generation. According to the findings, the most accurate estimates are obtained when the fitted models correspond to the generating models. Comparable results are obtained when the fA-M is retrofitted to other data or when the MIRT model is retrofitted to fA-M data. However, the results are poor when the pG-DINA is retrofitted to other data or the MIRT is retrofitted to pG-DINA data. Among the conditions used in the study, test length and item discrimination have the greatest influence on the person parameter estimation accuracy. Variation in the simple to complex item ratio has a notable influence when the MIRT model is used. Although the impact on the person parameter estimation accuracy of the correlation between skills is limited, its effect on MIRT data is more significant., Bu araştırmada çok kategorili bilişsel tanı ve çok boyutlu madde tepki kuramı (ÇBMTK) modellerinin birbiri yerine kullanımı durumunda birey parametrelerinin tekrar elde edilebilirliği incelenmiştir. Bu amaç doğrultusunda çok kategorili bilişsel tanı modellerinden polytomous generalized deterministic input noisy and gate (pG-DINA) ve fully-additive model (fA-M) ve ÇBMTK modellerinden ise telafi edici 2PL modeli için veriler üretilmiştir. Veriler, madde ayırt edicilik indeksi, maddelerin yapılarına göre testteki oranı, test uzunluğu ve yetenekler arası korelasyon değerleri farklılaştırılarak toplamda 54 koşul kullanılarak 25 tekrar ile üretilmiştir. Verinin her üç modelle de kestirimi sonucu ortaya çıkan birey parametreleri ile veri üretiminde kullanılan birey parametrelerinin karşılaştırılması ile bulgular elde edilmiştir. Bulgulara göre tüm veri türlerinin en yüksek doğrulukta kestirimi ait oldukları modeller tarafından gerçekleştirilmiştir. Uyarlama içeren analizlerde ise fA-M diğer iki model verilerini, ÇBMTK ise fA-M verisini, verinin ait olduğu model kestirimine yakın bir doğruluk oranında kestirmiştir. PG-DINA’nın diğer iki model verilerini, ÇBMTK’nin ise pG-DINA verisini kestirmede düşük performansa sahip olduğu gözlenmiştir. Araştırmada kullanılan koşullardan sırasıyla test uzunluğu ve madde ayırt ediciliğinin birey parametre doğruluğuna etki eden en kuvvetli faktörler olduğu görülmüştür. Madde yapısı oranı koşulunun ise ÇBMTK verisi analizlerinde ve uyarlamalarında çok daha etkili olduğu görülmüştür. Yetenekler arası korelasyonun varlığının birey parametre doğruluğuna etkisinin ise ÇBMTK verisinin analizlerinde daha belirgin ancak yine de sınırlı olduğu görülmüştür

To expand the use of cognitive diagnosis models (CDMs) to longitudinal assessments, this study proposes a bias-corrected three-step estimation approach for latent transition CDMs with covariates by integrating a general CDM and a latent transition model. The proposed method can be used to assess changes in attribute mastery status and attribute profiles and to evaluate the covariate effects on both the initial state and transition probabilities over time using latent (multinomial) logistic regression. Because stepwise approaches generally yield biased estimates, correction for classification error probabilities is considered in this study. The results of the simulation study showed that the proposed method yielded more accurate parameter estimates than the uncorrected approach. The use of the proposed method is also illustrated using a set of real data.

The Q-matrix identifies the subset of attributes measured by each item in the cognitive diagnosis modelling framework. Usually constructed by domain experts, the Q-matrix might contain some misspecifications, disrupting classification accuracy. Empirical Q-matrix validation methods such as the general discrimination index (GDI) and Wald have shown promising results in addressing this problem. However, a cut-off point is used in both methods, which might be suboptimal. To address this limitation, the Hull method is proposed and evaluated in the present study. This method aims to find the optimal balance between fit and parsimony, and it is flexible enough to be used either with a measure of item discrimination (the proportion of variance accounted for, PVAF) or a coefficient of determination (pseudo-R2 ). Results from a simulation study showed that the Hull method consistently showed the best performance and shortest computation time, especially when used with the PVAF. The Wald method also performed very well overall, while the GDI method obtained poor results when the number of attributes was high. The absence of a cut-off point provides greater flexibility to the Hull method, and it places it as a comprehensive solution to the Q-matrix specification problem in applied settings. This proposal is illustrated using real data.

This study proposes a multiple-group cognitive diagnosis model to account for the fact that students in different groups may use distinct attributes or use the same attributes but in different manners (e.g., conjunctive, disjunctive, and compensatory) to solve problems. Based on the proposed model, this study systematically investigates the performance of the likelihood ratio (LR) test and Wald test in detecting differential item functioning (DIF). A forward anchor item search procedure was also proposed to identify a set of anchor items with invariant item parameters across groups. Results showed that the LR and Wald tests with the forward anchor item search algorithm produced better calibrated Type I error rates than the ordinary LR and Wald tests, especially when items were of low quality. A set of real data were also analyzed to illustrate the use of these DIF detection procedures.

Cognitive diagnosis models (CDMs) have been studied as a means of providing detailed diagnostic information about the skills that have been mastered, and the skills that have not, by examinees. Pri...

Cognitive diagnosis models (CDMs) have gained increasing popularity recently because of their potential to provide diagnostic inferences that can inform learning and teaching. However, the developm...

In research applications, mental health problems such as alcohol-related problems and depression are commonly assessed and evaluated using scale scores or latent trait scores derived from factor analysis or item response theory models. This tutorial paper demonstrates the use of cognitive diagnosis models (CDMs) as an alternative approach to characterizing mental health problems of young adults when item-level data are available. Existing measurement approaches focus on estimating the general severity of a given mental health problem at the scale level as a unidimensional construct without accounting for other symptoms of related mental health problems. The prevailing approaches may ignore clinically meaningful presentations of related symptoms at the item level. The current study illustrates CDMs using item-level data from college students (40 items from 719 respondents; 34.6% men, 83.9% White, and 16.3% first-year students). Specifically, we evaluated the constellation of four postulated domains (i.e., alcohol-related problems, anxiety, hostility, and depression) as a set of attribute profiles using CDMs. After accounting for the impact of each attribute (i.e., postulated domain) on the estimates of attribute profiles, the results demonstrated that when items or attributes have limited information, CDMs can utilize item-level information in the associated attributes to generate potentially meaningful estimates and profiles, compared to analyzing each attribute independently. We introduce a novel visual inspection aid, the lens plot, for quantifying this gain. CDMs may be a useful analytical tool to capture respondents’ risk and resilience for prevention research.

Because the validity of diagnostic information generated by cognitive diagnosis models (CDMs) depends on the appropriateness of the estimated attribute profiles, it is imperative to ensure the accurate measurement of students’ test performance by conducting person fit (PF) evaluation to avoid flawed remediation measures. The standardized log-likelihood statistic lZ has been extended to the CDM framework. However, its null distribution is found to be negatively skewed. To address this issue, this study applies different methods of adjusting the skewness of lZ that have been proposed in the item response theory context, namely, χ2-approximation, Cornish-Fisher expansion, and Edgeworth expansion to bring its null distribution closer to the standard normal distribution. The skewness-corrected PF statistics are investigated by calculating their type I error and detection rates using a simulation study. Fraction-subtraction data are also used to illustrate the application of these PF statistics.

Formatif de g erlendirme icin hazirlanmi s olan olcme araclari, amacli bir s ekilde tasarlandi g inda, tanilayici bilgi toplamamiza imkân sa g larlar. Bu tur bilgilere dayanarak yapilan cikarimlar e g itimcilerin telafi edici eylemler gercekle s tirmelerini sa g lar. Son zamanlarda geli s tirilen formatif de g erlendirme yontemleri o g rencilere ve o g retmenlere o g rencilerin hangi bilgi ve beceri parcaciklarina (bili s sel niteliklere) sahip olduklari ya da olmadiklariyla ilgili ayrintili geri bildirim sa g layabilmektedir. Bu geribildirimlerin o g retimi ve o g renmeyi optimize etmek amaciyla kullanilmasi beklenmektedir. Formatif olcme ve de g erlendirmelerin sonuclarindan tanisal bilgi edinebilmek icin iyi tasarlanmi s sinavlarin uygulanmasinin yani sira ce s itli bili s sel tani modellerinin kullanimina da ihtiyac vardir. Ancak, bu modellerin alan yazina yeni kazandirilmi s olmasi ve e g itimcilerin henuz yeterince a s ina olmamalari nedeniyle, bili s sel tani modellemelerinin (BTM) parametre kestirimlerinin ve di g er ilgili analizlerin uygulamalari yeterince yayginla s mami s tir. Bu makalenin amaci BTM’ye a s ina olmayan e g itimci ve ara s tirmacilara BTM’nin temel prensiplerini tanitmak ve ucretsiz bir yazilim olan GDINA R paketi kullanilarak yapilabilecek cesitli BTM uygulamalarini yeterince detayli olarak gostermektir. BTM analizleri yapan bazi yazilim programlarinin mevcut olmasina ragmen, alanla ilgili en kapsamli paketlerden biri olan GDINA R paketinin kullaniminin saglayacagi avantajlar arasinda sunlar sayilabilir: (1) Bircok bilissel tani modelinin kestiriminin R ile mumkun olmasi; (2) Diger bircok yazilim programlarinin tek tip modelin kestirimine imkan veriyor olmasi; (3) Diger programlarin cogunun ticari olmasi veya ancak yazarin kendisiyle irtibat kurularak temin edilebilir olmasi; ve (4) Sozdizimi (sintaks) hazirlamanin bazi programlarda oldukca zahmet verici olmasi. Bu makalede oncelikle BTM analizinin girdi ve ciktilari da dahil olmak uzere BTM terminolojisi tartisilmakta, sonrasinda GDINA model yapisi tanitilmakta ve GDINA R paketi sunulmaktadir. Sonrasinda ise numerik bir ornek veri setinden yola cikilarak, GDINA R paketi ve bu paketin sundu g u grafiksel kullanici ara yuzu (GUI: graphical user interface) kullanilarak yapilabilecek analiz turleri adim adim takip edilebilecek s ekilde sunulmu s tur. Ayrica gerekli gorulen noktalarda yapilan i s ve i s lemlerin teorik bilgisine ve elde edilen sonuclarin yorumlanmasina dair bilgiler verilmi s tir. I lerleyen bolumlerde, GUI ile yapilamayan ancak GDINA paketinin sundu g u ve R kullanimina a s ina olan kullanicilarin sozdizimi yazarak yapabilecekleri ilave analizlere ve paketin ek ozelliklerine yer verilmi s tir. Makalenin son bolumunde ise bazi hatirlatma ve tespitler yapilmi s tir. G-DINA modeli ve GDINA paketi Genel ya da kisitlanmi s olarak bircok bili s sel tani modelinin ayrimlari o g rencinin maddeleri cevaplama surecinde bili s sel niteliklerin (olculen bilgi ya da becerilerin) etkile s imlerine ba g li olarak yapilir. G-DINA gibi genel modellerde butun ana etkiler ve etkile s im etkileri madde tepki fonksiyonuna katkida bulunur. Bu tur genel modellerin parametrelerinde yapilacak kisitlamalarla daha kisitli ya da daha sade modeller elde edilebilir. Bunun yani sira, genel modellerde birden farkli link fonksiyonlarinin kullanimi da soz konusudur. Orne g in, G-DINA model identity, logit ve log linkleriyle farkli isimlerle sunulabilir, ancak farkli linklerle ortaya konulan genel modeller ozde s model-veri uyumuna sahiptirler. GDINA paketi GUI aracili g iyla farkli linkler altinda olu s turulabilecek olan GDINA, logit GDINA, log GDINA’nin yani sira, bu genel modellerde ortaya koyulacak kisitlamalarla olu s turulabilen DINA, DINO, ACDM, LLM, ve RRUM modellerinin kestirimine imkan verir. Ayrica, GUI aracili g iyla olmasa da, R kullanimina a s ina olan ve sozdizimi yazabilen kullanicilar GDINA R paketini kullanarak farkli stratejilerin kullanimi durumunu goz onunde bulunduran MS-DINA ve G-DINA’nin farkli uzantilarindan olu s an modellerin (or. siniflama ve siralama olcekleriyle elde edilen veriler icin sequential G-DINA ve kismi puanlamali nitelikleri icin polytomous G-DINA) kestirimini yapabilirler. GDINA'yi kullanarak, testi alanlarin nitelikleri kazanmi s ya da kazanamami s olma durumlariyla ilgili tanilayici bilgi; belirli bir niteli g in kazandirilma orani; ve belirli bir profile sahip olan ki s ilerin orani ile ilgili bilgiler elde edilebilir. Bunlarin di s inda, temel CDM analizlerinin yani sira, madde ve nitelikler arasindaki etkile s imi en iyi aciklayan BTM’nin kullanilabilmesi adina GDINA paketi ara s tirmacilarin test- ve madde-duzeyinde model secimi yapabilmelerine imkan tanir. GDINA paketi ayni zamanda madde-nitelik e s le s tirme matrisinde yapilabilecek muhtemel yanli s lari ortaya cikarmaya yardimci olmak adina ampirik olarak Q-matrisin uygunlu g unun do g rulamasini sa g layan metotlarin kullanilmasina imkan sa g lar. Son olarak, bu paket, ce s itli kolay ve kullani s li grafikler ortaya koyar. Bu grafikler elde edilen analiz sonuclarinda onemli noktalarin vurgulanmasina ve elde edilen istatistik ve parametrelerin kolaylikla kar s ila s tirilmasina olanak sa g lar.

This article introduces a blocked-design procedure for cognitive diagnosis computerized adaptive testing (CD-CAT), which allows examinees to review items and change their answers during test administration. Four blocking versions of the new procedure were proposed. In addition, the impact of several factors, namely, item quality, generating model, block size, and test length, on the classification rates was investigated. Three popular item selection indices in CD-CAT were used and their efficiency compared using the new procedure. An additional study was carried out to examine the potential benefit of item review. The results showed that the new procedure is promising in that allowing item review resulted only in a small loss in attribute classification accuracy under some conditions. Moreover, using a blocked-design CD-CAT is beneficial to the extent that it alleviates the negative impact of test anxiety on examinees’ true performance.

As a core component of most cognitive diagnosis models, the Q-matrix, or item and attribute association matrix, is typically developed by domain experts, and tends to be subjective. It is critical to validate the Q-matrix empirically because a misspecified Q-matrix could result in erroneous attribute estimation. Most existing Q-matrix validation procedures are developed for dichotomous responses. However, in this paper, we propose a method to empirically detect and correct the misspecifications in the Q-matrix for graded response data based on the sequential generalized deterministic inputs, noisy 'and' gate (G-DINA) model. The proposed Q-matrix validation procedure is implemented in a stepwise manner based on the Wald test and an effect size measure. The feasibility of the proposed method is examined using simulation studies. Also, a set of data from the Trends in International Mathematics and Science Study (TIMSS) 2011 mathematics assessment is analysed for illustration.

OBJECTIVE: Integrative data analysis was used to combine existing data from nine trials of cognitive-behavioral therapy (CBT) for anxious youth (N = 832) and identify trajectories of symptom change and predictors of trajectories.METHOD: Youth- and parent-reported anxiety symptoms were combined using item-response theory models. Growth mixture modeling assessed for trajectories of treatment response across pre-, mid-, and posttreatment and 1-year follow-up. Pretreatment client demographic and clinical traits and treatment modality (individual- and family-based CBT) were examined as predictors of trajectory classes.RESULTS: Growth mixture modeling supported three trajectory classes based on parent-reported symptoms: steady responders, rapid responders, and delayed improvement. A 4-class model was supported for youth-reported symptoms: steady responders, rapid responders, delayed improvement, and low-symptom responders. Delayed improvement classes were predicted by higher number of diagnoses (parent and youth report). Receiving family CBT predicted membership in the delayed improvement class compared to all other classes and membership in the steady responder class compared with rapid responders (youth report). Rapid responders were predicted by older age (parent report) and higher number of diagnoses (parent report). Low-symptom responders were more likely to be male (youth report).CONCLUSIONS: Integrative data analysis identified distinct patterns of symptom change. Diagnostic complexity, age, gender, and treatment modality differentiated response classes. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

In this paper, the slice-within-Gibbs sampler has been introduced as a method for estimating cognitive diagnosis models (CDMs). Compared with other Bayesian methods, the slice-within-Gibbs sampler can employ a wide-range of prior specifications; moreover, it can also be applied to complex CDMs with the aid of auxiliary variables, especially when applying different identifiability constraints. To evaluate its performances, two simulation studies were conducted. The first study confirmed the viability of the slice-within-Gibbs sampler in estimating CDMs, mainly including G-DINA and DINA models. The second study compared the slice-within-Gibbs sampler with other commonly used Markov Chain Monte Carlo algorithms, and the results showed that the slice-within-Gibbs sampler converged much faster than the Metropolis-Hastings algorithm and more flexible than the Gibbs sampling in choosing the distributions of priors. Finally, a fraction subtraction dataset was analyzed to illustrate the use of the slice-within-Gibbs sampler in the context of CDMs.

In the context of cognitive diagnosis models (CDMs), a Q-matrix reflects the correspondence between attributes and items. The Q-matrix construction process is typically subjective in nature, which may lead to misspecifications. All this can negatively affect the attribute classification accuracy. In response, several methods of empirical Q-matrix validation have been developed. The general discrimination index (GDI) method has some relevant advantages such as the possibility of being applied to several CDMs. However, the estimation of the GDI relies on the estimation of the latent group sizes and success probabilities, which is made with the original (possibly misspecified) Q-matrix. This can be a problem, especially in those situations in which there is a great uncertainty about the Q-matrix specification. To address this, the present study investigates the iterative application of the GDI method, where only one item is modified at each step of the iterative procedure, and the required cutoff is updated considering the new parameter estimates. A simulation study was conducted to test the performance of the new procedure. Results showed that the performance of the GDI method improved when the application was iterative at the item level and an appropriate cutoff point was used. This was most notable when the original Q-matrix misspecification rate was high, where the proposed procedure performed better 96.5% of the times. The results are illustrated using Tatsuoka’s fraction-subtraction data set., The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was partially supported by Ministerio de Ciencia, Innovación y Universidades, Spain (Grant No. PSI2017-85022-P) and Cátedra de Modelos y Aplicaciones Psicométricas (Instituto de Ingeniería del Conocimiento and Universidad Autónoma de Madrid)

A number of parametric and nonparametric methods for estimating cognitive diagnosis models (CDMs) have been developed and applied in a wide range of contexts. However, in the literature, a wide chasm exists between these two families of methods, and their relationship to each other is not well understood. In this paper, we propose a unified estimation framework to bridge the divide between parametric and nonparametric methods in cognitive diagnosis to better understand their relationship. We also develop iterative joint estimation algorithms and establish consistency properties within the proposed framework. Lastly, we present comprehensive simulation results to compare different methods, and provide practical recommendations on the appropriate use of the proposed framework in various CDM contexts.

This study aims to conduct differential item functioning analyses in the context of cognitive diagnosis assessments using various formulations of the Wald test. In implementing the Wald test, two scenarios are considered: one where the underlying reduced model can be assumed; and another where a saturated CDM is used. Illustration of the different Wald test to detect DIF in CDM data was based on the items’ performance of the Proportional Reasoning test among low- and high-performing school students. A benchmark simulation study was included to compare the performance of the Wald test in each scenario. The agreement of the latent attribute classification based on different cognitive diagnosis models was also discussed.

Cognitive diagnosis models (CDMs) have attracted increasing attention in educational measurement because of their potential to provide diagnostic feedback about students' strengths and weaknesses. This article introduces the feature-rich R package GDINA for conducting a variety of CDM analyses. Built upon a general model framework, a number of CDMs can be calibrated using the GDINA package. Functions are also available for evaluating model-data fit, detecting differential item functioning, validating the item and attribute association, and examining classification accuracy. A grapical user interface is also provided for researchers who are less familar with R. This paper contains both technical details about model estimation and illustrations about how to use the package for data analysis. The GDINA package is also used to replicate published results, showing that it could provide comparable model parameter estimation.

Currently, there are two predominant approaches in adaptive testing. One, referred to as cognitive diagnosis computerized adaptive testing (CD-CAT), is based on cognitive diagnosis models, and the other, the traditional CAT, is based on item response theory. The present study evaluates the performance of two item selection rules (ISRs) originally developed in the CD-CAT framework, the double Kullback-Leibler information (DKL) and the generalized deterministic inputs, noisy “and” gate model discrimination index (GDI), in the context of traditional CAT. The accuracy and test security associated with these two ISRs are compared to those of the point Fisher information and weighted KL using a simulation study. The impact of the trait level estimation method is also investigated. The results show that the new ISRs, particularly DKL, could be used to improve the accuracy of CAT. Better accuracy for DKL is achieved at the expense of higher item overlap rate. Differences among the item selection rules become smaller as the test gets longer. The two CD-CAT ISRs select different types of items: items with the highest possible a parameter with DKL, and items with the lowest possible c parameter with GDI. Regarding the trait level estimator, expected a posteriori method is generally better in the first stages of the CAT, and converges with the maximum likelihood method when a medium to large number of items are involved. The use of DKL can be recommended in low-stakes settings where test security is less of a concern, This research was supported by Grant PSI2017-85022-P (Ministerio de Ciencia, Innovación y Universidades, Spain) and the UAMIIC Chair «Psychometric Models and Applications"

Cognitive diagnosis models (CDMs) are latent class models that hold great promise for providing diagnostic information about student knowledge profiles. The increasing use of computers in classrooms enhances the advantages of CDMs for more efficient diagnostic testing by using adaptive algorithms, referred to as cognitive diagnosis computerized adaptive testing (CD-CAT). When multiple-choice items are involved, CD-CAT can be further improved by using polytomous scoring (i.e., considering the specific options students choose), instead of dichotomous scoring (i.e., marking answers as either right or wrong). In this study, the authors propose and evaluate the performance of the Jensen–Shannon divergence (JSD) index as an item selection method for the multiple-choice deterministic inputs, noisy “and” gate (MC-DINA) model. Attribute classification accuracy and item usage are evaluated under different conditions of item quality and test termination rule. The proposed approach is compared with the random selection method and an approximate approach based on dichotomized responses. The results show that under the MC-DINA model, JSD improves the attribute classification accuracy significantly by considering the information from distractors, even with a very short test length. This result has important implications in practical classroom settings as it can allow for dramatically reduced testing times, thus resulting in more targeted learning opportunities.

Solving a constructed-response item usually requires successfully performing a sequence of tasks. Each task could involve different attributes, and those required attributes may be “condensed” in various ways to produce the responses. The sequential generalized deterministic input noisy “and” gate model is a general cognitive diagnosis model (CDM) for graded response items of this type. Although a host of dichotomous CDMs with different condensation rules can be used to parameterize the success probability of each task, specifying the most appropriate one remains challenging. If the CDM specified for each task is not in accordance with the underlying cognitive processes, the validity of the inference could be questionable. This study aims to evaluate whether several hypothesis tests, namely, the Wald test using various variance–covariance matrices, the likelihood ratio (LR) test, and the LR test using approximated parameters, can be used to select the appropriate CDMs for each task of graded response items. Simulation studies are conducted to examine the Type I error and power of the hypothesis tests under varied conditions. A data set from the Trends in International Mathematics and Science Study 2007 mathematics assessment is analyzed as an illustration.

Cognitive diagnosis models (CDMs) allow for the extraction of fine-grained, multidimensional diagnostic information from appropriately designed tests. In recent years, interest in such models has g...

We propose a general cognitive diagnosis model framework to diagnose mental disorders using item scores obtained from clinical measurement instruments. This framework can be used to validate the extent to which the items measure the specific disorders. The method is illustrated using data obtained with the Dutch version of Millon Clinical Multiaxial Inventory-III.

This study aims to investigate the mastery profiles of digital literacy skills of Hong Kong primary students using a general cognitive diagnosis model (CDM) framework. In particular, the relationship between the mastery of each digital skill and a number of students' background characteristics is explored using a three-step approach. The current study analyzes data collected from 642 Grade 3 students in Hong Kong using a newly developed digital literacy assessment (DLA). CDMs are fitted to the data to determine students' mastery profiles of five digital skills, as well as test properties; subsequently latent logistic regression analyses were implemented to determine the relationship between skill mastery and the covariates. Results indicate that CDM analysis is an appropriate method to analyze the DLA performance data, which exhibited measurement invariance across gender and socioeconomic status (SES). Despite low mastery proportions for all digital skills, students' skill mastery can be accurately classified. Finally, the latent logistic regression results indicate that children's background characteristics (i.e., gender, educational aspiration, home language, SES, and access to digital devices) are differentially related to their mastery of each digital skill.

Cognitive diagnosis models (CDMs) are an increasingly popular method to assess mastery or nonmastery of a set of fine-grained abilities in educational or psychological assessments. Several inference techniques are available to quantify the uncertainty of model parameter estimates, to compare different versions of CDMs, or to check model assumptions. However, they require a precise estimation of the standard errors (or the entire covariance matrix) of the model parameter estimates. In this article, it is shown analytically that the currently widely used form of calculation leads to underestimated standard errors because it only includes the item parameters but omits the parameters for the ability distribution. In a simulation study, we demonstrate that including those parameters in the computation of the covariance matrix consistently improves the quality of the standard errors. The practical importance of this finding is discussed and illustrated using a real data example.

Research related to the fit evaluation at the item level involving cognitive diagnosis models (CDMs) has been scarce. According to the parsimony principle, balancing goodness of fit against model complexity is necessary. General CDMs require a larger sample size to be estimated reliably, and can lead to worse attribute classification accuracy than the appropriate reduced models when the sample size is small and the item quality is poor, which is typically the case in many empirical applications. The main purpose of this study was to systematically examine the statistical properties of four inferential item-fit statistics: [Formula: see text], the likelihood ratio (LR) test, the Wald (W) test, and the Lagrange multiplier (LM) test. To evaluate the performance of the statistics, a comprehensive set of factors, namely, sample size, correlational structure, test length, item quality, and generating model, is systematically manipulated using Monte Carlo methods. Results show that the [Formula: see text] statistic has unacceptable power. Type I error and power comparisons favor LR and W tests over the LM test. However, all the statistics are highly affected by the item quality. With a few exceptions, their performance is only acceptable when the item quality is high. In some cases, this effect can be ameliorated by an increase in sample size and test length. This implies that using the above statistics to assess item fit in practical settings when the item quality is low remains a challenge.

In this article, we describe and illustrate the process by which we developed and validated short, multiple-choice, reliable tests to assess undergraduate students’ comprehension of three mathematical proofs. We discuss the purpose for each stage and how it benefited the design of our instruments. We also suggest ways in which this process could be employed by other researchers to develop and validate their own reliable proof comprehension tests.

Nondichotomous response models have been of greater interest in recent years due to the increasing use of different scoring methods and various performance measures. As an important alternative to dichotomous scoring, the use of continuous response formats has been found in the literature. To assess finer-grained skills or attributes and to extract information with diagnostic value from continuous response data, a multidimensional skills diagnosis model for continuous response is proposed. An expectation-maximization implementation of marginal maximum likelihood estimation is developed to estimate its parameters. The viability of the proposed model is shown via a simulation study and a real data example. The proposed model is also shown to provide a substantial improvement in attribute classification when compared to a model based on dichotomized continuous responses.

In this paper, we are discussing the case of offering retired assessment items as practice problems for the purposes of learning in a system called ACT Academy. In contrast to computer-assisted learning platforms, where students consistently focus on small sets of skills they practice till mastery, in our case, students are free to explore the whole subject domain. As a result, they have significantly lower attempt counts per individual skill. We have developed and evaluated a student modeling approach that differs from traditional approaches to modeling skill acquisition by leveraging the hierarchical relations in the skill taxonomy used for indexing practice problems. Results show that when applied in systems like ACT Academy, this approach offers significant improvements in terms of predicting student performance.

Over the past few decades, cognitive diagnostic models have generated a lot of interest due in large part to the call made by the No Child Left Behind Act of 2001 (No Child Left Behind, Act of 2001 Public Law No. 107–110, § 115. Stat, 1425, 2002) for more formative assessments in learning systems. In this chapter, we provide an overview of learning and assessment systems, including the rise in popularity of online and personalized learning systems; we contrast the role of summative and formative assessments in learning systems; and we provide a review of cognitive diagnostic models and the challenges of retrofitting models to data not designed for cognitive diagnostic models.

The development of cognitive diagnosis models (CDMs) has been prolific since the turn of the century; however, they have often been developed in such a way that they lack an overall connective framework. The purpose of this chapter is to review the G-DINA framework. As a general model, it subsumes several simpler and widely-known CDMs; as a general framework, it has also served as the foundation for a variety of model extensions and new methodological developments. We will also discuss associated topics, which include model estimation, Q-matrix validation, computerized adaptive testing, and model selection as they relate to the reviewed models.

At present, there are only a limited number of studies examining how to optimally construct cognitive diagnostic tests. The cognitive diagnostic index (CDI) and attribute-level discrimination index (ADI) have been proposed to assemble such tests. The CDI and ADI have been shown to be instrumental in constructing cognitive diagnostic tests when the attribute relationships are assumed to be nonhierarchical. For greater generality when designing cognitive diagnostic assessment, attribute hierarchy and the ratio of test length to the number of attributes (RTA) are two important factors to be considered. This article proposes modified indices that take into account attribute hierarchy and RTA. Simulation studies show that, under the deterministic input, noisy, “and” gate model (DINA) and the reduced version of the reparameterized unified model (rRUM), the proposed indices provide higher attribute and attribute pattern correct classification rates than the original indices.

Selecting the most appropriate cognitive diagnosis model (CDM) for an item is a challenging process. Although general CDMs provide better model-data fit, specific CDMs have more straightforward interpretations, are more stable, and can provide more accurate classifications when used correctly. Recently, the Wald test has been proposed to determine at the item level whether a general CDM can be replaced by specific CDMs without a significant loss in model-data fit. The current study examines the practical consequence of the test by evaluating whether the attribute-vector classification based on CDMs selected by the Wald test is better than that based on general CDMs. Although the Wald test can detect the true underlying model for certain CDMs, it is yet unclear how effective it is at distinguishing among the wider range of CDMs found in the literature. This study investigates the relative similarity of the various CDMs through the use of the newly developed dissimiliarity index, and explores the implications for the Wald test. Simulations show that the Wald test cannot distinguish among additive models due to their inherent similarity, but this does not impede the ability of the test to provide higher correct classification rates than general CDMs, particularly when the sample size is small and items are of low quality. An empirical example is included to demonstrate the viability of the procedure.