Search

Cognitive Diagnostic Assessment (CDA) is an alternative assessment which can give a clear picture of pupils’ learning process and cognitive structures to education stakeholders so that appropriate instructional strategies can be designed to tailored pupils’ needs. Coincide with this function, the Ordered Multiple-Choice (OMC) items were incorporated into the CDA developed in this study. This paper describes the process of developing and validating a Cognitive Diagnostic Assessment with OMC items for assessing pupils’ attribute mastery on ‘Addition of Time’ and reports the psychometric properties of the items, the model-data fit, the attribute reliability and the overall reliability of the assessment. The sample of the study consisted of 30 Year Four National School (SK) pupils, 48 Year Four National-Type Chinese School (SJKC) pupils and 12 Year Four National-Type Tamil School (SJKT) pupils in Penang state, Malaysia. The data was analysed using two measurement models, namely Classical Test Theory and Attribute Hierarchy Method (AHM). The findings of the study indicated that the instrument developed consisted of good-quality OMC items with appropriate difficulty and high discrimination power. With the satisfactory model-data fit in AHM, the inference made about pupils’ attributes mastery based on their performance in CDA with OMC items was valid. The CDA with OMC items developed in this study was found to be reliable at both attribute level and assessment level. Perhaps, this instrument could be used in the mathematics classrooms for supporting teachers in diagnosing pupils’ cognitive strengths and weaknesses for ‘Addition of Time’.

This paper describes the process of developing and validating an online cognitive diagnostic assessment (CDA) with ordered multiple-choice (OMC) items for assessing pupils’ mastery of attributes related to ‘Multiplication of Time’ and reports the empirical evidence regarding item quality, validity of the cognitive models and the reliability of the online CDA with OMC items. The sample of the study consisted of 90 Grade Four students in Penang state, Malaysia. The data were analysed using two measurement models, namely classical test theory and attribute hierarchy method. The findings of the study indicated that the instrument developed consisted of good quality OMC items with robust psychometric properties. The linear hierarchical structure of the three cognitive models was supported by moderate correlations among the attributes in each cognitive model as well as the decrease of mean attribute mastery probability with the increase of attribute complexity. With the satisfactory model-data fit, the inferences made about pupils’ attribute mastery based on their performance in the online CDA with OMC items were valid. The online CDA with OMC items developed in this study was found to be reliable at both the attribute and assessment levels. The online CDA with OMC items could be used in the mathematics classrooms for supporting teachers in diagnosing pupils’ cognitive strengths and weaknesses for ‘Multiplication of Time’.

Cognitive model [CM] is the description of individual’s problem-solving method on selected tasks at a precise level to make meaningful inference on his/her strengths and weaknesses. CM serves as a foundation in developing a cognitive diagnostic assessment [CDA] that links the interpretation of test score to the attributes. This study employed the CDA involving “Time” which was developed under the framework of assessment triangle and Attribute Hierarchy Method [AHM]. In this study, the expert-based cognitive model was generated by 7 experienced primary mathematics teachers while the student-based cognitive model was generated based on 259 Year 6 pupils’ responses in the CDA. The consistency between these cognitive models was calculated by using Hierarchical Consistency Index [HCI] and triangulated by pupils’ interview transcript. Hence, this paper aims to discuss the consistency of structure between the expert-based cognitive models and the student-based cognitive models. The extent of match between the expert-based and the student-based cognitive models is important as it determines the validity of the expert-based cognitive model from students’ perspective. In general, all the 3 expert-based cognitive models were excellently fit with the student-based cognitive models. However, the result shows that the cognitive models of those pupils who scored 50% and below exhibited poor fit to the expert-based cognitive models. Results of the study implied that the proposed CMs are able to predict pupils’ responses and there is a need to construct CM which is suitable for low ability pupils in order to help them in studying the topic of “Time.”

Artificial Neural Networks (ANNs) have been proposed as a promising approach for the classification of students into different levels of a psychological attribute hierarchy. Unfortunately, because ...

Nowadays, to investigate students’ potentiality, any exam has to contain a lot of questions. Therefore, using attribution hierarchy model will be able to reduce the number of questions. This study addresses the following research question: Which attributes are needed to be included in the diagnostic test and How much is the level of quality of the diagnostic test? The purpose of this study was to develop a diagnostic test and to quantify its quality. The study will concentrate in students’ mathematical problem-solving skill. Moreover, the created diagnostic test will limit to the skill of solving adding and subtracting fractions questions only. The diagnostic test will be developed based on the attribute hierarchy model. To quantify the level of ability of students, the Bayes’ Theorem was employed to determine the diagnostic score. Moreover, to quantifying the quality of the diagnostic test, 2-parameter item response model, content validity, Hoyt's reliability, and inter-rater reliability are used. The diagnostic test contained only 17 questions which are relatively low comparing to the current test. It is found that the difficulty parameter is between -2.14 and 1.06, the item discrimination parameter is 0.77 – 3.07, the content validity is 1.00, Hoyt's reliability is 0.84 and the inter-rater reliability is 0.90.

This paper discusses how cognitive diagnostic assessment can be used as an alternative assessment for learning the topic of “time” in primary mathematics. Cognitive diagnostic assessment is an assessment that can provide meaningful feedback on student’s cognitive strengths and weaknesses. The one described here was initially developed by the researchers and validated by a panel of seven expert primary mathematics teachers. It was then administrated to 127 Grade Six (12 years old) students. In this paper, several examples are given to illustrate how cognitive diagnostic assessment could be used as an assessment for learning. The limitations of using this approach as well as future research directions are also discussed.

Tujuan penelitian ini untuk: (1) mengembangkan instrumen ordered multiple-choice untuk diagnosis kesulitan belajar Stoikiometri kelas X, (2) menetapkan karakteristik dari instrumen, dan (3) membuat profil diagnostik peserta didik sebagai laporan yang informatif. Penelitian pengembangan ini menggunakan model Borg & Gall. Preliminary study menghasilkan pokok bahasan tersulit Stoikiometri. Penetapan atribut dan hierarki atribut dengan focus group discussion dilanjutkan dua putaran teknik Delphi. Indikator dan rumusan butir soal ditelaah expert untuk membuktikan validasi isi dengan rumus Aiken . Ujicoba terbatas dan ujicoba menengah pada sekolah kategori tinggi, sedang dan rendah. Ujicoba kelayakan dilakukan pada sekolah-sekolah di wilayah Surakarta, Karanganyar, Boyolali, dan Sragen. Kesimpulan: (1) penelitian berhasil mengembangkan tiga paket soal Ordered Multiple Choice (OMC) untuk mendeteksi kesulitan belajar peserta didik pada materi Stoikiometri kelas X; (2) butir-butir soal OMC pada paket A, B, dan C memiliki validitas konstruk yang baik dengan Goodness of Fit (GoF) berdasar Smart PLS > 0,36 yaitu 0,437; 0,466 dan 0,433. Instrumen memiliki reliabilitas tinggi berturut-turut 0,79; 0,81 dan 0,75; (3) profil peserta didik berupa laporan diagnostik tentang atribut yang sudah dikuasai dan belum dikuasai peserta didik. Kata kunci: Attribute Hierarchy Method, Ordered Multiple Choice, Graded Response Model ______________________________________________________________ DEVELOPING A DIAGNOSTIC INSTRUMENT FOR LEARNING DIFFICULTIES IN CHEMISTRY IN SECONDARY HIGH SCHOOL Abstract The purposes of this study are to: (1) develop Ordered Multiple Choice (OMC) instruments with a model of Attribute Hierarchy Method (AHM) for the diagnosis of stoichiometry learning difficulties in chemistry learning in secondary high school, (2) determine the characteristic of instruments which have been developed based on the Graded Response Model (GRM), (3) create a diagnostic profile of learners as an informative report. This development research used the Borg & Gall model. The development of the instrument is done using the AHM with the form of OMC. The determination of attributes and attribute hierarchy was done by the focus group discussion (FGD) by three experts, six teachers, and two measurement experts and continued with the Delphi technique by three experts with two rounds. A limited try out was conducted in the high, medium, and low category schools. A feasibility try out was conducted to the high, medium, and low category schools in the regions of Surakarta, Karanganyar, Boyolali, and Sragen. The results are as follows. 1). This research has developed three packages of OMC to detect the learning difficulties of students in the subjects of Chemistry, especially in Stoichiometry of class X. 2). The OMC test items on packages A, B, and C have good construct validity with the Goodness of Fit (GoF) greater than 0.36 namely 0.437, 0.466, and 0.433. 3). The learners’ profiles are created in the form of diagnostic report about the attributes which have been mastered and have not been mastered by the learners. Keywords: Attribute Hierarchy Method, Ordered Multiple Choice, Graded Response Model

The purpose of this study was to evaluate the adequacy of three cognitive models, one developed by content experts and two generated from student verbal reports for explaining examinee performance on a grade 3 diagnostic mathematics test. For this study, the items were developed to directly measure the attributes in the cognitive model. The performance of each cognitive model was evaluated by examining its fit to different data samples: verbal report, total, high-, moderate-, and low ability using the Hierarchy Consistency Index (Cui & Leighton, 2009), a model-data fit index. This study utilized cognitive diagnostic assessments developed under the framework of construct-centered test design and analyzed using the Attribute Hierarchy Method (Gierl, Wang, & Zhou, 2008; Leighton, Gierl, & Hunka, 2004). Both the expert-based and the student-based cognitive models provided excellent fit to the verbal report and high ability samples, but moderate to poor fit to the total, moderate and low ability samples. Impli...

This study aims to determine the learning difficulties of learners using the Attribute Hierarchy Method (AHM) with ordered multiple choices (OMC) model on chemical equilibrium. The Instruments develop used ADDIE (Analyze, Design, Development, Implementation and Evaluation) model. The research instrument used a multiple-choice test that has a score that all correct answer options. There are 8 types of attributes concerning chemical equilibrium in the instrument. The sample was high school students in Pekanbaru. Instrument validity includes material, construction, and language aspects. Based on the validity test results obtained that the developed instrument has been in valid category. Material validity aspect value 3.50, validity of the construction obtained value 3.7 and aspects of language value of 3.59. Instrument reliability test results using the formula of KR-20 obtained a value of 0.717 in high category. The learning difficulties of learners obtained that learners have the greatest learning attribute 6, which is the equilibrium constant of pressure (Kp) which has not been mastered by 58% of students 2.) Attribute 7 is the relationship between Kp and Kc that has not been mastered by 63.75% students, 3.) Attribute 4 is Le Chatelier’s principle which has not been mastered by 80% of students and 4.) Attribute 8 is the degree of dissociation that has not been mastered by 77.5% of students.

Diagnostic test has gained attention for its potentiality to produce fine-grained information about examinees. The dependency among attributes (i.e. attribute structure) is one of the most important factors affecting diagnostic test design. This article introduces four types of attribute structures and examines the effects of the attribute number, structure and level on classification accuracy and reliability. Results from the study help researchers and practitioners understand factors that affect classification when specifying attributes, and design diagnostic tests that provide accurate information about examinees.

Two new similarity indexes are introduced to explain the logic behind the classification methods A and B with attribute hierarchy model. Several kinds of new classification methods proposed are based on variations of the similarity between the expected response pattern and the observed response pattern. The results of simulation indicate that the new methods are better than methods A and B. And with the slips increasing the advantage is especially obvious.

The purpose of this study is to apply the attribute hierarchy method (AHM) to a subset of SAT critical reading items and illustrate how the method can be used to promote cognitive diagnostic inferences. The AHM is a psychometric procedure for classifying examinees’ test item responses into a set of attribute mastery patterns associated with different components from a cognitive model. The study was conducted in two steps. In step 1, three cognitive models were developed by reviewing selected literature in reading comprehension as well as research related to SAT Critical Reading. Then, the cognitive models were validated by having a sample of students think aloud as they solved each item. In step 2, psychometric analyses were conducted on the SAT critical reading cognitive models by evaluating the model-data fit between the expected and observed response patterns produced from two random samples of 2,000 examinees who wrote the items. The model that provided best data-model fit was then used to calculate attribute probabilities for 15 examinees to illustrate our diagnostic testing procedure.

Resumen:Dado que en la mayoría de las pruebas en el ámbito educativo las personas se involucran en algún tipo de resolución cognitiva de problemas, los enfoques psicométricos tradicionales tienen el inconveniente de no aportar información acerca de las estructuras de conocimiento ni de las habilidades subyacentes a la resolución de los ítems de una prueba. Por ello, se necesita de enfoques de evaluación que se orienten a aspectos cognitivos y que permitan diseñar pruebas e interpretar resultados en términos de los procesos y las habilidades propias de la resolución de problemas. En este artículo se presenta una de las opciones más recientes de la evaluación cognitiva diagnóstica: el Método de Jerarquía de Atributos (MJA), que puede ser visto como un modelo cognitivo de desempeño que resume un conjunto de conocimientos y habilidades interconectadas que las personas utilizan para responder correctamente los ítems de una prueba en un dominio específico. Este enfoque posibilita contar con métodos de evaluación orientados al mejoramiento del aprendizaje y proponer diversas técnicas de enseñanza basadas en la psicología del desempeño en la resolución de tareas.Abstract: Since most of educational tests require examinees to engage in some form of cognitive problem solving, traditional psychometric approaches have the disadvantage of not provide information about knowledge structures nor underlying skills to solve test items. Therefore, cognitive assessment approaches are required, which lead to design tests and to interpret results in terms of process and skills related to problem solving. This article presents one of the most recent options of cognitive diagnostic assessment: the Attribute Hierarchy Method (AHM), which can be viewed as a cognitive model of task performance; it summarizes the set of interconnected knowledge and skills that students use to respond correctly to test items within a specific domain. This approach allows for methods of assessment aimed to improve learning and to propose teaching situations based on the psychology of performance in problem solving.

The purpose of this study is to apply the attribute hierarchy method in an operational diagnostic mathematics program at Grades 3 and 6 to promote cognitive inferences about students’ problem-solving skills. The attribute hierarchy method is a psychometric procedure for classifying examinees’ test item responses into a set of structured attribute patterns associated with a cognitive model. Principled test design procedures were used to design the exam and evaluate the student response data. To begin, cognitive models were created by content specialists outlining the knowledge and skills required to solve mathematical tasks in Grades 3 and 6. Then, items were written specifically to measure the skills in the cognitive models. Finally, confirmatory psychometric analyses were used to evaluate the student response data by estimating model-data fit, attribute probabilities for diagnostic score reporting, and attribute reliabilities. Future directions for score reporting and adaptive diagnostic testing are disc...

This paper presents a framework to provide a structured approach for developing score reports for cognitive diagnostic assessments (CDAs). Guidelines for reporting and presenting diagnostic scores are based on a review of current educational test score reporting practices and literature from the area of information design. A sample diagnostic report is presented to illustrate application of the reporting framework in the context of one CDA procedure called the Attribute Hierarchy Method. Integration and application of interdisciplinary techniques from education, information design, and technology are required for effective score reporting. While the AHM is used in this paper, this framework is applicable to any attribute-based diagnostic testing method.

The attribute hierarchy method (AHM) is a psychometric procedure for classifying examinees’ test item responses into a set of structured attribute patterns associated with different components from a cognitive model of task performance. Results from an AHM analysis yield information on examinees’ cognitive strengths and weaknesses. Hence, the AHM can be used for cognitive diagnostic assessment. The purpose of this study is to introduce and evaluate a new concept for assessing attribute reliability using the ratio of true score variance to observed score variance on items that probe specific cognitive attributes. This reliability procedure is evaluated and illustrated using both simulated data and student response data from a sample of algebra items taken from the March 2005 administration of the SAT. The reliability of diagnostic scores and the implications for practice are also discussed.

The objective of the present investigation was to compare the adequacy of two cognitive models for predicting examinee performance on a sample of algebra I and II items from the March 2005 administration of the SAT™. The two models included one generated from verbal reports provided by 21 examinees as they solved the SAT™ items, and the other generated from the judgment of a content expert. Using the attribute hierarchy method (Leighton, Gierl, & Hunka, 2004) and the hierarchy consistency index (Cui, Leighton, Gierl, & Hunka, 2006), the predictive adequacy of the two cognitive models was tested with three samples of varying size. Results indicated that the student-based model predicted the test performance of examinees of moderately high ability with excellent accuracy but not the test performance of examinees of moderate (or average) ability. In contrast, the expert-based model was more consistent, predicting the test performance of both moderately high ability examinees and moderate ability examinees wi...

The attribute hierarchy method (AHM) applied to assessment engineering is described. It is a psychometric method for classifying examinees’ test item responses into a set of attribute mastery patterns associated with different components in a cognitive model of task performance. Attribute probabilities, computed using a neural network, can be estimated for each examinee thereby providing specific information about the examinee’s attribute-mastery level. The pattern recognition approach described in this study relies on an explicit cognitive model to produce the expected response patterns. The expected response patterns serve as the input to the neural network. The model also yields the cognitive test specifications. These specifications identify the examinees’ attribute patterns which are used as output for the neural network. The purpose of the statistical pattern recognition analysis is to estimate the probability that an examinee possess specific attribute combinations based on their observed item response patterns. Two examples using student response data from a sample of algebra items on the SAT illustrate our pattern recognition approach.

The purpose of this study is to describe how the attribute hierarchy method (AHM) can be used to evaluate differential group performance at the cognitive attribute level. The AHM is a psychometric method for classifying examinees’ test item responses into a set of attribute-mastery patterns associated with different components in a cognitive model of task performance. Attribute probabilities, computed using a neural network, can be estimated on each attribute for each examinee thereby providing specific information about the examinee’s attribute-mastery level. These probabilities can also be compared across groups. We describe a four-step procedure for estimating and interpreting group differences using the AHM. We also provide an example using student response data from a sample of algebra items on the SAT to illustrate our pattern recognition approach for studying group differences. Assessment engineering (AE) is emerging as a new research area in educational and psychological measurement (Luecht, 2006a, 2006b). AE is an innovative approach to measurement, where engineering-like principles are used to direct the design and the analysis of assessments as well as the scoring and the reporting of the results. With this approach, an assessment begins with specific, empirically derived cognitive models (e.g., Leighton & Gierl, 2007). Next, assessment task templates are created using established frameworks derived from the cognitive model to produce test items. Finally, psychometric methods are applied to the examinee response data, typically in a confirmatory mode, to produce scores that are interpretable (Luecht, Gierl, Tan, & Huff, 2006). AE differs from more traditional approaches to test design and analysis in three fundamental ways. First, cognitive models guide task design and item development, rather than content-based test specifications. While the categories in content blueprints can be included in the task templates, the assessment principles used to develop items are based on cognitive principles and, thus, provide more specific information for measuring problem-solving skills. Second, task templates are created to control and manipulate both the content and cognitive attributes of the items. Item writers are required to use the templates during development, thereby producing items that adhere to strict quality controls and that meet high psychometric standards. Third, psychometric models are employed in a confirmatory, versus exploratory, manner to assess the model-data fit relative to the intended underlying structure of the constructs or traits the test is designed to measure. The outcomes from these model-data fit analyses also provide developers with guidelines for

The purpose of this paper is to describe the logic and identify key assumptions associated with making cognitive inferences using two attribute-based psychometric methods. The first method is Kikumi Tatsuoka's rule-space model. This model provides a strong point of reference for studying the nature of diagnostic inferences because it is important in the evolution of skills diagnostic testing and it is well documented. The second method is a new procedure called the attribute hierarchy method that was developed from the rule-space approach. Although the attribute hierarchy method shares many commonalities with rule space, it represents an extension by including an attribute hierarchy that serves as an explicit cognitive model of task performance designed to link psychometric practices with contemporary cognitive theories. In this paper, we describe and compare these two attribute-based psychometric methods and identify new directions for research and practice in skills diagnostic testing.

The purpose of this paper is to define and evaluate the categories of cognitive models underlying at least three types of educational tests. We argue that while all educational tests may be based—explicitly or implicitly—on a cognitive model, the categories of cognitive models underlying tests often range in their development and in the psychological evidence gathered to support their value. For researchers and practitioners, awareness of different cognitive models may facilitate the evaluation of educational measures for the purpose of generating diagnostic inferences, especially about examinees' thinking processes, including misconceptions, strengths, and/or abilities. We think a discussion of the types of cognitive models underlying educational measures is useful not only for taxonomic ends, but also for becoming increasingly aware of evidentiary claims in educational assessment and for promoting the explicit identification of cognitive models in test development. We begin our discussion by defining the term cognitive model in educational measurement. Next, we review and evaluate three categories of cognitive models that have been identified for educational testing purposes using examples from the literature. Finally, we highlight the practical implications of “blending” models for the purpose of improving educational measures.

A cognitive item response theory model called the attribute hierarchy method (AHM) is introduced and illustrated. This method represents a variation of Tatsuoka's rule-space approach. The AHM is designed explicitly to link cognitive theory and psychometric practice to facilitate the development and analyses of educational and psychological tests. The following are described: cognitive properties of the AHM; psychometric properties of the AHM, as well as a demonstration of how the AHM differs from Tatsuoka's rule-space approach; and application of the AHM to the domain of syllogistic reasoning to illustrate how this approach can be used to evaluate the cognitive competencies required in a higher-level thinking task. Future directions for research are also outlined.

In educational assessment, we observe what students say, do, or make in a few particular circumstances and attempt to infer what they know, can do, or have accomplished more generally. A web of inference connects the two. Some connections depend on theories and experience concerning the targeted knowledge in the domain, how it is acquired, and the circumstances under which people bring their knowledge to bear. Other connections may depend on statistical models and probability-based reasoning. Still others concern the elements and processes involved in test construction, administration, scoring, and reporting. This article describes a framework for assessment that makes explicit the interrelations among substantive arguments, assessment designs, and operational processes. The work was motivated by the need to develop assessments that incorporate purposes, technologies, and psychological perspectives that are not well served by familiar forms of assessments. However, the framework is equally applicable to a...

The neural networks based attribute hierarchy method (NN-AHM) is promoted in this study, it could be applied in cognitive diagnosis. AHM could classify examinees' responses into a set of attribute mastery patterns. Very large sample is needed for AHM, and it is hard to describe the examinee master degree for each attribution. The generalized regression neural network is used in this study to estimate the attribute probabilities in small sample to provide information about the examinee's attribute-mastery level.

Contents: Preface. S.F. Chipman, P.D. Nichols, R.L. Brennan, Introduction. A.T. Corbett, J.R. Anderson, A.T. O'Brien, Student Modeling in the ACT Programming Tutor. R.J. Mislevy, Probability-Based Inference in Cognitive Diagnosis. D.H. Gitomer, L.S. Steinberg, R.J. Mislevy, Diagnostic Assessment of Troubleshooting Skill in an Intelligent Tutoring System. K.L. Draney, P. Pirolli, M. Wilson, A Measurement Model for a Complex Cognitive Skill. T.A. Polk, K. VanLehn, D. Kalp, ASPM2: Progress Toward the Analysis of Symbolic Parameter Models. J. Martin, K. VanLehn, A Bayesian Approach to Cognitive Assessment. G. Biswas, S.R. Goldman, D. Fisher, B. Bhuva, G. Glewwe, Assessing Design Activity in Complex CMOS Circuit Design. D. DuBois, V.L. Shalin, Adapting Cognitive Methods to Real-World Objectives: An Application to Job Knowledge Testing. P.J. Johnson, T.E. Goldsmith, K.W. Teague, Similarity, Structure, and Knowledge: A Representational Approach to Assessment. B.K. Britton, P. Tidwell, Cognitive Structure Testing: A Computer System for Diagnosis of Expert-Novice Differences. M. Naveh-Benjamin, Y-G. Lin, W.J. McKeachie, Inferring Students' Cognitive Structures and Their Development Using the "Fill-in-the-Structure" (FITS) Technique. J.E. Corter, Using Clustering Methods to Explore the Structure of Diagnostic Tests. K.K. Tatsuoka, Architecture of Knowledge Structures and Cognitive Diagnosis: A Statistical Pattern Recognition and Classification Approach. L.V. DiBello, W.F. Stout, L.A. Roussos, Unified Cognitive/Psychometric Diagnostic Assessment Likelihood-Based Classification Techniques. F. Samejima, A Cognitive Diagnosis Method Using Latent Trait Models: Competency Space Approach and Its Relationship With DiBello and Stout's Unified Cognitive-Psychometric Diagnosis Model. E. Hunt, Where and When to Represent Students This Way and That Way: An Evaluation of Approaches to Diagnostic Assessment. S.P. Marshall, Some Suggestions for Alternative Assessments.

Cognitive diagnostic assessment (CDA) for education can be recognized as an effective data mining approach in education research. In CDA, one of the critical steps is how to classify correctly the examinee in the most probable of the attribute mastery patterns or classes. The article mainly discussed and compared two classification methods: method A based on the attribute hierarchy method and the maximum a posteriori estimation of the DINA model. The Monte Carlo simulation results showed that the DINA model outperforms the AHM in terms of the correct classification rate for the entire patterns; however, for some patterns with larger test information conditioned on associated abilities, AHM performs better. Further, a rule of combining the results of two methods was proposed. Thus, one can exploit their individual advantages in order to reach an overall better performance than could be achieved by using each of them separately.

Rule space model (RSM) and attribute hierarchy method (AHM) have been research focuses in cognitive diagnosis. Reduced Q-matrix (incidence matrix between attributes and items), which named Qr matrix, plays an important role in RSM and AHM. The column vector of Qr matrix should satisfy an attribute hierarchy, and the vector is called valid item. Studying attribute hierarchy, the judgment method of valid item is proposed. The method is applied to RSM and AHM, and empirical evidence showed that it performed very well.

INTRODUCTION Cognitive diagnostic assessment (CDA) is increasingly a major focus in psychological and educational measurement. Instead of inferring a general response tendency or behavior consistency of an examinee over a target domain of measurement, diagnostic assessment results provide a detailed account of the underlying cognitive basis of the examinee's performance by mining the richer information that is afforded by specific response patterns. Sophisticated measurement procedures, such as the rule-space methodology (Tatsuoka, 1995), the attribute hierarchy method (Leighton, Gierl, & Hunka, 2004), the tree-based regression approach (Sheehan, 1997a, 1997b), and the knowledge space theory (Doignon & Falmagne, 1999), as well as specially parameterized psychometric models (De La Torre & Douglas, 2004; DiBello, Stout, & Roussos, 1995; Draney, Pirolli, & Wilson, 1995; Hartz, 2002; Junker & Sijtsma, 2001; Maris, 1999), have been developed for inferring diagnostic information. Although measurement models for diagnostic testing have become increasingly available, cognitive diagnosis must be evaluated by the same measurement criteria (e.g., construct validity) as traditional trait measures. With the goal of inferring more detailed information about an individual's skill profile, we are not just concerned about how many items have been correctly solved by an examinee. We are also concerned about the pattern of responses to items that differ in the knowledge, skills, or cognitive processes required for solution. Similar to traditional tests, empirical evidence and theoretical rationales that elaborate the underlying basis of item responses are required to support the inferences and interpretations made from diagnostic assessments.

With the current push toward educational reform, there is great potential for innovation and change, particularly in large scale testing. One area where change is possible is in cognitive diagnostic assessment. Researchers in educational measurement and cognitive psychology are finally in a position to design tests targeted specifically for providing valuable information about students' cognitive strengths and weaknesses. This self-contained volume organizes what is known about cognitive diagnostic assessment in education, including its conceptual and philosophical basis, methods, and applications. The complete list of topics includes educational demand, philosophical rationale, construct validity, cognitive methods, test construction, statistical models, and unresolved issues (e.g., how to best translate diagnostic information into teaching practices). Leighton and Gierl present a comprehensive and up-to-date examination of cognitive diagnostic assessment in education.

In this paper, combined with Tatsuoka 's Rule Space Model and the Attribute hierarchy Method (AHM), Formal Concept Analysis (FCA) is firstly applied into Computerized Adaptive Testing (CAT). Based on FCA, an initiatory and systemic CAT with cognitive diagnosis (CD) is designed, which diagnoses examinees real time. The result of Monte Carlo study shows that examinees ' knowledge states are well diagnosed and the precision in examinees ' abilities estimation is satisfied.

Publisher Summary A cognitive diagnostic assessment (CDA) is designed to model examinees' cognitive performances and yield specific information about their strengths and weaknesses. While most psychometric models are based on latent trait theories, CDA requires a cognitive information processing approach to model the psychology of test performance because the score inference is specifically targeted to the examinees' cognitive skills. This chapter describes the attribute hierarchy method (AHM) to illustrate the explicit modeling of cognitive performance using an information processing approach. The AHM is a psychometric method for classifying examinees' test item responses into a set of structured attribute patterns associated with different components specified in a cognitive model of task performance. The cognitive model is presented as a hierarchy of structured attributes. The hierarchy defines the psychological ordering among the attributes required to solve a test problem, and thus serves as the cognitive model of task performance. However, if the challenges in linking cognitively-based models and psychometric methods are to be realized, then a more productive relationship between cognitive psychologists and psychometricians is also required.

Several authors from various disciplines such as cognitive psychology, artificial intelligence, and psychometrics have developed rule assessment methods for diagnosing students' misconceptions. Siegler's (1976, 1978) binary decision tree method was illustrated in the context of the problem of balance scales and Anderson's (1974, 1981) functional method was developed for diagnosing errors occurring during the operation of integrating functional rules. A group of artificial intelligence researchers (Brown & Burton, 1978; Brown & VanLehn, 1980; VanLehn, 1981) developed a computer program "DEBUGGY" that can diagnose a number of erroneous rules resulting from misconceptions ("bugs") in whole number subtraction problems. These investigations were motivated by an interest in the basic foundations of knowledge structure and development in human cognition. Tatsuoka and her associates (Tatsuoka, Baillie, & Yamamoto, 1982; Tatsuoka, Birenbaum, Tatsuoka, & Baillie, 1980) also developed a computer program that can diagnose a number of erroneous rules in signed-number addition and subtraction problems, but they were motivated primarily by the exploration of psychometric properties of bugs such as changes in error types or the stability of misconceptions committed by a student throughout a test (Birenbaum & Tatsuoka, 1980; Tatsuoka, 1981; Tatsuoka & Tatsuoka, 1983). Analysis of misconceptions can provide useful information in evaluating instruction or instructional materials as well as specific prescriptions for planning remediation for a student. For example, the source of many of the misconceptions committed by students is often the ambiguity of explanations or the lack of precise, accurate instructions in teaching material. It is useful to have such computer programs in educational practice. However, constructing a DEBUGGY-type system for domains more general than arithmetic is extremely difficult and time consuming. On the other hand, personal indices that are designed to summarize response patterns can be used for detecting anomalous patterns (Drasgow, 1982; Harnisch & Linn, 1981; Levine & Rubin, 1979; Sato, 1975; Tatsuoka & Tatsuoka, 1982; Wright & Stone, 1979) which may result from applications of some erroneous rules, but they have only limited power. For instance, these indices cannot diagnose sources of misconceptions or provide prescriptive information for remediating them. Nevertheless, the use of personal indices is economical and applicable to general areas because such indices can first be used to spot candidates among students who may possibly possess misconceptions and may hence need personal attention from teachers. This procedure seems to be widely used in Japan (Sato, 1982). This study develops a model to fill the gap between the rule-assessing methods and the personal-index approach. The number of different erroneous rules already discovered in