Search

One critical issue with cognitive diagnostic assessment (CDA) lies in its lack of research evidence that shows how diagnostic feedback from CDA is interpreted and used by young students. This mixed methods research examined how holistic diagnostic feedback (HDF) is processed by young learners with different profiles of reading skills, goal orientations, and perceived ability. HDF provides three learner profiles: learners’ current skill mastery levels; self-assessed skill proficiency; and goal orientations. It also has a section for plans for future learning. A total of 44 Grades 5 and 6 students (aged 11–12) from two classrooms, their parents and teacher received individually customized HDF reports. Students’ reading skill mastery profiles were determined based on the application of cognitive diagnostic modeling to their performance on a provincial reading achievement measure, while their perceived ability and goal orientation profiles were created by using self-assessment and goal-orientation questionnaires. Students and parents provided written responses to their HDF reports.The study findings show the dynamic influence of young students’ profiles on the ways in which they perceive, interpret and use HDF. Students’ responses to diagnostic feedback did not differ substantially across reading mastery levels; however, psychological factors most strongly impacted the efficacy of learner feedback processing. Furthermore, the result that it was not students’ actual goal orientations but their perceived parent goal orientations that showed significant relationships with their skill mastery levels strongly indicates that young students’ responses to HDF are likely to be influenced by broader learning environments, and such influences are further filtered through their own perceptions. Understanding students’ interactions with diagnostic feedback is critical for maximizing its effect because their perceptions about ability and orientations to learning strongly influence the ways in which they process diagnostic feedback on their learning. [ABSTRACT FROM PUBLISHER]

Two examples demonstrate an argument-based approach to validation of diagnostic assessment using automated writing evaluation (AWE). Criterion®, was developed by Educational Testing Service to analyze students’ papers grammatically, providing sentence-level error feedback. An interpretive argument was developed for its use as part of the diagnostic assessment process in undergraduate university English for academic purposes (EAP) classes. The Intelligent Academic Discourse Evaluator (IADE) was developed for use in graduate EAP university classes, where the goal was to help students improve their discipline-specific writing. The validation for each was designed to support claims about the intended purposes of the assessments. We present the interpretive argument for each and show some of the data that have been gathered as backing for the respective validity arguments, which include the range of inferences that one would make in claiming validity of the interpretations, uses, and consequences of diagnostic AWE-based assessments. [ABSTRACT FROM PUBLISHER]

Alderson, Brunfaut and Harding (2014) recently investigated how diagnosis is practised across a range of professions in order to develop a tentative framework for a theory of diagnosis in second or foreign language (SFL) assessment. In articulating this framework, a set of five broad principles were proposed, encompassing the entire enterprise of diagnostic assessment. However, there remain questions about how best to implement these principles in practice, particularly in identifying learners’ strengths and weaknesses in the less well-documented areas of SFL reading and listening. In this paper, we elaborate on the set of principles by first outlining the stages of a diagnostic process built on these principles, and then discussing the implications of this process for the diagnostic assessment of reading and listening. In doing so, we will not only elaborate on the theory of diagnosis with respect to its application in the assessment of these skills, but also discuss the ways in which each construct might be defined and operationalized for diagnostic purposes. [ABSTRACT FROM PUBLISHER]

Diagnostic language assessment (DLA) is gaining a lot of attention from language teachers, testers, and applied linguists. With a recent surge of interest in DLA, there seems to be an urgent need to assess where the field of DLA stands at the moment and develop a general sense of where it should be moving in the future. The current article, as the first article in this special issue, aims to provide a general theoretical background for discussion of DLA and address some fundamental issues surrounding DLA. More specifically, the article (a) examines some of the defining characteristics of DLA and its major components, (b) reviews the current state of DLA in conjunction with these components, and (c) identifies some promising areas of future research and development of DLA where important breakthroughs can be made in the future. Some of the major obstacles and challenges facing DLA are identified and discussed, along with some possible solutions to them. [ABSTRACT FROM PUBLISHER]

Alderson (2005) suggests that diagnostic tests should identify strengths and weaknesses in learners' use of language and focus on specific elements rather than global abilities. However, rating scales used in performance assessment have been repeatedly criticized for being imprecise and therefore often resulting in holistic marking by raters (Weigle, 2002). The aim of this study is to compare two rating scales for writing in an EAP context; one `a priori' developed scale with less specific descriptors of the kind commonly used in proficiency tests and one empirically developed scale with detailed level descriptors. The validation process involved 10 trained raters applying both sets of descriptors to the rating of 100 writing scripts yielded from a large-scale diagnostic assessment administered to both native and non-native speakers of English at a large university. A quantitative comparison of rater behaviour was undertaken using FACETS. Questionnaires and interviews were administered to elicit the raters' perceptions of the efficacy of the two types of scales. The results indicate that rater reliability was substantially higher and that raters were able to better distinguish between different aspects of writing when the more detailed descriptors were used. Rater feedback also showed a preference for the more detailed scale. The findings are discussed in terms of their implications for rater training and rating scale development. [ABSTRACT FROM AUTHOR]

With recent statistical advances in cognitive diagnostic assessment (CDA), the CDA approach has been increasingly applied to non-diagnostic tests partly to meet accountability demands for student achievement. The study aimed to evaluate critically the validity of the CDA application to an existing non-diagnostic L2 reading comprehension test and to provide information about challenges and conditions for the CDA approach. Based on Jang's study (2005), this paper focuses on the dependability of the Fusion Model's skill profiling, the characteristics of resulting L2 skill profiles, and the diagnostic capacity of LanguEdge™ test items. In addition, the paper examines the validity arguments from the users' perspective by focusing on the usefulness of the diagnostic feedback. The results suggest that the CDA approach can provide more fine-grained diagnostic information about the level of competency in reading skills than traditional aggregated-test scoring can. While various empirical evidence supported the dependability of the skill profiling process, the results also raised some concerns about the application of the CDA approach to a test developed for non-diagnostic purposes, most significantly, a lack of diagnostic capacity of some of the test items with extremely easy or difficult levels. The results offer useful information about the potential challenges and conditions for future application of cognitive diagnostic assessment. [ABSTRACT FROM AUTHOR]