Search

Many university leaders and faculty have the goal of promoting learning that connects across domains and prepares students with skills for their whole lives. However, as assessment emerges in higher education, many assessments focus on knowledge and skills that are specific to a single domain. Reworking assessment in higher education to focus on more robust learning is an important step towards making assessment match the goals of the context where it is being applied. In particular, assessment should focus on whether learning is robust (Koedinger, Corbett, & Perfetti, 2012), whether learning occurs in a way that transfers, prepares students for future learning, and is retained over time; and also on skills and meta--competencies that generalize across domains. By doing so, we can measure the outcomes that we as educators want to create, and increase the chance that our assessments help us to improve the outcomes we wish to create. In this article, we discuss and compare both traditional test--based methods for assessing robust learning, and new ways of inferring robustness of learning while the learning itself is occurring, comparing the methods within the domain of college genetics.

Despite the accumulation of substantial cognitive science research relevant to education, there remains confusion and controversy in the application of research to educational practice. In support of a more systematic approach, we describe the Knowledge-Learning-Instruction (KLI) framework. KLI promotes the emergence of instructional principles of high potential for generality, while explicitly identifying constraints of and opportunities for detailed analysis of the knowledge students may acquire in courses. Drawing on research across domains of science, math, and language learning, we illustrate the analyses of knowledge, learning, and instructional events that the KLI framework affords. We present a set of three coordinated taxonomies of knowledge, learning, and instruction. For example, we identify three broad classes of learning events (LEs): (a) memory and fluency processes, (b) induction and refinement processes, and (c) understanding and sense-making processes, and we show how these can lead to different knowledge changes and constraints on optimal instructional choices. (Contains 9 notes, 6 tables, and 1 figure.)

This paper evaluates the student modeling procedure in the ACT Programming Tutor (APT). APT is a practice environment that provides assistance to students as they write short programs. The tutor is constructed around a set of several hundred programming rules called the ideal student model, that allows the program to solve exercises along with the student. As the student works the tutor maintains an estimate of the probability that the student has learned the rules in the ideal model, in a process we call knowledge tracing. The cognitive model, and the learning and performance assumptions that underlie knowledge tracing are described. The assumptions that underlie knowledge tracing also yield performance predictions. These predictions provide a good fit to students' performance in completing tutor exercises, but a more important issue is how well the model predicts students' performance outside the tutor enviroimient. A previous study showed that the model provides a good fit to average posttest performance across students, but is less sensitive to individual differences. This paper describes a method of individualizing learning and performance estimates on-line in the tutor and assesses the validity of the resulting performance predictions.

The ACT Programming Tutor provides assistance to students as they write short computer programs. The tutor is constructed around a set of several hundred programming rules that allows the tutor to solve exercises step-by-step along with the student. This p^)er evaluates the Oitor's student modeling jx^ocedure that is used to guide remediation. This procedure, termed knowledge tracing, employs an overlay of the tutor's programming rules. In knowledge tracing, the tutor maintains an estimate of the probability that the student has learned each of the rules. The probability for a rule is updated at each opportunity to apply the rule, based on the student's performance. The predictive validity of the modeling procedure for tutw performance accuracy and posttest performance accuracy is assessed. Individual differences in learning parameters and cognitive rules are discussed, along with possible improvements in the modeling procedure.

Control and content of feedback was manipulated as students practiced coding functions with the Lisp Four feedback conditions were employed: (1) immediate error feedback and correction, (2) immediate error flagging but immediate correction not required (3) feedback on demand aixl (4) no tutorial assistance. The wide range in feedback conditions did not affect mean learning rate as measured by individual production firing, time to complete the exercises or post-test performance. However, post-test results were more highly correlated with student ability as tutorial assistance decreased across conditions. Feedback conditions also affected students' monitoring of the learning process. Across groups, students found the material was easier and bcUeved they had learned it better as assistance decreased across conditions. However, students w h o received more assistance estimated their mastery of the material more accurately. Finally, students reported relatively little preference for one tutoring condition over the others. Students w h o could exercise the most control over feedback reacted fairly passively to the tutoring conditions; students in condition 3 tended not to ask for much help and students in condition 2 tended to correct error immediately although it was not required.

Recent research has extended student modeling to infer not just whether a student knows a skill or set of skills, but also whether the student has achieved robust learning--learning that enables the student to transfer their knowledge and prepares them for future learning (PFL). However, a student may fail to have robust learning in two fashions: they may have no learning, or they may have shallow learning (learning that applies only to the current skill, and does not support transfer or PFL). Within this paper, we present automated detectors which identify shallow learners, who are likely to need different intervention than students who have not yet learned at all. These detectors are developed using K* machine learned models, with data from college students learning introductory genetics from an intelligent tutoring system.

Recently, there has been growing emphasis on supporting robust learning within intelligent tutoring systems, assessed by measures such as transfer to related skills, preparation for future learning, and longer term retention. It has been shown that different pedagogical strategies promote robust learning to different degrees. However, the student modeling methods embedded within intelligent tutoring systems remain focused on assessing basic skill learning rather than robust learning. Recent work has proposed models, developed using educational data mining, that infer whether students are acquiring learning that transfers to related skills, and prepares the student for future learning (PFL). In this earlier work, evidence was presented that these models achieve superior prediction of robust learning to what can be achieved by traditional methods for student modeling. However, using these models to drive intervention by educational software depends on evidence that these models remain effective within new populations. To this end, we analyze the degree to which these detectors remain accurate for an entirely new population of high school students. We find limited evidence of degradation for transfer. More degradation is seen for PFL. This degradation appears to occur in part because it is generally more difficult to infer this construct within the new population.

Algebra students studied either static-table, static-graphics, or interactive-graphics instructional worked examples that alternated with Algebra Cognitive Tutor practice problems. A control group did not study worked examples but solved both the instructional and practice problems on the Cognitive Tutor (CT). Students in the control group requested fewer hints and made fewer errors on the CT practice problems but required more learning time on the instructional examples. There was no difference among the four groups in constructing equations on a paper-and-pencil posttest or on a delayed test that included training and transfer problems. However, students who studied worked examples with a table were best at identifying the meaning of the equation components. The concept of transfer-appropriate processing (the overlap between instructional task and assessment task) aided our interpretation of the findings. Although the CT had a short-term effect on reducing errors and hint requests on CT practice problems, the worked examples were as effective on delayed paper-and-pencil tests. The subsequent construction of a new module for the Animation Tutor (Reed and Hoffman, Animation Tutor: Mixtures. Instructional software, 2011) used both the interactive-graphics and static-table worked examples to take advantage of the complementary strengths of different representations (Ainsworth, Learn Instr 16:183-198, 2006).

Genetics is a unifying theme of biology that poses a major challenge for students across a wide range of post-secondary institutions, because it entails complex problem solving. This article reports a new intelligent learning environment called the Genetics Cognitive Tutor, which supports genetics problem solving. The tutor presents complex, multi-step problems and is constructed around a "cognitive model" of the knowledge needed to solve the problems. This embedded cognitive model enables the tutor to provide step-by-step assistance, and to maintain a model of the student's problem-solving knowledge. The tutor consists of 16 modules with about 125 total problems, spanning five general topics: Mendelian inheritance, pedigree analysis, genetic mapping, gene regulation, and population genetics. This article reports two evaluations of the tutor. A pretest/posttest evaluation of student learning gains for individual tutor modules across multiple colleges and universities yielded average gains equivalent to almost two letter grades, and the accuracy of student modeling in predicting students' test performance was empirically validated. (Contains 3 tables, 4 figures and 1 footnote.)

In recent years there has been increasing interest in the phenomena of "gaming the system," where a learner attempts to succeed in an educational environment by exploiting properties of the system's help and feedback rather than by attempting to learn the material. Developing environments that respond constructively and effectively to gaming depends upon understanding why students choose to game. In this article, we present three studies, conducted with two different learning environments, which present evidence on which student behaviors, motivations, and emotions are associated with the choice to game the system.We also present a fourth study to determine how teachers' perspectives on gaming behavior are similar to, and different from, researchers' perspectives and the data from our studies. We discuss what motivational and attitudinal patterns are associated with gaming behavior across studies, and what the implications are for the design of interactive learning environment. (Contains 3 figures and 2 tables.)

A year-long study of 131 second and third graders in 12 classrooms compared three daily 20-minute treatments. a) Fifty-eight students in six classrooms used the 1999-2000 version of Project LISTEN's Reading Tutor, a computer program that uses automated speech recognition to listen to a child read aloud, and gives spoken and graphical assistance. Students took daily turns using one shared Reading Tutor in their classroom while the rest of their class received regular instruction. b) Thirty-four students in the other six classrooms were pulled out daily for one-on-one tutoring by certified teachers. To control for materials, the human tutors used the same set of stories as the Reading Tutor. c) Thirty-nine students served as in-classroom controls, receiving regular instruction without tutoring. We compared students' pre- to post-test gains on the Word Identification, Word Attack, Word Comprehension, and Passage Comprehension subtests of the Woodcock Reading Mastery Test, and in oral reading fluency. Surprisingly, the human-tutored group significantly outgained the Reading Tutor group only in Word Attack (main effects p less than 0.02, effect size 0.55). Third graders in both the computer- and human-tutored conditions outgained the control group significantly in Word Comprehension (p less than 0.02, respective effect sizes 0.56 and 0.72) and suggestively in Passage Comprehension (p = 0.14, respective effect sizes 0.48 and 0.34). No differences between groups on gains in Word Identification or fluency were significant. These results are consistent with an earlier study in which students who used the 1998 version of the Reading Tutor outgained their matched classmates in Passage Comprehension (p = 0.11, effect size 0.60), but not in Word Attack, Word Identification, or fluency. To shed light on outcome differences between tutoring conditions and between individual human tutors, we compared process variables. Analysis of logs from all 6,080 human and computer tutoring sessions showed that human tutors included less rereading and more frequent writing than the Reading Tutor. Micro-analysis of 40 videotaped sessions showed that students who used the Reading Tutor spent considerable time waiting for it to respond, requested help more frequently, and picked easier stories when it was their turn. Human tutors corrected more errors, focused more on individual letters, and provided assistance more interactively, for example getting students to sound out words rather than sounding out words for students as the Reading Tutor did. (Contains 10 tables.)