Search

An important component of many Adaptive Instructional Systems (AIS) is a 'Learner Model' intended to track student learning and predict future performance. Predictions from learner models are frequently used in combination with mastery criterion decision rules to make pedagogical decisions. Important aspects of learner models, such as learning rate and item difficulty, can be estimated from prior data. A critical function of AIS is to have students practice new content once the AIS predicts that they have 'mastered' current content or learned it to some criterion. For making this prediction, individual student parameters (e.g., for learning rate) are frequently unavailable due to having no prior data about a student, and thus population-level parameters or rules-of-thumb are typically applied instead. In this paper, we will argue and demonstrate via simulation and data analysis that even in best-case scenarios, learner models assuming equal learning rates for students will inevitably lead to systematic errors that result in suboptimal pedagogical decisions for "most" learners. This finding leads us to conclude that systematic errors should be expected, and mechanisms to adjust predictions to account for them should be included in AIS. We introduce two solutions that can adjust for student differences "online" in a running system: one that tracks systemic errors of the learner model (not the student) and adjusts accordingly, and a student-level performance adaptive feature. We demonstrate these solutions' efficacy and practicality on six large educational datasets and show that these features improved model accuracy in all tested datasets. [For the corresponding Grantee Submission, see ED629726.]

The Mobile Fact and Concept Textbook System (MoFaCTS) is an individualized online tutoring system designed to increase information comprehension and retention. It is being implemented in community college anatomy and physiology (A&P) courses for further system development. A&P was selected because it is a very challenging and highly in demand course. MoFaCTS was used to create Cloze (fill-in-the-blank) questions from the course text which students use to practice the course material. The system provides correction and feedback to the student, repeating questions to improve recall. The system also produces detailed progress reports for both faculty and students. A survey given to students showed a moderately positive impression of the systems, with A&P II students responding more positively on some survey items.

Adaptive learning technology solutions often use a learner model to trace learning and make pedagogical decisions. The present research introduces a formalized methodology for specifying learner models, logistic knowledge tracing (LKT), that consolidates many extant learner modeling methods. The strength of LKT is the specification of a symbolic notation system for alternative logistic regression models that is powerful enough to specify many extant models in the literature and many new models. To demonstrate the generality of LKT, we fit 12 models, some variants of well-known models and some newly devised, to six learning technology datasets. The results indicated that no single learner model was best in all cases, further justifying a broad approach that considers multiple learner model features and the learning context. We introduce features to stand in for student-level intercepts and argue that to be maximally applicable, a learner model needs to adapt to student differences. The results of our comparisons show the general importance of modeling recent learning for all datasets, with special importance for terms that model memory in datasets involving fact learning. [This article was published in "IEEE Transactions on Learning Technologies" (EJ1324365).]

Despite considerable advances in knowledge tracing algorithms, educational technologies that use this technology typically continue to use older algorithms, such as Bayesian Knowledge Tracing. One key reason for this is that contemporary knowledge tracing algorithms primarily infer next-problem correctness in the learning system, but do not attempt to infer the knowledge the student can carry out of the system, information more useful for teachers. The ability of knowledge tracing algorithms to predict problem correctness using data from intelligent tutoring systems has been extensively researched, but data from outcomes other than next-problem correctness have received less attention. In addition, there has been limited use of knowledge tracing algorithms in games, because algorithms that do attempt to infer knowledge from answer correctness are often too simple to capture the more complex evidence of learning within games. In this study, data from a digital learning game, (anonymized), was used to compare ten knowledge tracing algorithms' ability to predict students' knowledge carried outside the learning system--measured here by posttest scores--given their game activity. All Opportunities Averaged (AOA), a method proposed by Authors (2020) was used to convert correctness predictions to knowledge estimates, which were also compared to the built-in estimates from algorithms that produced them. Although statistical testing was not feasible for these data, three algorithms tended to perform better than the others: Dynamic Key-Value Memory Networks, Logistic Knowledge Tracing, and a multivariate version of Elo. Algorithms' built-in estimates of student ability underperformed estimates produced by AOA, suggesting that some algorithms may be better at estimating performance than ability. Theoretical and methodological challenges related to comparing knowledge estimates with hypothesis testing are also discussed.

Lectures containing information presented in verbal and visual form are widely used in educational settings. To date, little is known about how the visual display in a video lecture influences immediate and delayed recall. In two experiments, participants listened to a pre-recorded lecture while viewing an animated concept map, sequential listing of bulleted text, or there was no visual display. Results indicated that the concept map lecture enhanced immediate recall accuracy; however, the type of ideas recalled is also important to consider. Compared to the listen-only lecture, the concept map and bulleted text lectures promoted more recall of main ideas rather than details. Finally, the concept map lecture mainly benefitted those with high prior science achievement. These findings suggest that the visual display in a video lecture may affect the type of ideas remembered, and prior knowledge should be considered when choosing how to present information in a video lecture.

Decades of research has shown that spacing practice trials over time can improve later memory, but there are few concrete recommendations concerning how to optimally space practice. We show that existing recommendations are inherently suboptimal due to their insensitivity to time costs and individual- and item-level differences. We introduce an alternative approach that optimally schedules practice with a computational model of spacing in tandem with microeconomic principles. We simulated conventional spacing schedules and our adaptive model-based approach. Simulations indicated that practicing according to microeconomic principles of "efficiency" resulted in substantially better memory retention than alternatives. The simulation results provided quantitative estimates of optimal difficulty that differed markedly from prior recommendations but still supported a desirable difficulty framework. Experimental results supported simulation predictions, with up to 40% more items recalled in conditions where practice was scheduled optimally according to the model of practice. Our approach can be readily implemented in online educational systems that adaptively schedule practice and has significant implications for millions of students currently learning with educational technology. [Published in partnership with The University of Queensland.]

In recent years, there has been a proliferation of adaptive learner models that seek to predict student correctness. Improvements on earlier models have shown that separate predictors for prior successes, failures, and recent performance further improve fit while remaining interpretable. However, students who engage in "gaming" or other off-task behaviors may reduce the predictiveness of learner models that treat counts of prior performance equivalently across gaming and non-gaming student populations. The present research evaluated how sub-groups of students that varied in their potential gaming behavior were differently fit by a logistic learner model, and whether any observed differences between sub-groups could inspire the creation of new predictors that might improve model fit. Student data extracted from a college-level online learning application were clustered according to speed and accuracy using Gaussian mixture modeling. Distinct clusters were found, with similar cluster patterns detected in three separate datasets. Subsequently, each cluster was separately fit to a Performance Factors Analysis model (PFA). Significantly different parameter coefficients across clusters implied that students more likely to have been gaming benefitted less from prior failures. These differences inspired new and modified predictors that were found to improve overall model fit -- an improvement that varied in magnitude across clusters. The present findings indicate that incorporating trial duration into counts of prior failures can improve the predictive power of learning models.

There is some research indicating that the presence of music adversely impacts academic task performance. While most of this research involves individuals reading text passages, few studies have explored how graphical representations contribute to the auditory distraction literature. The aim of our study was to investigate if concept maps, a graphical representation that depicts relations among concepts, and linear text differentially affect recall when they are studied in the presence of music. Participants studied a preconstructed concept map or text summary while listening to verbal or nonverbal music. Results indicated that participants who studied the concept map with verbal music recalled significantly more ideas than those who studied the text summary. This result was particularly robust for those with low to moderate prior knowledge in the domain being studied. These findings suggest that the novel structure of concept maps may induce greater concentration, which could provide protection against auditory distraction.

This paper describes the development of a dynamical systems model of motivation and metacognition during learning, which explains some of the practically and theoretically important relationships among three student engagement constructs and performance metrics during learning. In order to better calibrate and understand the model, the model was also fit with additional fixed factor predictors determined from the factor scores from a factor analysis of the pre-survey given to students. This work mined data from computerized adaptive flashcard learning system to create the dynamical systems model. This flashcard practice included pop-up survey questions on the students experience of recent easiness, strategy use, and usefulness, in addition to the correctness performance data for the practice. The dynamical systems model of this data was then used to simulate various student profiles to predict how they would experience the flashcard system. These simulations show how strategy use in this task is crucial because of the ways it influences performance and perceived usefulness. In the model, this result is shown by a bifurcation for higher and lower strategy use, where the higher strategy use equilibrium is accompanied by performance predictions suggesting learning that is more efficient. In addition, we examined the implications of our data for the flow theory of optimal experience by testing models of this theory and comparing it to a Vygotskian perspective on the results.

Background: This study investigated learning outcomes and user perceptions from interactions with a hybrid intelligent tutoring system created by combining the AutoTutor conversational tutoring system with the Assessment and Learning in Knowledge Spaces (ALEKS) adaptive learning system for mathematics. This hybrid intelligent tutoring system (ITS) uses a service-oriented architecture to combine these two web-based systems. Self-explanation tutoring dialogs were used to talk students through step-by-step worked examples to algebra problems. These worked examples presented an isomorphic problem to the preceding algebra problem that the student could not solve in the adaptive learning system. Results: Due to crossover issues between conditions, experimental versus control condition assignment did not show significant differences in learning gains. However, strong dose-dependent learning gains were observed that could not be otherwise explained by either initial mastery or time-on-task. User perceptions of the dialog-based tutoring were mixed, and survey results indicate that this may be due to the pacing of dialog-based tutoring using voice, students judging the agents based on their own performance (i.e., the quality of their answers to agent questions), and the students' expectations about mathematics pedagogy (i.e., expecting to solving problems rather than talking about concepts). Across all users, learning was most strongly influenced by time spent studying, which correlated with students' self-reported tendencies toward effort avoidance, effective study habits, and beliefs about their ability to improve in mathematics with effort. Conclusions: Integrating multiple adaptive tutoring systems with complementary strengths shows some potential to improve learning. However, managing learner expectations during transitions between systems remains an open research area. Finally, while personalized adaptation can improve learning efficiency, effort and time-on-task for learning remains a dominant factor that must be considered by interventions.

The current study aimed to replicate the results of previous studies examining the spacing and testing effect by showing a benefit of spaced practice and repeated testing on inductive learning. Seventy-four participants practiced diagnosing 36 case studies of six psychological disorders and tested in a posttest phase. Although learning occurred, there were no significant differences found in posttest scores between the stimuli that were practiced in a massed versus spaced format. There were also no differences found in posttest scores between stimuli that were practiced as study versus testing trials. The results of the current study necessitate a discussion about how spacing and testing can be most effective and if the effectiveness is conditional on the material being studied. Keywords: spacing effect, testing effect, inductive learning, retention, study habits, Many academics who teach complex and related concepts struggle to help students retain information, and previous research on the topic of learning and memory fails to provide efficient methods for [...]

Recent studies have used Coh-Metrix, an automated text analyzer, to assess differences in language characteristics across different genres and academic disciplines (Graesser, McNamara, & Kulikowich, 2011; McNamara, Graesser, McCarthy, & Cai, 2014). Coh-Metrix analyzes text on many constructs at different levels, including Word Concreteness (vs. abstractness), Narrativity (vs. informational), Deep Cohesion, Referential Cohesion, and Syntactic Simplicity. In previous research, texts in the natural sciences had lower Narrativity and Word Concreteness than texts in the language arts, but were higher in Syntactic Simplicity and Referential Cohesion. This pattern suggests a form of compensation in which difficulty on one dimension (e.g., Word Concreteness) is compensated for by increasing text ease on another dimension (e.g., Syntactic Simplicity). In the present study, we provide a further test of this compensation idea by analyzing oral language use across humanities and natural science lectures. We demonstrate that decreases in Word Concreteness across lectures are associated with increases in Narrativity, Deep Cohesion, and Syntactic Simplicity. In addition, within lectures, decreases in Word Concreteness are associated with increases in Syntactic Simplicity. Compensatory mechanisms are discussed in this article at different levels of language and discourse.

Efforts to improve instructional task design often make reference to the mental structures, such as "schemas" (e.g., Gick & Holyoak, 1983) or "identical elements" (Thorndike & Woodworth, 1901), that are common to both the instructional and target tasks. This component based (e.g., Singley & Anderson, 1989) approach has been employed in psychometrics (Tatsuoka, 1983), cognitive science (Koedinger & MacLaren, 2002), and most recently in educational data mining (Cen, Koedinger, & Junker, 2006). A typical assumption of these theory based models is that an itemization of "knowledge components" shared between tasks is sufficient to predict transfer between these tasks. In this paper we step back from these more cognitive theory based models of transfer and suggest a psychometric measurement model that removes most cognitive assumptions, thus allowing us to understand the data without the bias of a theory of transfer or domain knowledge. The goal of this work is to help provide a methodology that allows researchers to analyse complex data without the theoretical assumptions clearly part of other methods. Our experimentally controlled examples illustrate the non-intuitive nature of some transfer situations which motivates the necessity of the unbiased analysis that our model provides. We explain how to use this Contextual Performance Factors Analysis (CPFA) model to measure learning progress of related skills at a fine granularity. This CPFA analysis then allows us to answer questions regarding the best order of practice for related skills and the appropriate amount of repetition depending on whether students are succeeding or failing with each individual practice problem. We conclude by describing how the model allows us to test theories, in which we discuss how well two different cognitive theories agree with the qualitative results of the model.

Statistical learning refers to the ability to identify structure in the input based on its statistical properties. For many linguistic structures, the relevant statistical features are distributional: They are related to the frequency and variability of exemplars in the input. These distributional regularities have been suggested to play a role in many different aspects of language learning, including phonetic categories, using phonemic distinctions in word learning, and discovering non-adjacent relations. On the surface, these different aspects share few commonalities. Despite this, we demonstrate that the same computational framework can account for learning in all of these tasks. These results support two conclusions. The first is that much, and perhaps all, of distributional statistical learning can be explained by the same underlying set of processes. The second is that some aspects of language can be learned due to domain-general characteristics of memory. (Contains 4 tables and 7 figures.)

By balancing the spacing effect against the effects of recency and frequency, this paper explains how practice may be scheduled to maximize learning and retention. In an experiment, an optimized condition using an algorithm determined with this method was compared with other conditions. The optimized condition showed significant benefits with large effect sizes for both improved recall and recall latency. The optimization method achieved these benefits by using a modeling approach to develop a quantitative algorithm, which dynamically maximizes learning by determining for each item when the balance between increasing temporal spacing (that causes better long-term recall) and decreasing temporal spacing (that reduces the failure related time cost of each practice) means that the item is at the spacing interval where long-term gain per unit of practice time is maximal. As practice repetitions accumulate for each item, items become stable in memory and this optimal interval increases. (Contains 7 figures and 3 tables.)

Two different methods of practice are available in the learning of simple information, test practice or study practice. Of these two methods of learning, research has generally shown that test practice is superior to study practice. However, this research has not considered the testing advantage with respect to the fact that test learning is uncertain (i.e. if recall fails, nothing appears to be learned) or with respect to the fact that study learning depends greatly on the duration of the study event. The following work clarifies these issues by presenting an integrated computational model of the relative costs and benefits of testing compared to study presentation [based on the ACT-R theory of declarative memory; Anderson, J.R. & Schooler, L.J. (1991). "Psychological Science" 2: 396-408; Pavlik Jr., P.I. & Anderson, J.R. (2005). "Cognitive Science" 29: 559-586]. This model was applied to determine how test and study practice can be optimally employed to improve learning and provides a framework for understanding the effects of mnemonic strategies in simple memory tasks.

An experiment was performed to investigate the effects of practice and spacing on retention of Japanese-English vocabulary paired associates. The relative benefit of spacing increased with increased practice and with longer retention intervals. Data were fitted with an activation-based memory model, which proposes that each time an item is practiced it receives an increment of strength but that these increments decay as a power function of time. The rate of decay for each presentation depended on the activation at the time of the presentation. This mechanism limits long-term benefits from further practice at higher levels of activation and produces the spacing effect and its observed interactions with practice and retention interval. The model was compared with another model of the spacing effect (Raaijmakers, 2003) and was fit to some results from the literature on spacing and memory.