Search

Understanding how students with varying capabilities think about problem solving can greatly help in improving personalized education which can have significantly better learning outcomes. Here, we present the details of a system we call NeTra that we developed for discovering strategies that students follow in the context of Math learning. Specifically, we developed this system from large-scale data from MATHia that contains millions of student-tutor interactions. The goal of this system is to provide a visual interface for educators to understand the likely strategy the student will follow for problems that students are yet to attempt. This predictive interface can help educators/tutors to develop interventions that are personalized for students. Underlying the system is a powerful AI model based on Neuro-Symbolic learning that has shown promising results in predicting both strategies and the mastery over concepts used in the strategy.

Using archived student data for middle and high school students' mathematics-focused intelligent tutoring system (ITS) learning collected across a school year, this study explores situational, achievement-goal latent profile membership and the stability of these profiles with respect to student demographics and dispositional achievement goal scores. Over 65\% of students changed situational profile membership at some time during the school year. Start-of-year dispositional motivation scores were not related to whether students remained in the same profile across all unit-level measurements. Grade level was predictive of profile stability. Findings from the present study should shed light on how in-the-moment student motivation fluctuates while students are engaged in ITS math learning. Present findings have potential to inform motivation interventions designed for ITS math learning.

In this study we will explore the use of a personalization intervention in order to improve middle school students' sense of belonging and performance in math. Students will take a pre and post measure of belonging, and work through MATHia as normal. In the experimental condition, some word problems will replace the names of characters with a name from the student's school district. We predict that seeing familiar names in math problems will boost student sense of belonging and mathematics performance.

Using UpGrade integrated with MATHia to determine the impact of personalizing the names of characters in mathematics word problems focuses on a substantive element of mathematics instruction and practice, one that we believe could be vital to better understanding belonging and equity in math instruction. We hope to illuminate more general learning science principles, building on related work by Cordova & Lepper (1996) and Walkington (2013), among others, extending ideas from this previous work to better understand whether such personalization interventions focusing on representation (and the extent to which problem content is “localized” in this way) can contribute to improving students’ sense of belonging in a way that impacts their interest and performance in mathematics. High-quality outcome measures include surveys of students’ sense of belonging (drawing from the Mathematics Classroom Connectedness Scale and the Panorama Student Survey) and internal process measures from MATHia, including performance on fine-grained MATHia knowledge components as well as aggregate performance in MATHia workspaces. Knowledge component models are iteratively refined in light of both empirical student performance data (e.g., using so-called learning curve analysis) as well as subject matter experts’ judgment, and in some cases have been subject to such refinement over nearly two decades. Aggregate student performance over many MATHia workspaces has been found to be highly correlated with performance on high-quality external measures like standardized test scores across tens of thousands of students in multiple states over many years (Zheng et al., 2019; Fancsali et al., 2016; Ritter et al., 2013). References Cordova, D. I., & Lepper, M. R. (1996). Intrinsic motivation and the process of learning: Beneficial effects of contextualization, personalization, and choice. Journal of Educational Psychology, 88(4), 715–730. https://doi.org/10.1037/0022-0663.88.4.715 Fancsali, S. E., Ritter, S., Yudelson, M., Sandbothe, M., & Berman, S. R. (2016). Implementation factors and outcomes for intelligent tutoring systems: A case study of time and efficiency with Cognitive Tutor Algebra. In Proceedings of the 29th International Florida Artificial Intelligence Research Society Conference. FLAIRS 2016. (pp. 473–478). AAAI Press. Ritter, S., Joshi, A., Fancsali, S., & Nixon, T. (2013). Predicting Standardized Test Scores from Cognitive Tutor Interactions. In S. K. D’Mello, R. A. Calvo, & A. Olney (Eds.), Proceedings of the 6th International Conference on Educational Data Mining, Memphis, Tennessee, USA, July 6-9, 2013 (pp. 169–176). http://www.educationaldatamining.org/EDM2013/papers/rn_paper_25.pdf Walkington, C. A. (2013). Using adaptive learning technologies to personalize instruction to student interests: The impact of relevant contexts on performance and learning outcomes. Journal of Educational Psychology, 105(4), 932–945. https://doi.org/10.1037/a0031882 Zheng, G., Fancsali, S. E., Ritter, S., & Berman, S. (2019). Using Instruction-Embedded Formative Assessment to Predict State Summative Test Scores and Achievement Levels in Mathematics. Journal of Learning Analytics, 6(2), 153–174. https://doi.org/10.18608/jla.2019.62.11

We propose a half-day workshop for researchers interested in (or pursuing) inter-, multi-, and trans-disciplinary convergence research in educational data science. This workshop builds on the two previous workshops held at EDM 2020 and EDM 2021, which each attracted nearly 60 attendees for productive sessions summarizing the work of the LDI and its mission, invited talks, and contributed paper presentations. LDI is a U.S. National Science Foundation (NSF) funded initiative based at the Institute for Intelligent Systems, University of Memphis, in collaboration with Carnegie Learning, Inc., and additional partners spanning academia, government, and industry. LDI has sought to foster and support science convergence to address major challenges in online and blended learning with technology through an expanding, global network of experts, school-based practitioners, interdisciplinary research teams, and task-oriented special interest groups. LDI participants have iteratively refined mechanisms and processes for building shared understanding, identified investment opportunities for science convergence research that harness massive, often multi-modal datasets generated by current computer-supported and adaptive instructional systems. The workshop provides a forum, through contributions of research, work-in-progress, and/or position papers and presentations, for the educational data science community to present multi-disciplinary or trans-disciplinary research that "harnesses the data revolution" with "big" educational data and/or grapples with problems of translating the results of such research into action in real-world educational settings to make the learning ecosystem that incorporates learning technologies and computer-supported, adaptive instructional systems more effective, efficient, and/or equitable. Workshop participants are encouraged to explore the future of convergence research utilizing big data in education.

Philosophy

Multi-angle question answering models have recently been proposed that promise to perform related tasks like question generation. However, performance on related tasks has not been thoroughly studied. We investigate a leading model called Macaw on the task of multiple choice question generation and evaluate its performance on three angles that systematically reduce the complexity of the task. Our results indicate that despite the promise of generalization, Macaw performs poorly on untrained angles. Even on a trained angle, Macaw fails to generate four distinct multiple-choice options on 17% of inputs. We propose augmenting multiple- choice options by paraphrasing angle input and show this increases overall success to 97.5%. A human evaluation comparing the augmented multiple-choice questions with textbook questions on the same topic reveals that Macaw questions broadly score highly but below human questions.

Transfer, the application of knowledge to new problem-solving situations, and preparation for future learning are often considered among the primary goals of education, but it is difficult to achieve. This paper presents an initial exploration of young children's ability to transfer the specific knowledge and skills that they have learned from a game-based adaptive instructional system to the types of math problems they typically see in school. We describe a regression tree model predicting young children's far transfer performance of specific knowledge and skills following their performance in a game-based adaptive instructional system called My Math Academy. Findings suggest that My Math Academy gameplay empirically supports positive performance on a task that involves elements of both far transfer of learning and preparation for future learning.

This paper provides an update of the Learner Data Institute (LDI; www.learnerdatainstitute.org) which is now in its third year since conceptualization. Funded as a conceptualization project, the LDI's first two years had two major goals: (1) develop, implement, evaluate, and refine a framework for data-intensive science and engineering and (2) use the framework to start developing prototype solutions, based on data, data science, and science convergence, to a number of core challenges in learning science and engineering. One major focus in the third, current year is synthesizing efforts from the first two years to identify new opportunities for future research by various mutual interest groups within LDI, which have focused on developing a particular prototype solution to one or more related core challenges in learning science and engineering. In addition to highlighting emerging data-intensive solutions and innovations from the LDI's first two years, including places where LDI researchers have received additional funding for future research, we highlight here various core challenges our team has identified as being at a "tipping point." Tipping point challenges are those for which timely investment in data-intensive approaches has the maximum potential for a transformative effect.

In intelligent tutoring systems (ITS) abundant supportive messages are provided to learners. One implicit assumption behind this design is that learners would actively process and benefit from feedback messages when interacting with ITS individually. However, this is not true for all learners; some gain little after numerous practice opportunities. In the current research, we assume that if the learner invests enough cognitive effort to review feedback messages provided by the system, the learner's performance should be improved as practice opportunities accumulate. We expect that the learner's cognitive effort investment could be reflected to some extent by the response latency, then the learner's improvement should also be correlated with the response latency. Therefore, based on this core hypothesis, we conduct a cluster analysis by exploring features relevant to learners' response latency. We expect to find several features that could be used as indicators of the feedback usage of learners; consequently, these features may be used to predict learners' learning gain in future research. Our results suggest that learners' prior knowledge level plays a role when interacting with ITS and different patterns of response latency. Learners with higher prior knowledge levels tend to interact flexibly with the system and use feedback messages more effectively. The quality of their previous attempts influences their response latency. However, learners with lower prior knowledge perform two opposite patterns, some tend to respond more quickly, and some tend to respond more slowly. One common characteristic of these learners is their incorrect response latency is not influenced by the quality of their previous performance. One interesting result is that those quick responders forget faster. Thus, we concluded that for learners with lower prior knowledge, it is better for them not to react hastily to obtain a more durable memory.

A longstanding goal of learner modeling and educational data mining is to improve the domain model of knowledge that is used to make inferences about learning and performance. In this report we present a tool for finding domain models that is built into an existing modeling framework, logistic knowledge tracing (LKT). LKT allows the flexible specification of learner models in logistic regression by allowing the modeler to select whatever features of the data are relevant to prediction. Each of these features (such as the count of prior opportunities) is a function computed for a component of data (such as a student or knowledge component). In this context, we have developed the “autoKC” component, which clusters knowledge components and allows the modeler to compute features for the clustered components. For an autoKC, the input component (initial KC or item assignment) is clustered prior to computing the feature and the feature is a function of that cluster. Another recent new function for LKT, which allows us to specify interactions between the logistic regression predictor terms, is combined with autoKC for this report. Interactions allow us to move beyond just assuming the cluster information has additive effects to allow us to model situations where a second factor of the data moderates a first factor.

We report work-in-progress that aims to better understand prediction performance differences between Deep Knowledge Tracing (DKT) and Bayesian Knowledge Tracing (BKT) as well as "gaming the system" behavior by considering variation in features and design across individual pieces of instructional content. Our "non-monolithic" analysis considers hundreds of "workspaces" in Carnegie Learning's MATHia intelligent tutoring system and the extent to which two relatively simple features extracted from MATHia logs, potentially related to gaming the system behavior, are correlated with differences in DKT and BKT prediction performance. We then take a closer look at a set of six MATHia workspaces, three of which represent content in which DKT out-performs BKT and three of which represent content in which BKT out-performs DKT or there is little difference in performance between the approaches. We present some preliminary findings related to the extent to which students game the system in these workspaces, across two school years, as well as other facets of variability across these pieces of instructional content. We conclude with a road map for scaling these analyses over much larger sets of MATHia workspaces and learner data.