Search

In numerous studies, intelligent tutoring systems (ITSs) have proven effective in helping students learn mathematics. Prior work posits that their effectiveness derives from efficiently providing eventually-correct practice opportunities. Yet, there is little empirical evidence on how learning processes with ITSs compare to other forms of instruction. The current study compares problem-solving with an ITS versus solving the same problems on paper. We analyze the learning process and pre-post gain data from N = 97 middle school students practicing linear graphs in three curricular units. We find that (i) working with the ITS, students had more than twice the number of eventually-correct practice opportunities than when working on paper and (ii) omission errors on paper were associated with lower learning gains. Yet, contrary to our hypothesis, tutor practice did not yield greater learning gains, with tutor and paper comparing differently across curricular units. These findings align with tutoring allowing students to grapple with challenging steps through tutor assistance but not with eventually-correct opportunities driving learning gains. Gaming-the-system, lack of transfer to an unfamiliar test format, potentially ineffective tutor design, and learning affordances of paper can help explain this gap. This study provides first-of-its-kind quantitative evidence that ITSs yield more learning "opportunities" than equivalent paper-and-pencil practice and reveals that the relation between opportunities and learning gains emerges only when the instruction is effective.

This paper explores how learning analytics data provided by a math problem-solving educational technology platform informed 5th and 6th grade teachers' instructional decisions around socioemotional learning (SEL). MathSpring is an educational technology tool that provides teachers with data on students' effort, progress, and emotions while students are solving math problems. This paper presents findings from qualitative analyses of 11 classroom observations, 10 observation follow-up chats, and 35 teacher interviews that were collected as part of a randomized control trial conducted in the northeastern United States. The analyses include teachers who used the program with their students (treatment group) and teachers who implemented math instruction as usual (control group). Teachers in the treatment group viewed analytic data on students' effort and progress, which informed the discussions they had with their students around SEL topics including setting goals, the importance of effort, and using productive problem-solving strategies. Findings from these analyses suggest that analytic data on effort and progress can inform teachers' discussions of math-related SEL concepts with their students. [This paper was published in: "The 14th Learning Analytics and Knowledge Conference (LAK '24), March 18-22, 2024, Kyoto, Japan," ACM, 2024.]

Intelligent tutoring systems (ITS) can help students learn successfully, yet little work has explored the role of caregivers in shaping that success. Past interventions to support caregivers in supporting their child's homework have been largely disjunct from educational technology. The paper presents prototyping design research with nine middle school caregivers. We ask: (1) what are caregivers' preferences for different prototypes incorporating data-driven recommendations into their math homework support? Integrating caregivers' preferences, we then ask: (2) what are caregivers' perceptions when interacting with a prototype of an intelligent chatbot tool to support students' homework? We found caregivers reported feeling comfortable integrating AI into their practices and appreciated chat-based support for understanding content and effective ITS use. Our results highlight the affordances of ITS data and AI to assist caregivers who would otherwise not be able to support their child's homework, paving the way for more effective and equitable mathematics learning. [This paper will be published in the ISLS2024 proceedings.]

Predictive analytics is widely used in various domains, including education, to inform decision-making and improve outcomes. However, many predictive models are proprietary and inaccessible for evaluation or modification by researchers and practitioners, limiting their accountability and ethical design. Moreover, predictive models are often opaque and incomprehensible to the officials who use them, reducing their trust and utility. Furthermore, predictive models may introduce or exacerbate bias and inequity, as they have done in many sectors of society. Therefore, there is a need for transparent, interpretable, and fair predictive models that can be easily adopted and adapted by different stakeholders. In this paper, we propose a fair predictive model based on multivariate adaptive regression splines (MARS) that incorporates fairness measures in the learning process. MARS is a non-parametric regression model that performs feature selection, handles non-linear relationships, generates interpretable decision rules, and derives optimal splitting criteria on the variables. Specifically, we integrate fairness into the knot optimization algorithm and provide theoretical and empirical evidence of how it results in a fair knot placement. We apply our "fair"MARS model to real-world data and demonstrate its effectiveness in terms of accuracy and equity. Our paper contributes to the advancement of responsible and ethical predictive analytics for social good. [This paper was presented at an Association for the Advancement of Artificial Intelligence conference.]

The manner in which gender is portrayed in materials used to teach children conveys messages about people's roles in society. In this paper, we measure the gendered depiction of central domains of social life in 100 years of highly influential children's books. We make two main contributions: (1) we find that the portrayal of gender in these books reproduces traditional gender norms in society; and (2) we publish StoryWords 1.0, the first word embeddings trained on such a large body of children's literature. We find that, relative to males, females are more likely to be represented in relation to their appearance than in relation to their competence; second, they are more likely to be represented in relation to their role in the family than their role in business. Finally, we find that non-binary or gender-fluid individuals are rarely mentioned. Our analysis advances understanding of the different messages contained in content commonly used to teach children, with immediate applications for practice, policy, and research. [This paper was published in: "Proceedings of the 29th International Conference on Computational Linguistics," 2022, pp. 3086-3097.]

In the effort to improve student outcomes, teachers are usually the focus of professional development efforts. However, research increasingly demonstrates the significance of administrators in influencing student outcomes, raising questions about how they too should be involved in such programs. The study involved interviews conducted with 12 administrators engaged in a large-scale professional development program dedicated to equitable math instruction for diverse learners. The paper reports on the different understandings they had regarding their role in supporting the innovation, including their perspective on diverse learners, perceptions of teacher needs, and approaches for teacher support. While conventional professional development programs might overlook administrators, this paper argues for a situated approach for engaging administrators in the complex process of leading change.

This paper presents a tool for creating student models in logistic regression. Creating student models has typically been done by expert selection of the appropriate terms, beginning with models as simple as IRT or AFM but more recently with highly complex models like BestLR. While alternative methods exist to select the appropriate predictors for the regression-based models (e.g., step-wise selection or LASSO), we are unaware of their application to student modeling. Such automatic methods of model creation offer the possibility of better student models with either reduced com-plexity or better fit, in addition to relieving experts from the bur-den of searching for better models by hand with possible human error. Our new functions are now part of the preexisting R pack-age LKT. We explain our search methods with two datasets demonstrating the advantages of using the tool with stepwise re-gression and regularization (LASSO) methods to aid in feature selection. For the stepwise method using BIC, the models are simpler (due to the BIC penalty for parameters) than alternatives like BestLR with little lack of fit. For the LASSO method, the models can be made simpler due to the fitting procedure involving a regularization parameter that penalizes large absolute coefficient values. However, LASSO also offers the possibility of highly complex models with exceptional fit. [This paper was published in: "Proceedings of the 16th International Conference on Educational Data Mining," edited by M. Feng et. al., International Educational Data Mining Society, 2023, pp.17-27.]

Solving mathematical problems is cognitively complex, involving strategy formulation, solution development, and the application of learned concepts. However, gaps in students' knowledge or weakly grasped concepts can lead to errors. Teachers play a crucial role in predicting and addressing these difficulties, which directly influence learning outcomes. However, preemptively identifying misconceptions leading to errors can be challenging. This study leverages historical data to assist teachers in recognizing common errors and addressing gaps in knowledge through feedback. We present a longitudinal analysis of incorrect answers from the 2015-2020 academic years on two curricula, Illustrative Math and EngageNY, for grades 6, 7, and 8. We find consistent errors across 5 years despite varying student and teacher populations. Based on these Common Wrong Answers (CWAs), we designed a crowdsourcing platform for teachers to provide Common Wrong Answer Feedback (CWAF). This paper reports on an in vivo randomized study testing the effectiveness of CWAFs in two scenarios: next-problem-correctness within-skill and next-problem-correctness within-assignment, regardless of the skill. We find that receiving CWAF leads to a significant increase in correctness for consecutive problems within-skill. However, the effect was not significant for all consecutive problems within-assignment, irrespective of the associated skill. This paper investigates the potential of scalable approaches in identifying Common Wrong Answers (CWAs) and how the use of crowdsourced CWAFs can enhance student learning through remediation. [This paper was published in: "Proceedings of the Tenth ACM Conference on Learning @ Scale (L@S '23)," July 20-22, 2023.]

Images in children's books convey messages about society and the roles that people play in it. Understanding these messages requires systematic measurement of who is represented. Computer vision face detection tools can provide such measurements; however, state-of-the-art face detection models were trained with photographs, and 80\% of images in children's books are illustrated; thus existing methods both misclassify and miss classifying many faces. In this paper, we introduce a new approach to analyze images using AI tools, resulting in data that can assess representation of race, gender, and age in both illustrations and photographs in children's books. We make four primary contributions to the fields of deep learning and social sciences: (1) We curate an original face detection data set (IllusFace 1.0) by manually labeling 5,403 illustrated faces with bounding boxes. (2) We train two AutoML-based face detection models for illustrations: (i) using IllusFace 1.0 (FDAI); (ii) using iCartoon, a publicly available data set (FDAI_iC), each optimized for illustrated images, detecting 2.5 times more faces in our testing data than the established face detector using Google Vision (FDGV). (3) We curate a data set of the race, gender, and age of 980 faces manually labeled by three different raters (CBFeatures 1.0). (4) We train an AutoML feature classification model (FCA) using CBFeatures 1.0. We compare FCA with the performance of another AutoML model that we trained on UTKFace, a public data set (FCA_UTK) and of an established model using FairFace (FCF). Finally, we examine distributions of character identities over the last century across the models. We find that FCA is 34% more accurate than FCF in its race predictions. These contributions provide tools to educators, caregivers, and curriculum developers to assess the representation contained in children's content. [This paper was published in: "Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)," 2022, pp. 462-471.]

AutoTutor-ARC (adult reading comprehension) is an intelligent tutoring system that uses conversational agents to help adult learners improve their comprehension skills. However, in such a system, not all lessons and items optimally serve the same purposes. In this paper, we describe a method for classifying items that are "instructive, evaluative, motivational," versus "potentially flawed" based on analyses of items' psychometric properties. Further, there is no a priori way of determining which lessons are optimal given the learner's reading profile needs. To address this, we evaluate how assessing learner component reading skills can inform various aspects of learner needs on AutoTutor lessons. More specifically, we compare learners who were classified as "proficient, underengaged, conscientious," versus "struggling" readers based on their experiences with AutoTutor. Together, these analyses suggest the utility of integrating assessments with instruction: efficient, adaptive learning at the lesson level, more efficient and valid post-testing, and consequently, recommendations for more targeted, adaptive pathways through the instructional program/system. [This paper was published in: "AIED 2022, LNCS 13356," 2022, pp. 520-523.]

It is particularly important to identify and address issues of fairness and equity in educational contexts as academic performance can have large impacts on the types of opportunities that are made available to students. While it is always the hope that educators approach student assessment with these issues in mind, there are a number of factors that likely impact how a teacher approaches the scoring of student work. Particularly in cases where the assessment of student work requires subjective judgment, as in the case of open-ended answers and essays, contextual information such as how the student has performed in the past, general perceptions of the student, and even other external factors such as fatigue may all influence how a teacher approaches assessment. While such factors exist, however, it is not always clear how these may introduce bias, nor is it clear whether such bias poses measurable risks to fairness and equity. In this paper, we examine these factors in the context of the assessment of student answers to open response questions from middle school mathematics learners. We observe how several factors such as context and fatigue correlate with teacher-assigned grades and discuss how learning systems may support fair assessment. [This paper was published in: "Proceedings of the 30th International Conference on Computers in Education," edited by S. Iyer et al., Asia-Pacific Society for Computers in Education, 2022.]

The use of technology as a facilitator in learning environments has become increasingly prevalent with the global pandemic caused by COVID-19. As such, computer-supported collaborative learning (CSCL) gains a wider adoption in contrast to traditional learning methods. At the same time, the need for automated tools capable of assessing and stimulating collaboration between participants has become more stringent, as human monitoring of the increasing volume of conversations becomes overwhelming. This paper introduces a method grounded in dialogism for evaluating students' involvement in chat conversations based on semantic chains computed using language models. These semantic chains reflect emergent voices from dialogism that span and interact throughout the conversation. Our integrated method uses contextual information captured by BERT transformer models to identify links in a chain that connects semantically related concepts from a voice uttered by one or more participants. Two types of visualizations were generated to depict the longitudinal propagation and the transversal inter-animation of voices within the conversation. In addition, a list of handcrafted features derived from the constructed chains and computed for each participant is introduced. Several machine learning algorithms were tested using these features to evaluate the extent to which semantic chains are predictive of student involvement in chat conversations. [This paper was published in: "Ludic, Co-design and Tools Supporting Smart Learning Ecosystems and Smart Education, Proceedings of the 6th International Conference on Smart Learning Ecosystems and Regional Development," edited by Ó. Mealha et al., Springer Nature Singapore Pte Ltd., 2022, pp. 67-78.]

In this paper, we present iSTART-Early, an intelligent tutoring system that provides automated instruction and practice on higher-order reading comprehension strategies to 3rd and 4th grade students. iSTART-Early provides personalized, interactive, game-based strategy instruction and practice on comprehension strategies (i.e., Ask It, Reword It, Find It, Explain It, and Summarize It) with grade-appropriate informational texts. Natural language processing (NLP) combined with automated speech recognition (ASR) and text-to-speech technologies enable immediate formative and summative feedback. A teacher interface allows teachers to assign texts and monitor students' performance so that they can provide additional support and feedback when necessary, creating blended-learning opportunities. We describe the interface and the development of iSTART-Early, as well as our plans to examine the intelligent tutoring system for usability, feasibility and promise in improving reading comprehension strategies and outcomes for young readers. [This paper was published in: S. Crossley and E. Popescu, Eds., "ITS 2022, LNCS 13284," Springer Nature, 2022, pp.371-379.]

Cloze items are commonly used for both assessing learning and as a learning activity. This paper investigates the selection of sentences for cloze item creation by comparing methods ranging from simple heuristics to deep learning summarization models. An evaluation using human-generated cloze items from three different science texts indicates that simple heuristics substantially outperform summarization models, including state-of-the-art deep learning models. These results suggest that sentence selection for cloze item generation should be considered a distinct task from summarization and that continued advances on this task will require large datasets of human-generated cloze items. [This paper was published in: "Joint Proceedings of the Workshops at the 14th International Conference on Educational Data Mining," Vol. 3051, LDI-6, edited by T. W. Price and S. San Pedro, CEUR-WS.org, 2021.]

Teachers often rely on curricular materials as support for implementing reform-based science practices and as educative materials to continue developing ways in which they teach and approach science. Therefore, it is important to understand how teachers develop and use curricular materials and how they learn from their experiences. However, we lack deeper dives into longitudinal relationships that teachers have with curricular materials. This study explores the ways a teacher designs and adapts curriculum over three years. The study tracks the teacher's involvement in co-designing curricular materials alongside other teachers, researchers, and curriculum designers and adapting and using these materials in her own classroom. This study will trace curricular values that the teacher uses to guide her relationships with the curriculum and the adaptations she chooses to make for her own classroom context. The study focuses on a moment in class that the teacher was particularly proud of because it created a palpable shift in the students' engagement with the content she was teaching. This study aims to draw a narrative of this teacher's journey across her three years of writing and adapting materials she was using. [This paper was published in: "Hawaii International Conference on Education Proceedings 2024."]

This paper explores three simple data manipulation techniques (synthesis, augmentation, curriculum) for improving abstractive summarization models without the need for any additional data. We introduce a method of data synthesis with paraphrasing, a data augmentation technique with sample mixing, and curriculum learning with two new difficulty metrics based on specificity and abstractiveness. We conduct experiments to show that these three techniques can help improve abstractive summarization across two summarization models and two different small datasets. Furthermore, we show that these techniques can improve performance when applied in isolation and when combined. [This paper was published in: "Findings of the Association for Computational Linguistics: EMNLP 2021," Association for Computational Linguistics, 2021, pp. 2043-2052.]

This paper explores the effect of using multitask learning for abstractive summarization in the context of small training corpora. In particular, we incorporate four different tasks (extractive summarization, language modeling, concept detection, and paraphrase detection) both individually and in combination, with the goal of enhancing the target task of abstraction summarization via multitask learning. We show that for many task combinations, a model trained in a multitask setting outperforms a model trained only for abstractive summarization, with no additional summarization data introduced. Additionally, we do a comprehensive search and find that certain tasks (e.g. paraphrase detection) consistently benefit abstractive summarization, not only when combined with other tasks but also when using different architectures and training corpora. [This paper was published in: "Findings of the Association for Computational Linguistics: EMNLP 2021," Association for Computational Linguistics, 2021, pp. 1652-1661.]

As evidence grows supporting the importance of non-cognitive factors in learning, computer-assisted learning platforms increasingly incorporate non-academic interventions to influence student learning and learning related-behaviors. Non-cognitive interventions often attempt to influence students' mindset, motivation, or metacognitive reflection to impact learning behaviors and outcomes. In the current paper, we analyze data from five experiments, involving seven treatment conditions embedded in mastery-based learning activities hosted on a computer-assisted learning platform focused on middle school mathematics. Each treatment condition embodied a specific non-cognitive theoretical perspective. Over seven school years, 20,472 students participated in the experiments. We estimated the effects of each treatment condition on students' response time, hint usage, likelihood of mastering knowledge components, learning efficiency, and post-tests performance. Our analyses reveal a mix of both positive and negative treatment effects on student learning behaviors and performance. Few interventions impacted learning as assessed by the post-tests. These findings highlight the difficulty in positively influencing student learning behaviors and outcomes using non-cognitive interventions. [This paper was published in: "LAK23: 13th International Learning Analytics and Knowledge Conference Proceedings," March 13-17, 2023.]

An intelligent textbook may be defined as an interaction layer between the text and the student, helping the student master the content in the text. The Mobile Fact and Concept Training System (MoFaCTS) is an adaptive instructional system for simple content that has been developed into an interaction layer to mediate textbook instruction and so is being transformed into the Mobile Fact and Concept Textbook System (MoFaCTS). In this paper, we document the several terms of the logistic regression model we use to track performance adaptively. We then examine the contribution of each component of our model when it is fit to 4 semesters of Anatomy and Physiology course practice data. Following this documentation of the model, we explain how it is applied in the MoFaCTS system to schedule performance by targeting practice for each item at an optimal efficiency threshold. [This paper was published in the CEUR workshop proceedings (Vol. 2895).]

This paper explores the concept of dynamically generating definitions using a deep-learning model. We do this by creating a dataset that contains definition entries and contexts associated with each definition. We then fine-tune a GPT-2 based model on the dataset to allow the model to generate contextual definitions. We evaluate our model with human raters by generating definitions using two context types: short-form (the word used in a sentence) and long-form (the word used in a sentence along with the prior and following sentences). Results indicate that the model performed significantly better when generating definitions using short-form contexts. Additionally, we evaluate our model against human-generated definitions. The results show promise for the model, showing that the model was able to match human-level fluency. However, while it was able to reach human-level accuracy in some instances, it failed in others. [This paper was published in: "Proceedings of the Third International Workshop on Intelligent Textbooks," Vol. 2895, CEUR-WS.org, 2021, pp. 74-83.]

This paper explores a general approach to paraphrase generation using a pre-trained seq2seq model fine-tuned using a back-translated anatomy and physiology textbook. Human ratings indicate that the paraphrase model generally preserved meaning and grammaticality/fluency: 70% of meaning ratings were above 75, and 40% of paraphrases were considered more grammatical/fluent than the originals. An error analysis suggests potential avenues for future work. [This paper was published in: "Proceedings of the 22nd International Conference on Artificial Intelligence in Education," edited by I. Roll, D. McNamara, S. Sosnovsky, R. Luckin, and V. Dimitrova, Springer International Publishing, 2021, pp. 279-284.]

This paper presents WikiMorph, a tool that automatically breaks down words into morphemes, etymological compounds (morphemes from root languages), and generates contextual definitions for each component. It comes in two flavors: a dataset and a deep-learning-based model. The dataset was extracted from Wiktionary and contains over 450k entries. We then used this dataset to train a GPT-2 model to generalize and decompose any word into morphemes and their definitions. We find that the model accurately generates complex breakdowns when given a high-quality initial definition. [This paper was published in: "Proceedings of the 22nd International Conference on Artificial Intelligence in Education," edited by I. Roll, D. McNamara, S. Sosnovsky, R. Luckin, and V. Dimitrova, Springer International Publishing, 2021, pp. 406-11.]

Exploring new or emerging research domains or subdomains can become overwhelming due to the magnitude of available resources and the high speed at which articles are published. As such, a tool that curates the information and underlines central entities, both authors and articles from a given research context, is highly desirable. Starting from the articles of the International Conference of Learning Analytics & Knowledge (LAK) in its first decade, this paper proposes a novel method grounded in Cohesion Network Analysis (CNA) to analyze subcommunities of authors based on the semantic similarities between authors and papers, and estimate their global impact. Paper abstracts are represented as embeddings using a fine-tuned SciBERT language model, alongside a custom trained LSA model. The extrapolation between the local LAK community to a worldwide importance was also underlined by the comparison between the rankings obtained from our method and statistics from ResearchGate. The accuracies for binary classifications in terms of high/low impact predictions were around 70% for authors, and around 80% for articles. Our method can guide researchers by providing valuable information on the interactions between the members of a knowledge community and by highlighting central local authors who may potentially have a high global impact.

Computer-assisted learning platforms (CALPS) increasingly include gamified elements to improve student outcomes by enhancing their engagement with content. Although evidence exists that gamified programs increase engagement and learning outcomes, there is little causal research on what programmatic mechanisms drive the effect between engagement and learning. In the following paper, we explore this relationship through a method of causal moderation known as fully latent principal stratification. Using data from a large-scale randomized control trial assessing gamified and traditional CALP systems' effects on algebraic knowledge, we estimate the impact of using the gamified CALP on students who engage with one of its key gamification elements--replaying a problem after a suboptimal attempt. The gamified CALP asks students to manipulate algebraic expressions from start to goal states and provides feedback based on the efficiency of these manipulations, allowing students to replay the problems when their efficiency can be improved. We find that the effect of gamification is greater for students with a higher propensity to replay problems. This finding suggests that gamification elements that provide students with opportunities to retry problems are driving the game's efficacy and provide evidence for a scalable mechanism of gamification that can improve students' learning. [This paper was published in: "Proceedings of the Tenth ACM Conference on Learning @ Scale (L@S '23), July 20-22, 2023, Copenhagen, Denmark," ACM, 2023, pp.164-173.]

Reading comprehension is essential for both knowledge acquisition and memory reinforcement. Automated modeling of the comprehension process provides insights into the efficacy of specific texts as learning tools. This paper introduces an improved version of the Automated Model of Comprehension, version 3.0 (AMoC v3.0). AMoC v3.0 is based on two theoretical models of the comprehension process, namely the Construction-Integration and the Landscape models. In addition to the lessons learned from the previous versions, AMoC v3.0 uses Transformer-based contextualized embeddings to build and update the concept graph as a simulation of reading. Besides taking into account generative language models and presenting a visual walkthrough of how the model works, AMoC v3.0 surpasses the previous version in terms of the Spearman correlations between our activation scores and the values reported in the original Landscape Model for the presented use case. Moreover, features derived from AMoC significantly differentiate between high-low cohesion texts, thus arguing for the model's capabilities to simulate different reading conditions. [This paper was published in: "AIED 2023: Artificial Intelligence in Education," edited by N. Wang et al., Springer, Switzerland, 2023, pp. 229-241, 2023.]

This paper systematically explores how Large Language Models (LLMs) generate explanations of code examples of the type used in intro-to-programming courses. As we show, the nature of code explanations generated by LLMs varies considerably based on the wording of the prompt, the target code examples being explained, the programming language, the temperature parameter, and the version of the LLM. Nevertheless, they are consistent in two major respects for Java and Python: the readability level, which hovers around 7-8 grade, and lexical density, i.e., the relative size of the meaninful words with respect to the total explanation size. Furthermore, the explanations score very high in correctness but less on three other metrics: completeness, conciseness, and contextualization. [This paper is in: Proceedings of the workshop on Generative AI for Education(GAIED): Advances, Opportunities, and Challenges, 2003.]

Multi-angle question answering models have recently been proposed that promise to perform related tasks like question generation. However, performance on related tasks has not been thoroughly studied. We investigate a leading model called Macaw on the task of multiple choice question generation and evaluate its performance on three angles that systematically reduce the complexity of the task. Our results indicate that despite the promise of generalization, Macaw performs poorly on untrained angles. Even on a trained angle, Macaw fails to generate four distinct multiple-choice options on 17% of inputs. We propose augmenting multiple-choice options by paraphrasing angle input and show this increases overall success to 97.5%. A human evaluation comparing the augmented multiple-choice questions with textbook questions on the same topic reveals that Macaw questions broadly score highly but below human questions. [This paper was published in: "Proceedings of the 3rd Workshop of the Learner Data Institute," edited by S. E. Fancsali and V. Rus, Learner Data Institute, 2022, pp. 18-23.]

A longstanding goal of learner modeling and educational data mining is to improve the domain model of knowledge that is used to make inferences about learning and performance. In this report we present a tool for finding domain models that is built into an existing modeling framework, logistic knowledge tracing (LKT). LKT allows the flexible specification of learner models in logistic regression by allowing the modeler to select whatever features of the data are relevant to prediction. Each of these features (such as the count of prior opportunities) is a function computed for a component of data (such as a student or knowledge component). In this context, we have developed the "autoKC" component, which clusters knowledge components and allows the modeler to compute features for the clustered components. For an autoKC, the input component (initial KC or item assignment) is clustered prior to computing the feature and the feature is a function of that cluster. Another recent new function for LKT, which allows us to specify interactions between the logistic regression predictor terms, is combined with autoKC for this report. Interactions allow us to move beyond just assuming the cluster information has additive effects to allow us to model situations where a second factor of the data moderates a first factor. [This paper was published in: S. E. Fancsali and V. Rus (Eds.), "Proceedings of the 3rd Workshop of the Learner Data Institute," pp. 7-12, 2022. Learner Data Institute.]

As online learning platforms become more ubiquitous throughout various curricula, there is a growing need to evaluate the effectiveness of these platforms and the different methods used to structure online education and tutoring. Towards this endeavor, some platforms have performed randomized controlled experiments to compare different user experiences, curriculum structures, and tutoring strategies in order to ensure the effectiveness of their platform and personalize the education of the students using it. These experiments are typically analyzed on an individual basis in order to reveal insights on a specific aspect of students' online educational experience. In this work, the data from 50,752 instances of 30,408 students participating in 50 different experiments conducted at scale within the online learning platform ASSISTments were aggregated and analyzed for consistent trends across experiments. By combining common experimental conditions and normalizing the dependent measures between experiments, this work has identified multiple statistically significant insights on the impact of various skill mastery requirements, strategies for personalization, and methods for tutoring in an online setting. This work can help direct further experimentation and inform the design and improvement of new and existing online learning platforms. The anonymized data compiled for this work are hosted by the Open Science Foundation and can be found at https://osf.io/59shv/. [This paper was published in: "Proceedings of the 15th International Conference on Educational Data Mining," edited by A. Mitrovic and N. Bosch, International Educational Data Mining Society, 2022, pp. 27-38.]

Studies have shown that on-demand assistance, additional instruction given on a problem per student request, improves student learning in online learning environments. Students may have opinions on whether an assistance was effective at improving student learning. As students are the driving force behind the effectiveness of assistance, there could exist a correlation between students' perceptions of effectiveness and the computed effectiveness of the assistance. This work conducts a survey asking secondary education students on whether a given assistance is effective in solving a problem in an online learning platform. It then provides a cursory glance at the data to view whether a correlation exists between student perception and the measured effectiveness of an assistance. Over a three year period, approximately twenty-two thousand responses were collected across nearly four thousand, four hundred students. Initial analyses of the survey suggest no significance in the relationship between student perception and computed effectiveness of an assistance, regardless of if the student participated in the survey. All data and analysis conducted can be found on the Open Science Foundation website. [This paper was published in: "Proceedings of the 15th International Conference on Educational Data Mining," edited by A. Mitrovic and N. Bosch, International Educational Data Mining Society, 2022, pp. 734-37.]

Personalized learning stems from the idea that students benefit from instructional material tailored to their needs. Many online learning platforms purport to implement some form of personalized learning, often through on-demand tutoring or self-paced instruction, but to our knowledge none have a way to automatically explore for specific opportunities to personalize students' education nor a transparent way to identify the effects of personalization on specific groups of students. In this work we present the Automatic Personalized Learning Service (APLS). The APLS uses multi-armed bandit algorithms to recommend the most effective support to each student that requests assistance when completing their online work, and is currently used by ASSISTments, an online learning platform. The first empirical study of the APLS found that Beta-Bernoulli Thompson Sampling, a popular and effective multi-armed bandit algorithm, was only slightly more capable of selecting helpful support than randomly selecting from the relevant support options. Therefore, we also present Decision Tree Thompson Sampling (DTTS), a novel contextual multi-armed bandit algorithm that integrates the transparency and interpretability of decision trees into Thomson sampling. In simulation, DTTS overcame the challenges of recommending support within an online learning platform and was able to increase students' learning by as much as 10% more than the current algorithm used by the APLS. We demonstrate that DTTS is able to identify qualitative interactions that not only help determine the most effective support for students, but that also generalize well to new students, problems, and support content. The APLS using DTTS is now being deployed at scale within ASSISTments and is a promising tool for all educational learning platforms. [This paper was published in: "Proceedings of the Ninth ACM Conference on Learning @ Scale (L@S '22), June 1-3, 2022, New York City, NY, USA," ACM, 2022.]

Students often have limited opportunities in school to grapple with the ramifications of the increasing influence technology companies exert over their lives. Digital citizenship curricula could be the exception, but they typically emphasize personal safety and respectful behavior online, ignoring more participatory or justice-oriented notions of citizenship. The scholarship on critical pedagogy helps to envision a different definition of digital citizenship that challenges the status quo to achieve social justice. Critical digital citizenship curricula are therefore a means for educators and students to use technology and interrogate it in order to effect systemic change. This brief paper examines initial results from the Young People's Race, Power, and Technology Project (YPRPT), an out-of-school program that integrates technology "under the hood" investigations with social justice topics and documentary filmmaking. Youth-produced documentaries on topics such as artificial intelligence in healthcare, the problems as well as the possibilities of automation, and facial recognition technology demonstrate how the young people are developing critical digital citizenship through exploring knotty questions of civics and technology with questions and ideas that seek broad systemic change at institutional levels. If traditional digital citizenship curricula struggle to address complex social problems, then YPRPT exemplifies one learning experience designed to foster students' critical digital citizenship and encourage them to cultivate justice-oriented civic identities.

This paper demonstrates how to apply Machine Learning (ML) techniques to analyze student interaction data collected in an online mathematics game. We examined: (1) how different ML algorithms influenced the precision of middle-school students' (N = 359) performance prediction; and (2) what types of in-game features were associated with student math knowledge scores. The results indicated that the Random Forest algorithm showed the best performance in predicting posttest math knowledge scores among the seven algorithms employed. Out of 37 features included in the model, the validity of the students' first mathematical transformation was the most predictive of their math knowledge scores. Implications for game learning analytics and supporting students' algebraic learning are discussed based on the findings.

Comfort and preparedness are dispositions that along with teacher knowledge, comprise teacher quality (cf. Clarke & Hollingsworth, 2002). Professional development programs that help teachers understand and respond to individual students' strengths and needs can support equity in the classroom by building teachers' preparedness and comfort, which enhances the likelihood of teachers' engaging in differentiated instruction to support all students. This paper draws from two randomized controlled trials (RCTs) of "Math for All," which show consistent, positive results of the professional development (PD) program on elementary school teachers' reports of comfort and preparedness to teach mathematics to diverse students. We found comfort and preparedness to be modestly but consistently related to self-reported mathematical instructional practices across both studies. The fact that the results were similar across the two RCTs even though the modes of delivery differed (PD led by developers in RCT #1; by local staff developers and teacher leaders in RCT #2) also speaks to the scalability of the "Math for All" program.

Multiple choice questions are traditionally expensive to produce. Recent advances in large language models (LLMs) have led to fine-tuned LLMs that generate questions competitive with human-authored questions. However, the relative capabilities of ChatGPT-family models have not yet been established for this task. We present a carefully-controlled human evaluation of three conditions: a fine-tuned, augmented version of Macaw, instruction-tuned Bing Chat with zero-shot prompting, and human-authored questions from a college science textbook. Our results indicate that on six of seven measures tested, both LLM's performance was not significantly different from human performance. Analysis of LLM errors further suggests that Macaw and Bing Chat have different failure modes for this task: Macaw tends to repeat answer options whereas Bing Chat tends to not include the specified answer in the answer options. For Macaw, removing error items from analysis results in performance on par with humans for all metrics; for Bing Chat, removing error items improves performance but does not reach human-level performance. [This paper was published in the "CEUR Workshop Proceedings," 2023.]

Developing models and using mathematics are two key practices in internationally recognized science education standards, such as the Next Generation Science Standards (NGSS). However, students often struggle with these two intersecting practices, particularly when developing mathematical models about scientific phenomena. Formative performance-based assessments designed to elicit fine-grained data about students' competencies on these practices can be leveraged to develop embedded AI scaffolds to support students' learning. In this paper, we present the design and initial classroom test of virtual labs that automatically assess fine-grained sub-components of students' mathematical modeling competencies based on their actions within the learning environment. We describe how we leveraged underlying machine-learned and knowledge-engineered algorithms to trigger scaffolds, delivered proactively by a pedagogical agent, that address students' individual difficulties as they work. Results show that the students who received automated scaffolds for a given practice on their first virtual lab improved on that practice for the next virtual lab on the same science topic in a different scenario (a near-transfer task). These findings suggest that real-time automated scaffolds based on fine-grained assessment can foster students' mathematical modeling competencies related to the NGSS.

This full paper concerns an exploratory study that investigates students' reasoning about torsion. Mechanical reasoning is critical to engineering applications and yet students still struggle to accurately predict, analyze, and model mechanical systems using formal symbolic notations (i.e., formalizations). To understand the nature of students' reasoning, we analyzed students' discourse to explore two competing hypotheses: (H1) The "Formalisms First" (FF) hypothesis that students report their mechanical reasoning predominantly using mathematical formalisms that take on a disembodied, allocentric (observer) point-of-view; or (H2) the "Grounded and Embodied Cognition" (GEC) hypothesis that students predominantly use independent speech which includes analogy and imagery to simulate the physical structure and function of an object(s) using an embodied, egocentric (first-person) point-of-view in addition to an allocentric point-of-view. Qualitative results from discourse analysis of two student dyads showed that students' mechanical reasoning revealed through their speech included both analogy and imagery, as predicted by H2. Students generated analogies and imagery that described dynamic behaviors, such as how torque caused displacement, stored and released energy, and fractured. Usage of analogies and imagery supports that students' mechanical reasoning often drew upon simulations of torsion-related sensorimotor experiences. Students' egocentric and allocentric imagery invoked sensorial experiences in their speech, with allocentric viewpoints being more common, as predicted by H1 and H2. Student discourse included many references to formalisms, also consistent with the H1. Data from students' verbal discourse on mechanical reasoning suggests they employ both GEC and FF viewpoints of torsion, which has implications for designing effective learning experiences and for assessing students' knowledge.

Developing models and using mathematics are two key practices in internationally recognized science education standards such as the Next Generation Science Standards (NGSS, 2013). In this paper, we used a virtual performance-based formative assessment to capture students' competencies at both "developing" and "evaluating" mathematical models in science inquiry contexts aligned with the NGSS (2013). Results show that model development and evaluation competencies are correlated, but students who demonstrate proficiency with model development often struggle with evaluation. Nuanced data illustrate how components of modeling competencies differ and how they may be related.

This work proposes Dynamic Linear Epsilon-Greedy, a novel contextual multi-armed bandit algorithm that can adaptively assign personalized content to users while enabling unbiased statistical analysis. Traditional A/B testing and reinforcement learning approaches have trade-offs between empirical investigation and maximal impact on users. Our algorithm seeks to balance these objectives, allowing platforms to personalize content effectively while still gathering valuable data. Dynamic Linear Epsilon-Greedy was evaluated via simulation and an empirical study in the ASSISTments online learning platform. In simulation, Dynamic Linear Epsilon-Greedy performed comparably to existing algorithms and in ASSISTments, slightly increased students' learning compared to A/B testing. Data collected from its recommendations allowed for the identification of qualitative interactions, which showed high and low knowledge students benefited from different content. Dynamic Linear Epsilon-Greedy holds promise as a method to balance personalization with unbiased statistical analysis. All the data collected during the simulation and empirical study are publicly available at https://osf.io/zuwf7/. [This paper was published in: "UMAP '23: Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization," June 26-29, 2023.]

Response Process Validity (RPV) reflects the degree to which items are interpreted as intended by item developers. In this study, teacher responses to constructed response (CR) items to assess pedagogical content knowledge (PCK) of middle school mathematics teachers were evaluated to determine what types of teacher responses signaled weak RPV. We analyzed 38 CR pilot items on proportional reasoning across up to 13 middle school mathematics teachers per item. By coding teacher responses and using think-alouds, we found teachers' responses deemed indicative of low item RPV often had one of the following characteristics: vague answers, unanticipated assumptions, a focus on unintended topics, and paraphrasing. To develop a diverse pool of items with strong RPV, we suggest it is helpful to be aware of these symptoms, use them to consider how to improve items, and then revise and retest items accordingly. [This paper was published in: "Proceedings of the forty-fourth annual meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education," edited by A. E. Lischka et al., Middle Tennessee State University, 2022, pp. 671-75.]

Providing high-level support to students on NGSS inquiry practices can be challenging; however, teacher dashboards can help teachers provide just-in-time instruction to students, both in-person and online. Prior work has shown some success with a dashboard that alerts teachers in real time on students' science inquiry difficulties, but teachers differed in their use of the alerts. To further support teachers, we designed a second iteration, in which the alerts included actionable, evidence-based Teacher Inquiry Practice Supports (TIPS), a series of suggested scaffolds that teachers can use to support students on the practices with which they are struggling. In this study, we investigate how the discursive support patterns from one teacher differed when using the dashboard alerts "without" TIPS followed by "with" TIPS. Findings suggest that TIPS influenced how the teacher incorporated different types of support for her students, and further, that the support given varied across different virtual lab stages. [This paper was published in: "ICLS2022 Proceedings," International Society of the Learning Sciences, 2022, pp. 297-304.]

Historically, learning for young students has occurred in formal, in-person classroom environments, but the distance learning context has opened a myriad of learning modalities. To this end, we aim to better understand how deploying learning experience design (LXD) approach supports or hinders children's engagement while participating in an online, video based math course. This study operationalized LXD through the integration of evidence-based pedagogical instructional design and human-centered user experience (UX) design. Findings suggest that students' situational interest and mind-wandering significantly mediate the relationship between user experiences and online engagement. These results provide practical implications for how researchers, designers, and instructors can intentionally iterate the learning experience to reduce mind-wandering and sustain children's online engagement with learning theories as we consider the future of online teaching and learning modalities. [This paper was published in: "ICLS2022 Proceedings," International Society of the Learning Sciences (ICLS), 2022, pp. 472-479.]

Integrating visual representations in an interactive learning activity effectively scaffolds performance and learning. However, it is unclear whether and how "sustaining" or "interleaving" visual scaffolding helps learners solve problems efficiently and learn from problem solving. We conducted a classroom study with 63 middle-school students in which we tested whether sustaining or interleaving a particular form of visual scaffolding, called anticipatory diagrammatic self-explanation in an Intelligent Tutoring System, helps students' learning and performance in the domain of early algebra. Sustaining visual scaffolding during problem solving helped students solve problems efficiently with no negative effects on learning. However, in-depth log data analyses suggest that interleaving visual scaffolding allowed students to practice important skills that may help them in later phases of algebra learning. This paper extends scientific understanding that sustaining visual scaffold does not over-scaffold student learning in the early phase of skill acquisition in algebra. [This paper was published in: "Proceedings of the 44th Annual Conference of the Cognitive Science Society," edited by J. Culbertson et al., Cognitive Science Society, 2022, pp. 1751-58.]