Search

In numerous studies, intelligent tutoring systems (ITSs) have proven effective in helping students learn mathematics. Prior work posits that their effectiveness derives from efficiently providing eventually-correct practice opportunities. Yet, there is little empirical evidence on how learning processes with ITSs compare to other forms of instruction. The current study compares problem-solving with an ITS versus solving the same problems on paper. We analyze the learning process and pre-post gain data from N = 97 middle school students practicing linear graphs in three curricular units. We find that (i) working with the ITS, students had more than twice the number of eventually-correct practice opportunities than when working on paper and (ii) omission errors on paper were associated with lower learning gains. Yet, contrary to our hypothesis, tutor practice did not yield greater learning gains, with tutor and paper comparing differently across curricular units. These findings align with tutoring allowing students to grapple with challenging steps through tutor assistance but not with eventually-correct opportunities driving learning gains. Gaming-the-system, lack of transfer to an unfamiliar test format, potentially ineffective tutor design, and learning affordances of paper can help explain this gap. This study provides first-of-its-kind quantitative evidence that ITSs yield more learning "opportunities" than equivalent paper-and-pencil practice and reveals that the relation between opportunities and learning gains emerges only when the instruction is effective.

This paper explores how learning analytics data provided by a math problem-solving educational technology platform informed 5th and 6th grade teachers' instructional decisions around socioemotional learning (SEL). MathSpring is an educational technology tool that provides teachers with data on students' effort, progress, and emotions while students are solving math problems. This paper presents findings from qualitative analyses of 11 classroom observations, 10 observation follow-up chats, and 35 teacher interviews that were collected as part of a randomized control trial conducted in the northeastern United States. The analyses include teachers who used the program with their students (treatment group) and teachers who implemented math instruction as usual (control group). Teachers in the treatment group viewed analytic data on students' effort and progress, which informed the discussions they had with their students around SEL topics including setting goals, the importance of effort, and using productive problem-solving strategies. Findings from these analyses suggest that analytic data on effort and progress can inform teachers' discussions of math-related SEL concepts with their students. [This paper was published in: "The 14th Learning Analytics and Knowledge Conference (LAK '24), March 18-22, 2024, Kyoto, Japan," ACM, 2024.]

Intelligent tutoring systems (ITS) can help students learn successfully, yet little work has explored the role of caregivers in shaping that success. Past interventions to support caregivers in supporting their child's homework have been largely disjunct from educational technology. The paper presents prototyping design research with nine middle school caregivers. We ask: (1) what are caregivers' preferences for different prototypes incorporating data-driven recommendations into their math homework support? Integrating caregivers' preferences, we then ask: (2) what are caregivers' perceptions when interacting with a prototype of an intelligent chatbot tool to support students' homework? We found caregivers reported feeling comfortable integrating AI into their practices and appreciated chat-based support for understanding content and effective ITS use. Our results highlight the affordances of ITS data and AI to assist caregivers who would otherwise not be able to support their child's homework, paving the way for more effective and equitable mathematics learning. [This paper will be published in the ISLS2024 proceedings.]

Recent research seeks to develop more comprehensive learner models for adaptive learning software. For example, models of reading comprehension built using data from students' use of adaptive instructional software for mathematics have recently been developed. These models aim to deliver experiences that consider factors related to learning beyond performance in the target domain for instruction. We investigate the extent to which generalization is possible for a recently developed predictive model that seeks to infer students' reading comprehension ability (as measured by end-of-year standardized test scores) using an introductory learning experience in Carnegie Learning's MATHia intelligent tutoring system for mathematics. Building on a model learned on data from middle school students in a single school district in a mid-western U.S. state, using that state's end-of-year English Language Arts (ELA) standardized test score as an outcome, we consider data from a school district in a south-eastern U.S. state as well as that state's end-of-year ELA standardized test outcome. Generalization is explored by considering prediction performance when training and testing models on data from each of the individual school districts (and for their respective state's test outcomes) as well as pooling data from both districts together. We conclude with discussion of investigations of some algorithmic fairness characteristics of the learned models. The results suggest that a model trained on data from the smaller of the two school districts considered may achieve greater fairness in its predictions over models trained on data from the other district or both districts, despite broad, overall similarities in some demographic characteristics of the two school districts. This raises interesting questions for future research on generalizing these kinds of models as well as on ensuring algorithmic fairness of resulting models for use in real-world adaptive systems for learning. [This paper was published in: "Proceedings of the 16th International Conference on Educational Data Mining," edited by M. Feng et al., International Educational Data Mining Society, 2023, pp. 207-16.]

School rating websites are increasingly used by parents to assess the quality and fit of U.S. K-12 schools for their children. These online reviews often contain detailed descriptions of a school's strengths and weaknesses, which both reflect and inform perceptions of a school. Existing work on these text reviews has focused on finding words or themes that underlie these perceptions, but has stopped short of using the textual reviews as leading indicators of school performance. In this paper, we investigate to what extent the language used in online reviews of a school is predictive of changes in the attributes of that school, such as its socio-economic makeup and student test scores. Using over 300K reviews of 70K U.S. schools from a popular ratings website, we apply language processing models to predict whether schools will significantly increase or decrease in an attribute of interest over a future time horizon. We find that using the text improves predictive performance significantly over a baseline model that does not include text but only the historical time-series of the indicators themselves, suggesting that the review text carries predictive power. A qualitative analysis of the most predictive terms and phrases used in the text reviews indicates a number of topics that serve as leading indicators, such as diversity, changes in school leadership, a focus on testing, and school safety.

It is particularly important to identify and address issues of fairness and equity in educational contexts as academic performance can have large impacts on the types of opportunities that are made available to students. While it is always the hope that educators approach student assessment with these issues in mind, there are a number of factors that likely impact how a teacher approaches the scoring of student work. Particularly in cases where the assessment of student work requires subjective judgment, as in the case of open-ended answers and essays, contextual information such as how the student has performed in the past, general perceptions of the student, and even other external factors such as fatigue may all influence how a teacher approaches assessment. While such factors exist, however, it is not always clear how these may introduce bias, nor is it clear whether such bias poses measurable risks to fairness and equity. In this paper, we examine these factors in the context of the assessment of student answers to open response questions from middle school mathematics learners. We observe how several factors such as context and fatigue correlate with teacher-assigned grades and discuss how learning systems may support fair assessment. [This paper was published in: "Proceedings of the 30th International Conference on Computers in Education," edited by S. Iyer et al., Asia-Pacific Society for Computers in Education, 2022.]

Solving mathematical problems is cognitively complex, involving strategy formulation, solution development, and the application of learned concepts. However, gaps in students' knowledge or weakly grasped concepts can lead to errors. Teachers play a crucial role in predicting and addressing these difficulties, which directly influence learning outcomes. However, preemptively identifying misconceptions leading to errors can be challenging. This study leverages historical data to assist teachers in recognizing common errors and addressing gaps in knowledge through feedback. We present a longitudinal analysis of incorrect answers from the 2015-2020 academic years on two curricula, Illustrative Math and EngageNY, for grades 6, 7, and 8. We find consistent errors across 5 years despite varying student and teacher populations. Based on these Common Wrong Answers (CWAs), we designed a crowdsourcing platform for teachers to provide Common Wrong Answer Feedback (CWAF). This paper reports on an in vivo randomized study testing the effectiveness of CWAFs in two scenarios: next-problem-correctness within-skill and next-problem-correctness within-assignment, regardless of the skill. We find that receiving CWAF leads to a significant increase in correctness for consecutive problems within-skill. However, the effect was not significant for all consecutive problems within-assignment, irrespective of the associated skill. This paper investigates the potential of scalable approaches in identifying Common Wrong Answers (CWAs) and how the use of crowdsourced CWAFs can enhance student learning through remediation. [This paper was published in: "Proceedings of the Tenth ACM Conference on Learning @ Scale (L@S '23)," July 20-22, 2023.]

Prior studies identified effective, but mainly non-digital, homework aids. This research involved 18 middle school students in a lo-fi prototyping study to integrate traditional homework support tools with intelligent tutoring systems (ITS), leveraging rich log data for personalized learning. Feature investigations in standardized diaries, goal setting, homework graphing, and reminders revealed a preference for goal-setting, informing a refined prototype exploring students' attitudes towards self-set versus system-generated goals. Results indicate students prefer system-recommended goals, express goals by problem count and time, and value goal difficulty feedback to foster autonomy. Providing data-driven feedback and fading system-set goals in ITS could improve self-regulation during homework practices and equity.

This paper demonstrates how to apply Machine Learning (ML) techniques to analyze student interaction data collected in an online mathematics game. We examined: (1) how different ML algorithms influenced the precision of middle-school students' (N = 359) performance prediction; and (2) what types of in-game features were associated with student math knowledge scores. The results indicated that the Random Forest algorithm showed the best performance in predicting posttest math knowledge scores among the seven algorithms employed. Out of 37 features included in the model, the validity of the students' first mathematical transformation was the most predictive of their math knowledge scores. Implications for game learning analytics and supporting students' algebraic learning are discussed based on the findings.

This paper investigates two specific probabilistic biases which middle graders usually exhibit when reasoning about probability and randomness on assessment items. We discuss how students' reasoning about key probability concepts undergirds statistics literacy related to randomness, independence, and the likelihood of future events based on past results. We examine factors evoking misconceptions and students' (in)consistency in exhibiting them. Findings indicate that misconceptions can be evoked based on three types of factors including (1) students' particular understandings of probability and randomness, (2) general item characteristics, and (3) aspects of probability in items. Moreover, possession of a specific misconception will most likely result in exhibiting the bias again on other occasions including the same evoking factors (consistency). [This paper was published in: "Proceedings of the Satellite Conference of the International Association for Statistical Education (IASE): Statistics Education in the Era of Data Science," edited by R. Helenius and E. Falck, International Association for Statistical Education, 2021, pp. 1-6.]

As evidence grows supporting the importance of non-cognitive factors in learning, computer-assisted learning platforms increasingly incorporate non-academic interventions to influence student learning and learning related-behaviors. Non-cognitive interventions often attempt to influence students' mindset, motivation, or metacognitive reflection to impact learning behaviors and outcomes. In the current paper, we analyze data from five experiments, involving seven treatment conditions embedded in mastery-based learning activities hosted on a computer-assisted learning platform focused on middle school mathematics. Each treatment condition embodied a specific non-cognitive theoretical perspective. Over seven school years, 20,472 students participated in the experiments. We estimated the effects of each treatment condition on students' response time, hint usage, likelihood of mastering knowledge components, learning efficiency, and post-tests performance. Our analyses reveal a mix of both positive and negative treatment effects on student learning behaviors and performance. Few interventions impacted learning as assessed by the post-tests. These findings highlight the difficulty in positively influencing student learning behaviors and outcomes using non-cognitive interventions. [This paper was published in: "LAK23: 13th International Learning Analytics and Knowledge Conference Proceedings," March 13-17, 2023.]

This paper systematically explores how Large Language Models (LLMs) generate explanations of code examples of the type used in intro-to-programming courses. As we show, the nature of code explanations generated by LLMs varies considerably based on the wording of the prompt, the target code examples being explained, the programming language, the temperature parameter, and the version of the LLM. Nevertheless, they are consistent in two major respects for Java and Python: the readability level, which hovers around 7-8 grade, and lexical density, i.e., the relative size of the meaninful words with respect to the total explanation size. Furthermore, the explanations score very high in correctness but less on three other metrics: completeness, conciseness, and contextualization. [This paper is in: Proceedings of the workshop on Generative AI for Education(GAIED): Advances, Opportunities, and Challenges, 2003.]

Providing high-level support to students on NGSS inquiry practices can be challenging; however, teacher dashboards can help teachers provide just-in-time instruction to students, both in-person and online. Prior work has shown some success with a dashboard that alerts teachers in real time on students' science inquiry difficulties, but teachers differed in their use of the alerts. To further support teachers, we designed a second iteration, in which the alerts included actionable, evidence-based Teacher Inquiry Practice Supports (TIPS), a series of suggested scaffolds that teachers can use to support students on the practices with which they are struggling. In this study, we investigate how the discursive support patterns from one teacher differed when using the dashboard alerts "without" TIPS followed by "with" TIPS. Findings suggest that TIPS influenced how the teacher incorporated different types of support for her students, and further, that the support given varied across different virtual lab stages. [This paper was published in: "ICLS2022 Proceedings," International Society of the Learning Sciences, 2022, pp. 297-304.]

Response Process Validity (RPV) reflects the degree to which items are interpreted as intended by item developers. In this study, teacher responses to constructed response (CR) items to assess pedagogical content knowledge (PCK) of middle school mathematics teachers were evaluated to determine what types of teacher responses signaled weak RPV. We analyzed 38 CR pilot items on proportional reasoning across up to 13 middle school mathematics teachers per item. By coding teacher responses and using think-alouds, we found teachers' responses deemed indicative of low item RPV often had one of the following characteristics: vague answers, unanticipated assumptions, a focus on unintended topics, and paraphrasing. To develop a diverse pool of items with strong RPV, we suggest it is helpful to be aware of these symptoms, use them to consider how to improve items, and then revise and retest items accordingly. [This paper was published in: "Proceedings of the forty-fourth annual meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education," edited by A. E. Lischka et al., Middle Tennessee State University, 2022, pp. 671-75.]

Developing models and using mathematics are two key practices in internationally recognized science education standards, such as the Next Generation Science Standards (NGSS). However, students often struggle with these two intersecting practices, particularly when developing mathematical models about scientific phenomena. Formative performance-based assessments designed to elicit fine-grained data about students' competencies on these practices can be leveraged to develop embedded AI scaffolds to support students' learning. In this paper, we present the design and initial classroom test of virtual labs that automatically assess fine-grained sub-components of students' mathematical modeling competencies based on their actions within the learning environment. We describe how we leveraged underlying machine-learned and knowledge-engineered algorithms to trigger scaffolds, delivered proactively by a pedagogical agent, that address students' individual difficulties as they work. Results show that the students who received automated scaffolds for a given practice on their first virtual lab improved on that practice for the next virtual lab on the same science topic in a different scenario (a near-transfer task). These findings suggest that real-time automated scaffolds based on fine-grained assessment can foster students' mathematical modeling competencies related to the NGSS.

Teacher learning in asynchronous online professional development (PD) is often assessed through self-reported instruments, despite uncertainties regarding teachers' ability to accurately self-assess their learning gains in such settings. This study compared middle school mathematics teachers' self-perceived and observed learning gains in content knowledge (CK) and pedagogical content knowledge (PCK) after participating in an asynchronous online PD program. By analyzing data collected from 57 teachers, we found a weak and nonsignificant correlation between teachers' perceived learning gains and observed learning gains. Teachers who frequently monitored their understanding of the learning content tended to underestimate their improvement of CK from the PD program while those using cognitive learning strategies (i.e., organize and elaborate) often overestimated it, controlling for their baseline knowledge level. These findings highlight methodological concerns in using self-reports to evaluate online learning outcomes and the potential distortion in identifying effective learning strategies.

As more educators integrate their curricula with online learning, it is easier to crowdsource content from them. Crowdsourced tutoring has been proven to reliably increase students' next problem correctness. In this work, we confirmed the findings of a previous study in this area, with stronger confidence margins than previously, and revealed that only a portion of crowdsourced content creators had a reliable benefit to students. Furthermore, this work provides a method to rank content creators relative to each other, which was used to determine which content creators were most effective overall, and which content creators were most effective for specific groups of students. When exploring data from TeacherASSIST, a feature within the ASSISTments learning platform that crowdsources tutoring from teachers, we found that while overall this program provides a benefit to students, some teachers created more effective content than others. Despite this finding, we did not find evidence that the effectiveness of content reliably varied by student knowledge-level, suggesting that the content is unlikely suitable for personalizing instruction based on student knowledge alone. These findings are promising for the future of crowdsourced tutoring as they help provide a foundation for assessing the quality of crowdsourced content and investigating content for opportunities to personalize students' education. [This paper was published in: " L@S '21, June 22-25, 2021, Virtual Event, Germany," ACM, 2021.]

nstruction has forced teachers to navigate unexpected constraints while providing meaningful learning experiences for their students. This transition has drastically changed how teachers orchestrate learning for their students. To explore these unique orchestration challenges, we used needs finding and validation activities to explore middle school teachers' emergent needs and constraints during the unplanned shift to remote instruction. Our findings highlight the need for informative, real-time tools, issues with workload and burnout, and concerns with students feeling disconnected. The contribution of this work includes insights from the early stages of our design process and reflections on how we might support teachers during remote learning and in navigating future emergency shifts. [This paper was published in: "Proceedings of the European Conference on Technology-Enhanced Learning, EC-TEL 2021," edited by T. De Laet et al., Cham: Springer, 2021, pp. 347-51.]

Despite the critical role of teachers in the educational process, few advanced learning technologies have been developed to support teacher-instruction or professional development. This lack of support is particularly acute for middle school math teachers, where only 37% felt well prepared to scaffold instruction to address the needs of diverse students in a national sample. To address this gap, the Advancing Middle School Teachers' Understanding of Proportional Reasoning project is researching techniques to apply pedagogical virtual agents and dialog-based tutoring to enhance teachers' content knowledge and pedagogical content knowledge. This paper describes the design of a conversational, agent-based intelligent tutoring system to support teachers' professional development. Pedagogical strategies are presented that leverage a virtual human facilitator to tutor pedagogical content knowledge (how to teach proportions to students), as opposed to content knowledge (understanding proportions). The roles for different virtual facilitator capabilities are presented, including embedding actions into virtual agent dialog, open-response versus choice-based tutoring, ungraded pop-up sub-activities (e.g. whiteboard, calculator, notetaking). Usability feedback for a small cohort of instructors pursuing graduate studies was collected. In this feedback, teachers rated the system ease of use and perceived usefulness moderately well, but also reported confusion about what to expect from the system in terms of flow between lessons and support by the facilitator.

Math performance continues to be an important focus for improvement. The most recent National Report Card in the U.S. suggested student math scores declined in the past two years possibly due to COVID-19 pandemic and related school closures. We report on the implementation of a math homework program that leverages AI-based one-to-one technology, in 32 schools for two years as a part of a randomized controlled trial in diverse settings of the state of North Carolina in the US. The program, called "ASSISTments," provides feedback to students as they solve homework problems and automatically prepares reports for teachers about student performance on daily assignments. The paper describes the sample, the study design, the implementation of the intervention, including the recruitment effort, the training and support provided to teachers, and the approaches taken to assess teacher's progress and improve implementation fidelity. Analysis of data collected during the study suggest that (a) treatment teachers changed their homework review practices as they used ASSISTments, and (b) the usage of ASSISTments was positively correlated with student learning outcome.

Adaptive educational software is likely to better support broader and more diverse sets of learners by considering more comprehensive views (or models) of such learners. For example, recent work proposed making inferences about "non-math" factors like reading comprehension while students used adaptive software for mathematics to better support and adapt to learners. We build on this proposed approach to more comprehensive learning modeling by providing an empirical basis for making inferences about students' reading ability from their performance on activities in adaptive software for mathematics. We lay out an approach to predicting middle school students' reading ability using their performance on activities within Carnegie Learning's MATHia, a widely used intelligent tutoring system for mathematics. We focus on how performance in an early, introductory activity as an especially powerful place to consider instruction-embedded assessment of non-math factors like reading comprehension to guide adaptation based on factors like reading ability. We close by discussing opportunities to extend this work by focusing on particular knowledge components or skills tracked by MATHia that may provide important "levers" for driving adaptation based on students' reading ability while they learn and practice mathematics. [This paper was published in: "LAK23: 13th International Learning Analytics and Knowledge Conference," ACM, 2023.]

Integrating visual representations in an interactive learning activity effectively scaffolds performance and learning. However, it is unclear whether and how "sustaining" or "interleaving" visual scaffolding helps learners solve problems efficiently and learn from problem solving. We conducted a classroom study with 63 middle-school students in which we tested whether sustaining or interleaving a particular form of visual scaffolding, called anticipatory diagrammatic self-explanation in an Intelligent Tutoring System, helps students' learning and performance in the domain of early algebra. Sustaining visual scaffolding during problem solving helped students solve problems efficiently with no negative effects on learning. However, in-depth log data analyses suggest that interleaving visual scaffolding allowed students to practice important skills that may help them in later phases of algebra learning. This paper extends scientific understanding that sustaining visual scaffold does not over-scaffold student learning in the early phase of skill acquisition in algebra. [This paper was published in: "Proceedings of the 44th Annual Conference of the Cognitive Science Society," edited by J. Culbertson et al., Cognitive Science Society, 2022, pp. 1751-58.]

When designing an instructional tool and using it in pedagogical activities, it is essential that designers and users understand what pedagogical affordances and constraints the tool provides to support its successful integration into targeted pedagogical activities. Toward this end, we developed "Pedagogical Affordance Analysis (PAA)." PAA involves analyzing teachers' Pedagogical Content Knowledge and/or Technological Pedagogical Content Knowledge to elicit pedagogical affordances and constraints that are specific to a given instructional goal. Information obtained through PAA can help in designing, refining, and/or evaluating instructional tools. We present a case study in which we used PAA to successfully design a visual representation for middle-school algebra. To the best of our knowledge, PAA is the only available systematic method that leverages teachers' pedagogical knowledge in identifying pedagogical affordances and constraints. PAA can be used across a wide range of existing tools and prototypes of to-be-designed tools. [This paper was published in: "ICLS 2020 Proceedings," ISLS, 2020, pp. 1561-64.]

Large language models have recently been able to perform well in a wide variety of circumstances. In this work, we explore the possibility of large language models, specifically GPT-3, to write explanations for middle-school mathematics problems, with the goal of eventually using this process to rapidly generate explanations for the mathematics problems of new curricula as they emerge, shortening the time to integrate new curricula into online learning platforms. To generate explanations, two approaches were taken. The first approach attempted to summarize the salient advice in tutoring chat logs between students and live tutors. The second approach attempted to generate explanations using few-shot learning from explanations written by teachers for similar mathematics problems. After explanations were generated, a survey was used to compare their quality to that of explanations written by teachers. We test our methodology using the GPT-3 language model. Ultimately, the synthetic explanations were unable to outperform teacher written explanations. In the future more powerful large language models may be employed, and GPT-3 may still be effective as a tool to augment teachers' process for writing explanations, rather than as a tool to replace them. The explanations, survey results, analysis code, and a dataset of tutoring chat logs are all available at https://osf.io/wh5n9/.

Interactive Strategy Training for Active Reading and Thinking en Español, or iSTART-E, is a new intelligent tutoring system (ITS) that provides reading comprehension strategy training for Spanish speakers. This paper reports on studies evaluating the efficacy of iSTART-E in real-world classrooms in two different Spanish-speaking countries. In Study 1, Chilean high school students (n = 22) who practiced with iSTART-E showed significant gains on a standardized comprehension assessment (LECTUM) from pretest to posttest. In Study 2 (n = 85), Argentinian middle school students who practiced with iSTART-E showed greater gains on the ECOMPLEC.Sec comprehension test compared to those in control classrooms. Together these results suggest that iSTART-E is an effective means of enhancing Spanish speakers' reading comprehension, with demonstrated transfer of training to standardized reading tests. [This paper was published in: "Artificial Intelligence in Education. AIED 2018, LNAI 10948," edited by C. Penstein Rosé et al., Springer International Publishing AG, 2018, pp. 215-19.]

We apply an advanced data visualization technique, "Sankey diagram," to explore how middle-school students (N = 343) solved problems in a game-based algebraic notation tool. The results indicate that there is a large variation in the types of students' strategies to solve the problems, with some approaches being more efficient than others. The findings suggest that Sankey diagrams can be used both in research and practice to unpack our understanding of variability in mathematical problem-solving. [This paper was published in: "Proceedings of the 15th International Conference of the Learning Sciences--ICLS 2021," 2021, pp. 887-888.]

We study the usage of a self-guided online tutoring platform called Algebra Nation, which is widely by middle school and high school students who take the End-of-Course Algebra I exam at the end of the school year. This article aims to study how the platform contributes to increasing students' exam scores by examining users' logs over a three year period. The platform under consideration was used by more than 36,000 students in the first year, to nearly 67,000 by the third year, thus enabling us to examine how usage patterns evolved and influenced students' performance at scale. We first identify which Algebra Nation usage factors in conjunction with math overall preparation and socioeconomic factors contribute to the students' exam performance. Subsequently, we investigate the effect of increased teacher familiarity level with the Algebra Nation on students' scores across different grades through mediation analysis. The results show that the indirect effect of teacher's familiarity with the platform through increasing student's usage dosage is more significant in higher grades. [This paper was published in: "The 9th International Learning Analytics & Knowledge Conference (LAK19)" (Tempe, AZ, March 4-8, 2019). New York, NY: Association for Computing Machinery (ACM). (978-1-4503-6256-6).]

Mission HydroSci (MHS) teaches water systems and scientific argumentation towards meeting Next Generation Science Standards. MHS is a game-based 3D virtual environment for enacting transformational role-playing, wherein students must learn new knowledge and competencies in order to successfully complete the game missions. MHS was developed for middle school science as a replacement unit of about 6 to 8 hours and uses analytics and a teacher dashboard to help support teachers support their students. [This paper was published in: "CSCL 2019 Proceedings," p.1017-1020. ISLS.]

Prior studies on scaffolding for investigative inquiry practices (i.e. forming a question/hypothesis, collecting data, and analyzing and interpreting data [21]) revealed that students who received scaffolding were better able to both learn practices and transfer these competencies to new topics than were students who did not receive scaffolding. Prior studies have also shown that after removing scaffolding, students continued to demonstrate improved inquiry performance on a variety of practices across new driving questions over time. However, studies have not examined the relationship between the "amount of scaffolding received" and transfer of inquiry performance; this is the focus of the present study. 107 middle school students completed four virtual lab activities (i.e. driving questions) in Inq-ITS. Students received scaffolding when needed from an animated pedagogical computer agent for the first three driving questions for the Animal Cell virtual lab. Then they completed the fourth driving question without access to scaffolding in a different topic, Plant Cell. Results showed that students' performances increased even with fewer scaffolds for the inquiry practices of hypothesizing, collecting data, interpreting data, and warranting claims; furthermore, these results were robust as evidenced by the finding that students required less scaffolding as they completed subsequent inquiry activities. These data provide evidence of near and far transfer as a result of adaptive scaffolding of science inquiry practices. [This paper was published in: "Proceedings of the Sixth Annual ACM Conference on Learning at Scale."]

Understanding equivalence is fundamental to STEM disciplines, but many students struggle with the concept. We present a novel method for students to explore ideas of equivalence where students dynamically transform expressions from initial states to goal states in the web-based app Graspable Math. The structure of the goal state activities implies that any initial and goal state pair represent the same quantity (or varying quantity). We propose that for students, the physical experience of moving algebraic objects and observing how the initial state transforms into the goal state helps generalize notation mechanics. In fall of 2019, we will test this supposition in a randomized control trial of 1500 students in which student performance in pre- and post-tests will be compared to an online problem set control. [This paper was published in: "Proceedings of the forty-first annual meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education," edited by S. Otten et al., University of Missouri, 2019, pp. 1738-43.]

Mission HydroSci is a game-based 3D virtual environment for enacting transformational role-playing for middle school science students. Student-players will be engaged in a narrative about needing to investigate water resources and use scientific argumentation to complete missions critical to the survival and accomplishments of the members of their scientific enterprise. Our poster presents our progress in years 1 and 2 of a funded project to integrate pedagogical and gameplay objectives, and build mechanisms for purposeful player engagement and activity. The paper describes the argumentation engine as an example of developing a mechanism to enable learning. In additions, several innovative efforts have been made to support learning argumentation processes in games. MHS is being developed through a set of evaluations that support decision-making about design and development choices and eventually lead to a field test to assess outcomes. [This paper was published in: "Proceedings of the 12th International Conference on the Foundations of Digital Games" (Article 47). New York, NY: ACM.]

The results of educational research studies are only as accurate as the data used to produce them. Drawing on experiences conducting large-scale efficacy studies of classroom-based algebra interventions for community college and middle school students, I am developing practice-based data cleaning procedures to support scholars in conducting rigorous research. The paper identifies common sources of data errors in mathematics education research and offers a framework and related data cleaning process designed to address these errors. [This paper was published in: Weinberg, A., Rasmussen, C., Rabin, J., Wawro, M., and Brown, S. (Eds.), "Proceedings of the 20th Annual Conference on Research in Undergraduate Mathematics Education," p129-140. San Diego, CA (2017).]

Prior research shows that self-explanation promotes understanding by helping learners connect new knowledge with prior knowledge. However, despite ample evidence supporting the effectiveness of self-explanation, an instructional design challenge emerges in how best to scaffold self-explanation. In particular, it is an open challenge to design self- explanation support that simultaneously facilitates performance and learning outcomes. Towards this goal, we designed anticipatory diagrammatic self-explanation, a novel form of self-explanation embedded in an Intelligent Tutoring System (ITS). In our ITS, anticipatory diagrammatic self-explanation scaffolds learners by providing visual representations to help learners predict an upcoming strategic step in algebra problem solving. A classroom experiment with 108 middle-school students found that anticipatory diagrammatic self-explanation helped students learn formal algebraic strategies and significantly improve their problem-solving performance. This study contributes to understanding of how self-explanation can be scaffolded to support learning and performance.

We investigated the effects of proximal grouping of numbers, problem-solving goals to make 100, and prior knowledge on students' solution strategies in an online mathematics game. Logistic regression on 857 problem-level data points from 227 middle-school students showed that students were more likely to use productive solution strategies on addition and multiplication problems when proximity supported number grouping, 100 was the problem-solving goal, and students had high prior knowledge. Furthermore, when proximity and number goals did not support problem-solving, students with low prior knowledge were less likely to use productive solution strategies on multiplication problems than students with high prior knowledge. Findings demonstrate that the effects of perceptual and number features on solution strategies vary by students' prior knowledge.

This study used comparative interrupted times series analyses to assess the impact of a continuous improvement-focused family engagement intervention on ninth grade attendance and course passing rates. The intervention, conducted in an urban district, sought to improve middle and high school family engagement practices during the transition to high school. The initiative created a networked learning community of school teams that received training and coaching in applying the continuous improvement process to family engagement planning and implementation. After four years of implementation, there was no evidence of a significant positive effect on ninth grade student outcomes. Lack of positive effects could be due both to implementation issues and to other ninth grade interventions in the comparison district.

Feedback in educational technologies can teach and engage students in math, but questions remain on how to present failure feedback that supports positive learning behaviors. We explore how error- and reward-based feedback influenced students' choices to replay completed problems in "From Here to There!," a math game-based educational technology. We conducted a multilevel logistic regression in 1,031 seventh-grade students predicting whether a problem was replayed, using whether an error was made during the first problem attempt and how many clovers were earned after solving the problem (with more clovers earned for more efficient solutions). Preliminary results show that making an error or earning two clovers increased the likelihood of replays, whereas earning one or three clovers decreased the likelihood. Our findings suggest that clovers and errors may be "micro-failures" that encourage replays by supporting feelings of competency and/or autonomy, which has implications for game design elements for learning and engagement. [This paper was published in: ICLS2022 Proceedings, International Society of the Learning Sciences (ISLS), 2022, pp. 1633-1636.]

Learners' choices as to whether and how to use visual representations during learning are an important yet understudied aspect of self-regulated learning. To gain insight, we developed a "choice-based" intelligent tutor in which students can choose whether and when to use diagrams to aid their problem solving in algebra. In an exploratory classroom study with 26 students, we investigated how learners choose diagrams and how their choice behaviors relate to learning outcomes. Students who "proactively" chose to use diagrams achieved higher learning outcomes than those who "reactively" used diagrams when they made incorrect attempts. This study contributes to understanding of self-regulated use of visual representations during problem solving. [This paper was published in: "ICLS2022 Proceedings," International Society of the Learning Sciences, 2022, pp. 1649-52.]

One pedagogical technique that promotes conceptual understanding in mathematics learners is self-explanation integrated with worked examples (e.g., Rittle-Johnson et al., 2017). In this work, we implemented self-explanations with worked examples (correct and erroneous) in a software-based Intelligent Tutoring System (ITS) for learning algebra. We developed an approach to eliciting self-explanations in which the ITS guided students to select explanations that were conceptually rich in nature. Students who used the ITS with self-explanations scored higher on a posttest that included items tapping both conceptual and procedural knowledge than did students who used a version of the ITS that included only traditional problem-solving practice. This study replicates previous findings that self-explanation and worked examples in an ITS can foster algebra learning (Booth et al., 2013). Further, this study extends prior work to show that guiding students towards conceptual explanations is beneficial. [This paper was published in: "Proceedings of the 44th Annual Conference of the Cognitive Science Society," edited by J. Culbertson et al., Cognitive Science Society, 2022, pp. 3466-72.]

New challenges in today's world have contributed to increased attention toward evaluating individuals' collaborative problem solving (CPS) skills. One difficulty with this work is identifying evidence of individuals' CPS capabilities, particularly when interacting in digital spaces. Often human-driven approaches are used but are limited in scale. Machine-driven approaches can save time and money, but their reliability relative to human approaches can be a challenge. In the current study, we compare CPS skill profiles derived from human and semi-automated annotation methods across two tasks. Results showed that the same clusters emerged for both tasks and annotation methods, with the annotation methods showing agreement on labeling most students according to the same profile membership. Additionally, validation of cluster results using external survey measures yielded similar results across annotation methods. [This paper was published in: "CSCL2022 Proceedings," International Society of the Learning Sciences, 2022, pp. 363-366.]

This preliminary study examined whether distinct student profiles (N = 760) emerged based on their behavioral patterns in an online algebraic learning game. We applied k-means clustering analysis to clickstream data collected in the game and then examined how students' behavioral patterns varied across the clusters using data visualization. The results identified four groups of students based on their in-game behaviors, showing that there was a large variation in their behavioral patterns for engaging with the game.

Examining the interaction between content knowledge, inquiry proficiency, and writing proficiency is central to understanding the relative contribution of each proficiency on students' written communication about their science inquiry. Previous studies, however, have only analyzed one of these primary types of knowledge/proficiencies (i.e. content knowledge, inquiry proficiency, and writing proficiency) at a time. This study investigated the extent to which these proficiencies predicted students' written claims, evidence for their claims, and reasoning linking their claims to the evidence. Results showed that all three types of proficiencies significantly predicted students' claims, but only writing proficiency significantly predicted performance on evidence and reasoning statements. These findings indicate the challenges students face when constructing claim, evidence, and reasoning statements, and can inform scaffolding to support these challenges. [This paper was published in: "L@S '18: Proceedings of the Fifth Annual ACM Conference on Learning at Scale." ACM. (978-1-4503-5886-6).]