Search

In numerous studies, intelligent tutoring systems (ITSs) have proven effective in helping students learn mathematics. Prior work posits that their effectiveness derives from efficiently providing eventually-correct practice opportunities. Yet, there is little empirical evidence on how learning processes with ITSs compare to other forms of instruction. The current study compares problem-solving with an ITS versus solving the same problems on paper. We analyze the learning process and pre-post gain data from N = 97 middle school students practicing linear graphs in three curricular units. We find that (i) working with the ITS, students had more than twice the number of eventually-correct practice opportunities than when working on paper and (ii) omission errors on paper were associated with lower learning gains. Yet, contrary to our hypothesis, tutor practice did not yield greater learning gains, with tutor and paper comparing differently across curricular units. These findings align with tutoring allowing students to grapple with challenging steps through tutor assistance but not with eventually-correct opportunities driving learning gains. Gaming-the-system, lack of transfer to an unfamiliar test format, potentially ineffective tutor design, and learning affordances of paper can help explain this gap. This study provides first-of-its-kind quantitative evidence that ITSs yield more learning "opportunities" than equivalent paper-and-pencil practice and reveals that the relation between opportunities and learning gains emerges only when the instruction is effective.

Understanding key concepts in molecular biology requires reasoning about molecular processes that are not directly observable and, as such, presents a challenge to students and teachers. We ask whether novel interactive physical models and activities can help students understand key processes in viral replication. Our 3D tangible models are embedded with magnets that accurately represent chemical attractions and, in a study of 492 students, structured model use led to improved understanding of viral replication and self-assembly from pre- to posttest.

This paper explores how learning analytics data provided by a math problem-solving educational technology platform informed 5th and 6th grade teachers' instructional decisions around socioemotional learning (SEL). MathSpring is an educational technology tool that provides teachers with data on students' effort, progress, and emotions while students are solving math problems. This paper presents findings from qualitative analyses of 11 classroom observations, 10 observation follow-up chats, and 35 teacher interviews that were collected as part of a randomized control trial conducted in the northeastern United States. The analyses include teachers who used the program with their students (treatment group) and teachers who implemented math instruction as usual (control group). Teachers in the treatment group viewed analytic data on students' effort and progress, which informed the discussions they had with their students around SEL topics including setting goals, the importance of effort, and using productive problem-solving strategies. Findings from these analyses suggest that analytic data on effort and progress can inform teachers' discussions of math-related SEL concepts with their students. [This paper was published in: "The 14th Learning Analytics and Knowledge Conference (LAK '24), March 18-22, 2024, Kyoto, Japan," ACM, 2024.]

Intelligent tutoring systems (ITS) can help students learn successfully, yet little work has explored the role of caregivers in shaping that success. Past interventions to support caregivers in supporting their child's homework have been largely disjunct from educational technology. The paper presents prototyping design research with nine middle school caregivers. We ask: (1) what are caregivers' preferences for different prototypes incorporating data-driven recommendations into their math homework support? Integrating caregivers' preferences, we then ask: (2) what are caregivers' perceptions when interacting with a prototype of an intelligent chatbot tool to support students' homework? We found caregivers reported feeling comfortable integrating AI into their practices and appreciated chat-based support for understanding content and effective ITS use. Our results highlight the affordances of ITS data and AI to assist caregivers who would otherwise not be able to support their child's homework, paving the way for more effective and equitable mathematics learning. [This paper will be published in the ISLS2024 proceedings.]

Prisms VR, a secondary math learning application, allows for users to see, manipulate, and engage with mathematical concepts in an embodied way in Virtual Reality (VR) environment. We examine cases in which mathematics teachers and middle school students worked through Prisms and reflected upon their experiences. Findings indicate that VR environments can help teachers find alternative ways of presenting algebraic concepts in a hands-on, interactive, and embodied manner; and these findings were confirmed based on the student experience. VR allows teachers to make connections through scaffolding and personalized experiences, while also aiding students in the exploration and meaning-making. [This paper was published in: "Proceedings of the 18th International Conference of the Learning Sciences - ICLS 2024," edited by R. Lindgren et al., International Society of the Learning Sciences, 2024, pp. 1610-13.]

Recent research seeks to develop more comprehensive learner models for adaptive learning software. For example, models of reading comprehension built using data from students' use of adaptive instructional software for mathematics have recently been developed. These models aim to deliver experiences that consider factors related to learning beyond performance in the target domain for instruction. We investigate the extent to which generalization is possible for a recently developed predictive model that seeks to infer students' reading comprehension ability (as measured by end-of-year standardized test scores) using an introductory learning experience in Carnegie Learning's MATHia intelligent tutoring system for mathematics. Building on a model learned on data from middle school students in a single school district in a mid-western U.S. state, using that state's end-of-year English Language Arts (ELA) standardized test score as an outcome, we consider data from a school district in a south-eastern U.S. state as well as that state's end-of-year ELA standardized test outcome. Generalization is explored by considering prediction performance when training and testing models on data from each of the individual school districts (and for their respective state's test outcomes) as well as pooling data from both districts together. We conclude with discussion of investigations of some algorithmic fairness characteristics of the learned models. The results suggest that a model trained on data from the smaller of the two school districts considered may achieve greater fairness in its predictions over models trained on data from the other district or both districts, despite broad, overall similarities in some demographic characteristics of the two school districts. This raises interesting questions for future research on generalizing these kinds of models as well as on ensuring algorithmic fairness of resulting models for use in real-world adaptive systems for learning. [This paper was published in: "Proceedings of the 16th International Conference on Educational Data Mining," edited by M. Feng et al., International Educational Data Mining Society, 2023, pp. 207-16.]

School rating websites are increasingly used by parents to assess the quality and fit of U.S. K-12 schools for their children. These online reviews often contain detailed descriptions of a school's strengths and weaknesses, which both reflect and inform perceptions of a school. Existing work on these text reviews has focused on finding words or themes that underlie these perceptions, but has stopped short of using the textual reviews as leading indicators of school performance. In this paper, we investigate to what extent the language used in online reviews of a school is predictive of changes in the attributes of that school, such as its socio-economic makeup and student test scores. Using over 300K reviews of 70K U.S. schools from a popular ratings website, we apply language processing models to predict whether schools will significantly increase or decrease in an attribute of interest over a future time horizon. We find that using the text improves predictive performance significantly over a baseline model that does not include text but only the historical time-series of the indicators themselves, suggesting that the review text carries predictive power. A qualitative analysis of the most predictive terms and phrases used in the text reviews indicates a number of topics that serve as leading indicators, such as diversity, changes in school leadership, a focus on testing, and school safety.

Teachers often rely on curricular materials as support for implementing reform-based science practices and as educative materials to continue developing ways in which they teach and approach science. Therefore, it is important to understand how teachers develop and use curricular materials and how they learn from their experiences. However, we lack deeper dives into longitudinal relationships that teachers have with curricular materials. This study explores the ways a teacher designs and adapts curriculum over three years. The study tracks the teacher's involvement in co-designing curricular materials alongside other teachers, researchers, and curriculum designers and adapting and using these materials in her own classroom. This study will trace curricular values that the teacher uses to guide her relationships with the curriculum and the adaptations she chooses to make for her own classroom context. The study focuses on a moment in class that the teacher was particularly proud of because it created a palpable shift in the students' engagement with the content she was teaching. This study aims to draw a narrative of this teacher's journey across her three years of writing and adapting materials she was using. [This paper was published in: "Hawaii International Conference on Education Proceedings 2024."]

It is particularly important to identify and address issues of fairness and equity in educational contexts as academic performance can have large impacts on the types of opportunities that are made available to students. While it is always the hope that educators approach student assessment with these issues in mind, there are a number of factors that likely impact how a teacher approaches the scoring of student work. Particularly in cases where the assessment of student work requires subjective judgment, as in the case of open-ended answers and essays, contextual information such as how the student has performed in the past, general perceptions of the student, and even other external factors such as fatigue may all influence how a teacher approaches assessment. While such factors exist, however, it is not always clear how these may introduce bias, nor is it clear whether such bias poses measurable risks to fairness and equity. In this paper, we examine these factors in the context of the assessment of student answers to open response questions from middle school mathematics learners. We observe how several factors such as context and fatigue correlate with teacher-assigned grades and discuss how learning systems may support fair assessment. [This paper was published in: "Proceedings of the 30th International Conference on Computers in Education," edited by S. Iyer et al., Asia-Pacific Society for Computers in Education, 2022.]

Solving mathematical problems is cognitively complex, involving strategy formulation, solution development, and the application of learned concepts. However, gaps in students' knowledge or weakly grasped concepts can lead to errors. Teachers play a crucial role in predicting and addressing these difficulties, which directly influence learning outcomes. However, preemptively identifying misconceptions leading to errors can be challenging. This study leverages historical data to assist teachers in recognizing common errors and addressing gaps in knowledge through feedback. We present a longitudinal analysis of incorrect answers from the 2015-2020 academic years on two curricula, Illustrative Math and EngageNY, for grades 6, 7, and 8. We find consistent errors across 5 years despite varying student and teacher populations. Based on these Common Wrong Answers (CWAs), we designed a crowdsourcing platform for teachers to provide Common Wrong Answer Feedback (CWAF). This paper reports on an in vivo randomized study testing the effectiveness of CWAFs in two scenarios: next-problem-correctness within-skill and next-problem-correctness within-assignment, regardless of the skill. We find that receiving CWAF leads to a significant increase in correctness for consecutive problems within-skill. However, the effect was not significant for all consecutive problems within-assignment, irrespective of the associated skill. This paper investigates the potential of scalable approaches in identifying Common Wrong Answers (CWAs) and how the use of crowdsourced CWAFs can enhance student learning through remediation. [This paper was published in: "Proceedings of the Tenth ACM Conference on Learning @ Scale (L@S '23)," July 20-22, 2023.]

Studies have shown that on-demand assistance, additional instruction given on a problem per student request, improves student learning in online learning environments. Students may have opinions on whether an assistance was effective at improving student learning. As students are the driving force behind the effectiveness of assistance, there could exist a correlation between students' perceptions of effectiveness and the computed effectiveness of the assistance. This work conducts a survey asking secondary education students on whether a given assistance is effective in solving a problem in an online learning platform. It then provides a cursory glance at the data to view whether a correlation exists between student perception and the measured effectiveness of an assistance. Over a three year period, approximately twenty-two thousand responses were collected across nearly four thousand, four hundred students. Initial analyses of the survey suggest no significance in the relationship between student perception and computed effectiveness of an assistance, regardless of if the student participated in the survey. All data and analysis conducted can be found on the Open Science Foundation website. [This paper was published in: "Proceedings of the 15th International Conference on Educational Data Mining," edited by A. Mitrovic and N. Bosch, International Educational Data Mining Society, 2022, pp. 734-37.]

Prior studies identified effective, but mainly non-digital, homework aids. This research involved 18 middle school students in a lo-fi prototyping study to integrate traditional homework support tools with intelligent tutoring systems (ITS), leveraging rich log data for personalized learning. Feature investigations in standardized diaries, goal setting, homework graphing, and reminders revealed a preference for goal-setting, informing a refined prototype exploring students' attitudes towards self-set versus system-generated goals. Results indicate students prefer system-recommended goals, express goals by problem count and time, and value goal difficulty feedback to foster autonomy. Providing data-driven feedback and fading system-set goals in ITS could improve self-regulation during homework practices and equity.

Students often have limited opportunities in school to grapple with the ramifications of the increasing influence technology companies exert over their lives. Digital citizenship curricula could be the exception, but they typically emphasize personal safety and respectful behavior online, ignoring more participatory or justice-oriented notions of citizenship. The scholarship on critical pedagogy helps to envision a different definition of digital citizenship that challenges the status quo to achieve social justice. Critical digital citizenship curricula are therefore a means for educators and students to use technology and interrogate it in order to effect systemic change. This brief paper examines initial results from the Young People's Race, Power, and Technology Project (YPRPT), an out-of-school program that integrates technology "under the hood" investigations with social justice topics and documentary filmmaking. Youth-produced documentaries on topics such as artificial intelligence in healthcare, the problems as well as the possibilities of automation, and facial recognition technology demonstrate how the young people are developing critical digital citizenship through exploring knotty questions of civics and technology with questions and ideas that seek broad systemic change at institutional levels. If traditional digital citizenship curricula struggle to address complex social problems, then YPRPT exemplifies one learning experience designed to foster students' critical digital citizenship and encourage them to cultivate justice-oriented civic identities.

This paper demonstrates how to apply Machine Learning (ML) techniques to analyze student interaction data collected in an online mathematics game. We examined: (1) how different ML algorithms influenced the precision of middle-school students' (N = 359) performance prediction; and (2) what types of in-game features were associated with student math knowledge scores. The results indicated that the Random Forest algorithm showed the best performance in predicting posttest math knowledge scores among the seven algorithms employed. Out of 37 features included in the model, the validity of the students' first mathematical transformation was the most predictive of their math knowledge scores. Implications for game learning analytics and supporting students' algebraic learning are discussed based on the findings.

This paper investigates two specific probabilistic biases which middle graders usually exhibit when reasoning about probability and randomness on assessment items. We discuss how students' reasoning about key probability concepts undergirds statistics literacy related to randomness, independence, and the likelihood of future events based on past results. We examine factors evoking misconceptions and students' (in)consistency in exhibiting them. Findings indicate that misconceptions can be evoked based on three types of factors including (1) students' particular understandings of probability and randomness, (2) general item characteristics, and (3) aspects of probability in items. Moreover, possession of a specific misconception will most likely result in exhibiting the bias again on other occasions including the same evoking factors (consistency). [This paper was published in: "Proceedings of the Satellite Conference of the International Association for Statistical Education (IASE): Statistics Education in the Era of Data Science," edited by R. Helenius and E. Falck, International Association for Statistical Education, 2021, pp. 1-6.]

As evidence grows supporting the importance of non-cognitive factors in learning, computer-assisted learning platforms increasingly incorporate non-academic interventions to influence student learning and learning related-behaviors. Non-cognitive interventions often attempt to influence students' mindset, motivation, or metacognitive reflection to impact learning behaviors and outcomes. In the current paper, we analyze data from five experiments, involving seven treatment conditions embedded in mastery-based learning activities hosted on a computer-assisted learning platform focused on middle school mathematics. Each treatment condition embodied a specific non-cognitive theoretical perspective. Over seven school years, 20,472 students participated in the experiments. We estimated the effects of each treatment condition on students' response time, hint usage, likelihood of mastering knowledge components, learning efficiency, and post-tests performance. Our analyses reveal a mix of both positive and negative treatment effects on student learning behaviors and performance. Few interventions impacted learning as assessed by the post-tests. These findings highlight the difficulty in positively influencing student learning behaviors and outcomes using non-cognitive interventions. [This paper was published in: "LAK23: 13th International Learning Analytics and Knowledge Conference Proceedings," March 13-17, 2023.]

Computer-assisted learning platforms (CALPS) increasingly include gamified elements to improve student outcomes by enhancing their engagement with content. Although evidence exists that gamified programs increase engagement and learning outcomes, there is little causal research on what programmatic mechanisms drive the effect between engagement and learning. In the following paper, we explore this relationship through a method of causal moderation known as fully latent principal stratification. Using data from a large-scale randomized control trial assessing gamified and traditional CALP systems' effects on algebraic knowledge, we estimate the impact of using the gamified CALP on students who engage with one of its key gamification elements--replaying a problem after a suboptimal attempt. The gamified CALP asks students to manipulate algebraic expressions from start to goal states and provides feedback based on the efficiency of these manipulations, allowing students to replay the problems when their efficiency can be improved. We find that the effect of gamification is greater for students with a higher propensity to replay problems. This finding suggests that gamification elements that provide students with opportunities to retry problems are driving the game's efficacy and provide evidence for a scalable mechanism of gamification that can improve students' learning. [This paper was published in: "Proceedings of the Tenth ACM Conference on Learning @ Scale (L@S '23), July 20-22, 2023, Copenhagen, Denmark," ACM, 2023, pp.164-173.]

This paper systematically explores how Large Language Models (LLMs) generate explanations of code examples of the type used in intro-to-programming courses. As we show, the nature of code explanations generated by LLMs varies considerably based on the wording of the prompt, the target code examples being explained, the programming language, the temperature parameter, and the version of the LLM. Nevertheless, they are consistent in two major respects for Java and Python: the readability level, which hovers around 7-8 grade, and lexical density, i.e., the relative size of the meaninful words with respect to the total explanation size. Furthermore, the explanations score very high in correctness but less on three other metrics: completeness, conciseness, and contextualization. [This paper is in: Proceedings of the workshop on Generative AI for Education(GAIED): Advances, Opportunities, and Challenges, 2003.]

Providing high-level support to students on NGSS inquiry practices can be challenging; however, teacher dashboards can help teachers provide just-in-time instruction to students, both in-person and online. Prior work has shown some success with a dashboard that alerts teachers in real time on students' science inquiry difficulties, but teachers differed in their use of the alerts. To further support teachers, we designed a second iteration, in which the alerts included actionable, evidence-based Teacher Inquiry Practice Supports (TIPS), a series of suggested scaffolds that teachers can use to support students on the practices with which they are struggling. In this study, we investigate how the discursive support patterns from one teacher differed when using the dashboard alerts "without" TIPS followed by "with" TIPS. Findings suggest that TIPS influenced how the teacher incorporated different types of support for her students, and further, that the support given varied across different virtual lab stages. [This paper was published in: "ICLS2022 Proceedings," International Society of the Learning Sciences, 2022, pp. 297-304.]

Response Process Validity (RPV) reflects the degree to which items are interpreted as intended by item developers. In this study, teacher responses to constructed response (CR) items to assess pedagogical content knowledge (PCK) of middle school mathematics teachers were evaluated to determine what types of teacher responses signaled weak RPV. We analyzed 38 CR pilot items on proportional reasoning across up to 13 middle school mathematics teachers per item. By coding teacher responses and using think-alouds, we found teachers' responses deemed indicative of low item RPV often had one of the following characteristics: vague answers, unanticipated assumptions, a focus on unintended topics, and paraphrasing. To develop a diverse pool of items with strong RPV, we suggest it is helpful to be aware of these symptoms, use them to consider how to improve items, and then revise and retest items accordingly. [This paper was published in: "Proceedings of the forty-fourth annual meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education," edited by A. E. Lischka et al., Middle Tennessee State University, 2022, pp. 671-75.]

Developing models and using mathematics are two key practices in internationally recognized science education standards, such as the Next Generation Science Standards (NGSS). However, students often struggle with these two intersecting practices, particularly when developing mathematical models about scientific phenomena. Formative performance-based assessments designed to elicit fine-grained data about students' competencies on these practices can be leveraged to develop embedded AI scaffolds to support students' learning. In this paper, we present the design and initial classroom test of virtual labs that automatically assess fine-grained sub-components of students' mathematical modeling competencies based on their actions within the learning environment. We describe how we leveraged underlying machine-learned and knowledge-engineered algorithms to trigger scaffolds, delivered proactively by a pedagogical agent, that address students' individual difficulties as they work. Results show that the students who received automated scaffolds for a given practice on their first virtual lab improved on that practice for the next virtual lab on the same science topic in a different scenario (a near-transfer task). These findings suggest that real-time automated scaffolds based on fine-grained assessment can foster students' mathematical modeling competencies related to the NGSS.

Teacher learning in asynchronous online professional development (PD) is often assessed through self-reported instruments, despite uncertainties regarding teachers' ability to accurately self-assess their learning gains in such settings. This study compared middle school mathematics teachers' self-perceived and observed learning gains in content knowledge (CK) and pedagogical content knowledge (PCK) after participating in an asynchronous online PD program. By analyzing data collected from 57 teachers, we found a weak and nonsignificant correlation between teachers' perceived learning gains and observed learning gains. Teachers who frequently monitored their understanding of the learning content tended to underestimate their improvement of CK from the PD program while those using cognitive learning strategies (i.e., organize and elaborate) often overestimated it, controlling for their baseline knowledge level. These findings highlight methodological concerns in using self-reports to evaluate online learning outcomes and the potential distortion in identifying effective learning strategies.

Developing models and using mathematics are two key practices in internationally recognized science education standards such as the Next Generation Science Standards (NGSS, 2013). In this paper, we used a virtual performance-based formative assessment to capture students' competencies at both "developing" and "evaluating" mathematical models in science inquiry contexts aligned with the NGSS (2013). Results show that model development and evaluation competencies are correlated, but students who demonstrate proficiency with model development often struggle with evaluation. Nuanced data illustrate how components of modeling competencies differ and how they may be related.

As more educators integrate their curricula with online learning, it is easier to crowdsource content from them. Crowdsourced tutoring has been proven to reliably increase students' next problem correctness. In this work, we confirmed the findings of a previous study in this area, with stronger confidence margins than previously, and revealed that only a portion of crowdsourced content creators had a reliable benefit to students. Furthermore, this work provides a method to rank content creators relative to each other, which was used to determine which content creators were most effective overall, and which content creators were most effective for specific groups of students. When exploring data from TeacherASSIST, a feature within the ASSISTments learning platform that crowdsources tutoring from teachers, we found that while overall this program provides a benefit to students, some teachers created more effective content than others. Despite this finding, we did not find evidence that the effectiveness of content reliably varied by student knowledge-level, suggesting that the content is unlikely suitable for personalizing instruction based on student knowledge alone. These findings are promising for the future of crowdsourced tutoring as they help provide a foundation for assessing the quality of crowdsourced content and investigating content for opportunities to personalize students' education. [This paper was published in: " L@S '21, June 22-25, 2021, Virtual Event, Germany," ACM, 2021.]

The current study examined the extent to which the cohesion detected in readers' constructed responses to multiple documents was predictive of persuasive, source-based essay quality. Participants (N=95) completed multiple-documents reading tasks wherein they were prompted to think-aloud, self-explain, or evaluate the sources while reading a set of four texts. They were then asked to write a source-based essay based on their reading. Natural Language Processing techniques were used to automatically analyze the cohesion of the constructed responses at both within- and across-documents levels. Results indicated that within-document cohesion was "negatively" related to essay quality, whereas across-documents cohesion was positively related to essay quality. Further, these relations differed by instructional condition such that strategic instructions to either self-explain or evaluate sources seemed to promote across-text integration, compared to thinking aloud. Overall, this study indicates that the cohesion of constructed responses to text can provide insights into the coherence of the mental representations readers construct while reading multiple documents. [This paper was published in: "Proceedings of the Annual Meeting of the Cognitive Science," 2021, pp. 931-937 (ISSN- 1069-7977).]

This study looks at how students embody their ideas about geometry conjectures and how those ideas travel within and between student groups. In one classroom of a Title 1 high school, students participated in a three-part program in which they: (1) played "The Hidden Village," a motion-capture video game where they assess the veracity of geometric conjectures (i.e., if it is always true or ever false) while their intuitions, insights, and rationales (including their gestures) are video recorded, (2) designed their own directed actions (i.e., a sequence of movements that represents a body-based interpretation of the structure and transformation of a spatial configuration), and (3) re-played the game with a mixture of previous conjectures combined with the conjectures designed by their peers. Multiple cases revealed ways that simulated enactment and collaborative construction can convey mathematical ideas. [This paper was published in: "CSCL 2021 Proceedings," International Society of the Learning Sciences, 2021, pp. 177-180.]

Findings synthesized across five empirical laboratory- and classroom-based studies of high school and college students engaged in geometric reasoning and proof production during single- and multi-session investigations (346 participants overall) are presented. The findings converge on several design principles for computer technologies to support embodied mathematical thinking and learning: (a) the importance of integrating linguistic and body-based processes, especially via "dynamic gestures," which enable simulated action of mathematical transformations for hypothesizing, testing, and generalizing universal truths about space and shape; (b) the "cognitive relevance" of directed actions to the concepts under consideration so that action-cognition transduction elicits the bodily interactions needed to influence mathematical reasoning in the intended manner; (c) "collaborative contexts" that foster common ground and shared meaning construction, production of socially mediated gestures, and the emergence of extended and distributed knowledge and representations; (d) conditions that encourage "transfer" of sensorimotor experiences across contexts in order to predict and make sense of the world; and (e) the need to assess learners' knowledge and learning that is expressed nonverbally. "The Hidden Village" is presented as an instance of an embodied learning environment that uses the narrative context of a visual novel to instantiate these design principles and investigate their influences on mathematical thinking and learning. These design principles are placed in terms of larger efforts to articulate a broader theoretical framework of grounded and embodied learning. Together, the interplay of theoretical, empirical and design considerations of grounded and embodied learning reframes learning, transfer, and assessment, offering promising new pathways for an emerging class of learning environments that may propel educational practice and learning theory. [This paper was published in: "Proceedings of the 29th International Conference on Computers in Education," edited by M. M. T. Rodrigo et al., Asia-Pacific Society for Computers in Education, 2021, pp. 348-58.]

nstruction has forced teachers to navigate unexpected constraints while providing meaningful learning experiences for their students. This transition has drastically changed how teachers orchestrate learning for their students. To explore these unique orchestration challenges, we used needs finding and validation activities to explore middle school teachers' emergent needs and constraints during the unplanned shift to remote instruction. Our findings highlight the need for informative, real-time tools, issues with workload and burnout, and concerns with students feeling disconnected. The contribution of this work includes insights from the early stages of our design process and reflections on how we might support teachers during remote learning and in navigating future emergency shifts. [This paper was published in: "Proceedings of the European Conference on Technology-Enhanced Learning, EC-TEL 2021," edited by T. De Laet et al., Cham: Springer, 2021, pp. 347-51.]

Despite the critical role of teachers in the educational process, few advanced learning technologies have been developed to support teacher-instruction or professional development. This lack of support is particularly acute for middle school math teachers, where only 37% felt well prepared to scaffold instruction to address the needs of diverse students in a national sample. To address this gap, the Advancing Middle School Teachers' Understanding of Proportional Reasoning project is researching techniques to apply pedagogical virtual agents and dialog-based tutoring to enhance teachers' content knowledge and pedagogical content knowledge. This paper describes the design of a conversational, agent-based intelligent tutoring system to support teachers' professional development. Pedagogical strategies are presented that leverage a virtual human facilitator to tutor pedagogical content knowledge (how to teach proportions to students), as opposed to content knowledge (understanding proportions). The roles for different virtual facilitator capabilities are presented, including embedding actions into virtual agent dialog, open-response versus choice-based tutoring, ungraded pop-up sub-activities (e.g. whiteboard, calculator, notetaking). Usability feedback for a small cohort of instructors pursuing graduate studies was collected. In this feedback, teachers rated the system ease of use and perceived usefulness moderately well, but also reported confusion about what to expect from the system in terms of flow between lessons and support by the facilitator.

As intelligent textbooks become more ubiquitous in classrooms and educational settings, the need arises to automatically provide formative feedback to written responses provided by students in response to readings. This study develops models to automatically provide feedback to student summaries written at the end of intelligent textbook sections. The study builds on Botarleanu et al. (2022), who used the Longformer Large Language Model, a transformer Neural Network, to build a summary grading model. Their model explains around 55% of holistic summary score variance when compared to scores assigned by human raters on an analytic rubric. This study uses a principal component analysis to distill scores from the analytic rubric into two principal components -- content and wording. When training the models on the summaries and the sources using these principal components, we explained 79% and 66% of the score variance for content and wording, respectively. The developed models are freely available on HuggingFace and will allow formative feedback to users of intelligent textbooks to assess reading comprehension through summarization in real-time. The models can also be used for other summarization applications in learning systems. [This paper was published in: "AIED 2023, CCIS 1831," edited by N. Wang et al., Springer Nature Switzerland, 2023, pp. 484-489.]

Math performance continues to be an important focus for improvement. The most recent National Report Card in the U.S. suggested student math scores declined in the past two years possibly due to COVID-19 pandemic and related school closures. We report on the implementation of a math homework program that leverages AI-based one-to-one technology, in 32 schools for two years as a part of a randomized controlled trial in diverse settings of the state of North Carolina in the US. The program, called "ASSISTments," provides feedback to students as they solve homework problems and automatically prepares reports for teachers about student performance on daily assignments. The paper describes the sample, the study design, the implementation of the intervention, including the recruitment effort, the training and support provided to teachers, and the approaches taken to assess teacher's progress and improve implementation fidelity. Analysis of data collected during the study suggest that (a) treatment teachers changed their homework review practices as they used ASSISTments, and (b) the usage of ASSISTments was positively correlated with student learning outcome.

Adaptive educational software is likely to better support broader and more diverse sets of learners by considering more comprehensive views (or models) of such learners. For example, recent work proposed making inferences about "non-math" factors like reading comprehension while students used adaptive software for mathematics to better support and adapt to learners. We build on this proposed approach to more comprehensive learning modeling by providing an empirical basis for making inferences about students' reading ability from their performance on activities in adaptive software for mathematics. We lay out an approach to predicting middle school students' reading ability using their performance on activities within Carnegie Learning's MATHia, a widely used intelligent tutoring system for mathematics. We focus on how performance in an early, introductory activity as an especially powerful place to consider instruction-embedded assessment of non-math factors like reading comprehension to guide adaptation based on factors like reading ability. We close by discussing opportunities to extend this work by focusing on particular knowledge components or skills tracked by MATHia that may provide important "levers" for driving adaptation based on students' reading ability while they learn and practice mathematics. [This paper was published in: "LAK23: 13th International Learning Analytics and Knowledge Conference," ACM, 2023.]

This paper reports on a quasi-experimental pilot study of the impacts of a high school teacher professional development (PD) series. The 10-session series, "Engaging High School Students in Academic Work," was designed to equip teachers to deepen students' learning and engagement and thereby increase course-passing rates. The study took place in a district in the U.S. Southwest that selected two of its interested high schools with similar demographic characteristics to participate, one receiving the PD and the other serving as the comparison school. Longitudinal analyses found statistically and educationally significant impacts of the PD series on four of the five measures in the primary outcome domain (students' course passing) and on teachers' reports of providing extra help. The article discusses limitations of the study and the need for further exploration of how PD can lead teachers to adopt more engaging instructional practices. It offers readers access to all PD materials and invites them into dialogue about how to adapt the PD series and its implementation going forward.

Integrating visual representations in an interactive learning activity effectively scaffolds performance and learning. However, it is unclear whether and how "sustaining" or "interleaving" visual scaffolding helps learners solve problems efficiently and learn from problem solving. We conducted a classroom study with 63 middle-school students in which we tested whether sustaining or interleaving a particular form of visual scaffolding, called anticipatory diagrammatic self-explanation in an Intelligent Tutoring System, helps students' learning and performance in the domain of early algebra. Sustaining visual scaffolding during problem solving helped students solve problems efficiently with no negative effects on learning. However, in-depth log data analyses suggest that interleaving visual scaffolding allowed students to practice important skills that may help them in later phases of algebra learning. This paper extends scientific understanding that sustaining visual scaffold does not over-scaffold student learning in the early phase of skill acquisition in algebra. [This paper was published in: "Proceedings of the 44th Annual Conference of the Cognitive Science Society," edited by J. Culbertson et al., Cognitive Science Society, 2022, pp. 1751-58.]

When designing an instructional tool and using it in pedagogical activities, it is essential that designers and users understand what pedagogical affordances and constraints the tool provides to support its successful integration into targeted pedagogical activities. Toward this end, we developed "Pedagogical Affordance Analysis (PAA)." PAA involves analyzing teachers' Pedagogical Content Knowledge and/or Technological Pedagogical Content Knowledge to elicit pedagogical affordances and constraints that are specific to a given instructional goal. Information obtained through PAA can help in designing, refining, and/or evaluating instructional tools. We present a case study in which we used PAA to successfully design a visual representation for middle-school algebra. To the best of our knowledge, PAA is the only available systematic method that leverages teachers' pedagogical knowledge in identifying pedagogical affordances and constraints. PAA can be used across a wide range of existing tools and prototypes of to-be-designed tools. [This paper was published in: "ICLS 2020 Proceedings," ISLS, 2020, pp. 1561-64.]

Proof, though central to mathematical practice, is rarely explored through the lens of embodiment because of the centrality of abstraction and generalization. We use the case of a high school geometry student to investigate two research questions: (1) How do embodied processes facilitate mathematical learning? (2) How can generalized mathematical truths be manifest through embodied processes that are grounded in particular movements? To engage the body, researchers developed a motion-capture video game, "The Hidden Village," designed to elicit physical movement via in-game directed actions. In-game interactions complement logic and verbal forms of reasoning by promoting simulated actions, which are shown to recur during the student's proof and justification. Embodied theories of learning offer insights into how learners recruit body-based resources to foster meaning-making and generalization. Embodiment also offers new insights into re-characterizing mathematics curricula in terms of movement, and the promise of new forms of embodied learning technologies. [This paper was published in: "ICLS 2020 Proceedings," ISLS, 2020, pp. 175-182.]

Large language models have recently been able to perform well in a wide variety of circumstances. In this work, we explore the possibility of large language models, specifically GPT-3, to write explanations for middle-school mathematics problems, with the goal of eventually using this process to rapidly generate explanations for the mathematics problems of new curricula as they emerge, shortening the time to integrate new curricula into online learning platforms. To generate explanations, two approaches were taken. The first approach attempted to summarize the salient advice in tutoring chat logs between students and live tutors. The second approach attempted to generate explanations using few-shot learning from explanations written by teachers for similar mathematics problems. After explanations were generated, a survey was used to compare their quality to that of explanations written by teachers. We test our methodology using the GPT-3 language model. Ultimately, the synthetic explanations were unable to outperform teacher written explanations. In the future more powerful large language models may be employed, and GPT-3 may still be effective as a tool to augment teachers' process for writing explanations, rather than as a tool to replace them. The explanations, survey results, analysis code, and a dataset of tutoring chat logs are all available at https://osf.io/wh5n9/.

Interactive Strategy Training for Active Reading and Thinking en Español, or iSTART-E, is a new intelligent tutoring system (ITS) that provides reading comprehension strategy training for Spanish speakers. This paper reports on studies evaluating the efficacy of iSTART-E in real-world classrooms in two different Spanish-speaking countries. In Study 1, Chilean high school students (n = 22) who practiced with iSTART-E showed significant gains on a standardized comprehension assessment (LECTUM) from pretest to posttest. In Study 2 (n = 85), Argentinian middle school students who practiced with iSTART-E showed greater gains on the ECOMPLEC.Sec comprehension test compared to those in control classrooms. Together these results suggest that iSTART-E is an effective means of enhancing Spanish speakers' reading comprehension, with demonstrated transfer of training to standardized reading tests. [This paper was published in: "Artificial Intelligence in Education. AIED 2018, LNAI 10948," edited by C. Penstein Rosé et al., Springer International Publishing AG, 2018, pp. 215-19.]

We apply an advanced data visualization technique, "Sankey diagram," to explore how middle-school students (N = 343) solved problems in a game-based algebraic notation tool. The results indicate that there is a large variation in the types of students' strategies to solve the problems, with some approaches being more efficient than others. The findings suggest that Sankey diagrams can be used both in research and practice to unpack our understanding of variability in mathematical problem-solving. [This paper was published in: "Proceedings of the 15th International Conference of the Learning Sciences--ICLS 2021," 2021, pp. 887-888.]

This paper describes an initial pilot study of Rimac, a natural-language tutoring system for physics. Rimac uses a student model to guide decisions about "what content to discuss next" during reflective dialogues that are initiated after students solve quantitative physics problems, and "how much support to provide" during these discussions, that is, domain contingent scaffolding and instructional contingent scaffolding, respectively. The pilot study compared an experimental and control version of Rimac. The experimental version uses students' responses to pretest items to initialize the student model and dynamically updates the model based on students' responses to tutor questions during reflective dialogues. It then decides what and how to discuss the next question based on the model predictions. The control version initializes its student model based on students' pretest performance but does not update the model further and assigns students to a fixed line of reasoning level based on the student model predictions. We hypothesized that students who used the experimental version of Rimac would achieve higher learning gains than students who used the control version. Although we did not find a significant difference in learning between conditions, the experimental group took significantly less time to complete the pilot study dialogues than did the control group. That is, the experimental condition led to more efficient learning, for both low and high prior knowledge level learners. We discuss this finding and describe future work to improve the tutor's potential to support student learning.

This study investigated the effect of incorporating spelling and grammar checking tools within an automated writing tutoring system, Writing Pal. High school students (n = 119) wrote and revised six persuasive essays. After initial drafts, all students received formative feedback about writing strategies. Half of the participants were also given access to spelling and grammar checking tools during the writing and revision periods. Linear mixed effects models revealed that essay quality for students in both conditions improved from initial draft to revision in terms of all aspects except essay unity. The availability of spelling and grammar checking yielded added improvements from initial draft to revision for several aspects of essay quality (i.e., conclusion, organization, voice, grammar/mechanics, and word choice), but other aspects were unaffected (i.e., introduction, body, unity, and sentence structure). The availability of spelling and grammar checking tools had no effect on holistic essay scores. These results indicate that automated spelling and grammar feedback contribute to modest, incremental improvements in writing quality that may complement automated strategy feedback. [This paper was published in: S. Isotani et al. (Eds.), "AIED 2019" (pp. 270-282). Switzerland: Springer.]

We study the usage of a self-guided online tutoring platform called Algebra Nation, which is widely by middle school and high school students who take the End-of-Course Algebra I exam at the end of the school year. This article aims to study how the platform contributes to increasing students' exam scores by examining users' logs over a three year period. The platform under consideration was used by more than 36,000 students in the first year, to nearly 67,000 by the third year, thus enabling us to examine how usage patterns evolved and influenced students' performance at scale. We first identify which Algebra Nation usage factors in conjunction with math overall preparation and socioeconomic factors contribute to the students' exam performance. Subsequently, we investigate the effect of increased teacher familiarity level with the Algebra Nation on students' scores across different grades through mediation analysis. The results show that the indirect effect of teacher's familiarity with the platform through increasing student's usage dosage is more significant in higher grades. [This paper was published in: "The 9th International Learning Analytics & Knowledge Conference (LAK19)" (Tempe, AZ, March 4-8, 2019). New York, NY: Association for Computing Machinery (ACM). (978-1-4503-6256-6).]

Mission HydroSci (MHS) teaches water systems and scientific argumentation towards meeting Next Generation Science Standards. MHS is a game-based 3D virtual environment for enacting transformational role-playing, wherein students must learn new knowledge and competencies in order to successfully complete the game missions. MHS was developed for middle school science as a replacement unit of about 6 to 8 hours and uses analytics and a teacher dashboard to help support teachers support their students. [This paper was published in: "CSCL 2019 Proceedings," p.1017-1020. ISLS.]

Prior studies on scaffolding for investigative inquiry practices (i.e. forming a question/hypothesis, collecting data, and analyzing and interpreting data [21]) revealed that students who received scaffolding were better able to both learn practices and transfer these competencies to new topics than were students who did not receive scaffolding. Prior studies have also shown that after removing scaffolding, students continued to demonstrate improved inquiry performance on a variety of practices across new driving questions over time. However, studies have not examined the relationship between the "amount of scaffolding received" and transfer of inquiry performance; this is the focus of the present study. 107 middle school students completed four virtual lab activities (i.e. driving questions) in Inq-ITS. Students received scaffolding when needed from an animated pedagogical computer agent for the first three driving questions for the Animal Cell virtual lab. Then they completed the fourth driving question without access to scaffolding in a different topic, Plant Cell. Results showed that students' performances increased even with fewer scaffolds for the inquiry practices of hypothesizing, collecting data, interpreting data, and warranting claims; furthermore, these results were robust as evidenced by the finding that students required less scaffolding as they completed subsequent inquiry activities. These data provide evidence of near and far transfer as a result of adaptive scaffolding of science inquiry practices. [This paper was published in: "Proceedings of the Sixth Annual ACM Conference on Learning at Scale."]

Understanding equivalence is fundamental to STEM disciplines, but many students struggle with the concept. We present a novel method for students to explore ideas of equivalence where students dynamically transform expressions from initial states to goal states in the web-based app Graspable Math. The structure of the goal state activities implies that any initial and goal state pair represent the same quantity (or varying quantity). We propose that for students, the physical experience of moving algebraic objects and observing how the initial state transforms into the goal state helps generalize notation mechanics. In fall of 2019, we will test this supposition in a randomized control trial of 1500 students in which student performance in pre- and post-tests will be compared to an online problem set control. [This paper was published in: "Proceedings of the forty-first annual meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education," edited by S. Otten et al., University of Missouri, 2019, pp. 1738-43.]

Mission HydroSci is a game-based 3D virtual environment for enacting transformational role-playing for middle school science students. Student-players will be engaged in a narrative about needing to investigate water resources and use scientific argumentation to complete missions critical to the survival and accomplishments of the members of their scientific enterprise. Our poster presents our progress in years 1 and 2 of a funded project to integrate pedagogical and gameplay objectives, and build mechanisms for purposeful player engagement and activity. The paper describes the argumentation engine as an example of developing a mechanism to enable learning. In additions, several innovative efforts have been made to support learning argumentation processes in games. MHS is being developed through a set of evaluations that support decision-making about design and development choices and eventually lead to a field test to assess outcomes. [This paper was published in: "Proceedings of the 12th International Conference on the Foundations of Digital Games" (Article 47). New York, NY: ACM.]

This study describes the role that Rasch measurement played in the development of assessments aligned to the "Next Generation Science Standards," tasks that require students to use the three dimensions of science practices, disciplinary core ideas and cross-cutting concepts to make sense of energy-related phenomena. A set of 27 three-dimensional, multi-item tasks were developed and field tested along with a set of multiple-choice test items focused on disciplinary core ideas. Data were collected from elementary, middle, and high school students from across the U.S. Rasch modeling was used to investigate the fit of the data to the model, the dimensionality of the data, differential functioning of the tasks, the relative difficulties of the tasks, and the performance of students by grade band.

The results of educational research studies are only as accurate as the data used to produce them. Drawing on experiences conducting large-scale efficacy studies of classroom-based algebra interventions for community college and middle school students, I am developing practice-based data cleaning procedures to support scholars in conducting rigorous research. The paper identifies common sources of data errors in mathematics education research and offers a framework and related data cleaning process designed to address these errors. [This paper was published in: Weinberg, A., Rasmussen, C., Rabin, J., Wawro, M., and Brown, S. (Eds.), "Proceedings of the 20th Annual Conference on Research in Undergraduate Mathematics Education," p129-140. San Diego, CA (2017).]

Prior research shows that self-explanation promotes understanding by helping learners connect new knowledge with prior knowledge. However, despite ample evidence supporting the effectiveness of self-explanation, an instructional design challenge emerges in how best to scaffold self-explanation. In particular, it is an open challenge to design self- explanation support that simultaneously facilitates performance and learning outcomes. Towards this goal, we designed anticipatory diagrammatic self-explanation, a novel form of self-explanation embedded in an Intelligent Tutoring System (ITS). In our ITS, anticipatory diagrammatic self-explanation scaffolds learners by providing visual representations to help learners predict an upcoming strategic step in algebra problem solving. A classroom experiment with 108 middle-school students found that anticipatory diagrammatic self-explanation helped students learn formal algebraic strategies and significantly improve their problem-solving performance. This study contributes to understanding of how self-explanation can be scaffolded to support learning and performance.

We investigated the effects of proximal grouping of numbers, problem-solving goals to make 100, and prior knowledge on students' solution strategies in an online mathematics game. Logistic regression on 857 problem-level data points from 227 middle-school students showed that students were more likely to use productive solution strategies on addition and multiplication problems when proximity supported number grouping, 100 was the problem-solving goal, and students had high prior knowledge. Furthermore, when proximity and number goals did not support problem-solving, students with low prior knowledge were less likely to use productive solution strategies on multiplication problems than students with high prior knowledge. Findings demonstrate that the effects of perceptual and number features on solution strategies vary by students' prior knowledge.