Search

In numerous studies, intelligent tutoring systems (ITSs) have proven effective in helping students learn mathematics. Prior work posits that their effectiveness derives from efficiently providing eventually-correct practice opportunities. Yet, there is little empirical evidence on how learning processes with ITSs compare to other forms of instruction. The current study compares problem-solving with an ITS versus solving the same problems on paper. We analyze the learning process and pre-post gain data from N = 97 middle school students practicing linear graphs in three curricular units. We find that (i) working with the ITS, students had more than twice the number of eventually-correct practice opportunities than when working on paper and (ii) omission errors on paper were associated with lower learning gains. Yet, contrary to our hypothesis, tutor practice did not yield greater learning gains, with tutor and paper comparing differently across curricular units. These findings align with tutoring allowing students to grapple with challenging steps through tutor assistance but not with eventually-correct opportunities driving learning gains. Gaming-the-system, lack of transfer to an unfamiliar test format, potentially ineffective tutor design, and learning affordances of paper can help explain this gap. This study provides first-of-its-kind quantitative evidence that ITSs yield more learning "opportunities" than equivalent paper-and-pencil practice and reveals that the relation between opportunities and learning gains emerges only when the instruction is effective.

We compared students' performance on a paper-based test (PBT) and three computer-based tests (CBTs). The three computer-based tests used different test navigation and answer selection features, allowing us to examine how these features affect student performance. The study sample consisted of 9,698 fourth through twelfth grade students from across the U.S. who were randomly assigned to take a test in one of the four modes. CBT modes differed in whether students could skip questions and freely move through the test, and whether students could click directly on the answer choice or had to click on a radio button at the bottom of the screen. Rasch analysis was used to estimate item difficulties and student performance levels. Student performance level was then used as an outcome in hiearchal linear models to determine the mode effects. We found that student performance was unaffected by whether the test was paper-based or computer-based. A comparison of student performance on the three CBTs indicated that restricting test navigation did not affect student performance, but allowing students to select an answer choice by directly clicking on it improved student performance. Our findings show that CBTs can be considered equivalent to PBTs, and the results can also be used to inform best practices for the design of other CBTs.

This is an extended conference proceedings report [Long Paper] based on a shorter contributed report at the same conference. A variety of computerized learning platforms exist. In mathematics, most include sets of problems to complete. Feedback to users ranges from a single word like "Correct!" to offers of hints and partially- to fully-worked examples. Behind-the-scenes design of such systems also varies--from static dictionaries of problems to responsive programming that adapts assignments to users' demonstrated skills within the computerized environment. This report presents background on digital learning contexts and early results of a mixed-methods study that included a cluster randomized controlled trial design. The study was in community college algebra classes where the intervention was a particular type of web-based activity and testing system. [This paper was published in: Weinberg, A., Rasmussen, C., Rabin, J., Wawro, M., and Brown, S. (Eds.), "Proceedings of the 20th Annual Conference on Research in Undergraduate Mathematics Education," p638-645. San Diego, CA (2017).]

Can student performance on computer-based tests (CBT) and paper-and-pencil tests (PPT) be considered equivalent measures of student knowledge? States and school districts are grappling with this question, and although studies addressing this question are growing, additional research is needed. We report on the performance of students who took either a PPT or one of two different CBT containing multiple-choice items assessing science ideas. Propensity score matching was used to create equivalent demographic groups for each testing modality, and Rasch modelling was used to describe student performance. Performance was found to vary across testing modalities by grade band, students' primary language, and the specific CBT system used. These results are discussed in terms of the current literature and the differences between the specific PPT and CBT systems.

Understanding key concepts in molecular biology requires reasoning about molecular processes that are not directly observable and, as such, presents a challenge to students and teachers. We ask whether novel interactive physical models and activities can help students understand key processes in viral replication. Our 3D tangible models are embedded with magnets that accurately represent chemical attractions and, in a study of 492 students, structured model use led to improved understanding of viral replication and self-assembly from pre- to posttest.

A web-based activity and testing system (WATS) has features such as adaptive problem sets, videos, and data-driven tools for instructors to monitor and scaffold student learning. Central to WATS adoption and use are questions about the implementation process: What constitutes "good" implementation and how far from "good" is "good enough"? This proceedings paper reports on and illustrates work to provide structure for such examination. The context is a study about implementation that is part of a state-wide randomized controlled trial examining student learning in community college algebra when a particular WATS suite of tools is used. Discussion questions for conference participants dug into the distinctions among intended, enacted, and achieved curriculum and the processes surrounding these as well as the challenges and opportunities in researching fidelity of implementation in the community college context, particularly the role of instructional practice as a contextual component of the research. [This paper was published in the "Proceedings of the 19th Annual Conference on Research in Undergraduate Mathematics Education," T. Fukawa-Connelly, N. Infante, K. Keene, and M. Zandieh (Eds), 2016.]

This paper explores how learning analytics data provided by a math problem-solving educational technology platform informed 5th and 6th grade teachers' instructional decisions around socioemotional learning (SEL). MathSpring is an educational technology tool that provides teachers with data on students' effort, progress, and emotions while students are solving math problems. This paper presents findings from qualitative analyses of 11 classroom observations, 10 observation follow-up chats, and 35 teacher interviews that were collected as part of a randomized control trial conducted in the northeastern United States. The analyses include teachers who used the program with their students (treatment group) and teachers who implemented math instruction as usual (control group). Teachers in the treatment group viewed analytic data on students' effort and progress, which informed the discussions they had with their students around SEL topics including setting goals, the importance of effort, and using productive problem-solving strategies. Findings from these analyses suggest that analytic data on effort and progress can inform teachers' discussions of math-related SEL concepts with their students. [This paper was published in: "The 14th Learning Analytics and Knowledge Conference (LAK '24), March 18-22, 2024, Kyoto, Japan," ACM, 2024.]

Intelligent tutoring systems (ITS) can help students learn successfully, yet little work has explored the role of caregivers in shaping that success. Past interventions to support caregivers in supporting their child's homework have been largely disjunct from educational technology. The paper presents prototyping design research with nine middle school caregivers. We ask: (1) what are caregivers' preferences for different prototypes incorporating data-driven recommendations into their math homework support? Integrating caregivers' preferences, we then ask: (2) what are caregivers' perceptions when interacting with a prototype of an intelligent chatbot tool to support students' homework? We found caregivers reported feeling comfortable integrating AI into their practices and appreciated chat-based support for understanding content and effective ITS use. Our results highlight the affordances of ITS data and AI to assist caregivers who would otherwise not be able to support their child's homework, paving the way for more effective and equitable mathematics learning. [This paper will be published in the ISLS2024 proceedings.]

Predictive analytics is widely used in various domains, including education, to inform decision-making and improve outcomes. However, many predictive models are proprietary and inaccessible for evaluation or modification by researchers and practitioners, limiting their accountability and ethical design. Moreover, predictive models are often opaque and incomprehensible to the officials who use them, reducing their trust and utility. Furthermore, predictive models may introduce or exacerbate bias and inequity, as they have done in many sectors of society. Therefore, there is a need for transparent, interpretable, and fair predictive models that can be easily adopted and adapted by different stakeholders. In this paper, we propose a fair predictive model based on multivariate adaptive regression splines (MARS) that incorporates fairness measures in the learning process. MARS is a non-parametric regression model that performs feature selection, handles non-linear relationships, generates interpretable decision rules, and derives optimal splitting criteria on the variables. Specifically, we integrate fairness into the knot optimization algorithm and provide theoretical and empirical evidence of how it results in a fair knot placement. We apply our "fair"MARS model to real-world data and demonstrate its effectiveness in terms of accuracy and equity. Our paper contributes to the advancement of responsible and ethical predictive analytics for social good. [This paper was presented at an Association for the Advancement of Artificial Intelligence conference.]

The manner in which gender is portrayed in materials used to teach children conveys messages about people's roles in society. In this paper, we measure the gendered depiction of central domains of social life in 100 years of highly influential children's books. We make two main contributions: (1) we find that the portrayal of gender in these books reproduces traditional gender norms in society; and (2) we publish StoryWords 1.0, the first word embeddings trained on such a large body of children's literature. We find that, relative to males, females are more likely to be represented in relation to their appearance than in relation to their competence; second, they are more likely to be represented in relation to their role in the family than their role in business. Finally, we find that non-binary or gender-fluid individuals are rarely mentioned. Our analysis advances understanding of the different messages contained in content commonly used to teach children, with immediate applications for practice, policy, and research. [This paper was published in: "Proceedings of the 29th International Conference on Computational Linguistics," 2022, pp. 3086-3097.]

In the effort to improve student outcomes, teachers are usually the focus of professional development efforts. However, research increasingly demonstrates the significance of administrators in influencing student outcomes, raising questions about how they too should be involved in such programs. The study involved interviews conducted with 12 administrators engaged in a large-scale professional development program dedicated to equitable math instruction for diverse learners. The paper reports on the different understandings they had regarding their role in supporting the innovation, including their perspective on diverse learners, perceptions of teacher needs, and approaches for teacher support. While conventional professional development programs might overlook administrators, this paper argues for a situated approach for engaging administrators in the complex process of leading change.

This paper presents a tool for creating student models in logistic regression. Creating student models has typically been done by expert selection of the appropriate terms, beginning with models as simple as IRT or AFM but more recently with highly complex models like BestLR. While alternative methods exist to select the appropriate predictors for the regression-based models (e.g., step-wise selection or LASSO), we are unaware of their application to student modeling. Such automatic methods of model creation offer the possibility of better student models with either reduced com-plexity or better fit, in addition to relieving experts from the bur-den of searching for better models by hand with possible human error. Our new functions are now part of the preexisting R pack-age LKT. We explain our search methods with two datasets demonstrating the advantages of using the tool with stepwise re-gression and regularization (LASSO) methods to aid in feature selection. For the stepwise method using BIC, the models are simpler (due to the BIC penalty for parameters) than alternatives like BestLR with little lack of fit. For the LASSO method, the models can be made simpler due to the fitting procedure involving a regularization parameter that penalizes large absolute coefficient values. However, LASSO also offers the possibility of highly complex models with exceptional fit. [This paper was published in: "Proceedings of the 16th International Conference on Educational Data Mining," edited by M. Feng et. al., International Educational Data Mining Society, 2023, pp.17-27.]

Solving mathematical problems is cognitively complex, involving strategy formulation, solution development, and the application of learned concepts. However, gaps in students' knowledge or weakly grasped concepts can lead to errors. Teachers play a crucial role in predicting and addressing these difficulties, which directly influence learning outcomes. However, preemptively identifying misconceptions leading to errors can be challenging. This study leverages historical data to assist teachers in recognizing common errors and addressing gaps in knowledge through feedback. We present a longitudinal analysis of incorrect answers from the 2015-2020 academic years on two curricula, Illustrative Math and EngageNY, for grades 6, 7, and 8. We find consistent errors across 5 years despite varying student and teacher populations. Based on these Common Wrong Answers (CWAs), we designed a crowdsourcing platform for teachers to provide Common Wrong Answer Feedback (CWAF). This paper reports on an in vivo randomized study testing the effectiveness of CWAFs in two scenarios: next-problem-correctness within-skill and next-problem-correctness within-assignment, regardless of the skill. We find that receiving CWAF leads to a significant increase in correctness for consecutive problems within-skill. However, the effect was not significant for all consecutive problems within-assignment, irrespective of the associated skill. This paper investigates the potential of scalable approaches in identifying Common Wrong Answers (CWAs) and how the use of crowdsourced CWAFs can enhance student learning through remediation. [This paper was published in: "Proceedings of the Tenth ACM Conference on Learning @ Scale (L@S '23)," July 20-22, 2023.]

Images in children's books convey messages about society and the roles that people play in it. Understanding these messages requires systematic measurement of who is represented. Computer vision face detection tools can provide such measurements; however, state-of-the-art face detection models were trained with photographs, and 80\% of images in children's books are illustrated; thus existing methods both misclassify and miss classifying many faces. In this paper, we introduce a new approach to analyze images using AI tools, resulting in data that can assess representation of race, gender, and age in both illustrations and photographs in children's books. We make four primary contributions to the fields of deep learning and social sciences: (1) We curate an original face detection data set (IllusFace 1.0) by manually labeling 5,403 illustrated faces with bounding boxes. (2) We train two AutoML-based face detection models for illustrations: (i) using IllusFace 1.0 (FDAI); (ii) using iCartoon, a publicly available data set (FDAI_iC), each optimized for illustrated images, detecting 2.5 times more faces in our testing data than the established face detector using Google Vision (FDGV). (3) We curate a data set of the race, gender, and age of 980 faces manually labeled by three different raters (CBFeatures 1.0). (4) We train an AutoML feature classification model (FCA) using CBFeatures 1.0. We compare FCA with the performance of another AutoML model that we trained on UTKFace, a public data set (FCA_UTK) and of an established model using FairFace (FCF). Finally, we examine distributions of character identities over the last century across the models. We find that FCA is 34% more accurate than FCF in its race predictions. These contributions provide tools to educators, caregivers, and curriculum developers to assess the representation contained in children's content. [This paper was published in: "Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)," 2022, pp. 462-471.]

AutoTutor-ARC (adult reading comprehension) is an intelligent tutoring system that uses conversational agents to help adult learners improve their comprehension skills. However, in such a system, not all lessons and items optimally serve the same purposes. In this paper, we describe a method for classifying items that are "instructive, evaluative, motivational," versus "potentially flawed" based on analyses of items' psychometric properties. Further, there is no a priori way of determining which lessons are optimal given the learner's reading profile needs. To address this, we evaluate how assessing learner component reading skills can inform various aspects of learner needs on AutoTutor lessons. More specifically, we compare learners who were classified as "proficient, underengaged, conscientious," versus "struggling" readers based on their experiences with AutoTutor. Together, these analyses suggest the utility of integrating assessments with instruction: efficient, adaptive learning at the lesson level, more efficient and valid post-testing, and consequently, recommendations for more targeted, adaptive pathways through the instructional program/system. [This paper was published in: "AIED 2022, LNCS 13356," 2022, pp. 520-523.]

It is particularly important to identify and address issues of fairness and equity in educational contexts as academic performance can have large impacts on the types of opportunities that are made available to students. While it is always the hope that educators approach student assessment with these issues in mind, there are a number of factors that likely impact how a teacher approaches the scoring of student work. Particularly in cases where the assessment of student work requires subjective judgment, as in the case of open-ended answers and essays, contextual information such as how the student has performed in the past, general perceptions of the student, and even other external factors such as fatigue may all influence how a teacher approaches assessment. While such factors exist, however, it is not always clear how these may introduce bias, nor is it clear whether such bias poses measurable risks to fairness and equity. In this paper, we examine these factors in the context of the assessment of student answers to open response questions from middle school mathematics learners. We observe how several factors such as context and fatigue correlate with teacher-assigned grades and discuss how learning systems may support fair assessment. [This paper was published in: "Proceedings of the 30th International Conference on Computers in Education," edited by S. Iyer et al., Asia-Pacific Society for Computers in Education, 2022.]

The use of technology as a facilitator in learning environments has become increasingly prevalent with the global pandemic caused by COVID-19. As such, computer-supported collaborative learning (CSCL) gains a wider adoption in contrast to traditional learning methods. At the same time, the need for automated tools capable of assessing and stimulating collaboration between participants has become more stringent, as human monitoring of the increasing volume of conversations becomes overwhelming. This paper introduces a method grounded in dialogism for evaluating students' involvement in chat conversations based on semantic chains computed using language models. These semantic chains reflect emergent voices from dialogism that span and interact throughout the conversation. Our integrated method uses contextual information captured by BERT transformer models to identify links in a chain that connects semantically related concepts from a voice uttered by one or more participants. Two types of visualizations were generated to depict the longitudinal propagation and the transversal inter-animation of voices within the conversation. In addition, a list of handcrafted features derived from the constructed chains and computed for each participant is introduced. Several machine learning algorithms were tested using these features to evaluate the extent to which semantic chains are predictive of student involvement in chat conversations. [This paper was published in: "Ludic, Co-design and Tools Supporting Smart Learning Ecosystems and Smart Education, Proceedings of the 6th International Conference on Smart Learning Ecosystems and Regional Development," edited by Ó. Mealha et al., Springer Nature Singapore Pte Ltd., 2022, pp. 67-78.]

In this paper, we present iSTART-Early, an intelligent tutoring system that provides automated instruction and practice on higher-order reading comprehension strategies to 3rd and 4th grade students. iSTART-Early provides personalized, interactive, game-based strategy instruction and practice on comprehension strategies (i.e., Ask It, Reword It, Find It, Explain It, and Summarize It) with grade-appropriate informational texts. Natural language processing (NLP) combined with automated speech recognition (ASR) and text-to-speech technologies enable immediate formative and summative feedback. A teacher interface allows teachers to assign texts and monitor students' performance so that they can provide additional support and feedback when necessary, creating blended-learning opportunities. We describe the interface and the development of iSTART-Early, as well as our plans to examine the intelligent tutoring system for usability, feasibility and promise in improving reading comprehension strategies and outcomes for young readers. [This paper was published in: S. Crossley and E. Popescu, Eds., "ITS 2022, LNCS 13284," Springer Nature, 2022, pp.371-379.]

Cloze items are commonly used for both assessing learning and as a learning activity. This paper investigates the selection of sentences for cloze item creation by comparing methods ranging from simple heuristics to deep learning summarization models. An evaluation using human-generated cloze items from three different science texts indicates that simple heuristics substantially outperform summarization models, including state-of-the-art deep learning models. These results suggest that sentence selection for cloze item generation should be considered a distinct task from summarization and that continued advances on this task will require large datasets of human-generated cloze items. [This paper was published in: "Joint Proceedings of the Workshops at the 14th International Conference on Educational Data Mining," Vol. 3051, LDI-6, edited by T. W. Price and S. San Pedro, CEUR-WS.org, 2021.]

Teachers often rely on curricular materials as support for implementing reform-based science practices and as educative materials to continue developing ways in which they teach and approach science. Therefore, it is important to understand how teachers develop and use curricular materials and how they learn from their experiences. However, we lack deeper dives into longitudinal relationships that teachers have with curricular materials. This study explores the ways a teacher designs and adapts curriculum over three years. The study tracks the teacher's involvement in co-designing curricular materials alongside other teachers, researchers, and curriculum designers and adapting and using these materials in her own classroom. This study will trace curricular values that the teacher uses to guide her relationships with the curriculum and the adaptations she chooses to make for her own classroom context. The study focuses on a moment in class that the teacher was particularly proud of because it created a palpable shift in the students' engagement with the content she was teaching. This study aims to draw a narrative of this teacher's journey across her three years of writing and adapting materials she was using. [This paper was published in: "Hawaii International Conference on Education Proceedings 2024."]

This paper explores three simple data manipulation techniques (synthesis, augmentation, curriculum) for improving abstractive summarization models without the need for any additional data. We introduce a method of data synthesis with paraphrasing, a data augmentation technique with sample mixing, and curriculum learning with two new difficulty metrics based on specificity and abstractiveness. We conduct experiments to show that these three techniques can help improve abstractive summarization across two summarization models and two different small datasets. Furthermore, we show that these techniques can improve performance when applied in isolation and when combined. [This paper was published in: "Findings of the Association for Computational Linguistics: EMNLP 2021," Association for Computational Linguistics, 2021, pp. 2043-2052.]

This paper explores the effect of using multitask learning for abstractive summarization in the context of small training corpora. In particular, we incorporate four different tasks (extractive summarization, language modeling, concept detection, and paraphrase detection) both individually and in combination, with the goal of enhancing the target task of abstraction summarization via multitask learning. We show that for many task combinations, a model trained in a multitask setting outperforms a model trained only for abstractive summarization, with no additional summarization data introduced. Additionally, we do a comprehensive search and find that certain tasks (e.g. paraphrase detection) consistently benefit abstractive summarization, not only when combined with other tasks but also when using different architectures and training corpora. [This paper was published in: "Findings of the Association for Computational Linguistics: EMNLP 2021," Association for Computational Linguistics, 2021, pp. 1652-1661.]

As evidence grows supporting the importance of non-cognitive factors in learning, computer-assisted learning platforms increasingly incorporate non-academic interventions to influence student learning and learning related-behaviors. Non-cognitive interventions often attempt to influence students' mindset, motivation, or metacognitive reflection to impact learning behaviors and outcomes. In the current paper, we analyze data from five experiments, involving seven treatment conditions embedded in mastery-based learning activities hosted on a computer-assisted learning platform focused on middle school mathematics. Each treatment condition embodied a specific non-cognitive theoretical perspective. Over seven school years, 20,472 students participated in the experiments. We estimated the effects of each treatment condition on students' response time, hint usage, likelihood of mastering knowledge components, learning efficiency, and post-tests performance. Our analyses reveal a mix of both positive and negative treatment effects on student learning behaviors and performance. Few interventions impacted learning as assessed by the post-tests. These findings highlight the difficulty in positively influencing student learning behaviors and outcomes using non-cognitive interventions. [This paper was published in: "LAK23: 13th International Learning Analytics and Knowledge Conference Proceedings," March 13-17, 2023.]

An intelligent textbook may be defined as an interaction layer between the text and the student, helping the student master the content in the text. The Mobile Fact and Concept Training System (MoFaCTS) is an adaptive instructional system for simple content that has been developed into an interaction layer to mediate textbook instruction and so is being transformed into the Mobile Fact and Concept Textbook System (MoFaCTS). In this paper, we document the several terms of the logistic regression model we use to track performance adaptively. We then examine the contribution of each component of our model when it is fit to 4 semesters of Anatomy and Physiology course practice data. Following this documentation of the model, we explain how it is applied in the MoFaCTS system to schedule performance by targeting practice for each item at an optimal efficiency threshold. [This paper was published in the CEUR workshop proceedings (Vol. 2895).]

This paper explores the concept of dynamically generating definitions using a deep-learning model. We do this by creating a dataset that contains definition entries and contexts associated with each definition. We then fine-tune a GPT-2 based model on the dataset to allow the model to generate contextual definitions. We evaluate our model with human raters by generating definitions using two context types: short-form (the word used in a sentence) and long-form (the word used in a sentence along with the prior and following sentences). Results indicate that the model performed significantly better when generating definitions using short-form contexts. Additionally, we evaluate our model against human-generated definitions. The results show promise for the model, showing that the model was able to match human-level fluency. However, while it was able to reach human-level accuracy in some instances, it failed in others. [This paper was published in: "Proceedings of the Third International Workshop on Intelligent Textbooks," Vol. 2895, CEUR-WS.org, 2021, pp. 74-83.]

This paper explores a general approach to paraphrase generation using a pre-trained seq2seq model fine-tuned using a back-translated anatomy and physiology textbook. Human ratings indicate that the paraphrase model generally preserved meaning and grammaticality/fluency: 70% of meaning ratings were above 75, and 40% of paraphrases were considered more grammatical/fluent than the originals. An error analysis suggests potential avenues for future work. [This paper was published in: "Proceedings of the 22nd International Conference on Artificial Intelligence in Education," edited by I. Roll, D. McNamara, S. Sosnovsky, R. Luckin, and V. Dimitrova, Springer International Publishing, 2021, pp. 279-284.]

This paper presents WikiMorph, a tool that automatically breaks down words into morphemes, etymological compounds (morphemes from root languages), and generates contextual definitions for each component. It comes in two flavors: a dataset and a deep-learning-based model. The dataset was extracted from Wiktionary and contains over 450k entries. We then used this dataset to train a GPT-2 model to generalize and decompose any word into morphemes and their definitions. We find that the model accurately generates complex breakdowns when given a high-quality initial definition. [This paper was published in: "Proceedings of the 22nd International Conference on Artificial Intelligence in Education," edited by I. Roll, D. McNamara, S. Sosnovsky, R. Luckin, and V. Dimitrova, Springer International Publishing, 2021, pp. 406-11.]

Exploring new or emerging research domains or subdomains can become overwhelming due to the magnitude of available resources and the high speed at which articles are published. As such, a tool that curates the information and underlines central entities, both authors and articles from a given research context, is highly desirable. Starting from the articles of the International Conference of Learning Analytics & Knowledge (LAK) in its first decade, this paper proposes a novel method grounded in Cohesion Network Analysis (CNA) to analyze subcommunities of authors based on the semantic similarities between authors and papers, and estimate their global impact. Paper abstracts are represented as embeddings using a fine-tuned SciBERT language model, alongside a custom trained LSA model. The extrapolation between the local LAK community to a worldwide importance was also underlined by the comparison between the rankings obtained from our method and statistics from ResearchGate. The accuracies for binary classifications in terms of high/low impact predictions were around 70% for authors, and around 80% for articles. Our method can guide researchers by providing valuable information on the interactions between the members of a knowledge community and by highlighting central local authors who may potentially have a high global impact.

Human essay grading is a laborious task that can consume much time and effort. Automated Essay Scoring (AES) has thus been proposed as a fast and effective solution to the problem of grading student writing at scale. However, because AES typically uses supervised machine learning, a human-graded essay corpus is still required to train the AES model. Unfortunately, such a graded corpus often does not exist, so creating a corpus for machine learning can also be a laborious task. This paper presents an investigation of replacing the use of human-labeled essay grades when training an AES system with two automatically available but weaker signals of essay quality: word count and topic distribution similarity. Experiments using two source-based essay scoring (evidence score) corpora show that while weak supervision does not yield a competitive result when training a neural source-based AES model, it can be used to successfully extract Topical Components (TCs) from a source text, which are required by a supervised feature-based AES model. In particular, results show that feature-based AES performance is comparable with either automatically or manually constructed TCs. [This paper was published in: "Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications," Association for Computational Linguistics, 2021, pp. 85-96.]

The paper discusses the use of "Productive Disciplinary Engagement" (PDE) for a curricular project that features a technology-based alternate reality game (ARG) with the objective of teaching undergraduate students about the collaborative nature of STEM careers. Much of the PDE research uses PDE as either a design-principle or as an analytics lens. This project does both. Most of this extant research focuses on spoken discourse to teach disciplinary knowledge. This project uses workplace documentary texts that are embedded within a semester-long undergraduate course designed to teach students collaboration skills using the context of natural disasters. A range of texts are used in this design from didactic to disciplinary. Students learn about professional work through educational renditions of professional cultural historical activity systems. This paper focuses on design decisions and illustrates some ways that workplace documents can be used in education.

Computer-assisted learning platforms (CALPS) increasingly include gamified elements to improve student outcomes by enhancing their engagement with content. Although evidence exists that gamified programs increase engagement and learning outcomes, there is little causal research on what programmatic mechanisms drive the effect between engagement and learning. In the following paper, we explore this relationship through a method of causal moderation known as fully latent principal stratification. Using data from a large-scale randomized control trial assessing gamified and traditional CALP systems' effects on algebraic knowledge, we estimate the impact of using the gamified CALP on students who engage with one of its key gamification elements--replaying a problem after a suboptimal attempt. The gamified CALP asks students to manipulate algebraic expressions from start to goal states and provides feedback based on the efficiency of these manipulations, allowing students to replay the problems when their efficiency can be improved. We find that the effect of gamification is greater for students with a higher propensity to replay problems. This finding suggests that gamification elements that provide students with opportunities to retry problems are driving the game's efficacy and provide evidence for a scalable mechanism of gamification that can improve students' learning. [This paper was published in: "Proceedings of the Tenth ACM Conference on Learning @ Scale (L@S '23), July 20-22, 2023, Copenhagen, Denmark," ACM, 2023, pp.164-173.]

Reading comprehension is essential for both knowledge acquisition and memory reinforcement. Automated modeling of the comprehension process provides insights into the efficacy of specific texts as learning tools. This paper introduces an improved version of the Automated Model of Comprehension, version 3.0 (AMoC v3.0). AMoC v3.0 is based on two theoretical models of the comprehension process, namely the Construction-Integration and the Landscape models. In addition to the lessons learned from the previous versions, AMoC v3.0 uses Transformer-based contextualized embeddings to build and update the concept graph as a simulation of reading. Besides taking into account generative language models and presenting a visual walkthrough of how the model works, AMoC v3.0 surpasses the previous version in terms of the Spearman correlations between our activation scores and the values reported in the original Landscape Model for the presented use case. Moreover, features derived from AMoC significantly differentiate between high-low cohesion texts, thus arguing for the model's capabilities to simulate different reading conditions. [This paper was published in: "AIED 2023: Artificial Intelligence in Education," edited by N. Wang et al., Springer, Switzerland, 2023, pp. 229-241, 2023.]

This paper systematically explores how Large Language Models (LLMs) generate explanations of code examples of the type used in intro-to-programming courses. As we show, the nature of code explanations generated by LLMs varies considerably based on the wording of the prompt, the target code examples being explained, the programming language, the temperature parameter, and the version of the LLM. Nevertheless, they are consistent in two major respects for Java and Python: the readability level, which hovers around 7-8 grade, and lexical density, i.e., the relative size of the meaninful words with respect to the total explanation size. Furthermore, the explanations score very high in correctness but less on three other metrics: completeness, conciseness, and contextualization. [This paper is in: Proceedings of the workshop on Generative AI for Education(GAIED): Advances, Opportunities, and Challenges, 2003.]

Numerous approaches have been introduced to automate the process of text summarization, but only few can be easily adapted to multiple languages. This paper introduces a multilingual text processing pipeline integrated in the open-source "ReaderBench" framework, which can be retrofit to cover more than 50 languages. While considering the extensibility of the approach and the problem of missing labeled data for training in various languages besides English, an unsupervised algorithm was preferred to perform extractive summarization (i.e., select the most representative sentences from the original document). Specifically, two different approaches relying on text cohesion were implemented:(1) a graph-based text representation derived from Cohesion Network Analysis that extends TextRank; and (2) a class of submodular set functions. Evaluations were performed on the DUC dataset and use as baseline the implementation of TextRank from Gensim. Our results using the submodular set functions outperform the baseline. In addition, two use cases on English and Romanian languages are presented, with corresponding graphical representations for the two methods. [This paper was published in: 22nd International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC) Proceedings, 2020, pp. 161-168 (ISBN 978-1-7281-7628-4).]

This paper investigates two specific probabilistic biases which middle graders usually exhibit when reasoning about probability and randomness on assessment items. We discuss how students' reasoning about key probability concepts undergirds statistics literacy related to randomness, independence, and the likelihood of future events based on past results. We examine factors evoking misconceptions and students' (in)consistency in exhibiting them. Findings indicate that misconceptions can be evoked based on three types of factors including (1) students' particular understandings of probability and randomness, (2) general item characteristics, and (3) aspects of probability in items. Moreover, possession of a specific misconception will most likely result in exhibiting the bias again on other occasions including the same evoking factors (consistency). [This paper was published in: "Proceedings of the Satellite Conference of the International Association for Statistical Education (IASE): Statistics Education in the Era of Data Science," edited by R. Helenius and E. Falck, International Association for Statistical Education, 2021, pp. 1-6.]

Adult learners with low literacy skills compose a highly heterogeneous population in terms of demographic variables, educational backgrounds, knowledge and skills in reading, self-efficacy, motivation etc. They also face various difficulties in consistently attending offline literacy programs, such as unstable worktime, transportation difficulties, and childcare issues. AutoTutor for Adult Reading Comprehension (AT-ARC), as an online conversation-based intelligent tutoring system that incorporated a theoretical model of reading comprehension, was developed with great efforts to meet adult learners' needs and be adaptive to their knowledge, skills, self-efficacy, and motivation. In this paper, we introduced the adaptive features of AT-ARC from four aspects: learning material selection, adaptive branching, trialogues, and interface, as well as the rationale behind these designs. In the end, we suggested further research on improving the adaptivity of AT-ARC. [This paper was published in: "HCII 2021," Lecture Notes in Computer Science (LNCS) Vol. 12792, edited by R. A. Sottilare and J. Schwarz, Springer Nature Switzerland AG, 2021, pp. 592-603.]

Reading comprehension is key to knowledge acquisition and to reinforcing memory for previous information. While reading, a mental representation is constructed in the reader's mind. The mental model comprises the words in the text, the relations between the words, and inferences linking to concepts in prior knowledge. The automated model of comprehension (AMoC) simulates the construction of readers' mental representations of text by building syntactic and semantic relations between words, coupled with inferences of related concepts that rely on various automated semantic models. This paper introduces the second version of AMoC that builds upon the initial model with a revised processing pipeline in Python leveraging state-of-the-art NLP models, additional heuristics for improved representations, as well as a new radiant graph visualization of the comprehension model. [This paper was published in: "AIED 2021," edited by I. Roll et al., Springer Nature Switzerland AG, 2021, pp. 119-123.]

The ability to objectively quantify the complexity of a text can be a useful indicator of how likely learners of a given level will comprehend it. Before creating more complex models of assessing text difficulty, the basic building block of a text consists of words and, inherently, its overall difficulty is greatly influenced by the complexity of underlying words. One approach is to measure a word's Age of Acquisition (AoA), an estimate of the average age at which a speaker of a language understands the semantics of a specific word. Age of Exposure (AoE) statistically models the process of word learning, and in turn an estimate of a given word's AoA. In this paper, we expand on the model proposed by AoE by training regression models that learn and generalize AoA word lists across multiple languages including English, German, French, and Spanish. Our approach allows for the estimation of AoA scores for words that are not found in the original lists, up to the majority of the target language's vocabulary. Our method can be uniformly applied across multiple languages though the usage of parallel corpora and helps bridge the gap in the size of AoA word lists available for non-English languages. This effort is particularly important for efforts toward extending AI to languages with fewer resources and benchmarked corpora. [This paper was published in: "AIED 2021," edited by I. Roll et al., Springer Nature Switzerland AG, 2021, pp. 77-87.]

Multi-angle question answering models have recently been proposed that promise to perform related tasks like question generation. However, performance on related tasks has not been thoroughly studied. We investigate a leading model called Macaw on the task of multiple choice question generation and evaluate its performance on three angles that systematically reduce the complexity of the task. Our results indicate that despite the promise of generalization, Macaw performs poorly on untrained angles. Even on a trained angle, Macaw fails to generate four distinct multiple-choice options on 17% of inputs. We propose augmenting multiple-choice options by paraphrasing angle input and show this increases overall success to 97.5%. A human evaluation comparing the augmented multiple-choice questions with textbook questions on the same topic reveals that Macaw questions broadly score highly but below human questions. [This paper was published in: "Proceedings of the 3rd Workshop of the Learner Data Institute," edited by S. E. Fancsali and V. Rus, Learner Data Institute, 2022, pp. 18-23.]

A longstanding goal of learner modeling and educational data mining is to improve the domain model of knowledge that is used to make inferences about learning and performance. In this report we present a tool for finding domain models that is built into an existing modeling framework, logistic knowledge tracing (LKT). LKT allows the flexible specification of learner models in logistic regression by allowing the modeler to select whatever features of the data are relevant to prediction. Each of these features (such as the count of prior opportunities) is a function computed for a component of data (such as a student or knowledge component). In this context, we have developed the "autoKC" component, which clusters knowledge components and allows the modeler to compute features for the clustered components. For an autoKC, the input component (initial KC or item assignment) is clustered prior to computing the feature and the feature is a function of that cluster. Another recent new function for LKT, which allows us to specify interactions between the logistic regression predictor terms, is combined with autoKC for this report. Interactions allow us to move beyond just assuming the cluster information has additive effects to allow us to model situations where a second factor of the data moderates a first factor. [This paper was published in: S. E. Fancsali and V. Rus (Eds.), "Proceedings of the 3rd Workshop of the Learner Data Institute," pp. 7-12, 2022. Learner Data Institute.]

As online learning platforms become more ubiquitous throughout various curricula, there is a growing need to evaluate the effectiveness of these platforms and the different methods used to structure online education and tutoring. Towards this endeavor, some platforms have performed randomized controlled experiments to compare different user experiences, curriculum structures, and tutoring strategies in order to ensure the effectiveness of their platform and personalize the education of the students using it. These experiments are typically analyzed on an individual basis in order to reveal insights on a specific aspect of students' online educational experience. In this work, the data from 50,752 instances of 30,408 students participating in 50 different experiments conducted at scale within the online learning platform ASSISTments were aggregated and analyzed for consistent trends across experiments. By combining common experimental conditions and normalizing the dependent measures between experiments, this work has identified multiple statistically significant insights on the impact of various skill mastery requirements, strategies for personalization, and methods for tutoring in an online setting. This work can help direct further experimentation and inform the design and improvement of new and existing online learning platforms. The anonymized data compiled for this work are hosted by the Open Science Foundation and can be found at https://osf.io/59shv/. [This paper was published in: "Proceedings of the 15th International Conference on Educational Data Mining," edited by A. Mitrovic and N. Bosch, International Educational Data Mining Society, 2022, pp. 27-38.]