Search

In numerous studies, intelligent tutoring systems (ITSs) have proven effective in helping students learn mathematics. Prior work posits that their effectiveness derives from efficiently providing eventually-correct practice opportunities. Yet, there is little empirical evidence on how learning processes with ITSs compare to other forms of instruction. The current study compares problem-solving with an ITS versus solving the same problems on paper. We analyze the learning process and pre-post gain data from N = 97 middle school students practicing linear graphs in three curricular units. We find that (i) working with the ITS, students had more than twice the number of eventually-correct practice opportunities than when working on paper and (ii) omission errors on paper were associated with lower learning gains. Yet, contrary to our hypothesis, tutor practice did not yield greater learning gains, with tutor and paper comparing differently across curricular units. These findings align with tutoring allowing students to grapple with challenging steps through tutor assistance but not with eventually-correct opportunities driving learning gains. Gaming-the-system, lack of transfer to an unfamiliar test format, potentially ineffective tutor design, and learning affordances of paper can help explain this gap. This study provides first-of-its-kind quantitative evidence that ITSs yield more learning "opportunities" than equivalent paper-and-pencil practice and reveals that the relation between opportunities and learning gains emerges only when the instruction is effective.

We compared students' performance on a paper-based test (PBT) and three computer-based tests (CBTs). The three computer-based tests used different test navigation and answer selection features, allowing us to examine how these features affect student performance. The study sample consisted of 9,698 fourth through twelfth grade students from across the U.S. who were randomly assigned to take a test in one of the four modes. CBT modes differed in whether students could skip questions and freely move through the test, and whether students could click directly on the answer choice or had to click on a radio button at the bottom of the screen. Rasch analysis was used to estimate item difficulties and student performance levels. Student performance level was then used as an outcome in hiearchal linear models to determine the mode effects. We found that student performance was unaffected by whether the test was paper-based or computer-based. A comparison of student performance on the three CBTs indicated that restricting test navigation did not affect student performance, but allowing students to select an answer choice by directly clicking on it improved student performance. Our findings show that CBTs can be considered equivalent to PBTs, and the results can also be used to inform best practices for the design of other CBTs.

This is an extended conference proceedings report [Long Paper] based on a shorter contributed report at the same conference. A variety of computerized learning platforms exist. In mathematics, most include sets of problems to complete. Feedback to users ranges from a single word like "Correct!" to offers of hints and partially- to fully-worked examples. Behind-the-scenes design of such systems also varies--from static dictionaries of problems to responsive programming that adapts assignments to users' demonstrated skills within the computerized environment. This report presents background on digital learning contexts and early results of a mixed-methods study that included a cluster randomized controlled trial design. The study was in community college algebra classes where the intervention was a particular type of web-based activity and testing system. [This paper was published in: Weinberg, A., Rasmussen, C., Rabin, J., Wawro, M., and Brown, S. (Eds.), "Proceedings of the 20th Annual Conference on Research in Undergraduate Mathematics Education," p638-645. San Diego, CA (2017).]

Can student performance on computer-based tests (CBT) and paper-and-pencil tests (PPT) be considered equivalent measures of student knowledge? States and school districts are grappling with this question, and although studies addressing this question are growing, additional research is needed. We report on the performance of students who took either a PPT or one of two different CBT containing multiple-choice items assessing science ideas. Propensity score matching was used to create equivalent demographic groups for each testing modality, and Rasch modelling was used to describe student performance. Performance was found to vary across testing modalities by grade band, students' primary language, and the specific CBT system used. These results are discussed in terms of the current literature and the differences between the specific PPT and CBT systems.

Understanding key concepts in molecular biology requires reasoning about molecular processes that are not directly observable and, as such, presents a challenge to students and teachers. We ask whether novel interactive physical models and activities can help students understand key processes in viral replication. Our 3D tangible models are embedded with magnets that accurately represent chemical attractions and, in a study of 492 students, structured model use led to improved understanding of viral replication and self-assembly from pre- to posttest.

A web-based activity and testing system (WATS) has features such as adaptive problem sets, videos, and data-driven tools for instructors to monitor and scaffold student learning. Central to WATS adoption and use are questions about the implementation process: What constitutes "good" implementation and how far from "good" is "good enough"? This proceedings paper reports on and illustrates work to provide structure for such examination. The context is a study about implementation that is part of a state-wide randomized controlled trial examining student learning in community college algebra when a particular WATS suite of tools is used. Discussion questions for conference participants dug into the distinctions among intended, enacted, and achieved curriculum and the processes surrounding these as well as the challenges and opportunities in researching fidelity of implementation in the community college context, particularly the role of instructional practice as a contextual component of the research. [This paper was published in the "Proceedings of the 19th Annual Conference on Research in Undergraduate Mathematics Education," T. Fukawa-Connelly, N. Infante, K. Keene, and M. Zandieh (Eds), 2016.]

This paper explores how learning analytics data provided by a math problem-solving educational technology platform informed 5th and 6th grade teachers' instructional decisions around socioemotional learning (SEL). MathSpring is an educational technology tool that provides teachers with data on students' effort, progress, and emotions while students are solving math problems. This paper presents findings from qualitative analyses of 11 classroom observations, 10 observation follow-up chats, and 35 teacher interviews that were collected as part of a randomized control trial conducted in the northeastern United States. The analyses include teachers who used the program with their students (treatment group) and teachers who implemented math instruction as usual (control group). Teachers in the treatment group viewed analytic data on students' effort and progress, which informed the discussions they had with their students around SEL topics including setting goals, the importance of effort, and using productive problem-solving strategies. Findings from these analyses suggest that analytic data on effort and progress can inform teachers' discussions of math-related SEL concepts with their students. [This paper was published in: "The 14th Learning Analytics and Knowledge Conference (LAK '24), March 18-22, 2024, Kyoto, Japan," ACM, 2024.]

Worked examples, which present an explained code for solving typical programming problems are among the most popular types of learning content in programming classes. Most approaches and tools for presenting these examples to students are based on line-by-line explanations of the example code. However, instructors rarely have time to provide explanations for many examples typically used in a programming class. In this paper, we assess the feasibility of using LLMs to generate code explanations for passive and active example exploration systems. To achieve this goal, we compare the code explanations generated by chatGPT with the explanations generated by both experts and students. [This paper was published in: "Proceedings of Machine Learning Research" (2024).]

Intelligent tutoring systems (ITS) can help students learn successfully, yet little work has explored the role of caregivers in shaping that success. Past interventions to support caregivers in supporting their child's homework have been largely disjunct from educational technology. The paper presents prototyping design research with nine middle school caregivers. We ask: (1) what are caregivers' preferences for different prototypes incorporating data-driven recommendations into their math homework support? Integrating caregivers' preferences, we then ask: (2) what are caregivers' perceptions when interacting with a prototype of an intelligent chatbot tool to support students' homework? We found caregivers reported feeling comfortable integrating AI into their practices and appreciated chat-based support for understanding content and effective ITS use. Our results highlight the affordances of ITS data and AI to assist caregivers who would otherwise not be able to support their child's homework, paving the way for more effective and equitable mathematics learning. [This paper will be published in the ISLS2024 proceedings.]

Predictive analytics is widely used in various domains, including education, to inform decision-making and improve outcomes. However, many predictive models are proprietary and inaccessible for evaluation or modification by researchers and practitioners, limiting their accountability and ethical design. Moreover, predictive models are often opaque and incomprehensible to the officials who use them, reducing their trust and utility. Furthermore, predictive models may introduce or exacerbate bias and inequity, as they have done in many sectors of society. Therefore, there is a need for transparent, interpretable, and fair predictive models that can be easily adopted and adapted by different stakeholders. In this paper, we propose a fair predictive model based on multivariate adaptive regression splines (MARS) that incorporates fairness measures in the learning process. MARS is a non-parametric regression model that performs feature selection, handles non-linear relationships, generates interpretable decision rules, and derives optimal splitting criteria on the variables. Specifically, we integrate fairness into the knot optimization algorithm and provide theoretical and empirical evidence of how it results in a fair knot placement. We apply our "fair"MARS model to real-world data and demonstrate its effectiveness in terms of accuracy and equity. Our paper contributes to the advancement of responsible and ethical predictive analytics for social good. [This paper was presented at an Association for the Advancement of Artificial Intelligence conference.]

As language technologies have become more sophisticated and prevalent, there have been increasing concerns about bias in natural language processing (NLP). Such work often focuses on the effects of bias instead of sources. In contrast, this paper discusses how normative language assumptions and ideologies influence a range of automated language tools. These underlying assumptions can inform (a) grammar and tone suggestions provided by commercial products, (b) language varieties (e.g., dialects and other norms) taught by language learning technologies, (c) language patterns used by chatbots and similar applications to interact with users. These tools demonstrate considerable technological advancement but are rarely interrogated with regard to the language ideologies they intentionally or implicitly reinforce. We consider prior research on language ideologies and how they may impact (at scale) the large language models (LLMs) that underlie many automated language technologies. Specifically, this paper draws on established theoretical frameworks for understanding how humans typically perceive or judge language varieties and patterns that may differ from their own or their perceived standard. We then discuss how language ideologies can perpetuate social hierarchies and stereotypes, even within seemingly impartial automation. In doing so, we contribute to the emerging literature on how the risks of language ideologies and assumptions can be better understood and mitigated in the design, testing, and implementation of automated language technologies. [This paper was published in: "Universal Access in Human-Computer Interaction (HCII 2024) Proceedings, LNCS 14696," edited by M. Antona and C. Stephanidis, 2024, pp. 271-290.]

The manner in which gender is portrayed in materials used to teach children conveys messages about people's roles in society. In this paper, we measure the gendered depiction of central domains of social life in 100 years of highly influential children's books. We make two main contributions: (1) we find that the portrayal of gender in these books reproduces traditional gender norms in society; and (2) we publish StoryWords 1.0, the first word embeddings trained on such a large body of children's literature. We find that, relative to males, females are more likely to be represented in relation to their appearance than in relation to their competence; second, they are more likely to be represented in relation to their role in the family than their role in business. Finally, we find that non-binary or gender-fluid individuals are rarely mentioned. Our analysis advances understanding of the different messages contained in content commonly used to teach children, with immediate applications for practice, policy, and research. [This paper was published in: "Proceedings of the 29th International Conference on Computational Linguistics," 2022, pp. 3086-3097.]

Platform-enabled research services will control, manage, and measure learner experiences within that platform. In this paper, we consider the need for research services that examine learner experiences "outside" the platform. For example, we describe an effort to conduct an experiment on peer assessment in a college writing course, where Terracotta (a research service within the learning management system) randomly assigned students to either (1) write peer assessments themselves, (2) use a generative AI tool to provide the feedback, or (3) build upon and improve the feedback provided by generative AI. This research effort was not successful, and it stands as an example of the limitations of current platform-enabled research services, and the need for infrastructure to support research beyond existing platforms.

In the effort to improve student outcomes, teachers are usually the focus of professional development efforts. However, research increasingly demonstrates the significance of administrators in influencing student outcomes, raising questions about how they too should be involved in such programs. The study involved interviews conducted with 12 administrators engaged in a large-scale professional development program dedicated to equitable math instruction for diverse learners. The paper reports on the different understandings they had regarding their role in supporting the innovation, including their perspective on diverse learners, perceptions of teacher needs, and approaches for teacher support. While conventional professional development programs might overlook administrators, this paper argues for a situated approach for engaging administrators in the complex process of leading change.

Images in children's books convey messages about society and the roles that people play in it. Understanding these messages requires systematic measurement of who is represented. Computer vision face detection tools can provide such measurements; however, state-of-the-art face detection models were trained with photographs, and 80\% of images in children's books are illustrated; thus existing methods both misclassify and miss classifying many faces. In this paper, we introduce a new approach to analyze images using AI tools, resulting in data that can assess representation of race, gender, and age in both illustrations and photographs in children's books. We make four primary contributions to the fields of deep learning and social sciences: (1) We curate an original face detection data set (IllusFace 1.0) by manually labeling 5,403 illustrated faces with bounding boxes. (2) We train two AutoML-based face detection models for illustrations: (i) using IllusFace 1.0 (FDAI); (ii) using iCartoon, a publicly available data set (FDAI_iC), each optimized for illustrated images, detecting 2.5 times more faces in our testing data than the established face detector using Google Vision (FDGV). (3) We curate a data set of the race, gender, and age of 980 faces manually labeled by three different raters (CBFeatures 1.0). (4) We train an AutoML feature classification model (FCA) using CBFeatures 1.0. We compare FCA with the performance of another AutoML model that we trained on UTKFace, a public data set (FCA_UTK) and of an established model using FairFace (FCF). Finally, we examine distributions of character identities over the last century across the models. We find that FCA is 34% more accurate than FCF in its race predictions. These contributions provide tools to educators, caregivers, and curriculum developers to assess the representation contained in children's content. [This paper was published in: "Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)," 2022, pp. 462-471.]

AutoTutor-ARC (adult reading comprehension) is an intelligent tutoring system that uses conversational agents to help adult learners improve their comprehension skills. However, in such a system, not all lessons and items optimally serve the same purposes. In this paper, we describe a method for classifying items that are "instructive, evaluative, motivational," versus "potentially flawed" based on analyses of items' psychometric properties. Further, there is no a priori way of determining which lessons are optimal given the learner's reading profile needs. To address this, we evaluate how assessing learner component reading skills can inform various aspects of learner needs on AutoTutor lessons. More specifically, we compare learners who were classified as "proficient, underengaged, conscientious," versus "struggling" readers based on their experiences with AutoTutor. Together, these analyses suggest the utility of integrating assessments with instruction: efficient, adaptive learning at the lesson level, more efficient and valid post-testing, and consequently, recommendations for more targeted, adaptive pathways through the instructional program/system. [This paper was published in: "AIED 2022, LNCS 13356," 2022, pp. 520-523.]

It is particularly important to identify and address issues of fairness and equity in educational contexts as academic performance can have large impacts on the types of opportunities that are made available to students. While it is always the hope that educators approach student assessment with these issues in mind, there are a number of factors that likely impact how a teacher approaches the scoring of student work. Particularly in cases where the assessment of student work requires subjective judgment, as in the case of open-ended answers and essays, contextual information such as how the student has performed in the past, general perceptions of the student, and even other external factors such as fatigue may all influence how a teacher approaches assessment. While such factors exist, however, it is not always clear how these may introduce bias, nor is it clear whether such bias poses measurable risks to fairness and equity. In this paper, we examine these factors in the context of the assessment of student answers to open response questions from middle school mathematics learners. We observe how several factors such as context and fatigue correlate with teacher-assigned grades and discuss how learning systems may support fair assessment. [This paper was published in: "Proceedings of the 30th International Conference on Computers in Education," edited by S. Iyer et al., Asia-Pacific Society for Computers in Education, 2022.]

The use of technology as a facilitator in learning environments has become increasingly prevalent with the global pandemic caused by COVID-19. As such, computer-supported collaborative learning (CSCL) gains a wider adoption in contrast to traditional learning methods. At the same time, the need for automated tools capable of assessing and stimulating collaboration between participants has become more stringent, as human monitoring of the increasing volume of conversations becomes overwhelming. This paper introduces a method grounded in dialogism for evaluating students' involvement in chat conversations based on semantic chains computed using language models. These semantic chains reflect emergent voices from dialogism that span and interact throughout the conversation. Our integrated method uses contextual information captured by BERT transformer models to identify links in a chain that connects semantically related concepts from a voice uttered by one or more participants. Two types of visualizations were generated to depict the longitudinal propagation and the transversal inter-animation of voices within the conversation. In addition, a list of handcrafted features derived from the constructed chains and computed for each participant is introduced. Several machine learning algorithms were tested using these features to evaluate the extent to which semantic chains are predictive of student involvement in chat conversations. [This paper was published in: "Ludic, Co-design and Tools Supporting Smart Learning Ecosystems and Smart Education, Proceedings of the 6th International Conference on Smart Learning Ecosystems and Regional Development," edited by Ó. Mealha et al., Springer Nature Singapore Pte Ltd., 2022, pp. 67-78.]

In this paper, we present iSTART-Early, an intelligent tutoring system that provides automated instruction and practice on higher-order reading comprehension strategies to 3rd and 4th grade students. iSTART-Early provides personalized, interactive, game-based strategy instruction and practice on comprehension strategies (i.e., Ask It, Reword It, Find It, Explain It, and Summarize It) with grade-appropriate informational texts. Natural language processing (NLP) combined with automated speech recognition (ASR) and text-to-speech technologies enable immediate formative and summative feedback. A teacher interface allows teachers to assign texts and monitor students' performance so that they can provide additional support and feedback when necessary, creating blended-learning opportunities. We describe the interface and the development of iSTART-Early, as well as our plans to examine the intelligent tutoring system for usability, feasibility and promise in improving reading comprehension strategies and outcomes for young readers. [This paper was published in: S. Crossley and E. Popescu, Eds., "ITS 2022, LNCS 13284," Springer Nature, 2022, pp.371-379.]

Prisms VR, a secondary math learning application, allows for users to see, manipulate, and engage with mathematical concepts in an embodied way in Virtual Reality (VR) environment. We examine cases in which mathematics teachers and middle school students worked through Prisms and reflected upon their experiences. Findings indicate that VR environments can help teachers find alternative ways of presenting algebraic concepts in a hands-on, interactive, and embodied manner; and these findings were confirmed based on the student experience. VR allows teachers to make connections through scaffolding and personalized experiences, while also aiding students in the exploration and meaning-making. [This paper was published in: "Proceedings of the 18th International Conference of the Learning Sciences - ICLS 2024," edited by R. Lindgren et al., International Society of the Learning Sciences, 2024, pp. 1610-13.]

Access to body-based resources has been shown to augment cognitive processes, but not all movements equally aid reasoning. Interactive technologies, like dynamic geometry systems (DGS), potentially amplify the link between movement and geometric representation, thereby deepening students' understanding of geometric properties. This study investigated the influence of access to a DGS on students' geometric reasoning and gesture production compared to the impact of only spontaneous gestures. Access to simulations did not significantly increase the likelihood of students producing correct mathematical insights compared to spontaneous gestures. However, when controlling gesture production, DGS access became a significant predictor for transformational proof production. We also found that access to DGS influenced the type of gestures students produced during geometric reasoning tasks. This study underlines the vital role of gestures in geometric reasoning and suggests that while DGS offers benefits, spontaneous gestures remain a crucial embodied resource for exploring geometric concepts. [This paper was published in: "Proceedings of the 18th International Conference of the Learning Sciences - ICLS 2024," edited by R. Lindgren et al., International Society of the Learning Sciences, 2024, pp. 131-38.]

Perseverance is critical for students' achievement and may be particularly important after COVID-19. This paper includes analyses of teacher and principal interviews and student educational technology usage data to examine students' perseverance during math problem-solving across three cohorts of students during and after COVID-19. Data were collected as part of a randomized control trial examining the efficacy of an educational technology platform, MathSpring, in supporting students' math achievement. The results indicated that teachers and principals were concerned with their students' perseverance, and students' usage data similarly demonstrated that students may have reduced perseverance when solving math problems after COVID-19. Teachers and students may benefit from using tools that help foster perseverance in math class. MathSpring and other educational technology tools that support perseverance could be useful for encouraging students to put forth effort when solving math problems, which could in turn support students' math learning outcomes.

This paper presents a tool for creating student models in logistic regression. Creating student models has typically been done by expert selection of the appropriate terms, beginning with models as simple as IRT or AFM but more recently with highly complex models like BestLR. While alternative methods exist to select the appropriate predictors for the regression-based models (e.g., step-wise selection or LASSO), we are unaware of their application to student modeling. Such automatic methods of model creation offer the possibility of better student models with either reduced com-plexity or better fit, in addition to relieving experts from the bur-den of searching for better models by hand with possible human error. Our new functions are now part of the preexisting R pack-age LKT. We explain our search methods with two datasets demonstrating the advantages of using the tool with stepwise re-gression and regularization (LASSO) methods to aid in feature selection. For the stepwise method using BIC, the models are simpler (due to the BIC penalty for parameters) than alternatives like BestLR with little lack of fit. For the LASSO method, the models can be made simpler due to the fitting procedure involving a regularization parameter that penalizes large absolute coefficient values. However, LASSO also offers the possibility of highly complex models with exceptional fit. [This paper was published in: "Proceedings of the 16th International Conference on Educational Data Mining," edited by M. Feng et. al., International Educational Data Mining Society, 2023, pp.17-27.]

Solving mathematical problems is cognitively complex, involving strategy formulation, solution development, and the application of learned concepts. However, gaps in students' knowledge or weakly grasped concepts can lead to errors. Teachers play a crucial role in predicting and addressing these difficulties, which directly influence learning outcomes. However, preemptively identifying misconceptions leading to errors can be challenging. This study leverages historical data to assist teachers in recognizing common errors and addressing gaps in knowledge through feedback. We present a longitudinal analysis of incorrect answers from the 2015-2020 academic years on two curricula, Illustrative Math and EngageNY, for grades 6, 7, and 8. We find consistent errors across 5 years despite varying student and teacher populations. Based on these Common Wrong Answers (CWAs), we designed a crowdsourcing platform for teachers to provide Common Wrong Answer Feedback (CWAF). This paper reports on an in vivo randomized study testing the effectiveness of CWAFs in two scenarios: next-problem-correctness within-skill and next-problem-correctness within-assignment, regardless of the skill. We find that receiving CWAF leads to a significant increase in correctness for consecutive problems within-skill. However, the effect was not significant for all consecutive problems within-assignment, irrespective of the associated skill. This paper investigates the potential of scalable approaches in identifying Common Wrong Answers (CWAs) and how the use of crowdsourced CWAFs can enhance student learning through remediation. [This paper was published in: "Proceedings of the Tenth ACM Conference on Learning @ Scale (L@S '23)," July 20-22, 2023.]

Cloze items are commonly used for both assessing learning and as a learning activity. This paper investigates the selection of sentences for cloze item creation by comparing methods ranging from simple heuristics to deep learning summarization models. An evaluation using human-generated cloze items from three different science texts indicates that simple heuristics substantially outperform summarization models, including state-of-the-art deep learning models. These results suggest that sentence selection for cloze item generation should be considered a distinct task from summarization and that continued advances on this task will require large datasets of human-generated cloze items. [This paper was published in: "Joint Proceedings of the Workshops at the 14th International Conference on Educational Data Mining," Vol. 3051, LDI-6, edited by T. W. Price and S. San Pedro, CEUR-WS.org, 2021.]

This paper explores the effect of using multitask learning for abstractive summarization in the context of small training corpora. In particular, we incorporate four different tasks (extractive summarization, language modeling, concept detection, and paraphrase detection) both individually and in combination, with the goal of enhancing the target task of abstraction summarization via multitask learning. We show that for many task combinations, a model trained in a multitask setting outperforms a model trained only for abstractive summarization, with no additional summarization data introduced. Additionally, we do a comprehensive search and find that certain tasks (e.g. paraphrase detection) consistently benefit abstractive summarization, not only when combined with other tasks but also when using different architectures and training corpora. [This paper was published in: "Findings of the Association for Computational Linguistics: EMNLP 2021," Association for Computational Linguistics, 2021, pp. 1652-1661.]

This paper explores three simple data manipulation techniques (synthesis, augmentation, curriculum) for improving abstractive summarization models without the need for any additional data. We introduce a method of data synthesis with paraphrasing, a data augmentation technique with sample mixing, and curriculum learning with two new difficulty metrics based on specificity and abstractiveness. We conduct experiments to show that these three techniques can help improve abstractive summarization across two summarization models and two different small datasets. Furthermore, we show that these techniques can improve performance when applied in isolation and when combined. [This paper was published in: "Findings of the Association for Computational Linguistics: EMNLP 2021," Association for Computational Linguistics, 2021, pp. 2043-2052.]

An intelligent textbook may be defined as an interaction layer between the text and the student, helping the student master the content in the text. The Mobile Fact and Concept Training System (MoFaCTS) is an adaptive instructional system for simple content that has been developed into an interaction layer to mediate textbook instruction and so is being transformed into the Mobile Fact and Concept Textbook System (MoFaCTS). In this paper, we document the several terms of the logistic regression model we use to track performance adaptively. We then examine the contribution of each component of our model when it is fit to 4 semesters of Anatomy and Physiology course practice data. Following this documentation of the model, we explain how it is applied in the MoFaCTS system to schedule performance by targeting practice for each item at an optimal efficiency threshold. [This paper was published in the CEUR workshop proceedings (Vol. 2895).]

This paper explores the concept of dynamically generating definitions using a deep-learning model. We do this by creating a dataset that contains definition entries and contexts associated with each definition. We then fine-tune a GPT-2 based model on the dataset to allow the model to generate contextual definitions. We evaluate our model with human raters by generating definitions using two context types: short-form (the word used in a sentence) and long-form (the word used in a sentence along with the prior and following sentences). Results indicate that the model performed significantly better when generating definitions using short-form contexts. Additionally, we evaluate our model against human-generated definitions. The results show promise for the model, showing that the model was able to match human-level fluency. However, while it was able to reach human-level accuracy in some instances, it failed in others. [This paper was published in: "Proceedings of the Third International Workshop on Intelligent Textbooks," Vol. 2895, CEUR-WS.org, 2021, pp. 74-83.]

This paper explores a general approach to paraphrase generation using a pre-trained seq2seq model fine-tuned using a back-translated anatomy and physiology textbook. Human ratings indicate that the paraphrase model generally preserved meaning and grammaticality/fluency: 70% of meaning ratings were above 75, and 40% of paraphrases were considered more grammatical/fluent than the originals. An error analysis suggests potential avenues for future work. [This paper was published in: "Proceedings of the 22nd International Conference on Artificial Intelligence in Education," edited by I. Roll, D. McNamara, S. Sosnovsky, R. Luckin, and V. Dimitrova, Springer International Publishing, 2021, pp. 279-284.]

This paper presents WikiMorph, a tool that automatically breaks down words into morphemes, etymological compounds (morphemes from root languages), and generates contextual definitions for each component. It comes in two flavors: a dataset and a deep-learning-based model. The dataset was extracted from Wiktionary and contains over 450k entries. We then used this dataset to train a GPT-2 model to generalize and decompose any word into morphemes and their definitions. We find that the model accurately generates complex breakdowns when given a high-quality initial definition. [This paper was published in: "Proceedings of the 22nd International Conference on Artificial Intelligence in Education," edited by I. Roll, D. McNamara, S. Sosnovsky, R. Luckin, and V. Dimitrova, Springer International Publishing, 2021, pp. 406-11.]

Exploring new or emerging research domains or subdomains can become overwhelming due to the magnitude of available resources and the high speed at which articles are published. As such, a tool that curates the information and underlines central entities, both authors and articles from a given research context, is highly desirable. Starting from the articles of the International Conference of Learning Analytics & Knowledge (LAK) in its first decade, this paper proposes a novel method grounded in Cohesion Network Analysis (CNA) to analyze subcommunities of authors based on the semantic similarities between authors and papers, and estimate their global impact. Paper abstracts are represented as embeddings using a fine-tuned SciBERT language model, alongside a custom trained LSA model. The extrapolation between the local LAK community to a worldwide importance was also underlined by the comparison between the rankings obtained from our method and statistics from ResearchGate. The accuracies for binary classifications in terms of high/low impact predictions were around 70% for authors, and around 80% for articles. Our method can guide researchers by providing valuable information on the interactions between the members of a knowledge community and by highlighting central local authors who may potentially have a high global impact.

Human essay grading is a laborious task that can consume much time and effort. Automated Essay Scoring (AES) has thus been proposed as a fast and effective solution to the problem of grading student writing at scale. However, because AES typically uses supervised machine learning, a human-graded essay corpus is still required to train the AES model. Unfortunately, such a graded corpus often does not exist, so creating a corpus for machine learning can also be a laborious task. This paper presents an investigation of replacing the use of human-labeled essay grades when training an AES system with two automatically available but weaker signals of essay quality: word count and topic distribution similarity. Experiments using two source-based essay scoring (evidence score) corpora show that while weak supervision does not yield a competitive result when training a neural source-based AES model, it can be used to successfully extract Topical Components (TCs) from a source text, which are required by a supervised feature-based AES model. In particular, results show that feature-based AES performance is comparable with either automatically or manually constructed TCs. [This paper was published in: "Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications," Association for Computational Linguistics, 2021, pp. 85-96.]

The paper discusses the use of "Productive Disciplinary Engagement" (PDE) for a curricular project that features a technology-based alternate reality game (ARG) with the objective of teaching undergraduate students about the collaborative nature of STEM careers. Much of the PDE research uses PDE as either a design-principle or as an analytics lens. This project does both. Most of this extant research focuses on spoken discourse to teach disciplinary knowledge. This project uses workplace documentary texts that are embedded within a semester-long undergraduate course designed to teach students collaboration skills using the context of natural disasters. A range of texts are used in this design from didactic to disciplinary. Students learn about professional work through educational renditions of professional cultural historical activity systems. This paper focuses on design decisions and illustrates some ways that workplace documents can be used in education.

Recent research seeks to develop more comprehensive learner models for adaptive learning software. For example, models of reading comprehension built using data from students' use of adaptive instructional software for mathematics have recently been developed. These models aim to deliver experiences that consider factors related to learning beyond performance in the target domain for instruction. We investigate the extent to which generalization is possible for a recently developed predictive model that seeks to infer students' reading comprehension ability (as measured by end-of-year standardized test scores) using an introductory learning experience in Carnegie Learning's MATHia intelligent tutoring system for mathematics. Building on a model learned on data from middle school students in a single school district in a mid-western U.S. state, using that state's end-of-year English Language Arts (ELA) standardized test score as an outcome, we consider data from a school district in a south-eastern U.S. state as well as that state's end-of-year ELA standardized test outcome. Generalization is explored by considering prediction performance when training and testing models on data from each of the individual school districts (and for their respective state's test outcomes) as well as pooling data from both districts together. We conclude with discussion of investigations of some algorithmic fairness characteristics of the learned models. The results suggest that a model trained on data from the smaller of the two school districts considered may achieve greater fairness in its predictions over models trained on data from the other district or both districts, despite broad, overall similarities in some demographic characteristics of the two school districts. This raises interesting questions for future research on generalizing these kinds of models as well as on ensuring algorithmic fairness of resulting models for use in real-world adaptive systems for learning. [This paper was published in: "Proceedings of the 16th International Conference on Educational Data Mining," edited by M. Feng et al., International Educational Data Mining Society, 2023, pp. 207-16.]

School rating websites are increasingly used by parents to assess the quality and fit of U.S. K-12 schools for their children. These online reviews often contain detailed descriptions of a school's strengths and weaknesses, which both reflect and inform perceptions of a school. Existing work on these text reviews has focused on finding words or themes that underlie these perceptions, but has stopped short of using the textual reviews as leading indicators of school performance. In this paper, we investigate to what extent the language used in online reviews of a school is predictive of changes in the attributes of that school, such as its socio-economic makeup and student test scores. Using over 300K reviews of 70K U.S. schools from a popular ratings website, we apply language processing models to predict whether schools will significantly increase or decrease in an attribute of interest over a future time horizon. We find that using the text improves predictive performance significantly over a baseline model that does not include text but only the historical time-series of the indicators themselves, suggesting that the review text carries predictive power. A qualitative analysis of the most predictive terms and phrases used in the text reviews indicates a number of topics that serve as leading indicators, such as diversity, changes in school leadership, a focus on testing, and school safety.

Assessing students' answers and in particular natural language answers is a crucial challenge in the field of education. Advances in transformer-based models such as Large Language Models (LLMs), have led to significant progress in various natural language tasks. Nevertheless, amidst the growing trend of evaluating LLMs across diverse tasks, evaluating LLMs in the realm of automated answer assessment has not received much attention. To address this gap, we explore the potential of using LLMs for automated assessment of student's short and open-ended answers in program comprehension tasks. Particularly, we use LLMs to compare students' explanations with expert explanations in the context of line-by-line explanations of computer programs. For comparison purposes, we assess both decoder-only Large Language Models (LLMs) and encoder-based Semantic Textual Similarity (STS) models in the context of assessing the correctness of students' explanation of computer code. Our findings indicate that decoder-only LLMs, when prompted in few-shot and chain-of-thought setting perform comparable to fine-tuned encoder-based models in evaluating students' short answers in the programming domain. [[This paper was published in: "Proceedings of Machine Learning Research" (2024).]

The process of generating challenging and appropriate distractors for multiple-choice questions is a complex and time-consuming task. Existing methods for an automated generation have limitations in proposing challenging distractors, or they fail to effectively filter out incorrect choices that closely resemble the correct answer, share synonymous meanings, or imply the same information. To overcome these challenges, we propose a comprehensive toolkit that integrates various approaches for generating distractors, including leveraging a general knowledge base and employing a T5 LLM. Additionally, we introduce a novel strategy that utilizes natural language inference to increase the accuracy of the generated distractors by removing confusing options. Our models have zero-shot capabilities and achieve good results on the DGen dataset; moreover, the models were fine-tuned and outperformed state-of-the-art methods on the considered dataset. To further extend the analysis, we introduce human annotations with scores for 100 test questions with 1085 distractors in total. The evaluations indicated that our generated options are of high quality, surpass all previous automated methods, and are on par with the ground truth of human-defined alternatives. [This paper was published in: "AIED 2024, LNAI 14830," edited by A. M. Olney et al., Springer Nature Switzerland, 2024, pp. 242-50.]

Reading comprehension is essential for students' ability to build knowledge. Students' comprehension abilities can be enhanced by providing students with deliberate practice and formative feedback on reading comprehension strategies. iSTART is an Intelligent Tutoring System (ITS) that is designed to provide instruction in reading strategies with minimal teacher supervision -- affording the ability to teach reading strategies at scale. In the current study, undergraduate students received reading strategy instruction and opportunities for deliberate practice via the iSTART intelligent tutoring system or not (i.e., no-treatment control group). Participants' reading comprehension and psychology knowledge were assessed. The iSTART group demonstrated substantially greater scores than the control group on a post-training reading comprehension measure (Cohen's d > 1.0). The average psychology knowledge scores did not differ between iSTART (post-training) and control groups, but overall these scores were unexpectedly low. Within the iSTART group, there was no difference in reading comprehension and knowledge scores as a function of students' different behaviors in the system. Overall, the results indicate that iSTART is an effective tool to teach reading strategies at large scale. However, further work is required to test the extent to which iSTART supports knowledge building. [This paper was published in: "Proceedings of Learning @ Scale conference (L@S '24)," ACM, 2024.]

Teachers often rely on curricular materials as support for implementing reform-based science practices and as educative materials to continue developing ways in which they teach and approach science. Therefore, it is important to understand how teachers develop and use curricular materials and how they learn from their experiences. However, we lack deeper dives into longitudinal relationships that teachers have with curricular materials. This study explores the ways a teacher designs and adapts curriculum over three years. The study tracks the teacher's involvement in co-designing curricular materials alongside other teachers, researchers, and curriculum designers and adapting and using these materials in her own classroom. This study will trace curricular values that the teacher uses to guide her relationships with the curriculum and the adaptations she chooses to make for her own classroom context. The study focuses on a moment in class that the teacher was particularly proud of because it created a palpable shift in the students' engagement with the content she was teaching. This study aims to draw a narrative of this teacher's journey across her three years of writing and adapting materials she was using. [This paper was published in: "Hawaii International Conference on Education Proceedings 2024."]

Numerous approaches have been introduced to automate the process of text summarization, but only few can be easily adapted to multiple languages. This paper introduces a multilingual text processing pipeline integrated in the open-source "ReaderBench" framework, which can be retrofit to cover more than 50 languages. While considering the extensibility of the approach and the problem of missing labeled data for training in various languages besides English, an unsupervised algorithm was preferred to perform extractive summarization (i.e., select the most representative sentences from the original document). Specifically, two different approaches relying on text cohesion were implemented:(1) a graph-based text representation derived from Cohesion Network Analysis that extends TextRank; and (2) a class of submodular set functions. Evaluations were performed on the DUC dataset and use as baseline the implementation of TextRank from Gensim. Our results using the submodular set functions outperform the baseline. In addition, two use cases on English and Romanian languages are presented, with corresponding graphical representations for the two methods. [This paper was published in: 22nd International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC) Proceedings, 2020, pp. 161-168 (ISBN 978-1-7281-7628-4).]

This paper investigates two specific probabilistic biases which middle graders usually exhibit when reasoning about probability and randomness on assessment items. We discuss how students' reasoning about key probability concepts undergirds statistics literacy related to randomness, independence, and the likelihood of future events based on past results. We examine factors evoking misconceptions and students' (in)consistency in exhibiting them. Findings indicate that misconceptions can be evoked based on three types of factors including (1) students' particular understandings of probability and randomness, (2) general item characteristics, and (3) aspects of probability in items. Moreover, possession of a specific misconception will most likely result in exhibiting the bias again on other occasions including the same evoking factors (consistency). [This paper was published in: "Proceedings of the Satellite Conference of the International Association for Statistical Education (IASE): Statistics Education in the Era of Data Science," edited by R. Helenius and E. Falck, International Association for Statistical Education, 2021, pp. 1-6.]

Adult learners with low literacy skills compose a highly heterogeneous population in terms of demographic variables, educational backgrounds, knowledge and skills in reading, self-efficacy, motivation etc. They also face various difficulties in consistently attending offline literacy programs, such as unstable worktime, transportation difficulties, and childcare issues. AutoTutor for Adult Reading Comprehension (AT-ARC), as an online conversation-based intelligent tutoring system that incorporated a theoretical model of reading comprehension, was developed with great efforts to meet adult learners' needs and be adaptive to their knowledge, skills, self-efficacy, and motivation. In this paper, we introduced the adaptive features of AT-ARC from four aspects: learning material selection, adaptive branching, trialogues, and interface, as well as the rationale behind these designs. In the end, we suggested further research on improving the adaptivity of AT-ARC. [This paper was published in: "HCII 2021," Lecture Notes in Computer Science (LNCS) Vol. 12792, edited by R. A. Sottilare and J. Schwarz, Springer Nature Switzerland AG, 2021, pp. 592-603.]