Search

Mission HydroSci is a game-based 3D virtual environment for enacting transformational role-playing for middle school science students. Student-players will be engaged in a narrative about needing to investigate water resources and use scientific argumentation to complete missions critical to the survival and accomplishments of the members of their scientific enterprise. Our poster presents our progress in years 1 and 2 of a funded project to integrate pedagogical and gameplay objectives, and build mechanisms for purposeful player engagement and activity. The paper describes the argumentation engine as an example of developing a mechanism to enable learning. In additions, several innovative efforts have been made to support learning argumentation processes in games. MHS is being developed through a set of evaluations that support decision-making about design and development choices and eventually lead to a field test to assess outcomes. [This paper was published in: "Proceedings of the 12th International Conference on the Foundations of Digital Games" (Article 47). New York, NY: ACM.]

Adaptive learning technologies hold great promise for improving the reading skills of adults with low literacy, but adults with low literacy skills typically have low computer literacy skills. In order to determine whether adults with low literacy skills would be able to use an intelligent tutoring system for reading comprehension, we adapted a 44 task computer literacy assessment and delivered it to 114 adults with reading skills between 3rd and 8th grade levels. This paper presents four analyses on these data. First, we report the pass/fail data natively exported by the assessment for particular computer-based tasks. Second, we undertook a GOMS [Goals, Operators, Methods, & Selection rules] analysis of each computer-based task, to predict the task completion time for a skilled user, and found that it negatively correlated with proportion correct for each item, r(42) = -0.4, p = 0.01. Third, we used the GOMS task decomposition to develop a Q-matrix of component computer skills for each task, and using logistic mixed effects models on this matrix identified five component skills highly predictive of the success or failure of an individual on a computer task: function keys, typing, using icons, right clicking, and mouse dragging. And finally, we assessed the predictive value of all component skills using logistic lasso. [This paper was published in: "Proceedings of the 10th International Conference on Educational Data Mining" (p128-134).]

In this paper, we investigate two purported problems with Bayesian Knowledge Tracing (BKT), a popular statistical model of student learning: "identifiability" and "semantic model degeneracy." In 2007, Beck and Chang stated that BKT is susceptible to an "identifiability problem"--various models with different parameters can give rise to the same predictions about student performance. We show that the problem they pointed out was not an identifiability problem, and using an existing result from the identifiability of hidden Markov models, we show that under mild conditions on the parameters, BKT is actually identifiable. In the second part of the paper, we discuss a problem that has been conflated with identifiability, but which actually does arise when fitting BKT models, "semantic model degeneracy"--the model parameters that best fit the data are inconsistent with the conceptual assumptions underlying BKT. We give some intuition for why semantic model degeneracy may arise by showing that BKT models fit to data generated from alternative models of student learning can have semantically degenerate parameters. Finally, we discuss the potential implications of these insights. [This article was published in: Proceedings of the 10th International Conference on Educational Data Mining, Wuhan, China, June 25-27, 2017 (p143-149).]

Engaging in explanation while studying expository science texts can improve comprehension. The present study varied the timing of explanation activities and restudy opportunities before taking final comprehension tests on a set of 6 topics studied as part of a course in Introduction to Psychology. When students had the opportunity to restudy in the same session, then comprehension benefited from explanation activities. However, no benefit from explanation activities was seen when they were done in a session prior to the restudy opportunity. [This is online version of paper published in "Preprint Repository for the 32nd Annual Meeting of the Society for Text & Discourse" (No. 9587).]

As intelligent textbooks become more ubiquitous in classrooms and educational settings, the need arises to automatically provide formative feedback to written responses provided by students in response to readings. This study develops models to automatically provide feedback to student summaries written at the end of intelligent textbook sections. The study builds on Botarleanu et al. (2022), who used the Longformer Large Language Model, a transformer Neural Network, to build a summary grading model. Their model explains around 55% of holistic summary score variance when compared to scores assigned by human raters on an analytic rubric. This study uses a principal component analysis to distill scores from the analytic rubric into two principal components -- content and wording. When training the models on the summaries and the sources using these principal components, we explained 79% and 66% of the score variance for content and wording, respectively. The developed models are freely available on HuggingFace and will allow formative feedback to users of intelligent textbooks to assess reading comprehension through summarization in real-time. The models can also be used for other summarization applications in learning systems. [This paper was published in: "AIED 2023, CCIS 1831," edited by N. Wang et al., Springer Nature Switzerland, 2023, pp. 484-489.]

Self-efficacy, or the belief in one's ability to accomplish a task or achieve a goal, can significantly influence the effectiveness of various instructional methods to induce learning gains. The importance of self-efficacy is particularly pronounced in complex subjects like Computer Science, where students with high self-efficacy are more likely to feel confident in their ability to learn and succeed. Conversely, those with low self-efficacy may become discouraged and consider abandoning the field. The work presented here examines the relationship between self-efficacy and students learning computer programming concepts. For this purpose, we conducted a randomized control trial experiment with university-level students who were randomly assigned into two groups: a control group where participants read Java programs accompanied by explanatory texts (a passive strategy) and an experimental group where participants self-explain while interacting through dialogue with an intelligent tutoring system (an interactive strategy). We report here the findings of this experiment with a focus on self-efficacy, its relation to students' learning gains (to evaluate the effectiveness, we measure pre/post-test), and other important factors such as prior knowledge or experimental condition/instructional strategies as well as interaction effects. [This paper was published in: "Proceedings of the 7th Educational Data Mining in Computer Science Education (CSEDM) Workshop, In conjunction with The 13th International Conference on Learning Analytics Knowledge (LAK23)," 2023.]

Self-explanation practice is an effective method to support students in better understanding complex texts. This study focuses on automatically assessing the comprehension strategies employed by readers while understanding STEM texts. Data from 3 datasets (N = 11,833) with self-explanations annotated on different comprehension strategies (i.e., bridging, elaboration, and paraphrasing) and an overall quality score was used to train various machine learning models in both single-task and multi-task setups. Our end-to-end neural architecture considers RoBERTa as an encoder applied to the target and self-explanation texts, combined with handcrafted features for assessing text cohesion and filtering out low-quality examples. The best configuration obtained a 0.699 weighted F1-score for the overall self-explanation quality. [This paper was published in: "AIED 2023, CCIS 1831," edited by N. Wang et al., Springer Nature, 2023, pp. 695-700.]

Math performance continues to be an important focus for improvement. The most recent National Report Card in the U.S. suggested student math scores declined in the past two years possibly due to COVID-19 pandemic and related school closures. We report on the implementation of a math homework program that leverages AI-based one-to-one technology, in 32 schools for two years as a part of a randomized controlled trial in diverse settings of the state of North Carolina in the US. The program, called "ASSISTments," provides feedback to students as they solve homework problems and automatically prepares reports for teachers about student performance on daily assignments. The paper describes the sample, the study design, the implementation of the intervention, including the recruitment effort, the training and support provided to teachers, and the approaches taken to assess teacher's progress and improve implementation fidelity. Analysis of data collected during the study suggest that (a) treatment teachers changed their homework review practices as they used ASSISTments, and (b) the usage of ASSISTments was positively correlated with student learning outcome.

Adaptive educational software is likely to better support broader and more diverse sets of learners by considering more comprehensive views (or models) of such learners. For example, recent work proposed making inferences about "non-math" factors like reading comprehension while students used adaptive software for mathematics to better support and adapt to learners. We build on this proposed approach to more comprehensive learning modeling by providing an empirical basis for making inferences about students' reading ability from their performance on activities in adaptive software for mathematics. We lay out an approach to predicting middle school students' reading ability using their performance on activities within Carnegie Learning's MATHia, a widely used intelligent tutoring system for mathematics. We focus on how performance in an early, introductory activity as an especially powerful place to consider instruction-embedded assessment of non-math factors like reading comprehension to guide adaptation based on factors like reading ability. We close by discussing opportunities to extend this work by focusing on particular knowledge components or skills tracked by MATHia that may provide important "levers" for driving adaptation based on students' reading ability while they learn and practice mathematics. [This paper was published in: "LAK23: 13th International Learning Analytics and Knowledge Conference," ACM, 2023.]

In this investigation we propose new machine learning methods for automated scoring models that predict the vocabulary acquisition in science and social studies of second grade English language learners, based upon free-form spoken responses. We evaluate performance on an existing dataset and use transfer learning from a large pre-trained language model, reporting the influence of various objective function designs and the input-convex network design. In particular, we find that combining objective functions with varying properties, such as distance among scores, greatly improves the model reliability compared to human raters. Our models extend the current state of the art performance for assessing word definition tasks and sentence usage tasks in science and social studies, achieving excellent quadratic weighted kappa scores compared with human raters. However, human-human agreement still surpasses model-human agreement, leaving room for future improvement. Even so, our work highlights the scalability of automated vocabulary assessment of free-form spoken language tasks in early grades. [This paper was published in: "Proceedings of the Tenth ACM Conference on Learning @ Scale (L@S '23), July 20-22, 2023, Copenhagen, Denmark," ACM, 2023.]

Prior studies identified effective, but mainly non-digital, homework aids. This research involved 18 middle school students in a lo-fi prototyping study to integrate traditional homework support tools with intelligent tutoring systems (ITS), leveraging rich log data for personalized learning. Feature investigations in standardized diaries, goal setting, homework graphing, and reminders revealed a preference for goal-setting, informing a refined prototype exploring students' attitudes towards self-set versus system-generated goals. Results indicate students prefer system-recommended goals, express goals by problem count and time, and value goal difficulty feedback to foster autonomy. Providing data-driven feedback and fading system-set goals in ITS could improve self-regulation during homework practices and equity.

nstruction has forced teachers to navigate unexpected constraints while providing meaningful learning experiences for their students. This transition has drastically changed how teachers orchestrate learning for their students. To explore these unique orchestration challenges, we used needs finding and validation activities to explore middle school teachers' emergent needs and constraints during the unplanned shift to remote instruction. Our findings highlight the need for informative, real-time tools, issues with workload and burnout, and concerns with students feeling disconnected. The contribution of this work includes insights from the early stages of our design process and reflections on how we might support teachers during remote learning and in navigating future emergency shifts. [This paper was published in: "Proceedings of the European Conference on Technology-Enhanced Learning, EC-TEL 2021," edited by T. De Laet et al., Cham: Springer, 2021, pp. 347-51.]

Despite the critical role of teachers in the educational process, few advanced learning technologies have been developed to support teacher-instruction or professional development. This lack of support is particularly acute for middle school math teachers, where only 37% felt well prepared to scaffold instruction to address the needs of diverse students in a national sample. To address this gap, the Advancing Middle School Teachers' Understanding of Proportional Reasoning project is researching techniques to apply pedagogical virtual agents and dialog-based tutoring to enhance teachers' content knowledge and pedagogical content knowledge. This paper describes the design of a conversational, agent-based intelligent tutoring system to support teachers' professional development. Pedagogical strategies are presented that leverage a virtual human facilitator to tutor pedagogical content knowledge (how to teach proportions to students), as opposed to content knowledge (understanding proportions). The roles for different virtual facilitator capabilities are presented, including embedding actions into virtual agent dialog, open-response versus choice-based tutoring, ungraded pop-up sub-activities (e.g. whiteboard, calculator, notetaking). Usability feedback for a small cohort of instructors pursuing graduate studies was collected. In this feedback, teachers rated the system ease of use and perceived usefulness moderately well, but also reported confusion about what to expect from the system in terms of flow between lessons and support by the facilitator.

Lack of diversity in the science community is a serious concern for social justice, scientific productivity, equity and efficacy. The first year of undergraduate education is of critical importance in increasing diversity in these fields. Peer-Led Team Learning (PLTL) has previously been shown to be associated with higher student achievement in gateway courses, particularly among students from populations that have been underserved and excluded within STEM fields (often referred to as underrepresented minorities, or URMs). We sought to determine whether participation in PLTL in an undergraduate introductory biology course is associated long-term retention rates among URM students in STEM majors. We used institutional data related to student recruitment and retention rates as well as pertinent demographic information over three and a half years subsequent to the introductory biology course experience. These data were combined with data on PLTL participation from the introductory biology course. Among students who did not engage in PLTL, URM students were significantly less likely to remain in STEM fields than non-URM students. However, no significant difference in STEM retention rates between URM and non-URM students was observed among those students who engaged in PLTL. Additionally, we found that retention rates were significantly higher for URM students who engaged in PLTL versus those who did not. These findings identify PLTL as a potential strategy to improve URM student recruitment and retention in STEM majors and indicate a need for further studies to determine the important aspects of PLTL that may lead to improved outcomes for URM students. [This paper was published in: "Proceedings of the 13th Annual Research Symposium," National Association of Biology Teachers, 2021.]

Text summarization is an effective reading comprehension strategy. However, summary evaluation is complex and must account for various factors including the summary and the reference text. This study examines a corpus of approximately 3,000 summaries based on 87 reference texts, with each summary being manually scored on a 4-point Likert scale. Machine learning models leveraging Natural Language Processing (NLP) techniques were trained to predict the extent to which summaries capture the main idea of the target text. The NLP models combined both domain and language independent textual complexity indices from the ReaderBench framework, as well as state-of-the-art language models and deep learning architectures to provide semantic contextualization. The models achieve low errors -- normalized MAE ranging from 0.13-0.17 with corresponding R2 values of up to 0.46. Our approach consistently outperforms baselines that use TF-IDF vectors and linear models, as well as Transfomer-based regression using BERT. These results indicate that NLP algorithms that combine linguistic and semantic indices are accurate and robust, while ensuring generalizability to a wide array of topics. [This paper was published in: A. I. Cristea and C. Troussas (Eds.), "ITS 2021: Intelligent Tutoring Systems proceedings," pp. 321-332, 2021. Springer, Cham Switzerland.]

Reading comprehension is key to knowledge acquisition and to reinforcing memory for previous information. While reading, a mental representation is constructed in the reader's mind. The mental model comprises the words in the text, the relations between the words, and inferences linking to concepts in prior knowledge. The automated model of comprehension (AMoC) simulates the construction of readers' mental representations of text by building syntactic and semantic relations between words, coupled with inferences of related concepts that rely on various automated semantic models. This paper introduces the second version of AMoC that builds upon the initial model with a revised processing pipeline in Python leveraging state-of-the-art NLP models, additional heuristics for improved representations, as well as a new radiant graph visualization of the comprehension model.

Engineering students used ClassTranscribe, an accessible video player, in multiple engineering courses to view course videos and search for video content. The tool collected detailed timestamped student behavioral data from 1,894 students across 25 engineering courses that included what individual students searched for and when. A previous analysis, published in ASEE 2020 [1], found that using ClassTranscribe caption search significantly predicted improvement in final exam scores in a computer science course. In this paper we present how students used the search functionality based on a more detailed analysis of the log data. ClassTranscribe automatically created captions and transcripts for all lecture videos using an Azure speech-to-text system that was supplemented with crowd-sourced editing to fix captioning errors. The search functionality used the timestamped caption data to find specific video moments both within the current video or across the entire course. The number of search activities per person ranged from zero to 186 events. An in-depth analysis of the students (N=167) who performed 1,022 searches was conducted to gain insight into student search needs and behaviors. Based on the total number of searches performed, students were grouped into "Infrequent Searcher" (< 18 searches) and "Frequent Searcher" (18 to 110 searches) using clustering algorithms. The search queries used by each group were found to follow the Zipf's Law and were categorized into STEM-related terms, course logistics and others. Our study reports on students' search context, behaviors, strategies, and optimizations. Using Universal Design for Learning as a foundation, we discuss the implications for educators, designers, and developers who are interested in providing new learning pathways to support and enhance video-based learning environments.

Consistency of learning behaviors is known to play an important role in learners' engagement in a course and impact their learning outcomes. Despite significant advances in the area of learning analytics (LA) in measuring various self-regulated learning behaviors, using LA to measure consistency of online course engagement patterns remains largely unexplored. This study focuses on modeling consistency of learners in online courses to address this research gap. Toward this, we propose a novel unsupervised algorithm that combines sequence pattern mining and ideas from information retrieval with a clustering algorithm to first extract engagement patterns of learners, represent learners in a vector space of these patterns and finally group them into groups with similar consistency levels. Using clickstream data recorded in a popular learning management system over two offerings of a STEM course, we validate our proposed approach to detect learners that are inconsistent in their behaviors. We find that our method not only groups learners by consistency levels, but also provides reliable instructor support at an early stage in a course. [This paper was published in: "LAK21: 11th International Learning Analytics and Knowledge Conference (LAK21), April 12-16, 2021, Irvine, CA, USA." ACM, 2021, pp. 226-236.]

The current study examined the extent to which the cohesion detected in readers' constructed responses to multiple documents was predictive of persuasive, source-based essay quality. Participants (N=95) completed multiple-documents reading tasks wherein they were prompted to think-aloud, self-explain, or evaluate the sources while reading a set of four texts. They were then asked to write a source-based essay based on their reading. Natural Language Processing techniques were used to automatically analyze the cohesion of the constructed responses at both within- and across-documents levels. Results indicated that within-document cohesion was "negatively" related to essay quality, whereas across-documents cohesion was positively related to essay quality. Further, these relations differed by instructional condition such that strategic instructions to either self-explain or evaluate sources seemed to promote across-text integration, compared to thinking aloud. Overall, this study indicates that the cohesion of constructed responses to text can provide insights into the coherence of the mental representations readers construct while reading multiple documents. [This paper was published in: "Proceedings of the Annual Meeting of the Cognitive Science," 2021, pp. 931-937 (ISSN- 1069-7977).]

Dialogism is a philosophical theory centered on the idea that life involves a dialogue among multiple voices in a continuous exchange and interaction. Considering human language, different ideas or points of view take the form of voices, which spread throughout any discourse and influence it. From a computational point of view, voices can be operationalized as semantic chains that contain related words. This study introduces and evaluates a novel method of identifying semantic chains using BERT, a state-of-the-art language model for computational linguistics. The resulting model generalizes to multiple relations including repetitions, semantically related concepts from WordNet (i.e., synonyms, hypernyms, hyponyms, and siblings), as well as pronominal resolutions. By combining the attention scores between words, word pairs are merged into connected components that denote emerging voices from the discourse. The introduced visualization argues for a more dense capturing of inner semantic links between words and even compound words in contrast to classical methods of building lexical chains. [This paper was published in: "AIED 2021," edited by I. Roll et al., Springer Nature Switzerland AG, 2021, pp. 296-301.]

Speech and language development in children are crucial for ensuring effective skills in their long-term learning ability. A child's vocabulary size at the time of entry into kindergarten is an early indicator of their learning ability to read and potential long-term success in school. The preschool classroom is thus a promising venue for assessing growth in young children by measuring their interactions with teachers as well as classmates. However, to date limited studies have explored such naturalistic audio communications. Automatic Speech Recognition (ASR) technologies provide an opportunity for 'Early Childhood' researchers to obtain knowledge through automatic analysis of naturalistic classroom recordings in measuring such interactions. For this purpose, 208 hours of audio recordings across 48 daylong sessions are collected in a childcare learning center in the United States using Language Environment Analysis (LENA) devices worn by the preschool children. Approximately 29 hours of adult speech and 26 hours of child speech is segmented using manual transcriptions provided by CRSS transcription team. Traditional as well as End-to-End ASR models are trained on adult/child speech data subset. Factorized Time Delay Neural Network provides a best Word-Error-Rate (WER) of 35.05% on the adult subset of the test set. End-to-End transformer models achieve 63.5% WER on the child subset of the test data. Next, bar plots demonstrating the frequency of WH-question words in Science vs. Reading activity areas of the preschool are presented for sessions in the test set. It is suggested that learning spaces could be configured to encourage greater adult-child conversational engagement given such speech/audio assessment strategies. [This paper was published in: "ICMI '21 Companion, October 18-22, 2021, Montreal, QC, Canada," Association for Computing Machinery, 2021.]

Findings synthesized across five empirical laboratory- and classroom-based studies of high school and college students engaged in geometric reasoning and proof production during single- and multi-session investigations (346 participants overall) are presented. The findings converge on several design principles for computer technologies to support embodied mathematical thinking and learning: (a) the importance of integrating linguistic and body-based processes, especially via "dynamic gestures," which enable simulated action of mathematical transformations for hypothesizing, testing, and generalizing universal truths about space and shape; (b) the "cognitive relevance" of directed actions to the concepts under consideration so that action-cognition transduction elicits the bodily interactions needed to influence mathematical reasoning in the intended manner; (c) "collaborative contexts" that foster common ground and shared meaning construction, production of socially mediated gestures, and the emergence of extended and distributed knowledge and representations; (d) conditions that encourage "transfer" of sensorimotor experiences across contexts in order to predict and make sense of the world; and (e) the need to assess learners' knowledge and learning that is expressed nonverbally. "The Hidden Village" is presented as an instance of an embodied learning environment that uses the narrative context of a visual novel to instantiate these design principles and investigate their influences on mathematical thinking and learning. These design principles are placed in terms of larger efforts to articulate a broader theoretical framework of grounded and embodied learning. Together, the interplay of theoretical, empirical and design considerations of grounded and embodied learning reframes learning, transfer, and assessment, offering promising new pathways for an emerging class of learning environments that may propel educational practice and learning theory. [This paper was published in: "Proceedings of the 29th International Conference on Computers in Education," edited by M. M. T. Rodrigo et al., Asia-Pacific Society for Computers in Education, 2021, pp. 348-58.]

This study looks at how students embody their ideas about geometry conjectures and how those ideas travel within and between student groups. In one classroom of a Title 1 high school, students participated in a three-part program in which they: (1) played "The Hidden Village," a motion-capture video game where they assess the veracity of geometric conjectures (i.e., if it is always true or ever false) while their intuitions, insights, and rationales (including their gestures) are video recorded, (2) designed their own directed actions (i.e., a sequence of movements that represents a body-based interpretation of the structure and transformation of a spatial configuration), and (3) re-played the game with a mixture of previous conjectures combined with the conjectures designed by their peers. Multiple cases revealed ways that simulated enactment and collaborative construction can convey mathematical ideas. [This paper was published in: "CSCL 2021 Proceedings," International Society of the Learning Sciences, 2021, pp. 177-180.]

We describe a data mining pipeline to convert data from educational systems into knowledge component (KC) models. In contrast to other approaches, our approach employs and compares multiple model search methodologies (e.g., sparse factor analysis, covariance clustering) within a single pipeline. In this preliminary work, we describe our approach's results on two datasets when using 2 model search methodologies for inferring item or KCs relations (i.e., implied transfer). The first method uses item covariances which are clustered to determine related KCs, and the second method uses sparse factor analysis to derive the relationship matrix for clustering. We evaluate these methods on data from experimentally controlled practice of statistics items as well as data from the Andes physics system. We explain our plans to upgrade our pipeline to include additional methods of finding item relationships and creating domain models. We discuss advantages of improving the domain model that go beyond model fit, including the fact that models with clustered item KCs result in performance predictions transferring between KCs, enabling the learning system to be more adaptive and better able to track student knowledge. [This paper was published in: "Proceedings of the 14th International Conference on Educational Data Mining (EDM21)," International Educational Data Mining Society, 2021, pp. 672-76 (see ED615472).]

In contrast to simple feedback, which provides students with the correct answer, elaborated feedback provides an explanation of the correct answer with respect to the student's error. Elaborated feedback is thus a challenge for AI in education systems because it requires dynamic explanations, which traditionally require logical reasoning and knowledge engineering to generate. This study presents an alternative approach that formulates elaborated feedback in terms of long-form question answering (LFQA). An off-the-shelf LFQA system was evaluated by human raters in a 2x2x2x2 ablation design that manipulated the context documents given to the LFQA model and the post-processing of model output. Results indicate that context manipulations improve performance but that postprocessing can have detrimental results. [This paper was published in: "Proceedings of the Eighth ACM Conference on Learning @ Scale," 2021, pp. 27-36.]

Automated Essay Scoring (AES) can reliably grade essays at scale and reduce human effort in both classroom and commercial settings. There are currently three dominant supervised learning paradigms for building AES models: feature-based, neural, and hybrid. While feature-based models are more explainable, neural network models often outperform feature-based models in terms of prediction accuracy. To create models that are accurate and explainable, hybrid approaches combining neural network and feature-based models are of increasing interest. We compare these three types of AES models with respect to a different evaluation dimension, namely algorithmic fairness. We apply three definitions of AES fairness to an essay corpus scored by different types of AES systems with respect to upper elementary students' use of text evidence. Our results indicate that different AES models exhibit different types of biases, spanning students' gender, race, and socioeconomic status. We conclude with a step towards mitigating AES bias once detected. [This paper was published in: "AIED 2021," edited by I. Roll et al., Springer Nature Switzerland AG, 2021, pp. 255-67.]

When text datasets are very large, manually coding line by line becomes impractical. As a result, researchers sometimes try to use machine learning algorithms to automatically code text data. One of the most popular algorithms is topic modeling. For a given text dataset, a topic model provides probability distributions of words for a set of "topics" in the data, which researchers then use to interpret meaning of the topics. A topic model also gives each document in the dataset a score for each topic, which can be used as a non-binary coding for what proportion of a topic is in the document. Unfortunately, it is often difficult to interpret what the topics mean in a defensible way, or to validate document topic proportion scores as meaningful codes. In this study, we examine how keywords from codes developed by human experts were distributed in topics generated from topic modeling. The results show that (1) top keywords of a single topic often contain words from multiple human-generated codes; and conversely, (2) words from human-generated codes appear as high-probability keywords in multiple topic. These results explain why directly using topics from topic models as codes is problematic. However, they also imply that topic modeling makes it possible for researchers to discover codes from short word lists. [This paper was published in: "ICQE 2021," edited by A. R. Ruis and S. B. Lee, Springer Nature Switzerland AG, 2021, pp. 18-31.]

Educational content labeled with proper knowledge components (KCs) are particularly useful to teachers or content organizers. However, manually labeling educational content is labor intensive and error-prone. To address this challenge, prior research proposed machine learning based solutions to auto-label educational content with limited success. In this work, we significantly improve prior research by (1) expanding the input types to include KC descriptions, instructional video titles, and problem descriptions (i.e., three types of prediction task), (2) doubling the granularity of the prediction from 198 to 385 KC labels (i.e., more practical setting but much harder multinomial classification problem), (3) improving the prediction accuracies by 0.5-2.3% using Task-adaptive Pre-trained BERT, outperforming six baselines, and (4) proposing a simple evaluation measure by which we can recover 56-73% of mispredicted KC labels. All codes and data sets in the experiments are available at: https://github.com/tbs17/TAPT-BERT [This paper was published in: "AIED 2021, LNAI1 2748," edited by I. Roll et al., Springer Nature Switzerland AG, 2021, pp. 408-19.]

Teacher learning in asynchronous online professional development (PD) is often assessed through self-reported instruments, despite uncertainties regarding teachers' ability to accurately self-assess their learning gains in such settings. This study compared middle school mathematics teachers' self-perceived and observed learning gains in content knowledge (CK) and pedagogical content knowledge (PCK) after participating in an asynchronous online PD program. By analyzing data collected from 57 teachers, we found a weak and nonsignificant correlation between teachers' perceived learning gains and observed learning gains. Teachers who frequently monitored their understanding of the learning content tended to underestimate their improvement of CK from the PD program while those using cognitive learning strategies (i.e., organize and elaborate) often overestimated it, controlling for their baseline knowledge level. These findings highlight methodological concerns in using self-reports to evaluate online learning outcomes and the potential distortion in identifying effective learning strategies.

Increasing 4-year graduation rates among US students, particularly those historically disadvantaged in higher education, is a national imperative. The purpose of this study is to test the efficacy of the multifaceted Research Aligned Mentorship Program in advancing the academic success of low income, first generation college, and minority students at Farmingdale State College. The Program was launched in 2015 with a First in the World FIPSE Grant from the US Department of Education. In this rigorous study, Farmingdale conducts a Randomized Controlled Trial that annually selects 400 entering first-generation, minority, and/or low income first-year students (approximately 220 treatment and 180 control) by applying an algorithm to the database of all entering students who fall into these demographics: first-generation, minority, and/or low income. Using student data (N=1,462), this paper examines the impacts of the Research Aligned Mentorship Program on whether students are on track to graduate within four years. Qualitative and quantitative data and analytic approaches are integrated to explore significant outcomes, including year-to-year retention, annual grade-point average, annual credits earned, and whether students succeed in graduating at the end of four academic years. The central research questions guiding this study are: (1) What is the impact of participation in the program on annual retention, annual credits earned, GPA, and four-year graduation rates? (2) Do impacts vary by level of participation? The Program provides such interventions as a first-year experience seminar, collaborative learning workshops, intensive counseling, and mentored research experiences. The overriding goal of the Program is to increase the four-year graduation rate of students by 20 percentage points over the 2015 IPEDS 4-year rate of 20%. Data show that active engagement is positively and significantly associated with students having higher GPAs, annual credits earned, persistence from year-to-year, and 4-year graduation. The first cohort of 216 students arrived in 2016. 199 of those students entered as first-year students. Of those entering first-year students, 49% graduated within 4 years (29 percentage points over the IPEDS baseline). The Research Aligned Mentorship Program is a proven example that can be replicated by institutions across the US -- especially by second tier public institutions like Farmingdale that educate the majority of our nation's minority, low-income, and first-generation college students. The appendices include: 73 references, six tables of data, and six deidentified student testimonials.

This paper reports on a quasi-experimental pilot study of the impacts of a high school teacher professional development (PD) series. The 10-session series, "Engaging High School Students in Academic Work," was designed to equip teachers to deepen students' learning and engagement and thereby increase course-passing rates. The study took place in a district in the U.S. Southwest that selected two of its interested high schools with similar demographic characteristics to participate, one receiving the PD and the other serving as the comparison school. Longitudinal analyses found statistically and educationally significant impacts of the PD series on four of the five measures in the primary outcome domain (students' course passing) and on teachers' reports of providing extra help. The article discusses limitations of the study and the need for further exploration of how PD can lead teachers to adopt more engaging instructional practices. It offers readers access to all PD materials and invites them into dialogue about how to adapt the PD series and its implementation going forward.

Large-scale, classroom-based experiments using adaptive instructional software pose somewhat unique challenges for experimental design and deployment. One reason for this is that adaptive software allows students to advance through the curriculum at different rates and encounter content at different times, meaning that content targeted for experimentation is often reached asynchronously by students within the same classroom. In addition, many pedagogical approaches subject to experimentation will require multiple "touch points" with students throughout a course. Nimble experimental methods necessitate that experiments are able to take place at any time during the school year, so a robust experimentation system may need to be aware of the student's educational experience(s) prior to the experiment, both to permit students to be excluded from the experiment (if desired by the researcher) if they have previously been taught the target content, and to ensure that the pedagogical approach taken during the experiment is consistent with that which has been experienced by the student before and after the experimental period. The UpGrade A/B testing platform (Ritter et al., 2020) is designed to help experimenters navigate these issues. This paper discusses several key design considerations for conducting digital experiments within adaptive educational software: ordering and sequencing, coordination of experimental activities, and exclusion criteria. Following this is an illustration of how these principles were applied in two recently-conducted large-scale experiments.

Through a theoretical framework emphasizing the importance of fidelity of implementation (FOI), this paper explores how 3rd and 4th grade teachers implemented an early algebra intervention, and the extent to which the FOI related to student learning. The data for this report are taken from the first two years of an experimental research project. Videotaped classroom observations, our primary measure of FOI, were coded by adding to and adapting the Mathematical Quality of Instruction (MQI) instrument, and student performance was measured by overall score (correctness) on an algebra assessment. Results revealed a significant positive relationship between teachers' implementation and their students' performance. [This paper was published in: Galindo, E., & Newton, J., (Eds.), "Proceedings of the Thirty-Ninth Annual Meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education" (pp. 299-302). Hoosier Association of Mathematics Teacher Educators.]

This paper introduces StairStepper, a new addition to Interactive Strategy Training for Active Reading and Thinking (iSTART), an intelligent tutoring system (ITS) that provides adaptive self-explanation training and practice. Whereas iSTART focuses on improving comprehension at levels geared toward answering challenging questions associated with complex texts, StairStepper focuses on improving learners' performance when reading grade-level expository texts. StairStepper is designed as a scaffolded practice activity wherein text difficulty level and task are adapted according to learners' performance. This offers a unique module that provides reading comprehension tutoring through a combination of self-explanation practice and answering of multiple-choice questions representative of those found in standardized tests. [This paper was published in: E. Andre, R. Baker, X. Hu, M. M. T. Rodrigo, & B. du Boulay (Eds.), "Proceedings of the 18th International Conference on Artificial Intelligence in Education" (pp. 557-560). Wuhan, China: Springer.]

There is little empirical research available on the substantial problem of adult low literacy rates, and limited educational technologies are available to address distinct instructional needs of this population. This paper reports on development and testing of a version of Interactive Strategy Training for Active Reading and Thinking (iSTART) for Adult Literacy Learners (iSTART-ALL) We describe modifications of iSTART to accommodate adult literacy learners, including new practice modules (i.e., summarization, question asking), a new library of texts, and an interactive narrative for adult literacy learners to engage in extended practice of reading comprehension strategies. We report results of a study examining reactions to iSTART-ALL and performance data while engaging with the interactive narrative. The attitudinal study, conducted with 38 adult literacy learners, demonstrated generally positive reactions to the narrative. Results also revealed that task performance was strongly related to individual difference scores on reading comprehension assessments, and more so with higher-level comprehension skills than basic word-level skills, providing concurrent validity for the interactive narrative tasks. [This paper was published in: E. Andre, R. Baker, X. Hu, M. M. T. Rodrigo, & B. du Boulay (Eds.), "Proceedings of the 18th International Conference on Artificial Intelligence in Education" (pp. 125-136). Wuhan, China: Springer.]

iSTART is a web-based reading comprehension tutor. A recent translation of iSTART from English to Spanish has made the system available to a new audience. In this paper, we outline several challenges that arose during the development process, specifically focusing on the algorithms that drive the feedback. Several iSTART activities encourage students to use comprehension strategies to generate self-explanations in response to challenging texts. Unsurprisingly, analyzing responses in a new language required many changes, such as implementing Spanish natural language processing tools and rebuilding lists of regular expressions used to flag responses. We also describe our use of an algorithm inspired from genetics to optimize the Fischer Discriminant Function Analysis coefficients used to determine self-explanation scores. [This paper was published in: R. Baker & E. Andre (Eds.), "Proceedings of the 18th International Conference on Artificial Intelligence in Education" (pp. 485-489), Wuhan, China: Springer.]

The results of educational research studies are only as accurate as the data used to produce them. Drawing on experiences conducting large-scale efficacy studies of classroom-based algebra interventions for community college and middle school students, I am developing practice-based data cleaning procedures to support scholars in conducting rigorous research. The paper identifies common sources of data errors in mathematics education research and offers a framework and related data cleaning process designed to address these errors. [This paper was published in: Weinberg, A., Rasmussen, C., Rabin, J., Wawro, M., and Brown, S. (Eds.), "Proceedings of the 20th Annual Conference on Research in Undergraduate Mathematics Education," p129-140. San Diego, CA (2017).]

It has been shown in multiple studies that expert-created on-demand assistance, such as hint messages, improves student learning in online learning environments. However, there are also evident that certain types of assistance may be detrimental to student learning. In addition, creating and maintaining on-demand assistance are hard and time-consuming. In 2017-2018 academic year, 132,738 distinct problems were assigned inside ASSISTments, but only 38,194 of those problems had on-demand assistance. In order to take on-demand assistance to scale, we needed a system that is able to gather new on-demand assistance and allows us to test and measure its effectiveness. Thus, we designed and deployed TeacherASSIST inside ASSISTments. TeacherASSIST allowed teachers to create on-demand assistance for any problems as they assigned those problems to their students. TeacherASSIST then redistributed on-demand assistance by one teacher to students outside of their classrooms. We found that teachers inside ASSISTments had created 40,292 new instances of assistance for 25,957 different problems in three years. There were 14 teachers who created more than 1,000 instances of on-demand assistance. We also conducted two large-scale randomized controlled experiments to investigate how on-demand assistance created by one teacher affected students outside of their classes. Students who received on-demand assistance for one problem resulted in significant statistical improvement on the next problem performance. The students' improvement in this experiment confirmed our hypothesis that crowd-sourced on-demand assistance was sufficient in quality to improve student learning, allowing us to take on-demand assistance to scale. [This paper was published in: "L@S '20, August 12-14, 2020, Virtual Event, USA," ACM, 2020, pp. 115-124.]

An intelligent textbook may be considered to be an interaction layer that lies between the text and the student, helping the student to master the content in the text. The Mobile Fact and Concept Training System (MoFaCTS) is an adaptive instructional system for simple content that has been developed into an interaction layer to mediate textbook instruction and so is being transformed into the Mobile Fact and Concept Textbook System (MoFaCTS). In this project, MoFaCTS is being completely retooled to accept texts from a textbook and to automatically create cloze sentence practice content to help the student learn the material in the text. Additional features in the prototype stage include automatically generated refutational feedback for incorrect cloze responses and a dialog system, which will trigger a short conversation by a tutor to correct conceptual misunderstandings. MoFaCTS administers this content via a web browser, providing the teacher with score reports and class management tools. Because the "optimal practice" module is interchangeable and the cloze content can come from any text, the system is highly configurable for different grade levels, populations, and academic subjects. To foster faster research progress, data export supports the DataShop transaction format, which allows quick analysis of data using the DataShop tools. [This paper was published in the CEUR workshop proceedings (Vol. 2674).]

Participation in adult education and training opportunities over the entire life-course is necessary in international and technologically advanced economies. However, there is a dearth of literature on equitable access to lifelong education opportunities, particularly for at risk and underserved adults in the labor force. Furthermore, opportunities for financing lifelong learning remain minimal for middle-age and older adult populations. The purpose of this research is to explore strategies or alternative models to fund lifelong education with attention to the Individual Learning Account (ILA) schemes. We explore initiatives across four countries--Canada, Sweden, the United Kingdom, and the United States. This research uses a qualitative descriptive approach to compare strategies to financing lifelong education targeting adult learners. Data were collected from key informant interviews (N=14), government documents, and research reports. Two major themes emerged that are common across all key informant interviews and documentation: (1) ILA offers key lessons in informing the creation of effective lifelong learning funding; (2) Low-income and low-skilled adults are often excluded from participation in ILA schemes. The authors' findings move adult education beyond a focus on providers of education to considering additional strategies for financing individual learning. [This paper was published in: J. P. Egan (Ed.), "Proceedings of the Adult Education in Global Times Conference." Ottawa, ON: Canadian Association for the Study of Adult Education." (ISBN 978-0-920056-53-0).]