Search

Can student performance on computer-based tests (CBT) and paper-and-pencil tests (PPT) be considered equivalent measures of student knowledge? States and school districts are grappling with this question, and although studies addressing this question are growing, additional research is needed. We report on the performance of students who took either a PPT or one of two different CBT containing multiple-choice items assessing science ideas. Propensity score matching was used to create equivalent demographic groups for each testing modality, and Rasch modelling was used to describe student performance. Performance was found to vary across testing modalities by grade band, students' primary language, and the specific CBT system used. These results are discussed in terms of the current literature and the differences between the specific PPT and CBT systems.

This paper explores how learning analytics data provided by a math problem-solving educational technology platform informed 5th and 6th grade teachers' instructional decisions around socioemotional learning (SEL). MathSpring is an educational technology tool that provides teachers with data on students' effort, progress, and emotions while students are solving math problems. This paper presents findings from qualitative analyses of 11 classroom observations, 10 observation follow-up chats, and 35 teacher interviews that were collected as part of a randomized control trial conducted in the northeastern United States. The analyses include teachers who used the program with their students (treatment group) and teachers who implemented math instruction as usual (control group). Teachers in the treatment group viewed analytic data on students' effort and progress, which informed the discussions they had with their students around SEL topics including setting goals, the importance of effort, and using productive problem-solving strategies. Findings from these analyses suggest that analytic data on effort and progress can inform teachers' discussions of math-related SEL concepts with their students. [This paper was published in: "The 14th Learning Analytics and Knowledge Conference (LAK '24), March 18-22, 2024, Kyoto, Japan," ACM, 2024.]

Perseverance is critical for students' achievement and may be particularly important after COVID-19. This paper includes analyses of teacher and principal interviews and student educational technology usage data to examine students' perseverance during math problem-solving across three cohorts of students during and after COVID-19. Data were collected as part of a randomized control trial examining the efficacy of an educational technology platform, MathSpring, in supporting students' math achievement. The results indicated that teachers and principals were concerned with their students' perseverance, and students' usage data similarly demonstrated that students may have reduced perseverance when solving math problems after COVID-19. Teachers and students may benefit from using tools that help foster perseverance in math class. MathSpring and other educational technology tools that support perseverance could be useful for encouraging students to put forth effort when solving math problems, which could in turn support students' math learning outcomes.

In the effort to improve student outcomes, teachers are usually the focus of professional development efforts. However, research increasingly demonstrates the significance of administrators in influencing student outcomes, raising questions about how they too should be involved in such programs. The study involved interviews conducted with 12 administrators engaged in a large-scale professional development program dedicated to equitable math instruction for diverse learners. The paper reports on the different understandings they had regarding their role in supporting the innovation, including their perspective on diverse learners, perceptions of teacher needs, and approaches for teacher support. While conventional professional development programs might overlook administrators, this paper argues for a situated approach for engaging administrators in the complex process of leading change.

School rating websites are increasingly used by parents to assess the quality and fit of U.S. K-12 schools for their children. These online reviews often contain detailed descriptions of a school's strengths and weaknesses, which both reflect and inform perceptions of a school. Existing work on these text reviews has focused on finding words or themes that underlie these perceptions, but has stopped short of using the textual reviews as leading indicators of school performance. In this paper, we investigate to what extent the language used in online reviews of a school is predictive of changes in the attributes of that school, such as its socio-economic makeup and student test scores. Using over 300K reviews of 70K U.S. schools from a popular ratings website, we apply language processing models to predict whether schools will significantly increase or decrease in an attribute of interest over a future time horizon. We find that using the text improves predictive performance significantly over a baseline model that does not include text but only the historical time-series of the indicators themselves, suggesting that the review text carries predictive power. A qualitative analysis of the most predictive terms and phrases used in the text reviews indicates a number of topics that serve as leading indicators, such as diversity, changes in school leadership, a focus on testing, and school safety.

In this paper, we present iSTART-Early, an intelligent tutoring system that provides automated instruction and practice on higher-order reading comprehension strategies to 3rd and 4th grade students. iSTART-Early provides personalized, interactive, game-based strategy instruction and practice on comprehension strategies (i.e., Ask It, Reword It, Find It, Explain It, and Summarize It) with grade-appropriate informational texts. Natural language processing (NLP) combined with automated speech recognition (ASR) and text-to-speech technologies enable immediate formative and summative feedback. A teacher interface allows teachers to assign texts and monitor students' performance so that they can provide additional support and feedback when necessary, creating blended-learning opportunities. We describe the interface and the development of iSTART-Early, as well as our plans to examine the intelligent tutoring system for usability, feasibility and promise in improving reading comprehension strategies and outcomes for young readers. [This paper was published in: S. Crossley and E. Popescu, Eds., "ITS 2022, LNCS 13284," Springer Nature, 2022, pp.371-379.]

Solving mathematical problems is cognitively complex, involving strategy formulation, solution development, and the application of learned concepts. However, gaps in students' knowledge or weakly grasped concepts can lead to errors. Teachers play a crucial role in predicting and addressing these difficulties, which directly influence learning outcomes. However, preemptively identifying misconceptions leading to errors can be challenging. This study leverages historical data to assist teachers in recognizing common errors and addressing gaps in knowledge through feedback. We present a longitudinal analysis of incorrect answers from the 2015-2020 academic years on two curricula, Illustrative Math and EngageNY, for grades 6, 7, and 8. We find consistent errors across 5 years despite varying student and teacher populations. Based on these Common Wrong Answers (CWAs), we designed a crowdsourcing platform for teachers to provide Common Wrong Answer Feedback (CWAF). This paper reports on an in vivo randomized study testing the effectiveness of CWAFs in two scenarios: next-problem-correctness within-skill and next-problem-correctness within-assignment, regardless of the skill. We find that receiving CWAF leads to a significant increase in correctness for consecutive problems within-skill. However, the effect was not significant for all consecutive problems within-assignment, irrespective of the associated skill. This paper investigates the potential of scalable approaches in identifying Common Wrong Answers (CWAs) and how the use of crowdsourced CWAFs can enhance student learning through remediation. [This paper was published in: "Proceedings of the Tenth ACM Conference on Learning @ Scale (L@S '23)," July 20-22, 2023.]

Prior studies identified effective, but mainly non-digital, homework aids. This research involved 18 middle school students in a lo-fi prototyping study to integrate traditional homework support tools with intelligent tutoring systems (ITS), leveraging rich log data for personalized learning. Feature investigations in standardized diaries, goal setting, homework graphing, and reminders revealed a preference for goal-setting, informing a refined prototype exploring students' attitudes towards self-set versus system-generated goals. Results indicate students prefer system-recommended goals, express goals by problem count and time, and value goal difficulty feedback to foster autonomy. Providing data-driven feedback and fading system-set goals in ITS could improve self-regulation during homework practices and equity.

Comfort and preparedness are dispositions that along with teacher knowledge, comprise teacher quality (cf. Clarke & Hollingsworth, 2002). Professional development programs that help teachers understand and respond to individual students' strengths and needs can support equity in the classroom by building teachers' preparedness and comfort, which enhances the likelihood of teachers' engaging in differentiated instruction to support all students. This paper draws from two randomized controlled trials (RCTs) of "Math for All," which show consistent, positive results of the professional development (PD) program on elementary school teachers' reports of comfort and preparedness to teach mathematics to diverse students. We found comfort and preparedness to be modestly but consistently related to self-reported mathematical instructional practices across both studies. The fact that the results were similar across the two RCTs even though the modes of delivery differed (PD led by developers in RCT #1; by local staff developers and teacher leaders in RCT #2) also speaks to the scalability of the "Math for All" program.

This paper investigates two specific probabilistic biases which middle graders usually exhibit when reasoning about probability and randomness on assessment items. We discuss how students' reasoning about key probability concepts undergirds statistics literacy related to randomness, independence, and the likelihood of future events based on past results. We examine factors evoking misconceptions and students' (in)consistency in exhibiting them. Findings indicate that misconceptions can be evoked based on three types of factors including (1) students' particular understandings of probability and randomness, (2) general item characteristics, and (3) aspects of probability in items. Moreover, possession of a specific misconception will most likely result in exhibiting the bias again on other occasions including the same evoking factors (consistency). [This paper was published in: "Proceedings of the Satellite Conference of the International Association for Statistical Education (IASE): Statistics Education in the Era of Data Science," edited by R. Helenius and E. Falck, International Association for Statistical Education, 2021, pp. 1-6.]

This paper systematically explores how Large Language Models (LLMs) generate explanations of code examples of the type used in intro-to-programming courses. As we show, the nature of code explanations generated by LLMs varies considerably based on the wording of the prompt, the target code examples being explained, the programming language, the temperature parameter, and the version of the LLM. Nevertheless, they are consistent in two major respects for Java and Python: the readability level, which hovers around 7-8 grade, and lexical density, i.e., the relative size of the meaninful words with respect to the total explanation size. Furthermore, the explanations score very high in correctness but less on three other metrics: completeness, conciseness, and contextualization. [This paper is in: Proceedings of the workshop on Generative AI for Education(GAIED): Advances, Opportunities, and Challenges, 2003.]

Providing high-level support to students on NGSS inquiry practices can be challenging; however, teacher dashboards can help teachers provide just-in-time instruction to students, both in-person and online. Prior work has shown some success with a dashboard that alerts teachers in real time on students' science inquiry difficulties, but teachers differed in their use of the alerts. To further support teachers, we designed a second iteration, in which the alerts included actionable, evidence-based Teacher Inquiry Practice Supports (TIPS), a series of suggested scaffolds that teachers can use to support students on the practices with which they are struggling. In this study, we investigate how the discursive support patterns from one teacher differed when using the dashboard alerts "without" TIPS followed by "with" TIPS. Findings suggest that TIPS influenced how the teacher incorporated different types of support for her students, and further, that the support given varied across different virtual lab stages. [This paper was published in: "ICLS2022 Proceedings," International Society of the Learning Sciences, 2022, pp. 297-304.]

Historically, learning for young students has occurred in formal, in-person classroom environments, but the distance learning context has opened a myriad of learning modalities. To this end, we aim to better understand how deploying learning experience design (LXD) approach supports or hinders children's engagement while participating in an online, video based math course. This study operationalized LXD through the integration of evidence-based pedagogical instructional design and human-centered user experience (UX) design. Findings suggest that students' situational interest and mind-wandering significantly mediate the relationship between user experiences and online engagement. These results provide practical implications for how researchers, designers, and instructors can intentionally iterate the learning experience to reduce mind-wandering and sustain children's online engagement with learning theories as we consider the future of online teaching and learning modalities. [This paper was published in: "ICLS2022 Proceedings," International Society of the Learning Sciences (ICLS), 2022, pp. 472-479.]

Developing models and using mathematics are two key practices in internationally recognized science education standards, such as the Next Generation Science Standards (NGSS). However, students often struggle with these two intersecting practices, particularly when developing mathematical models about scientific phenomena. Formative performance-based assessments designed to elicit fine-grained data about students' competencies on these practices can be leveraged to develop embedded AI scaffolds to support students' learning. In this paper, we present the design and initial classroom test of virtual labs that automatically assess fine-grained sub-components of students' mathematical modeling competencies based on their actions within the learning environment. We describe how we leveraged underlying machine-learned and knowledge-engineered algorithms to trigger scaffolds, delivered proactively by a pedagogical agent, that address students' individual difficulties as they work. Results show that the students who received automated scaffolds for a given practice on their first virtual lab improved on that practice for the next virtual lab on the same science topic in a different scenario (a near-transfer task). These findings suggest that real-time automated scaffolds based on fine-grained assessment can foster students' mathematical modeling competencies related to the NGSS.

Computational thinking (CT) has emerged as a key topic of interest in K-12 education. Children that are exposed at an early age to STEM curriculum, such as computer programming and computational thinking, demonstrate fewer obstacles entering technical fields. Increased knowledge of programming and computation in early childhood is also associated with better problem solving, decision-making, basic number sense, language skills, and visual memory. As a digital competence, coding is explicitly regarded as a key 21st Century Skill, as the "literacy of today," such that its acquisition is regarded as essential to sustain economic development and competitiveness. Hence, the reliable evaluation of students' process data in context of problem solving tasks that require CT is of great importance. As opposed to product data, which only contain information about "what" the outcome of a problem solving process was (e.g. the final score), process data contain information about "how" the problem was solved (e.g. all the actions and problem solving steps). Students' coding processes are thus defined by their actions while coding, as evidenced by "process data," and are evaluated by comparing their action sequences to optimal action sequences. Prior research on process data analysis shows several inherent issues. Their approaches aggregate data and thus loses information which precludes them from being used in more detailed analyses of student behavior. Vector-based approaches often apply dimensionality reduction or normalization and require interpretation of the reduced dimensions, which is often not possible. Network-based or finite state visualizations that show transitions between states (i.e., actions or game-states), are aggregations over the student, game level, or time dimensions and thus lose detailed information along these dimensions. Additionally, these networks only model Markov processes of order one (current state and preceding state) and do not show the frequency of higher-order sequences such as transitions through more than one preceding state. Sequential pattern mining approaches can deal with higher-order sequences, but their results tend to be verbose and need tedious manual analysis. In summary, prior research has analyzed overall action sequences or code snapshots, but has not interpreted student actions in context of a situation during the problem solving process -- i.e. while students create the solution. A more fine-grained analysis of coding process data is needed, where relevant actions are interpreted as a part of the student's problem solving process. This paper addresses some of above issues and presents an approach to detect patterns related to computational thinking based on visualizations of students fine-grained actions in situational context.

Historically, learning for young students has occurred in formal, in-person classroom environments. But in just a matter of weeks, children were mandated to transition to a completely new mode of learning, facing new learning challenges with heightened anxieties. To this end, we aim to better understand how our learning experience design (LXD) efforts support or hinder children's engagement while participating in an online, video-based math course. This study operationalized LXD through the integration of e-learning instructional design (ID) as a lever for promoting students' situational interest (SI), emphasis on human centered design to support students' user experience (UX), and the combination of SI and UX to foster student engagement in an online environment. Results provide practical implications for how we can intentionally iterate our designs to sustain children's online engagement as we prepare for future instances of traditional, online and even hybrid models of instruction. [This paper was published in: "ICLS 2021 Proceedings," ISLS, 2021, pp. 521-524.]

Speech and language development in children are crucial for ensuring effective skills in their long-term learning ability. A child's vocabulary size at the time of entry into kindergarten is an early indicator of their learning ability to read and potential long-term success in school. The preschool classroom is thus a promising venue for assessing growth in young children by measuring their interactions with teachers as well as classmates. However, to date limited studies have explored such naturalistic audio communications. Automatic Speech Recognition (ASR) technologies provide an opportunity for 'Early Childhood' researchers to obtain knowledge through automatic analysis of naturalistic classroom recordings in measuring such interactions. For this purpose, 208 hours of audio recordings across 48 daylong sessions are collected in a childcare learning center in the United States using Language Environment Analysis (LENA) devices worn by the preschool children. Approximately 29 hours of adult speech and 26 hours of child speech is segmented using manual transcriptions provided by CRSS transcription team. Traditional as well as End-to-End ASR models are trained on adult/child speech data subset. Factorized Time Delay Neural Network provides a best Word-Error-Rate (WER) of 35.05% on the adult subset of the test set. End-to-End transformer models achieve 63.5% WER on the child subset of the test data. Next, bar plots demonstrating the frequency of WH-question words in Science vs. Reading activity areas of the preschool are presented for sessions in the test set. It is suggested that learning spaces could be configured to encourage greater adult-child conversational engagement given such speech/audio assessment strategies. [This paper was published in: "ICMI '21 Companion, October 18-22, 2021, Montreal, QC, Canada," Association for Computing Machinery, 2021.]

Automated Essay Scoring (AES) can reliably grade essays at scale and reduce human effort in both classroom and commercial settings. There are currently three dominant supervised learning paradigms for building AES models: feature-based, neural, and hybrid. While feature-based models are more explainable, neural network models often outperform feature-based models in terms of prediction accuracy. To create models that are accurate and explainable, hybrid approaches combining neural network and feature-based models are of increasing interest. We compare these three types of AES models with respect to a different evaluation dimension, namely algorithmic fairness. We apply three definitions of AES fairness to an essay corpus scored by different types of AES systems with respect to upper elementary students' use of text evidence. Our results indicate that different AES models exhibit different types of biases, spanning students' gender, race, and socioeconomic status. We conclude with a step towards mitigating AES bias once detected. [This paper was published in: "AIED 2021," edited by I. Roll et al., Springer Nature Switzerland AG, 2021, pp. 255-67.]

Math performance continues to be an important focus for improvement. The most recent National Report Card in the U.S. suggested student math scores declined in the past two years possibly due to COVID-19 pandemic and related school closures. We report on the implementation of a math homework program that leverages AI-based one-to-one technology, in 32 schools for two years as a part of a randomized controlled trial in diverse settings of the state of North Carolina in the US. The program, called "ASSISTments," provides feedback to students as they solve homework problems and automatically prepares reports for teachers about student performance on daily assignments. The paper describes the sample, the study design, the implementation of the intervention, including the recruitment effort, the training and support provided to teachers, and the approaches taken to assess teacher's progress and improve implementation fidelity. Analysis of data collected during the study suggest that (a) treatment teachers changed their homework review practices as they used ASSISTments, and (b) the usage of ASSISTments was positively correlated with student learning outcome.

In this investigation we propose new machine learning methods for automated scoring models that predict the vocabulary acquisition in science and social studies of second grade English language learners, based upon free-form spoken responses. We evaluate performance on an existing dataset and use transfer learning from a large pre-trained language model, reporting the influence of various objective function designs and the input-convex network design. In particular, we find that combining objective functions with varying properties, such as distance among scores, greatly improves the model reliability compared to human raters. Our models extend the current state of the art performance for assessing word definition tasks and sentence usage tasks in science and social studies, achieving excellent quadratic weighted kappa scores compared with human raters. However, human-human agreement still surpasses model-human agreement, leaving room for future improvement. Even so, our work highlights the scalability of automated vocabulary assessment of free-form spoken language tasks in early grades. [This paper was published in: "Proceedings of the Tenth ACM Conference on Learning @ Scale (L@S '23), July 20-22, 2023, Copenhagen, Denmark," ACM, 2023.]

Writing a good essay typically involves students revising an initial paper draft after receiving feedback. We present eRevise, a web-based writing and revising environment that uses natural language processing features generated for rubric-based essay scoring to trigger formative feedback messages regarding students' use of evidence in response-to-text writing. By helping students understand the criteria for using text evidence during writing, eRevise empowers students to better revise their paper drafts. In a pilot deployment of eRevise in 7 classrooms spanning grades 5 and 6, the quality of text evidence usage in writing improved after students received formative feedback then engaged in paper revision. [This paper was published in: "The Thirty-First AAAI Conference on Innovative Applications of Artificial Intelligence (IAAI-19)" (p. 9619-9625). Association for the Advancement of Artificial Intelligence.]

We sought to identify the extent to which an innovative educational technology called ELCII (Early Language Comprehension Individualized Instruction) improved inference making for various populations of kindergarten students. Analyses examined student performance based on individual-level demographic characteristics (i.e., gender, race/ethnicity, home language, English Language Learner status, and Free or Reduced Lunch eligibility) to shed light on whether ELCII addresses the needs of learners from marginalized communities. Results showed that all students, on average, improved in their inference making performance over time and that growth in inference making performance did not differ across subgroups. However, the results also highlighted persistent gaps that were present at the onset of the program, and unfortunately continued to persist. To bridge such gaps, the standard of 'working for all' is a necessary first step, but not sufficient. Future research should identify ways to accelerate growth of core skill development among students who need additional support.

Speech technology (automated speech recognition -- ASR and text-to-speech) offers great promise in the field of automated literacy and reading tutors for children. Students in third and fourth grades struggle with generating longer strings of text on a QWERTY keyboard because they still "hunt and peck" for AQ1 the letters and symbols rather than typing fluently. Thus, in addition to reading comprehension, students' performance is limited by their ability to translate their ideas into language and then transcribe those words into written text. Fourth grade students produce fewer words and recall less when typing or writing a response relative to speaking. Hence, speech technology is a crucial component in the development of iSTART-Early, an intelligent tutoring system that aims to provide online, automated reading strategy instruction and practice to improve deep comprehension for third and fourth graders. This paper discusses the key components and features of the speech technology incorporated into iSTART-Early. We also discuss some initial findings from pilot studies conducted with adults and youth for this ASR integrated tutoring system. Finally, we discuss considerations for future development of speech technology and integration with intelligent tutoring systems. [Paper published in: "ITS 2022." Lecture Notes in Computer Science (LNCS) Vol. 13284, edited by S. Crossley and E. Popescu, Springer Nature Switzerland AG, 2022, pp. 1-9.]

The "Framework" and NGSS bring to the forefront the role of language in doing science and in learning from doing science. Yet, most existing science assessments for elementary learners do not integrate or attend to aspects of scientific language and literacy that are essential components of science proficiency. Accordingly, there is a need for high-quality classroom resources capable of providing teachers with information on where students are in the process of developing proficiency in science and scientific-specific literacy. In this paper, we present our design process for developing assessment resources -- tasks, rubrics, and teacher support materials -- and describe initial insights from using the process to develop resources for elementary grade classrooms. We highlight challenges and opportunities that have emerged from our work attending to the role of literacy assessing multi-dimensional science learning, incorporating equity and inclusion considerations in assessment design, and embracing the affordances of technology. [This paper was published in: "ICLS2022 Proceedings," International Society of the Learning Sciences, 2022, pp. 1237-1240.]

Many studies have shown that visual representations can enhance student understanding of STEM concepts. However, prior research suggests that visual representations alone are not necessarily effective across a broad range of students. To address this problem, we created a novel, scaffolded form of diagrammatic self-explanation in which students "explain" their problem-solving steps in the form of diagrams. We used contrasting cases to support students' sense-making between algebraic equations and diagrams in the self-explanation activity. We conducted a classroom experiment with 41 students in grades 5 and 6 to test the effectiveness of this strategy when embedded in an Intelligent Tutoring System for algebra. We found that scaffolded diagrammatic self-explanation enhanced conceptual knowledge for students who did not have prior knowledge of formal equation-solving strategies. The study is the first experimental study showing that visual representations can enhance conceptual knowledge in early algebra. [This paper was published in: "ICLS 2020 Proceedings," ISLS, 2020, pp. 35-42.]

Understanding how young learners come to construct viable mathematical arguments about general claims is a critical objective in early algebra research. The study reported here characterizes empirically developed progressions in Grades K-1 students' thinking about arguments concerning sums of evens and odds. Data are drawn from classroom lessons of an early algebra instructional sequence and interviews conducted at four timepoints during the implementation of the sequence. Overall, students transitioned from unfamiliarity with the concepts of even or odd prior to instruction in Kindergarten to making valid parity arguments at the conclusion of instruction in Grade 1. Results of this study align with other research that shows young learners can develop viable arguments to justify mathematical generalizations.

The ability to automatically assess the quality of paraphrases can be very useful for facilitating literacy skills and providing timely feedback to learners. Our aim is twofold: a) to automatically evaluate the quality of paraphrases across four dimensions: lexical similarity, syntactic similarity, semantic similarity and paraphrase quality, and b) to assess how well models trained for this task generalize. The task is modeled as a classification problem and three different methods are explored: (a) manual feature extraction combined with an Extra Trees model, (b) GloVe embeddings and a Siamese neural network, and (c) using a pre-trained BERT model fine-tuned on our task. Starting from a dataset of 1998 paraphrases from the User Language Paraphrase Corpus (ULPC), we explore how the three models trained on the ULPC dataset generalize when applied on a separate, small paraphrase corpus based on children inputs. The best out-of-the-box generalization performance is obtained by the Extra Trees model with at least 75% average F1-scores for the three similarity dimensions. We also show that the Siamese neural network and BERT models can obtain an improvement of at least 5% after fine-tuning across all dimensions.

We apply an advanced data visualization technique, "Sankey diagram," to explore how middle-school students (N = 343) solved problems in a game-based algebraic notation tool. The results indicate that there is a large variation in the types of students' strategies to solve the problems, with some approaches being more efficient than others. The findings suggest that Sankey diagrams can be used both in research and practice to unpack our understanding of variability in mathematical problem-solving. [This paper was published in: "Proceedings of the 15th International Conference of the Learning Sciences--ICLS 2021," 2021, pp. 887-888.]

This demonstration introduces and presents an innovative online cognitive diagnostic assessment, developed to identify the types of cognitive processes that readers use during comprehension; specifically, processes that distinguish between subtypes of struggling comprehenders. Cognitive diagnostic assessments are designed to provide valuable information by measuring specific processes emphasized during learning, and can provide instructionally relevant results aligned with curriculum that other large-scale, standardized assessments cannot provide. This hands-on session includes in-formation behind how the technology of MOCCA was developed, as well as how a reader would experience taking this assessment, how a teacher/educator would find the results of a user's assessment, and which instructional techniques to then use. Interpretation of assessment results and instructional recommendations are obtainable online. Future directions for the continued development of online digital learning regarding how to generate appropriate cognitive processes (e.g., inferences) during reading are ongoing and discussed. [This paper was published in: "EC-TEL Proceedings," LNCS 11722, 2019, pp. 769-772.]

This study evaluated the efficacy of the Word Learning Strategies (WLS) supplementary program to improve elementary students' vocabulary skills and reading comprehension. The study used a multi-site cluster randomized, experimental design, which randomly assigned 92 4th grade classrooms (n=2558 students) from two cohorts to a treatment or control group. Results indicated that the program was positively associated with gains in students' vocabulary learning and knowledge as measured by the Word Learning Strategies Test and the VASE Assessment, and in students' reading comprehension as measured by the Gates-MacGrinitie Reading Test, after accounting for differences in baseline measures. The use of the WLS program also led to increases in teachers' awareness of strategies to support their students' vocabulary and reading comprehension. [This paper was published in the Proceedings of the 2019 Annual Meeting of the American Educational Research Association (AERA), Toronto, Canada.]

We study the usage of a self-guided online tutoring platform called Algebra Nation, which is widely by middle school and high school students who take the End-of-Course Algebra I exam at the end of the school year. This article aims to study how the platform contributes to increasing students' exam scores by examining users' logs over a three year period. The platform under consideration was used by more than 36,000 students in the first year, to nearly 67,000 by the third year, thus enabling us to examine how usage patterns evolved and influenced students' performance at scale. We first identify which Algebra Nation usage factors in conjunction with math overall preparation and socioeconomic factors contribute to the students' exam performance. Subsequently, we investigate the effect of increased teacher familiarity level with the Algebra Nation on students' scores across different grades through mediation analysis. The results show that the indirect effect of teacher's familiarity with the platform through increasing student's usage dosage is more significant in higher grades. [This paper was published in: "The 9th International Learning Analytics & Knowledge Conference (LAK19)" (Tempe, AZ, March 4-8, 2019). New York, NY: Association for Computing Machinery (ACM). (978-1-4503-6256-6).]

Prior studies on scaffolding for investigative inquiry practices (i.e. forming a question/hypothesis, collecting data, and analyzing and interpreting data [21]) revealed that students who received scaffolding were better able to both learn practices and transfer these competencies to new topics than were students who did not receive scaffolding. Prior studies have also shown that after removing scaffolding, students continued to demonstrate improved inquiry performance on a variety of practices across new driving questions over time. However, studies have not examined the relationship between the "amount of scaffolding received" and transfer of inquiry performance; this is the focus of the present study. 107 middle school students completed four virtual lab activities (i.e. driving questions) in Inq-ITS. Students received scaffolding when needed from an animated pedagogical computer agent for the first three driving questions for the Animal Cell virtual lab. Then they completed the fourth driving question without access to scaffolding in a different topic, Plant Cell. Results showed that students' performances increased even with fewer scaffolds for the inquiry practices of hypothesizing, collecting data, interpreting data, and warranting claims; furthermore, these results were robust as evidenced by the finding that students required less scaffolding as they completed subsequent inquiry activities. These data provide evidence of near and far transfer as a result of adaptive scaffolding of science inquiry practices. [This paper was published in: "Proceedings of the Sixth Annual ACM Conference on Learning at Scale."]

This study describes the role that Rasch measurement played in the development of assessments aligned to the "Next Generation Science Standards," tasks that require students to use the three dimensions of science practices, disciplinary core ideas and cross-cutting concepts to make sense of energy-related phenomena. A set of 27 three-dimensional, multi-item tasks were developed and field tested along with a set of multiple-choice test items focused on disciplinary core ideas. Data were collected from elementary, middle, and high school students from across the U.S. Rasch modeling was used to investigate the fit of the data to the model, the dimensionality of the data, differential functioning of the tasks, the relative difficulties of the tasks, and the performance of students by grade band.

Through a theoretical framework emphasizing the importance of fidelity of implementation (FOI), this paper explores how 3rd and 4th grade teachers implemented an early algebra intervention, and the extent to which the FOI related to student learning. The data for this report are taken from the first two years of an experimental research project. Videotaped classroom observations, our primary measure of FOI, were coded by adding to and adapting the Mathematical Quality of Instruction (MQI) instrument, and student performance was measured by overall score (correctness) on an algebra assessment. Results revealed a significant positive relationship between teachers' implementation and their students' performance. [This paper was published in: Galindo, E., & Newton, J., (Eds.), "Proceedings of the Thirty-Ninth Annual Meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education" (pp. 299-302). Hoosier Association of Mathematics Teacher Educators.]

This study used comparative interrupted times series analyses to assess the impact of a continuous improvement-focused family engagement intervention on ninth grade attendance and course passing rates. The intervention, conducted in an urban district, sought to improve middle and high school family engagement practices during the transition to high school. The initiative created a networked learning community of school teams that received training and coaching in applying the continuous improvement process to family engagement planning and implementation. After four years of implementation, there was no evidence of a significant positive effect on ninth grade student outcomes. Lack of positive effects could be due both to implementation issues and to other ninth grade interventions in the comparison district.

Feedback in educational technologies can teach and engage students in math, but questions remain on how to present failure feedback that supports positive learning behaviors. We explore how error- and reward-based feedback influenced students' choices to replay completed problems in "From Here to There!," a math game-based educational technology. We conducted a multilevel logistic regression in 1,031 seventh-grade students predicting whether a problem was replayed, using whether an error was made during the first problem attempt and how many clovers were earned after solving the problem (with more clovers earned for more efficient solutions). Preliminary results show that making an error or earning two clovers increased the likelihood of replays, whereas earning one or three clovers decreased the likelihood. Our findings suggest that clovers and errors may be "micro-failures" that encourage replays by supporting feelings of competency and/or autonomy, which has implications for game design elements for learning and engagement. [This paper was published in: ICLS2022 Proceedings, International Society of the Learning Sciences (ISLS), 2022, pp. 1633-1636.]

New challenges in today's world have contributed to increased attention toward evaluating individuals' collaborative problem solving (CPS) skills. One difficulty with this work is identifying evidence of individuals' CPS capabilities, particularly when interacting in digital spaces. Often human-driven approaches are used but are limited in scale. Machine-driven approaches can save time and money, but their reliability relative to human approaches can be a challenge. In the current study, we compare CPS skill profiles derived from human and semi-automated annotation methods across two tasks. Results showed that the same clusters emerged for both tasks and annotation methods, with the annotation methods showing agreement on labeling most students according to the same profile membership. Additionally, validation of cluster results using external survey measures yielded similar results across annotation methods. [This paper was published in: "CSCL2022 Proceedings," International Society of the Learning Sciences, 2022, pp. 363-366.]

Teacher quality, along with the school context, classroom practice, and student mathematics achievement, are the key components in Clarke and Hollingsworth's (2002) model that depicts teachers' professional growth as an ongoing, dynamic, and interactive enterprise. Comfort and preparedness are dispositions that, along with teacher knowledge, are aspects of teacher quality. This model provides a basis for proposing a special causal mechanism: professional development designed to help teachers understand and respond to individual students' strengths and needs should support classroom equity by increasing teachers' preparedness and comfort, which in turn enhances the likelihood of teachers' engaging in differentiated instruction to support all students. This paper draws from two randomized controlled trials of the "Math for All" professional development program, which show consistent, positive results of the program on elementary school teachers' reports of comfort and preparedness to teach mathematics to diverse students--across modes of delivery, grade levels, and, to a lesser extent, locales. We also found comfort and preparedness to be modestly but consistently related to self-reported mathematical instructional practices across both studies. These results point to the consistency of the positive effects across sub-samples, and to the scalability of the "Math for All" program.