Search

The synthetic control method (SCM) is a popular approach for estimating the impact of a treatment on a single unit with panel data. Two challenges arise with higher frequency data (e.g., monthly versus yearly): (1) achieving excellent pre-treatment fit is typically more challenging; and (2) overfitting to noise is more likely. Aggregating data over time can mitigate these problems but can also destroy important signal. In this paper, we bound the bias for SCM with disaggregated and aggregated outcomes and give conditions under which aggregating tightens the bounds. We then propose finding weights that balance both disaggregated and aggregated series. [This paper was published in: "AEA Papers and Proceedings," Volume 114, 2024.]

Despite increased efforts to assess the adoption rates of open science and robustness of reproducibility in sub-disciplines of education technology, there is a lack of understanding of why some research is not reproducible. Prior work has taken the first step toward assessing reproducibility of research, but has assumed certain constraints which hinder its discovery. Thus, the purpose of this study was to replicate previous work on papers within the proceedings of the "International Conference on Educational Data Mining" to accurately report on which papers are reproducible and why. Specifically, we examined 208 papers, attempted to reproduce them, documented reasons for reproducibility failures, and asked authors to provide additional information needed to reproduce their study. Our results showed that out of 12 papers that were potentially reproducible, only one successfully reproduced all analyses, and another two reproduced most of the analyses. The most common failure for reproducibility was failure to mention libraries needed, followed by non-seeded randomness. [For the complete proceedings, see ED630829. Additional funding for this paper was provided by the U.S. Department of Education's Graduate Assistance in Areas of National Need (GAANN).]

The profound advancements in technology have rendered novel forms of data and data visualisation increasingly accessible to individuals within society, thereby influencing daily decision-making processes. To address this change, this study sets out to review recent research on data-driven inquiries at the K-12 level from two perspectives: innovative data visualisation and non-traditional data sources. Our findings indicate that transnumeration of multiple data representations, along with data moves throughout the process of data visualisation, can potentially enhance the development of visual reasoning and data modelling skills.

The General Data Protection Regulation (GDPR) in the European Union contains directions on how user data may be collected, stored, and when it must be deleted. As similar legislation is developed around the globe, there is the potential for repercussions across multiple fields of research, including educational data mining (EDM). Over the past two decades, the EDM community has taken consistent steps to protect learner privacy within our research, whilst pursuing goals that will benefit their learning. However, recent privacy legislation may cause our practices to need to change. The right to be forgotten states that users have the right to request that all their data (including deidentified data generated by them) be removed. In this paper, we discuss the potential challenges of this legislation for EDM research, including impacts on Open Science practices, data modeling, and data sharing. We also consider changes to EDM best practices that may aid compliance with this new legislation. [For the complete proceedings, see ED630829.]

Students who take an online course, such as a MOOC, use the course's discussion forum to ask questions or reach out to instructors when encountering an issue. However, reading and responding to students' questions is difficult to scale because of the time needed to consider each message. As a result, critical issues may be left unresolved, and students may lose the motivation to continue in the course. To help address this problem, we build predictive models that automatically determine the urgency of each forum post, so that these posts can be brought to instructors' attention. This paper goes beyond previous work by predicting not just a binary decision cut-off but a post's level of urgency on a 7-point scale. First, we train and cross-validate several models on an original data set of 3,503 posts from MOOCs at University of Pennsylvania. Second, to determine the generalizability of our models, we test their performance on a separate, previously published data set of 29,604 posts from MOOCs at Stanford University. While the previous work on post urgency used only one data set, we evaluated the prediction across different data sets and courses. The best-performing model was a support vector regressor trained on the Universal Sentence Encoder embeddings of the posts, achieving an RMSE of 1.1 on the training set and 1.4 on the test set. Understanding the urgency of forum posts enables instructors to focus their time more effectively and, as a result, better support student learning. [For the complete proceedings, see ED630829.]

This paper presents a tool for creating student models in logistic regression. Creating student models has typically been done by expert selection of the appropriate terms, beginning with models as simple as IRT or AFM but more recently with highly complex models like BestLR. While alternative methods exist to select the appropriate predictors for the regression-based models (e.g., stepwise selection or LASSO), we are unaware of their application to student modeling. Such automatic methods of model creation offer the possibility of better student models with either reduced complexity or better fit, in addition to relieving experts from the burden of searching for better models by hand with possible human error. Our new functions are now part of the preexisting R package LKT. We explain our search methods with two datasets demonstrating the advantages of using the tool with stepwise regression and regularization (LASSO) methods to aid in feature selection. For the stepwise method using BIC, the models are simpler (due to the BIC penalty for parameters) than alternatives like BestLR with little lack of fit. For the LASSO method, the models can be made simpler due to the fitting procedure involving a regularization parameter that penalizes large absolute coefficient values. However, LASSO also offers the possibility of highly complex models with exceptional fit. [For the complete proceedings, see ED630829.]

Knowledge components (KCs) have many applications. In computing education, knowing the demonstration of specific KCs has been challenging. This paper introduces an entirely data-driven approach for: (1) discovering KCs; and (2) demonstrating KCs, using students' actual code submissions. Our system is based on two expected properties of KCs: (1) generate learning curves following the power law of practice; and (2) are predictive of response correctness. We train a neural architecture (named KC-Finder) that classifies the correctness of student code submissions and captures problem-KC relationships. Our evaluation on data from 351 students in an introductory Java course shows that the learned KCs can generate reasonable learning curves and predict code submission correctness. At the same time, some KCs can be interpreted to identify programming skills. We compare the learning curves described by our model to four baselines, showing that: (1) identifying KCs with naive methods is a difficult task; and (2) our learning curves exhibit a substantially better curve fit. Our work represents a first step in solving the data-driven KC discovery problem in computing education. [For the complete proceedings, see ED630829.]

This paper reports on students' experiences of describing and representing variation in hypothetical data. Fifty-six students (8-9 years-old) experienced collecting and working with quantitative data for two years as part of a STEM education project. The task described here was an end-of-year survey question, with three parts about a hypothetical context for surveying students in two different Australian cities: recording the data, describing the potential variation in the data, and creating a representation of what the data might look like when only a descriptive account of the context and variables were provided. The data analysis framework utilised provides a means of determining students' readiness for further development of statistical ideas.

Data literacy is the ability to collect, manage, evaluate, and apply data, in a critical manner. It is a relatively new field of study, dating only from the 2010s. It includes the skills necessary to discover and access data, manipulate data, evaluate data quality, conduct analysis using data, interpret results of analyses, and understand the ethics of using data. This paper considers data literacy education across three frameworks: the competency model defining data literacy, the assessment of data literacy competencies, and methods for the development of data literacy in an organization. These principles are applied to a discussion of the development of an open online course supporting the development of data literacy in the form of a corresponding data structure encompassing the three frameworks identified in the literature. [For the full proceedings, see ED636095.]

This paper presents a mobile application aimed at enhancing the financial literacy of college students by monitoring their spending patterns and promoting better decision-making. The application is developed using the agile methodology with Android Studio and Flutter as development tools and Firebase as a database. The app is divided into sub-applications, with the home page serving as the program's integration point, displaying a summary of the user's financial progress. The app generates valuable insights into the user's current and future financial success, utilizing data analytics and machine learning to provide detailed and summary insights into the user's financial progress. The machine-learning algorithm used in this app is linear regression, which predicts the user's income and expenses for the upcoming month based on their historical spending data. In addition, the app highlights deals and student discounts in the user's vicinity and links to financial articles that promote better financial planning and decision-making. By promoting responsible spending habits and providing valuable financial insights, this mobile application aims to help students become financially literate and make informed financial decisions for future. [For the full proceedings, see ED654100.]

In this study, we held a 12-hour workshop focused on training the Data-Driven Learning (DDL) approach for in-service Chinese teachers aiming to implement this pedagogy in Mandarin Chinese classrooms in the future. We analyzed data from a postworkshop questionnaire to understand how the individual-level traits of Chinese teachers (such as their experience, ability, and motivations) relate to their integration of corpus-based methods. The questionnaire results indicate that teacher confidence in using the corpus is critical for the future application of DDL in teaching Chinese as a second language. Teachers who use corpus-based pedagogies are more positive at designing and guiding DDL activities and are more willing to practice DDL in language classes. [For the complete volume, "Intelligent CALL, Granular Systems and Learner Data: Short Papers from EUROCALL 2022 (30th, Reykjavik, Iceland, August 17-19, 2022)," see ED624779.]

This paper explores the effect of using multitask learning for abstractive summarization in the context of small training corpora. In particular, we incorporate four different tasks (extractive summarization, language modeling, concept detection, and paraphrase detection) both individually and in combination, with the goal of enhancing the target task of abstraction summarization via multitask learning. We show that for many task combinations, a model trained in a multitask setting outperforms a model trained only for abstractive summarization, with no additional summarization data introduced. Additionally, we do a comprehensive search and find that certain tasks (e.g. paraphrase detection) consistently benefit abstractive summarization, not only when combined with other tasks but also when using different architectures and training corpora. [This paper was published in: "Findings of the Association for Computational Linguistics: EMNLP 2021," Association for Computational Linguistics, 2021, pp. 1652-1661.]

This paper explores three simple data manipulation techniques (synthesis, augmentation, curriculum) for improving abstractive summarization models without the need for any additional data. We introduce a method of data synthesis with paraphrasing, a data augmentation technique with sample mixing, and curriculum learning with two new difficulty metrics based on specificity and abstractiveness. We conduct experiments to show that these three techniques can help improve abstractive summarization across two summarization models and two different small datasets. Furthermore, we show that these techniques can improve performance when applied in isolation and when combined. [This paper was published in: "Findings of the Association for Computational Linguistics: EMNLP 2021," Association for Computational Linguistics, 2021, pp. 2043-2052.]

Recommender systems in educational contexts have proven effective to identify learning resources that fit the interests and needs of learners. Their usage has been of special interest in online self-learning scenarios to increase student retention and improve the learning experience. In current recommendation techniques, and in particular, in collaborative filtering recommender systems, the quality of the recommendation is largely based on the explicit or implicit information obtained about the learners. On free massive online learning platforms, however, the information available about learners may be limited and based mostly on logs from website analytics tools such as Google Analytics. In this paper, we address the challenge of recommending meaningful content with limited information from users by using rating estimation strategies from a log system. Our approach posits strategies to mine logs and generates effective ratings through the counting and temporal analysis of sessions. We evaluate different rating penalty strategies and compare the use of per-user and global metrics for rating estimation. The results show that using the average number of lessons viewed per-user is better than using global metrics with a p-value under 0.01 for 4 of our 5 hypotheses, showing statistical significance. Additionally, the results show that functions that penalize the rating to a lesser degree behave better and lead to a better recommendation. [For the full proceedings, see ED623995.]

This work explores how population-based engagement prediction can address cold-start at scale in large learning resource collections. The paper introduces: (1) VLE, a novel dataset that consists of content and video based features extracted from publicly available scientific video lectures coupled with implicit and explicit signals related to learner engagement; (2) two standard tasks related to predicting and ranking context-agnostic engagement in video lectures with preliminary baselines; and (3) a set of experiments that validate the usefulness of the proposed dataset. Our experimental results indicate that the newly proposed VLE dataset leads to building context-agnostic engagement prediction models that are significantly performant than ones based on previous datasets, mainly attributing to the increase of training examples. VLE dataset's suitability in building models towards Computer Science/ Artificial Intelligence education focused on e-learning/MOOC use-cases is also evidenced. Further experiments in combining the built model with a personalising algorithm show promising improvements in addressing the cold-start problem encountered in educational recommenders. This is the largest and most diverse publicly available dataset to our knowledge that deals with learner engagement prediction tasks. The dataset, helper tools, descriptive statistics and example code snippets are available publicly. [For the full proceedings, see ED623995.]

A comparative longitudinal data analysis between two online non-thesis master's programs--natural resource management and environmental science--in a college of natural resources to determine the relationship between student characteristics and disenrollment risks. Risks varied between the two programs, with significance found to increase the risk of disenrollment due to cumulative GPA, gender, time between degrees, and the number of terms not enrolled. [For the full proceedings, see ED648717.]

Computer-mediated social learning contexts have become increasingly popular over the last few years; yet existing models of students' cognitive-affective states have been slower to adopt dyadic interaction data for predictions. Here, we explore the possibility of capitalizing on the inherently social component of collaborative learning by using keystroke log data to make predictions across conversational partners (i.e., using person A's data to make prediction about if person B is mind wandering). Log files from 33 dyads (total N = 66) were used to examine: (a) how mind wandering (defined here as task-unrelated thought) during computer-mediated conversations is related to critical outcomes of the conversation (trust, likability, agreement); (b) if task-unrelated thought can be predicted by the keystrokes of one's partner; and (c) how much data is needed to make predictions by testing various window-sizes of data preceding task-unrelated thought reports. Results indicated a negative relationship between task-unrelated thought and perceptions of the conversation, suggesting that attention is an important factor during computer mediated chat conversations. Finally, in line with our hypothesis, results from mixed effects models showed that one's level of task-unrelated thought was predicted by the keystroke patterns of their conversational partner, but only using small window sizes (5s worth of data). [For the complete proceedings, see ED630829.]

Problem decomposition into sub-problems or subgoals and recomposition of the solutions to the subgoals into one complete solution is a common strategy to reduce difficulties in structured problem solving. In this study, we use a datadriven graph-mining-based method to decompose historical student solutions of logic-proof problems into Chunks. We design a new problem type where we present these chunks in a Parsons Problem fashion and asked students to reconstruct the complete solution from the chunks. We incorporated these problems within an intelligent logic tutor and called them Chunky Parsons Problems (CPP). These problems demonstrate the process of problem decomposition to students and require them to pay attention to the decomposed solution while they reconstruct the complete solution. The aim of introducing CPP was to improve students' problem-solving skills and performance by improving their decomposition-recomposition skills without significantly increasing training difficulty. Our analysis showed that CPPs could be as easy as Worked Examples (WE). And, students who received CPP with simple explanations attached to the chunks had marginally higher scores than those who received CPPs without explanation or did not receive them. Also, the normalized learning gain of these students shifted more towards the positive side than other students. Finally, as we looked into their proof-construction traces in posttest problems, we observed them to form identifiable chunks aligned with those found in historical solutions with higher efficiency. [For the complete proceedings, see ED630829.]

Due to the precautionary measures during the COVID-19 pandemic many universities offered unproctored take-home exams. We propose methods to detect potential collusion between students and apply our approach on event log data from take-home exams during the pandemic. We find groups of students with suspiciously similar exams. In addition, we compare our findings to a proctored comparison group. By this, we establish a rule of thumb for evaluating which cases are "outstandingly similar", i.e., suspicious cases. [For the complete proceedings, see ED630829.]

Within the last decade, different educational data mining techniques, particularly quantitative methods such as clustering, and regression analysis are widely used to analyze the data from educational games. In this research, we implemented a quantitative data mining technique (clustering) to further investigate students' feedback. Students played educational games within a week on the educational games platform, Legends of Learning and after a week, we asked them to fulfill the feedback survey about their feelings on the use of this platform. To analyze the collected data from students, firstly, we prepared clusters and selected one prototype student closest to the centroid of each cluster to interview. Interviews were held to explain the clusters more and due to time and resource limitations, we were unable to interview all (N=60) students, thus only the most representative students were interviewed. In addition to the students, we conducted an interview with the teacher as well to get her detailed feedback and observations on the usage of educational games. We also asked students to take an exam before and after the research to see the impact of games on their grades. Our results depict that though educational games can increase students' motivation, they may negatively impact some students' grades. And even though playing games made students feel interested and fun, they would not like to play them on a daily basis. Hence, using educational games for a certain duration such as subject revision weeks may positively influence students' grades and motivation. [For the complete proceedings, see ED630829.]

The Indian Institute of Science is proud to host the fully in-person sixteenth iteration of the International Conference on Educational Data Mining (EDM) during July 11-14, 2023. EDM is the annual flagship conference of the International Educational Data Mining Society. The theme of this year's conference is "Educational data mining for amplifying human potential." Not all students or seekers of knowledge receive the education necessary to help them realize their full potential, be it due to a lack of resources or lack of access to high quality teaching. The dearth in high-quality educational content, teaching aids, and methodologies, and non-availability of objective feedback on how they could become better teachers, deprive our teachers from achieving their full potential. The administrators and policy makers lack tools for making optimal decisions such as optimal class sizes, class composition, and course sequencing. All these handicap the nations, particularly the economically emergent ones, who recognize the centrality of education for their growth. EDM-2023 has striven to focus on concepts, principles, and techniques mined from educational data for amplifying the potential of all the stakeholders in the education system. The spotlights of EDM-2023 include: (1) Five keynote talks by outstanding researchers of eminence; (2) A plenary Test of Time award talk and a Banquet talk; (3) Five tutorials (foundational as well as advanced); (4) Four thought provoking panels on contemporary themes; (5) Peer reviewed technical paper and poster presentations; (6) Doctoral students consortium; and (7) An enchanting cultural programme. [Individual papers are indexed in ERIC.]

Assessing students' performance in online learning could be executed not only by the traditional forms of summative assessments such as using essays, assignments, and a final exam, etc. but also by more formative assessment approaches such as interaction activities, forum posts, etc. However, it is difficult for teachers to monitor and assess students' learning activities using the log data. To provide teachers with a more comprehensive view of students' distinct learning behaviour patterns, and to supply personalized interventions and support to meet the specific needs of each learning group, this study focuses on how to automatically acquire learning logs from Manaba, a Japanese commercial LMS, and how to cluster students' learning activities using the k-means algorithm. Firstly, we developed a program using Python to scrape students' learning activity log information from the Manaba web pages. We collected 56446 lines of clickstreams log data from 121 students in two computer literacy hybrid classes in the fall semester of 2022 (2022/9-2023/1). Secondly, we convert the raw logs into a structured dataset with 33 features which represent each student's learning activities. Then we extract and select 15 features representing three perspectives: raw activity, time on task, and learning frequency. Thirdly, we grouped students' learning activity patterns with the three perspectives into 5 clusters by the k-means clustering algorithm. As a result, this study identified five distinct learning activity patterns depending on how much, how long and how often the students learned online. For example, cluster 1 seldom learned but spent time on learning whom we considered the disengaged or struggling students, and cluster 5 had more learning activities with little time on each activity whom we considered the well-self-regulated students. The results of this study contribute to how to monitor students' learning activity in online learning and how to assess and support student's learning by their learning activity patterns. [For the full proceedings, see ED636095.]

The value of an instructor is that she exactly recognizes what the learner is struggling with and provides constructive feedback straight to the point. This work aims at a step towards this type of feedback in the context of an introductory programming course, where students perform program execution tracing to align their understanding of Java instructions with reality. The students' submissions are analyzed for repeating mistakes across different exercises by representing the context surrounding the error by a graph and applying graph mining techniques to discover their common grounds. The patterns need to be annotated only once and help to address misconceptions of individual students. They may also be used to select follow-up exercises automatically, that contain the same intricacy. [For the full proceedings, see ED623995.]

An online peer-assessment system typically allows students to give textual feedback to their peers, with the goal of helping the peers improve their work. The amount of help that students receive is highly dependent on the quality of the reviews. Previous studies have investigated using machine learning to detect characteristics of reviews (e.g., Do they mention a problem, make a suggestion, or tell the student where to make a change?). Machine-learning approaches to peer-assessment evaluation are heavily reliant on labeled data to learn how to identify review characteristics. However, attaining reliable labels for those characteristics is always time-consuming and labor-intensive. In this study, we propose to apply pseudo-labeling, a semi-supervised learning-based strategy, to improve the recognition of reviews that detect problems in the reviewed work. This is done by utilizing a small, reliably labeled dataset along with a large unlabeled dataset to train a text classifier. The ultimate goal of this research is to show that for peer assessment evaluation, we can utilize both unlabeled and labeled datasets to obtain a robust auto-labeling system and thereby save much effort in labeling the data. [For the full proceedings, see ED623995.]

Ten years after the announcement of the "rise of the super experiment" at Educational Data Mining 2012, challenges to implementing "internet scale" educational experiments often persist for educational technology providers, especially when they seek to test substantive instructional interventions. Studies that deploy and test interventions, when informed by data-driven modeling, are often described as "close the loop" studies. Studies that close the loop attempt to link improvements in statistical and machine learning models of learning to real-world learning outcomes. After first considering challenges to internet scale experiments, we review several educational data science/mining studies that close the loop between data-driven modeling and learning outcomes. Next, we describe UpGrade, an open source architecture that, when integrated with educational technologies, helps overcome challenges to large-scale field trials (or internet scale experiments) that close the loop between data-driven work and real-world learning outcomes. In addition to describing preliminary randomized experiments that have been conducted and will soon be conducted using the architecture in two educational technology platforms, we end with a "call for contributors and integrators." UpGrade contributors and integrators will be researchers and developers who seek to drive continuous, data-driven improvements in real-world settings where learning with technology occurs. [For the full proceedings, see ED623995.]

Although badges are among the most-used game elements in gamified education, studies about their optimal features to motivate learning are scarce. How should a badge be designed to represent an incentive for a specific goal like optimal exam preparation? This study examines usage data of a higher education learning app to determine whether the used badges have the intended motivational effect. The preliminary results suggest that the badges that were initially implemented in the app have the intended effect in most cases, but the stages of the multi-level badges could be optimized. The methodological framework used in this study can be transferred to usage data of other similar learning tools. With the help of easy-to-interpret outputs of decision trees, researchers and practitioners alike can work towards an optimal badge design. [For the full proceedings, see ED623995.]

The aim of this study is the postgraduate thesis studies in the field of social studies education in Turkey between 2015 and 2021; to examine according to descriptive, methodological and subject distributions. Document analysis technique, which is one of the qualitative research designs, was used in the research. The main data source of the research is postgraduate theses. These theses were obtained by scanning in YÖK Thesis Center, and a total of 785 postgraduate theses, of which 659 are master's and 126 are doctoral theses, were examined. In the research, "Thesis Classification Table" and "Methodical Classification Table" were used as data collection tools. Content analysis technique, which is one of the analysis techniques used for qualitative research, was used in the research. According to the descriptive distribution results of the research; When the thesis type distributions were examined, it was observed that the master's theses were much more than the doctoral theses and the universities where the most studies were conducted were Gazi and Marmara universities. When the research type distributions of the theses were examined, it was observed that the most qualitative research type was used, and the scanning design was mostly used as a design. When the sample distributions of the theses were examined, it was observed that they mostly worked with secondary school students. According to the subject distribution results of the research; It was observed that subjects belonging to the categories of attitude/opinion/concern/belief and teaching method/technique/strategy/approach were studied. Another result is; It is the result of the theses made in the field of social studies education that there are some methodological deficiencies. It has been observed that some of the theses examined do not have a method section, and some theses have a method section, but are very incomplete and faulty in content. [For the full proceedings, see ED630948.]

With the rapid development of science and technology in recent years, the application areas of fuzzy logic have also gained speed. Fuzzy logic is a frequently preferred approach in the educational process, and it can be said that scientific publications on this topic have recently gained momentum in the literature. In this context, the present study aims to shed light on the research forms of fuzzy logic approach in education. The Web of Science (WoS) database was used for data collection. The search reached 374 scientific publications on the topic after various exclusion procedures. The obtained data were visualized using the mapping technique. As a result of the research, it was found that the number of publications on this topic has generally increased from past to present. The analysis revealed that the most influential countries reporting on this topic are the United States, China, and Taiwan. In addition, National Cheng Kung University, Gazi University, Cornell University, and Marmara University have published the most publications. The authors with the most publications on the fuzzy logic approach are Reyna V.F., Castillo, O. and Huang, Y.M. The analysis of the journals showed that Computer Applications in Engineering Education and International Journal of Electrical Engineering Education are the journals that contribute the most to this topic. The concepts that appeared in the clusters of co-occurrences analysis are fuzzy logic, higher education, e-learning, data mining, matlab and education. [For the full proceedings, see ED630948.]

Typical data science instruction uses generic datasets like survival rates on the Titanic, which may not be motivating for students. Will introducing real-life data science problems fill this motivational deficit? To analyze this question, we contrasted learning with generic datasets and artificial problems (Phase 1) with a community-sourced dataset and authentic problems (Phase 2) in the context of an 8-week virtual internship. Retrospective survey questions indicated interns experienced increased motivation in Phase 2. Additionally, analysis of intern discourse using Linguistic Inquiry and Word Count (LIWC) indicated a significant difference in linguistic measures between the two phases. Phase 1 had significantly greater measures of pronouns with a small-medium effect size, 2nd person words with a medium-large effect size, positive emotion with a medium effect size, inter-rogations with a medium-large effect size, question marks with a medium-large effect size, risk with a medium-large effect size, and causal words with a medium effect size. These results in conjunction with a retrospective survey suggest that phase 1 had more questions asked, more causal relationships defined, and included linguistic features of success and failure. Results from Phase 2 indicated that community-sourced data and problems may increase motivation for learning data science. [For the full proceedings, see ED623995.]

The COVID-19 epidemic had caused one of the most significant disruptions to the global education system. Many educational institutions faced sudden pressure to switch from face-to-face to online delivery of courses. The conventional classes are no longer the primary means of delivery; instead, online education and resources have become the prominent approach. With the increasing demand for supplementary course materials to fulfill the needs of each area of study, students began to use search engines and online resources that contain discussions, practical demonstrations, and tutorial videos to aid students in their studies and course work. This study addresses the underlying challenges of retrieving relevant online educational materials by introducing an intelligent agent for semantic data mining. It works as middleware infrastructure that allow context-aware data processing and mining. YouTube was used to assess the consistency of the proposed model since it returns a large number of results in its search pool. The results showed that using the extraction of topics method, the similarities scores with the proposed model provided favorable results. Furthermore, an improvement in video ranking and sorting was realized. According to the findings, using this method provided users with a more productive and reliable study experience. [For the full proceedings, see ED639633.]

Randomized A/B tests allow causal estimation without confounding but are often under-powered. This paper uses a new dataset, including over 250 randomized comparisons conducted in an online learning platform, to illustrate a method combining data from A/B tests with log data from users who were not in the experiment. Inference remains exact and unbiased without additional assumptions, regardless of the deep-learning model's quality. In this dataset, incorporating auxiliary data improves precision consistently and, in some cases, substantially.

We interrogate the opportunities and challenges of using mixed methods (MM) within a developmental evaluation (DE) context by drawing on two illustrative cases that investigated educational change in Canada. Methods: Multi-case design and cross-case analysis, with a focus on examining common patterns across the two cases, enabling new ways of portraying and thinking about the data. Findings: We present a four-part framework for charting MM integration (why, what, when, and how) within DE contexts. Then, from the juxtaposition of these cases using this framework, we present three considerations for evaluation practice and future research: (1) balancing the need for data collection versus moving initiatives forward, (2) not assuming stakeholders want co-production, and (3) developing integration literacy.