Search

We propose an innovative, effective, and data-agnostic method to train a deep-neural network model with an extremely small training dataset, called VELR (Voting-based Ensemble Learning with Rejection). In educational research and practice, providing valid labels for a sufficient amount of data to be used for supervised learning can be very costly and often impractical. The shortage of training data often results in deep neural networks being overfitting. There are many methods to avoid overfitting such as data augmentation and regularization. Though, data augmentation is considerably data dependent and does not usually work well for natural language processing tasks. Moreover, regularization is often quite task specific and costly. To address this issue, we propose an ensemble of overfitting models with uncertainty-based rejection. We hypothesize that misclassification can be identified by estimating the distribution of the class-posterior probability P(y|x) as a random variable. The proposed VELR method is data independent, and it does not require changes to the model structure or the re-training of the model. Empirical studies demonstrated that VELR achieved classification accuracy of 0.7 with only 200 samples per class on the CIFAR-10 dataset, but 75% of input samples were rejected. VELR was also applied to a question generation task using a BERT language model with only 350 training data points, which resulted in generating questions that are indistinguishable from human-generated questions. The paper concludes that VELR has potential applications to a broad range of real-world problems where misclassification is very costly, which is quite common in the educational domain. [For the complete proceedings, see ED630829.]

Despite calls to increase the focus on explainability and interpretability in EDM and, in particular, student success prediction, so that it becomes useful for personalized intervention systems, only few efforts have been undertaken in that direction so far. In this paper, we argue that this is mainly due to the limitations of current Explainable Artificial Intelligence (XAI) approaches regarding interpretability. We further argue that the issue, thus, calls for a a combination of AI and social science methods utilizing the strengths of both. For this, we introduce a step-wise model of interpretability where the first step constitutes of knowing important features, the second step of understanding counterfactuals regarding a particular person's prediction, and the third step of uncovering causal relations relevant for a set of similar students. We show that LIME, a current XAI method, reaches the first but not subsequent steps. To reach step two, we propose an extension to LIME, Minimal Counterfactual-LIME, finding the smallest number of changes necessary to change a prediction. Reaching step three, however, is more involved and additionally requires theoretical and causal reasoning - to this end, we construct an easily applicable framework. Using artificial data, we showcase that our methods can recover connections among features; additionally, we demonstrate its applicability on real-life data. Limitations of our methods are discussed and collaborations with social scientists encouraged. [For the full proceedings, see ED623995.]

This work explores how population-based engagement prediction can address cold-start at scale in large learning resource collections. The paper introduces: (1) VLE, a novel dataset that consists of content and video based features extracted from publicly available scientific video lectures coupled with implicit and explicit signals related to learner engagement; (2) two standard tasks related to predicting and ranking context-agnostic engagement in video lectures with preliminary baselines; and (3) a set of experiments that validate the usefulness of the proposed dataset. Our experimental results indicate that the newly proposed VLE dataset leads to building context-agnostic engagement prediction models that are significantly performant than ones based on previous datasets, mainly attributing to the increase of training examples. VLE dataset's suitability in building models towards Computer Science/ Artificial Intelligence education focused on e-learning/MOOC use-cases is also evidenced. Further experiments in combining the built model with a personalising algorithm show promising improvements in addressing the cold-start problem encountered in educational recommenders. This is the largest and most diverse publicly available dataset to our knowledge that deals with learner engagement prediction tasks. The dataset, helper tools, descriptive statistics and example code snippets are available publicly. [For the full proceedings, see ED623995.]

Typical data science instruction uses generic datasets like survival rates on the Titanic, which may not be motivating for students. Will introducing real-life data science problems fill this motivational deficit? To analyze this question, we contrasted learning with generic datasets and artificial problems (Phase 1) with a community-sourced dataset and authentic problems (Phase 2) in the context of an 8-week virtual internship. Retrospective survey questions indicated interns experienced increased motivation in Phase 2. Additionally, analysis of intern discourse using Linguistic Inquiry and Word Count (LIWC) indicated a significant difference in linguistic measures between the two phases. Phase 1 had significantly greater measures of pronouns with a small-medium effect size, 2nd person words with a medium-large effect size, positive emotion with a medium effect size, inter-rogations with a medium-large effect size, question marks with a medium-large effect size, risk with a medium-large effect size, and causal words with a medium effect size. These results in conjunction with a retrospective survey suggest that phase 1 had more questions asked, more causal relationships defined, and included linguistic features of success and failure. Results from Phase 2 indicated that community-sourced data and problems may increase motivation for learning data science. [For the full proceedings, see ED623995.]

In higher education, predictive analytics can provide actionable insights to diverse stakeholders such as administrators, instructors, and students. Separate feature sets are typically used for different prediction tasks, e.g., student activity logs for predicting in-course performance and registrar data for predicting long-term college success. However, little is known about the overall utility of different data sources across prediction tasks and the fairness of their predictions with respect to different subpopulations. Using data from over 2,000 college students at a large public university, we examined the utility of institutional data, learning management system (LMS) data, and survey data for accurately and fairly predicting short-term and long-term student success. We found that institutional data and LMS data both have decent predictive power, but survey data shows very little predictive utility. Combining institutional data with LMS data leads to even higher accuracy than using either alone. In terms of fairness, using institutional data consistently underestimates historically disadvantaged student subpopulations more than their peers, whereas LMS data tend to overestimate some of these groups more often. Combining the two data sources does not fully neutralize the biases and still leads to high rates of underestimation among disadvantaged groups. Moreover, algorithmic biases affect not only demographic minorities but also students with acquired disadvantages. These analyses serve to inform more cost-effective and equitable use of student data for predictive analytics applications in higher education. [For the full proceedings, see ED607784.]

Supervised machine learning has become one of the most important methods for developing educational and intelligent tutoring software; it is the backbone of many educational data mining methods for estimating knowledge, emotion, and other aspects of learning. Hence, in order to ensure optimal utilization of computing resources and effective analysis of models, it is essential that researchers know which evaluation metrics are best suited to educational data. In this article, we focus on the problem of wrapper feature selection, where predictors are added to models based on how much they improve model accuracy in terms of a given metric. We compared commonly-used machine learning algorithms including naive Bayes, support vector machines, logistic regression, and random forests on 11 diverse learning-related datasets. We optimized feature selection based on nine different metrics, then evaluated each to address research questions about how effective each metric was in terms of the others (e.g., does optimizing for precision also result in good F1?) as well as calibration (i.e., are predictions produced by models accurate probabilities of correctness?). We provide empirical evidence that the Matthews correlation coefficient (MCC) produced the overall best results across the other metrics, but that root mean squared error (RMSE) selected the best-calibrated models. Finally, we also discuss issues related to the number of features selected when optimizing for each metric, as well as the types of datasets for which certain metrics were more effective. [For the full proceedings, see ED607784.]

The field of education is undergoing fundamental change with the growing use of data. Fine-scale data collection at the item-response level is now possible. Xerox has developed a system that bridges the paper-to-digital divide by providing the well-established and easy-to-use paper interface to students, but digitizes the responses for scoring, validating, reporting, and managing data using a range of digital technologies. The Ignite™ system supports written responses, shading, connecting lines, multiple choice selections, and other question types. For some users, monitoring is at a very fine-grain level in both time and skill, while for others the data is used for more summative evaluations and strategic planning; one users' details may be another user's overview. All the reports presented in this document use the same basic atomic data elements and associated meta-data. The hierarchical nature of the organization of users requires that these atomic elements be combined in different ways for specialized visual representations, dependent upon the needs of the user. [For complete proceedings, see ED560503.]

Intelligent Tutoring Systems (ITSs) have shown success in the domain of programming, in part by providing customized hints and feedback to students. However, many popular novice programming environments still lack these intelligent features. This is due in part to their use of open-ended programming assignments, which are difficult to support with existing hint generation techniques. In this paper, we present a new data-driven algorithm, based on the Hint Factory, to generate hints for these open-ended assignments. We evaluate our algorithm on historical student data and show that it can provide hints that successfully lead students to solutions from any state, help students achieve assignment objectives, and align with the student's future solution. [For the full proceedings, see ED592609.]

Course selection can be a daunting task, especially for first year students. Sub-optimal selection can lead to bad performance of students and increase the dropout rate. Given the availability of historic data about student performances, it is possible to aid students in the selection of appropriate courses. Here, we propose a method to compose a personalized curriculum for a given student. We develop a modular approach that combines a context-aware grade prediction with statistical information on the useful temporal ordering of courses. This allows for meaningful course recommendations, both for fresh and senior students. We demonstrate the approach using the data of the computer science Bachelor students at Saarland University. [For the full proceedings, see ED593090.]

Data-driven methods have been a successful approach to generating hints for programming problems. However, the majority of previous studies are focused on procedural hints that aim at moving students to the next closest state to the solution. In this paper, I propose a data-driven method to generate remedy hints for BOTS, a game that teaches programming through a block-moving puzzle. Remedy hints aim to help students out of dead-end states, which are states in the problem from where no student has ever derived a solution. To address this, my proposed work includes designing debugging activities and generating remedy hints from students' solutions to debugging activities. [For complete proceedings, see ED560503.]

We present an analysis of log data from a semester's use of the OpenDSA eTextbook system with the goal of determining the most difficult course topics in a data structures course. While experienced instructors can identify which topics students most struggle with, this often comes only after much time and effort, and does not provide real-time analysis that might benefit an intelligent tutoring system. Our factors included the fraction of wrong answers given by student, results from Item Response Theory, and the rate of model answer and hint use by students. We grouped exercises by topic covered to yield a list of topics associated with the harder exercises. We found that a majority of these exercises were related to algorithm analysis topics. We compared our results to responses given by a sample of experienced instructors, and found that the automated results match the expert opinions reasonably well. We investigated reasons that might explain the over-representation of algorithm analysis among the difficult topics, and hypothesize that visualizations might help to better present this material. [For the full proceedings, see ED592609.]

With the growing popularity of MOOCs and sharp trend of digitalizing education, there is a huge amount of free digital educational material on the web along with the activity logs of large number of participating students. However, this data is largely unstructured and there is hardly any information about the relationship between material from different sources. We propose a generic algorithm to use educational material and student activity data from heterogeneous sources to create a Prerequisite Structure Graph (PSG). A PSG is a directed acyclic graph, where the nodes are educational units and the edges specify the pairwise ordering of the units in effective teaching by instructors or for effective learning by students. We propose an unsupervised approach utilizing both text content and student data, which outperforms to supervised methods (utilizing only text content) on the task of estimating a PSG. [For the full proceedings, see ED592609.]

We have been incrementally adding data-driven methods into the Deep Thought logic tutor for the purpose of creating a fully data-driven intelligent tutoring system. Our previous research has shown that the addition of data-driven hints, worked examples, and problem assignment can improve student performance and retention in the tutor. In this study, we investigate how the addition of these methods affects students' demonstrative knowledge of logic proof solving using their post-tutor examination scores. We have used data collected from three test conditions with different combinations of our data-driven additions to determine which methods are most beneficial to students who demonstrate higher or lower knowledge of the subject matter. Our results show that students who are assigned problems based on profiling proficiency compared to prior exemplary students with similar problem-solving behavior show higher examination scores overall, and the use of proficiency profiling increases retention and reduces the amount of time taken in-tutor for lower performing students in particular. The results from this study also helps differentiate the behavior of higher and lower performing students in tutor, which can allow quicker interventions for lower proficiency students. [For the full proceedings, see ED592609.]

The first intelligent tutoring systems for computer programming have been proposed more than 30 years ago, mostly focusing on well defined programming tasks e.g. in the context of logic programming. Recent systems also teach complex programs, where explicit modelling of every possible program and mistake is no longer possible. Such systems are based on data-driven approaches, which focus on the syntax of a program or consider the output for example cases. However, the system's understanding of student programs could be enriched by a deeper focus on the actual execution of a program. This requires a suitable data representation which encodes information of programming style as well as its functionality in a suitable way, thus offering entry points for automated feedback generation. In this contribution we propose a representation of computer programs via execution traces for example input and demonstrate the power of this representation in three key challenges for intelligent tutoring systems: identifying the underlying solution strategy, identifying erroneous solutions and locating the errors in erroneous programs for feedback display. [For the full proceedings, see ED592609.]

The increasing use of educational technologies in classrooms is producing vast amounts of process data that capture rich information about learning as it unfolds. The field of educational data mining has made great progress in using log data to build models that improve instruction and advance the science of learning. Thus far, however, the predictive and explanatory power of such models has often been limited to the actions that educational technologies can log. A major challenge in incorporating more contextually rich data streams into models of learning is collecting and integrating data from different sources and at different grain sizes. We present our methodological advances in automating the integration of log data with additional multi-modal (e.g., audio, screen video, webcam video) data streams. We also demonstrate several examples of how integrating multiple streams of data into the knowledge component (KC) model refinement process improves the predictive fit of student models and yields important pedagogical implications. This work represents an important advancement in facilitating the integration of rich qualitative details of students' learning contexts into the quantitative approaches characteristic of EDM research. [For the full proceedings, see ED592609.]

This study investigates the issue of the goodness of fit of different skills assessment models using both synthetic and real data. Synthetic data is generated from the different skills assessment models. The results show wide differences of performances between the skills assessment models over synthetic data sets. The set of relative performances for the different models create a kind of "signature" for each specific data. We conjecture that if this signature is unique, it is a good indicator that the corresponding model is a good fit to the data. [For complete proceedings, see ED560503.]

How can we automatically determine which skills must be mastered for the successful completion of an online course? Large-scale online courses (e.g., MOOCs) often contain a broad range of contents frequently intended to be a semester's worth of materials; this breadth often makes it difficult to articulate an accurate set of skills and knowledge (i.e., a skill model, or the QMatrix). We have developed an innovative method to discover skill models from the data of online courses. Our method assumes that online courses have a pre-defined skill map for which skills are associated with formative assessment items embedded throughout the online course. Our method carefully exploits correlations between various parts of student performance, as well as in the text of assessment items, to build a superior statistical model that even outperforms human experts. To evaluate our method, we compare our method with existing methods (LFA) and human engineered skill models on three Open Learning Initiative (OLI) courses at Carnegie Mellon University. The results show that (1) our method outperforms human-engineered skill models, (2) skill models discovered by our method are interpretable, and (3) our method is remarkably faster than existing methods. These results suggest that our method provides a significant contribution to the evidence-based, iterative refinement of online courses with a promising scalability. [For complete proceedings, see ED560503.]

The 5th International Conference on Educational Data Mining (EDM 2012) is held in picturesque Chania on the beautiful Crete island in Greece, under the auspices of the International Educational Data Mining Society (IEDMS). The EDM 2012 conference is a leading international forum for high quality research that mines large data sets of educational data to answer educational research questions. These data sets may come from learning management systems, interactive learning environments, intelligent tutoring systems, or any system used in a learning context. The following papers are presented at the conference: (1) Stream Mining in Education? Dealing with Evolution (Myra Spiliopoulou); (2) From Text to Feedback: Leveraging Data Mining to Build Educational Technologies (Danielle S. McNamara); (3) Five Aspirations for Educational Data Mining (Bob Dolan and John Behrens); (4) Assisting Instructional Assessment of Undergraduate Collaborative Wiki and SVN Activities (Jihie Kim, Erin Shaw, Hao Xu and Adarsh G V); (5) Automated Student Model Improvement (Kenneth R. Koedinger, Elizabeth A. McLaughlin and John C. Stamper); (6) Automatic Discovery of Speech Act Categories in Educational Games (Vasile Rus, Arthur Graesser, Cristian Moldovan and Nobal Niraula); (7) Co-Clustering by Bipartite Spectral Graph Partitioning for Out-of-Tutor Prediction (Shubhendu Trivedi, Zachary Pardos, Gabor Sarkozy and Neil Heffernan); (8) Comparison of methods to trace multiple subskills: Is LR-DBN best? (Yanbo Xu and Jack Mostow); (9) Dynamic Cognitive Tracing: Towards Unified Discovery of Student and Cognitive Models (Jose Gonzalez-Brenes and Jack Mostow); (10) Identifying Learning Behaviors by Contextualizing Differential Sequence Mining with Action Features and Performance Evolution (John S. Kinnebrew and Gautam Biswas); (11) Identifying Students' Characteristic Learning Behaviors in an Intelligent Tutoring System Fostering Self-Regulated Learning (Francois Bouchet, John S. Kinnebrew, Gautam Biswas and Roger Azevedo); (12) Learner Differences in Hint Processing (Ilya Goldin, Kenneth R. Koedinger and Vincent Aleven); (13) Methods to find the number of latent skills (Behzad Beheshti, Michel C. Desmarais and Rhouma Naceur); (14) Mining Student Behavior Patterns in Reading Comprehension Tasks (Terry Peckham and Gordon McCalla); (15) Model-Based Collaborative Filtering Analysis of Student Response Data: Machine-Learning Item Response Theory (Yoav Bergner, Stefan Droschler, Gerd Kortemeyer, Saif Rayyan, Daniel Seaton and David E. Pritchard); (16) Predicting drop-out from social behaviour of students (Tomas Obsivac, Lubomir Popelinsky, Jaroslav Bayer, Jan Geryk and Hana Bydzovska); (17) Searching for Variables and Models to Investigate Mediators of Learning from Multiple Representations (Martina A. Rau and Richard Scheines); (18) The Impact on Individualizing Student Models on Necessary Practice Opportunities (Jung In Lee and Emma Brunskill); (19) Towards Sensor-Free Affect Detection in Cognitive Tutor Algebra (Ryan S.J.D. Baker, Sujith M. Gowda, Michael Wixon, Jessica Kalka, Angela Z. Wagner, Aatish Salvi, Vincent Aleven, Gail W. Kusbit, Jaclyn Ocumpaugh and Lisa Rossi); (20) Using Edit Distance to Analyse Errors in a Natural Language to Logic Translation Corpus (Dave Barker-Plummer, Robert Dale, and Richard Cox); (21) Calculating Probabilistic Distance to Solution in a Complex Problem Solving Domain (Leigh Ann Sudol, Kelly Rivers and Thomas K. Harris); (22) Classification via clustering for predicting final marks based on student participation in forums (M.I. Lopez, J.M. Luna, C. Romero, and S. Ventura); (23) Development of a Workbench to Address the Educational Data Mining Bottleneck (Ma. Mercedes T. Rodrigo, Ryan S. J. D. Baker, Bruce McLaren, Alejandra Jayme and Thomas T. Dy); (24) Early Prediction of Student Self-Regulation Strategies by Combining Multiple Models (Jennifer L. Sabourin, Bradford W. Mott and James C. Lester); (25) Identifying Successful Learners from Interaction Behaviour (Judi McCuaig and Julia Baldwin); (26) Interaction Networks: Generating High Level Hints Based on Network Community Clusterings (Michael Eagle, Matthew Johnson and Tiffany Barnes); (27) Interleaved Practice with Multiple Representations: Analyses with Knowledge Tracing Based Techniques (Martina A. Rau and Zachary A. Pardos); (28) Learning Gains for Core Concepts in a Serious Game on Scientific Reasoning (Carol Forsyth, Philip Pavlik Jr, Arthur C. Graesser, Zhiqiang Cai, Mae-Lynn Germany, Keith Millis, Heather Butler, Diane Halpern and Robert P. Dolan); (29) Leveraging First Response Time into the Knowledge Tracing Model (Yutao Wang and Neil T. Heffernan); (30) Meta-learning Approach for Automatic Parameter Tuning: A case of study with educational datasets (M.M. Molina, J.M. Luna, C. Romero, and S. Ventura); (31) Mining Concept Maps to Understand University Students' Learning (Jin Soung Yoo and Moon-Heum Cho); (32) Policy Building--An Extension To User Modeling (Michael V. Yudelson and Emma Brunskill); (33) The real world significance of performance prediction (Zachary A. Pardos, Qing Yang Wang and Shubhendu Trivedi); (34) The Rise of the Super Experiment (John C. Stamper, Derek Lomas, Dixie Ching, Steven Ritter, Kenneth R. Koedinger and Jonathan Steinhart); (35) Using Student Modeling to Estimate Student Knowledge Retention (Yutao Wang and Joseph Beck); (36) A promising classification method for predicting distance students' performance (Diego Garcia-Saiz and Marta Zorrilla); (37) Analyzing paths in a student database (Donatella Merlini, Renza Campagni and Renzo Sprugnoli); (38) Analyzing the behavior of a teacher network in a Web 2.0 environment (Eliana Scheihing, Carolina Aros and Daniel Guerra); (39) Automated Detection of Mentors and Players in an Educational Game (Fazel Keshtkar, Brent Morgan and Arthur Graesser); (40) Categorizing Students' Response Patterns using the Concept of Fractal Dimension (Rasil Warnakulasooriya and William Galen); (41) CurriM: Curriculum Mining (M. Pechenizkiy, N. Trcka, P. De Bra and Pedro A. Toledo); (42) Data mining techniques for design of ITS student models (Ritu Chaturvedi and C. I. Ezeife); (43) Deciding on Feedback Polarity and Timing (Stuart Johnson and Osmar Zaiane); (44) Finding Dependent Test Items: An Information Theory Based Approach (Xiaoxun Sun); (45) Fit-to-Model Statistics for Evaluating Quality of Bayesian Student Ability Estimation (Ling Tan); (46) Inferring learners' knowledge from observed actions (Anna N. Rafferty, Michelle M. Lamar and Thomas L. Griffiths); (47) Learning Paths in a Non-Personalizing e-Learning Environment (Agathe Merceron, Sebastian Schwarzrock, Margarita Elkina, Andreas Pursian, Liane Beuster, Albrecht Fortenbacher, Leonard Kappe, and Boris Wenzlaff); (48) Similarity Functions for Collaborative Master Recommendations (Alexandru Surpatean, Evgueni Smirnov and Nicolai Manie); (49) Social Networks Analysis for Quantifying Students' Performance in Teamwork (Pedro Crespo and Claudia Antunes); (50) Speaking (and touching) to learn: a method for mining the digital footprints of face-to-face collaboration (Roberto Martinez Maldonado, Kalina Yacef and Judy Kay); (51) Stress Analytics in Education (Rafal Kocielnik, Mykola Pechenizkiy and Natalia Sidorova); and (52) Variable Construction and Causal Discovery for Cognitive Tutor Log Data: Initial Results (Stephen E. Fancsali). Individual papers contain figures, tables, references and footnotes. [Support for this publication was provided by Carnegie Learning, Pearson and LearnLab.]

Interactive problem solving environments, such as intelligent tutoring systems and educational video games, produce large amounts of transactional data which make it a challenge for both researchers and educators to understand how students work within the environment. Researchers have modeled the student-tutor interactions using complex network representations to automatically derive next-step hints, derive high-level approaches, and create visualizations of student behavior. However, students do not explore the complete problem space. The nonuniform exploration of the problem results in smaller networks and less next-step hints. In this work we explore the possibility of using frequency estimation to uncover locations in the network with differing amounts of student-saturation. Identification of these regions can be used to locate specific problem approaches and strategies that would be most improved by additional student-data. [For complete proceedings, see ED560503.]