Search

Even highly motivated undergraduates drift off their STEM career pathways. In large introductory STEM classes, instructors struggle to identify and support these students. To address these issues, we developed co‐redesign methods in partnership with disciplinary experts to create high‐structure STEM courses that better support students and produce informative digital event data. To those data, we applied theory‐ and context‐relevant labels to reflect active and self‐regulated learning processes involving LMS‐hosted course materials, formative assessments, and help‐seeking tools. We illustrate the predictive benefits of this process across two cycles of model creation and reapplication. In cycle 1, we used theory‐relevant features from 3 weeks of data to inform a prediction model that accurately identified struggling students and sustained its accuracy when reapplied in future semesters. In cycle 2, we refit a model with temporally contextualized features that achieved superior accuracy using data from just two class meetings. This modelling approach can produce durable learning analytics solutions that afford scaled and sustained prediction and intervention opportunities that involve explainable artificial intelligence products. Those same products that inform prediction can also guide intervention approaches and inform future instructional design and delivery.Practitioner notesWhat is already known about this topicLearning analytics includes an evolving collection of methods for tracing and understanding student learning through their engagements with learning technologies.Prediction models based on demographic data can perpetuate systemic biases.Prediction models based on behavioural event data can produce accurate predictions of academic success, and validation efforts can enrich those data to reflect students' self‐regulated learning processes within learning tasks.What this paper addsLearning analytics can be successfully applied to predict performance in an authentic postsecondary STEM context, and the use of context and theory as guides for feature engineering can ensure sustained predictive accuracy upon reapplication.The consistent types of learning resources and cyclical nature of their provisioning from lesson to lesson are hallmarks of high‐structure active learning designs that are known to benefit learners. These designs also provide opportunities for observing and modelling contextually grounded, theory‐aligned and temporally positioned learning events that informed prediction models that accurately classified students upon initial and later reapplications in subsequent semesters.Co‐design relationships where researchers and instructors work together toward pedagogical implementation and course instrumentation are essential to developing unique insights for feature engineering and producing explainable artificial intelligence approaches to predictive modelling.Implications for practice and/or policyHigh‐structure course designs can scaffold student engagement with course materials to make learning more effective and products of feature engineering more explainable.Learning analytics initiatives can avoid perpetuation of systemic biases when methods prioritize theory‐informed behavioural data that reflect learning processes, sensitivity to instructional context and development of explainable predictors of success rather than relying on students' demographic characteristics as predictors.Prioritizing behaviours as predictors improves explainability in ways that can inform the redesign of courses and design of learning supports, which further informs the refinement of learning theories and their applications. [ABSTRACT FROM AUTHOR]

• Many students struggle to overcome poor performances early in STEM courses. • Evaluation of prior studying predicts plans to adapt in next learning cycle. • Plans to adapt predict an increase of desirable learning behaviors. • Desirable learning behaviors positively relate to subsequent exam score. • Natural language processing was used to measure evaluation and plans to adapt. Researchers and many educators agree that the ability to self-regulate learning is important for academic success. Yet, many students struggle to anticipate learning difficulties and adjust accordingly. Further, despite theorizing that self-regulated learning involves adaptation across learning cycles, few researchers have examined students' evaluative judgments, their implications for students' behavior in a subsequent learning cycle, or their effects on achievement. Utilizing data from a large, introductory college biology course, we examined how struggling students' evaluative judgments made after a first unit exam predicted changes in learning behaviors as well as how those changes predicted performance on a subsequent exam. We used natural language processing to analyze data from a reflective essay written after a first unit exam, identifying language that reflected evaluation of prior studying and plans to adapt learning. Then, we utilized digital traces of learning behaviors to measure students' actual adaptation of their use of learning resources. Results from a path analysis revealed students' evaluations predicted how extensively they discussed plans to adapt their learning process. Plans to adapt described in written reflections predicted an increase in the frequency of desirable learning behaviors, which in turn predicted higher subsequent exam scores, after controlling for previous exam performance. These findings provide empirical evidence of multiple theorized self-regulated learning processes, including how evaluations of learning at the end of a learning cycle can inform planning and behavior changes in a subsequent learning cycle, and that increases in the enactment of effective learning strategies predict improved performance in complex learning tasks. [ABSTRACT FROM AUTHOR]