Search

Background: Many learners struggle to productively self-regulate their learning. To support the learners' self-regulated learning (SRL) and boost their achievement, it is essential to understand the cognitive and metacognitive processes that underlie SRL. To measure these processes, contemporary SRL researchers have largely utilized think aloud or trace data, however, not without challenges. Objectives: In this paper, we present the findings of a study that investigated how concurrent analysis and integration of think aloud and trace data could advance the measurement of SRL and assist in better understanding the mechanisms of SRL processes, especially those details that remain obscured by observing each data channel individually. Methods: We concurrently collected think aloud and trace data generated by 44 university students in a laboratory setting and analysed those data relative to the same timeline. Results: We found that the two data channels could be interchangeably used to measure SRL processes for only 17.18% of all the time segments identified in a learning task. Moreover, SRL processes for around 45% of all the time segments could be detected via either trace data or think aloud data. For another 27.17% of all the time segments, different SRL processes were detected in both data channels. Conclusions: Our results largely suggest that the two data collection methods can be used to complement each other in measuring SRL. In particular, we found that think aloud and trace data could provide different perspectives on SRL. The integration of the two methods further allowed us to reveal a more complex and more comprehensive temporal associations among SRL processes compared to using a single data collection method. In future research, the integrated measurement of SRL can be used to improve the detection of SRL processes and provide a fuller picture of SRL.

Background: Assignments that involve writing based on several texts are challenging to many learners. Formative feedback supporting learners in these tasks should be informed by the characteristics of evolving written product and by the characteristics of learning processes learners enacted while developing the product. However, formative feedback in writing tasks based on multiple texts has almost exclusively focused on essay product and rarely included SRL processes. Objectives: We explored the viability of using product and process features to develop machine learning classifiers that identify low- and high-performing essays in a multi-text writing task. Methods: We examined learning processes and essay submissions of 163 graduate students working on an authentic multi-text writing assignment. We utilised learners' trace data to obtain process features and state-of-the-art natural language processing methods to obtain product features for our classifiers. Results and Conclusions: Of four popular classifiers examined in this study, Random Forest achieved the best performance (accuracy = 0.80 and recall = 0.77). The analysis of important features identified in the Random Forest classification model revealed one product (coverage of reading topics) and three process (elaboration/organisation, re-reading and planning) features as important predictors of writing quality. Major Takeaways: The classifier can be used as a part of a future automated writing evaluation system that will support at scale formative assessment in writing tasks based on multiple texts in different courses. Based on important predictors of essay performance, a guidance can be tailored to learners at the outset of a multi-text writing task to help them do well in the task.

In online courses, discussion forums play a key role in enhancing student interaction with peers and instructors. Due to large enrolment sizes, instructors often struggle to respond to students in a timely manner. To address this problem, both traditional machine learning (ML) (e.g., Random Forest) and deep learning (DL) approaches have been applied to classify educational forum posts (e.g., those that required urgent responses versus those that did not). However, there lacks an in-depth comparison between these two kinds of approaches. To better guide people to select an appropriate model, we aimed at providing a comparative study on the effectiveness of six frequently-used traditional ML and DL models across a total of seven different classification tasks centering around two datasets of educational forum posts. Through extensive evaluation, we showed that (1) the up-to-date DL approaches did not necessarily outperform traditional ML approaches; (2) the performance gap between the two kinds of approaches can be up to 3.68% (measured in F1 score); (3) the traditional ML approaches should be equipped with carefully-designed features, especially those of common importance across different classification tasks. Based on the derived findings, we further provided insights to help instructors and educators construct effective classifiers for characterizing educational forum discussions, which, ultimately, would enable them to provide students with timely and personalized learning support.

Background: Select and enact appropriate learning tactics that advance learning has been considered a critical set of skills to successfully complete highly flexible online courses, such as Massive open online courses (MOOCs). However, limited by analytic methods that have been used in the past, such as frequency distribution, sequence mining and process mining, we lack a deep, complete and detailed understanding of the learning tactics used by MOOC learners. Objectives: In the present study, we proposed four major dimensions to better interpret and understand learning tactics, which are frequency, continuity, sequentiality and role of learning actions within tactics. The aim of this study was to examine to what extent can a new analytic technique, the ordered network analysis (ONA), deepen the understanding of MOOC learning tactics compared to using other methods. Methods: In particular, we performed a fine-grained analysis of learning tactics detected from more than 4 million learning events in the behavioural trace data of 8788 learners who participated in a large-scale MOOC 'Flipped Classroom'. Results and Conclusions: We detected eight learning tactics, and then chose one typical tactic as an example to demonstrate how the ONA technique revealed all four dimensions and provided deeper insights into this MOOC learning tactic. Most importantly, based on the comparison with different methods such as process mining, we found that the ONA method provided a unique opportunity and novel insight into the roles of different learning actions in tactics which was neglected in the past. Takeaway: In summary, we conclude that ONA is a promising technique that can benefit the research on learning tactics, and ultimately benefit MOOC learners by strengthening the strategic support.

Self-regulation is an essential skill for lifelong learning. Research has shown that self-regulated learning (SRL) leads to greater academic achievement and sustainable education, but students often struggle with SRL. Scaffolds are widely reported as an effective and efficient support method for SRL. To further improve digital scaffolds' effectiveness, real-time detection of learning behavior can be used to personalize scaffolds. Therefore, the present study aimed to inform the field of scaffolding SRL by reporting on the design and evaluation of digital scaffolds. We present decisions made during the design process of personalized scaffolds to inform future scaffold designs. We evaluated how scaffolds were personalized based on real-time detection SRL, how university students respond to the scaffolds (i.e., compliance), and how this response is related to learning outcomes (i.e., quality of an essay). The research design was a pre-posttest with three conditions (no, generalized (same for all), or personalized scaffolds). A 45-minute reading and writing task was used, during which SRL processes were recorded in real-time. Findings revealed that different real-time SRL processes could be used to personalize scaffolds, meaning that we were able to personalize the content of scaffold based on students' actual learning behavior. In addition, students in the personalized condition complied more with the scaffolds than students in the other conditions. This compliance with the scaffolds was generally associated with better learning outcomes. To conclude, our approach showed how design decisions could be evaluated and provided insight into the personalization of scaffolds.

Contemporary research that looks at self-regulated learning (SRL) as processes of learning events derived from trace data has attracted increasing interest over the past decade. However, limited research has been conducted that looks into the validity of trace-based measurement protocols. In order to fill this gap in the literature, we propose a novel validation approach that combines "theory-driven" and "data-driven" perspectives to increase the validity of interpretations of SRL processes extracted from trace-data. The main contribution of this approach consists of three alignments between trace data and think aloud data to improve measurement validity. In addition, we define the match rate between SRL processes extracted from trace data and think aloud as a quantitative indicator together with other three indicators (sensitivity, specificity and trace coverage), to evaluate the "degree" of validity. We tested this validation approach in a laboratory study that involved 44 learners who learned individually about the topic of artificial intelligence in education with the use of a technology-enhanced learning environment for 45 minutes. Following this new validation approach, we achieved an improved match rate between SRL processes extracted from trace-data and think aloud data (training set: 54.24%; testing set: 55.09%) compared to the match rate before applying the validation approach (training set: 38.97%; test set: 34.54%). By considering think aloud data as "reference point", this improvement of the match rate quantified the extent to which validity can be improved by using our validation approach. In conclusion, the novel validation approach presented in this study used both empirical evidence from think aloud data and rationale from our theoretical framework of SRL, which now, allows testing and improvement of the validity of trace-based SRL measurements.

In recent years, unobtrusive measures of self-regulated learning (SRL) processes based on log data recorded by digital learning environments have attracted increasing attention. However, researchers have also recognised that simple navigational log data or time spent on pages are often not fine-grained enough to study complex SRL processes. Recent advances in data-capturing technologies enabled researchers to go beyond simple navigational logs to measure SRL processes with multi-channel data. What multi-channel data can reveal about SRL processes, and to what extent can the addition of peripheral and eye-tracking data with navigational log data change and improve the measurement of SRL are key questions that require further investigation. Hence, we conducted a study and collected learning trace data generated by 25 university students in a laboratory setting, that aimed to address this problem by enhancing navigational log data with peripheral and eye-tracking data. We developed a trace-based measurement protocol of SRL, which interpreted raw trace data from multi-channel data into SRL processes. Specifically, the study compared the frequency and duration of SRL processes detected, how much duration and times of occurrences of the detected SRL processes were affected or refined. We also used a process mining technique to analyses how temporal sequencing of the detected SRL processes changed by enriching navigational log data with peripheral and eye-tracking data. The results revealed that by adding new data channels, we improved the capture of learning actions and detected SRL processes while enhancing the granularity of the measurement. In comparison to the use of navigational logs only, the completeness of temporal sequencing relationships between SRL processes with multi-channel data improved. In addition, we concluded that eye-tracking data is valuable for measuring and extracting SRL processes, and it should receive more attention in the future.

Predictive modeling is a core technique used in tackling various tasks in learning analytics research, e.g., classifying educational forum posts, predicting learning performance, and identifying at-risk students. When applying a predictive model, it is often treated as the first priority to improve its prediction accuracy as much as possible. Class balancing, which aims to adjust the unbalanced data samples of different class labels before using them as input to train a predictive model, has been widely regarded as a powerful method for boosting prediction accuracy. However, its impact on algorithmic bias remains largely unexplored, i.e., whether the use of class balancing methods would alleviate or amplify the differentiated prediction accuracy received by different groups of students (e.g., female versus male). To fill this gap, our study selected three representative predictive tasks as the testbed, based on which we (1) applied two well known metrics (i.e., hardness bias and distribution bias) to measure data characteristics to which algorithmic bias might be attributed; and (2) investigated the impact of a total of eleven class balancing techniques on prediction fairness. Through extensive analysis and evaluation, we found that class balancing techniques, in general, tended to improve predictive fairness between different groups of students. Furthermore, class balancing techniques (e.g., SMOTE and ADASYN), which add samples to the minority group (i.e., oversampling) can enhance the predictive accuracy of the minority group while not negatively affecting the majority group. Consequently, both fairness and accuracy can be improved by applying these oversampling class balancing methods.

Data used in learning analytics rarely provide strong and clear signals about how learners process content. As a result, learning as a process is not clearly described for learners or for learning scientists. Gaševic, Dawson, and Siemens (2015) urged data be sought that more straightforwardly describe processes in terms of events within learning episodes. They recommended building on Winne's (1982) characterization of traces -- ambient data gathered as learners study that more clearly represent which operations learners apply to which information -- and his COPES model of a learning event -- conditions, operations, products, evaluations, standards (Winne, 1997). We designed and describe an open source, open access, scalable software system called nStudy that responds to their challenge. nStudy gathers data that trace cognition, metacognition, and motivation as processes that are operationally captured as learners operate on information using nStudy's tools. nStudy can be configured to support learners' evolving self-regulated learning, a process akin to personally focused, self-directed learning science.

Developing knowledge-transforming skills in writing may help students increase learning by actively building knowledge, regardless of the domain. However, many undergraduate students struggle to transform knowledge when drafting essays based on multiple sources. Writing analytics can be used to scaffold knowledge transforming as writers bring evidence to bear in supporting claims. We investigated how to automatically identify sentences representing knowledge transformation in argumentative essays. A synthesis of cognitive theories of writing and Bloom's typology identified 22 linguistic features to model processes of knowledge transforming in a corpus of 38 undergraduates' essays. Findings indicate undergraduates mostly paraphrase or copy information from multiple sources rather than engage deeply with sources' content. Eight linguistic features were important for discriminating evidential sentences as telling versus transforming source knowledge. We trained a machine learning algorithm that accurately classified nearly three of four evidential sentences as knowledge-telling or knowledge-transforming, offering potential for use in future research.

Self‐regulated learning (SRL) is an essential skill to achieve one's learning goals. This is particularly true for online learning environments (OLEs) where the support system is often limited compared to a traditional classroom setting. Likewise, existing research has found that learners often struggle to adapt their behaviour to the self‐regulatory demands of OLEs. Even so, existing SRL analysis tools have limited utility for real‐time or individualised support of a learner's SRL strategy during a study session. Accordingly, we explore a reinforcement learning based approach to learning optimal SRL strategies for a specific learning task. Specifically, we utilise the sequences of SRL processes acted by 44 participants, and their assessment scores for a prescribed learning task, in a purpose‐built OLE to develop a long short‐term memory (LSTM) network based reward function. This is used to train a reinforcement learning agent to find the optimal sequence of SRL processes for the learning task. Our findings indicate that the developed agents were able to effectively select SRL processes so as to maximise a prescribed learning goal as measured by predicted assessment score and predicted knowledge gains. The contributions of this work will facilitate the development of a tool which can detect sub‐optimal SRL strategy in real‐time and enable individualised SRL focused scaffolding. Practitioner notesWhat is already known about this topic Learners often fail to adequately adapt their behavior to the self‐regulatory demands of e‐Learning environments.In order to promote effective Self‐regulated learning (SRL) capabilities, researchers and educators need tools that are able to analyze and diagnose a learner's SRL strategy use.Current methods for SRL analysis are more often descriptive as opposed to prescriptive and have limited utility for real‐time analysis or support of a learner's SRL behavior.What this paper adds This paper proposes the use of Reinforcement Learning for prescriptive analytics of SRL. We train a Reinforcement Learning agent on sequences of SRL processes acted by learners in order to learn the optimal SRL strategy for a given learning task.Implications for practice and/or policy Our work will facilitate the development of a tool which can detect sub‐optimal SRL strategy in real‐time and enable individualized SRL focused scaffolding.The implications of our work can aid in course design by predicting the self‐regulatory load imposed by a given task.The ability to model SRL strategies using Reinforcement Learning can be extended to simulate or test SRL theories. [ABSTRACT FROM AUTHOR]

Scaffolds that support self‐regulated learning (SRL) have been found to improve learning outcomes. The effects of scaffolds can differ depending on how learners use them and how specific scaffolds might influence learning processes differently. Personalized scaffolds have been proposed to be more beneficial for learning due to their adaptivity to learning progress and individualized content to learning needs. The present study investigated finer‐grained effects of how personalized scaffolds driven by a rule‐based artificial intelligence system influenced SRL processes, especially how students learned with them. Using a pre‐post experimental design, we investigated personalized scaffolds based on university students' real‐time learning processes in a technologically enhanced learning environment. Students in the experimental group (n = 30) received personalized scaffolds, while the control group (n = 29) learned without scaffolds. All students completed a 45‐minute learning task with trace data recorded. Findings indicated scaffold effects on students' subsequent learning behaviour. Additionally, only scaffold interaction correlated to essay performance and suggests that the increase in frequencies of SRL activities alone does not contribute directly to learning outcomes. As guidelines for real‐time SRL support are lacking, this study provides valuable insights to enhance SRL support with adaptive learning technologies. Practitioner notesWhat is already known about this topic Self‐regulated learning scaffolds, especially adaptive scaffolds, improve learning.Personalized scaffolds have effects on self‐regulated learning activities.Past research focused on aggregated effects of scaffolds.What this paper adds Investigates how students learn with personalized scaffolds in terms of frequencies of learning activities and scaffold interaction.Takes a closer look at which learning activities and when the effects of personalized scaffolds occur.Examines how finer‐grained effects of personalized scaffolds correspond to learning outcomes.Implications for practice and/or policy Personalized scaffold effects vary across learning, and future research should consider finer‐grained investigations of SRL support in order to better understand their influence on learning.The number of personalized scaffolds provided should be reconsidered in the future as students only use some of the support provided, especially when task demands increase.Personalized scaffold interaction is linked to improvement in task performance, so future research should also focus on students' appropriate use of self‐regulated learning support. [ABSTRACT FROM AUTHOR]

Learning analytics are often formatted as visualisations developed from traced data collected as students study in online learning environments. Optimal analytics inform and motivate students' decisions about adaptations that improve their learning. We observe that designs for learning often neglect theories and empirical findings in learning science that explain how students learn. We present six learning analytics that reflect what is known in six areas (we call them cases) of theory and research findings in the learning sciences: setting goals and monitoring progress, distributed practice, retrieval practice, prior knowledge for reading, comparative evaluation of writing, and collaborative learning. Our designs demonstrate learning analytics can be grounded in research on self-regulated learning and self-determination. We propose designs for learning analytics in general should guide students toward more effective self-regulated learning and promote motivation through perceptions of autonomy, competence, and relatedness.