Search

Reflection plays a critical role in learning. Game-based learning environments have significant potential to elicit and support student reflection by prompting learners to think critically about their own learning processes and performance. Stealth assessment models, used for unobtrusively assessing student competencies from evidence of game interaction data and facilitating learning through adaptive feedback, can be enhanced by incorporating evidence from students' written reflections. We present a deep learning-based stealth assessment framework that predicts depth of student reflections and science content post-test scores during game-based learning. With the increasing adoption of AI techniques in decision-making processes, it is important to evaluate the fairness of these models. To address this concern, we investigate encoded bias in our stealth assessment model with respect to student gender and prior game-playing experience in deep learning-based stealth assessment models and examine the impact of debiasing on the models' predictive performance. We evaluate the predictive performance of the deep learning-based stealth assessment models and measure encoded bias with the Absolute Between-ROC Area (ABROCA) statistic using gameplay data from 119 students collected in a series of classroom studies with a reflection-enriched game-based learning environment for middle school microbiology, Crystal Island. The results demonstrate the effectiveness of deep learning-based stealth assessment models and multiple debiasing techniques for deriving algorithmically fair stealth assessment models. [ABSTRACT FROM AUTHOR]

Introduction: Self-regulated learning (SRL), or learners' ability to monitor and change their own cognitive, affective, metacognitive, and motivational processes, encompasses several operations that should be deployed during learning including Searching, Monitoring, Assembling, Rehearsing, and Translating (SMART). Scaffolds are needed within GBLEs to both increase learning outcomes and promote the accurate and efficient use of SRL SMART operations. This study aims to examine how restricted agency (i.e., control over one's actions) can be used to scaffold learners' SMART operations as they learn about microbiology with Crystal Island, a game-based learning environment. Methods: Undergraduate students (N=94) were randomly assigned to one of two conditions: (1) Full Agency, where participants were able to make their own decisions about which actions they could take; and (2) Partial Agency, where participants were required to follow a pre-defined path that dictated the order in which buildings were visited, restricting one's control. As participants played Crystal Island, participants' multimodal data (i.e., log files, eye tracking) were collected to identify instances where participants deployed SMART operations. Results: Results from this study support restricted agency as a successful scaffold of both learning outcomes and SRL SMART operations, where learners who were scaffolded demonstrated more efficient and accurate use of SMART operations. Discussion: This study provides implications for future scaffolds to better support SRL SMART operations during learning and discussions for future directions for future studies scaffolding SRL during game-based learning. [ABSTRACT FROM AUTHOR]

Pedagogical agents have been designed to support the significant challenges that learners face when self-regulating in advanced learning environments. Evidence suggests differences in learners' prior skills and abilities, in conjunction with excessive didactic support, can cause overreliance on these external aids, which in turn prevents deeper learning, and pedagogical agents can provide tailored scaffolding to accommodate learners' individual needs. However, there is less evidence about the impact of abstract scaffolding, such as the sharing of non-verbal metacognitive information via a pedagogical agent's facial expressions, on self-regulated learning. To assess factors in the passing of non-verbal metacognitive information via pedagogical agents in a multimedia learning environment, we used growth modeling with self-reports, eye-tracking, and log-file data collected from fifty (n = 50) undergraduates at a large North American university as they learned about human body systems while using MetaTutor-IVH, a multimedia learning environment with a pedagogical agent. We controlled for participant characteristics (perceived utility of emotions for self- and other-centered positive and negative emotions) and characteristics of the metacognitive monitoring information provided by a pedagogical agent (expression type and expression congruency) to assess factors in non-verbally communicating metacognitive information. Results suggest that learners attend to pedagogical agents less over time, but this rate of change is weaker when an agent is providing an expression that is congruent with the ground truth of the environment. Further, only the perceived information utility of other-centered negative emotions has a significant effect on this duration, suggesting learners are driven to consult pedagogical agents to avoid embarrassment or shame. We discuss design implications of these findings for technology-based learning environments. [ABSTRACT FROM AUTHOR]

Game‐based learning environments hold significant promise for facilitating learning experiences that are both effective and engaging. To support individualised learning and support proactive scaffolding when students are struggling, game‐based learning environments should be able to accurately predict student knowledge at early points in students' gameplay. Student knowledge is traditionally assessed prior to and after each student interacts with the learning environment with conventional methods, such as multiple choice content knowledge assessments. While previous student modelling approaches have leveraged machine learning to automatically infer students' knowledge, there is limited work that incorporates the fine‐grained content from each question in these types of tests into student models that predict student performance at early junctures in gameplay episodes. This work investigates a predictive student modelling approach that leverages the natural language text of the post‐gameplay content knowledge questions and the text of the possible answer choices for early prediction of fine‐grained individual student performance in game‐based learning environments. With data from a study involving 66 undergraduate students from a large public university interacting with a game‐based learning environment for microbiology, Crystal Island, we investigate the accuracy and early prediction capacity of student models that use a combination of gameplay features extracted from student log files as well as distributed representations of post‐test content assessment questions. The results demonstrate that by incorporating knowledge about assessment questions, early prediction models are able to outperform competing baselines that only use student game trace data with no question‐related information. Furthermore, this approach achieves high generalisation, including predicting the performance of students on unseen questions. Practitioner notesWhat is already known about this topic A distinctive characteristic of game‐based learning environments is their capacity to enable fine‐grained student assessment.Adaptive game‐based learning environments offer individualisation based on specific student needs and should be able to assess student competencies using early prediction models of those competencies.Word embedding approaches from the field of natural language processing show great promise in the ability to encode semantic information that can be leveraged by predictive student models.What this paper adds Investigates word embeddings of assessment question content for reliable early prediction of student performance.Demonstrates the efficacy of distributed word embeddings of assessment questions when used by early prediction models compared to models that use either no assessment information or discrete representations of the questions.Demonstrates the efficacy and generalisability of word embeddings of assessment questions for predicting the performance of both new students on existing questions and existing students on new questions.Implications for practice and/or policy Word embeddings of assessment questions can enhance early prediction models of student knowledge, which can drive adaptive feedback to students who interact with game‐based learning environments.Practitioners should determine if new assessment questions will be developed for their game‐based learning environment, and if so, consider using our student modelling framework that incorporates early prediction models pretrained with existing student responses to previous assessment questions and is generalisable to the new assessment questions by leveraging distributed word embedding techniques.Researchers should consider the most appropriate way to encode the assessment questions in ways that early prediction models are able to infer relationships between the questions and gameplay behaviour to make accurate predictions of student competencies. [ABSTRACT FROM AUTHOR]

This exploratory paper highlights how problem‐based learning (PBL) provided the pedagogical framework used to design and interpret learning analytics from Crystal Island: EcoJourneys, a collaborative game‐based learning environment centred on supporting science inquiry. In Crystal Island: EcoJourneys, students work in teams of four, investigate the problem individually and then utilize a brainstorming board, an in‐game PBL whiteboard that structured the collaborative inquiry process. The paper addresses a central question: how can PBL support the interpretation of the observed patterns in individual actions and collaborative interactions in the collaborative game‐based learning environment? Drawing on a mixed method approach, we first analyzed students' pre‐ and post‐test results to determine if there were learning gains. We then used principal component analysis (PCA) to describe the patterns in game interaction data and clustered students based on the PCA. Based on the pre‐ and post‐test results and PCA clusters, we used interaction analysis to understand how collaborative interactions unfolded across selected groups. Results showed that students learned the targeted content after engaging with the game‐based learning environment. Clusters based on the PCA revealed four main ways of engaging in the game‐based learning environment: students engaged in low to moderate self‐directed actions with (1) high and (2) moderate collaborative sense‐making actions, (3) low self‐directed with low collaborative sense‐making actions and (4) high self‐directed actions with low collaborative sense‐making actions. Qualitative interaction analysis revealed that a key difference among four groups in each cluster was the nature of verbal student discourse: students in the low to moderate self‐directed and high collaborative sense‐making cluster actively initiated discussions and integrated information they learned to the problem, whereas students in the other clusters required more support. These findings have implications for designing adaptive support that responds to students' interactions with in‐game activities. Practitioner notesWhat is already known about this topicLearning analytic methods have been effective for understanding student learning interactions for the purposes of assessment, profiling student behaviour and the effectiveness of interventions.However, the interpretation of analytics from these diverse data sets are not always grounded in theory and challenges of interpreting student data are further compounded in collaborative inquiry settings, where students work in groups to solve a problem.What this paper addsProblem‐based learning as a pedagogical framework allowed for the design to focus on individual and collaborative actions in a game‐based learning environment and, in turn, informed the interpretation of game‐based analytics as it relates to student's self‐directed learning in their individual investigations and collaborative inquiry discussions.The combination of principal component analysis and qualitative interaction analysis was critical in understanding the nuances of student collaborative inquiry.Implications for practice and/or policySelf‐directed actions in individual investigations are critical steps to collaborative inquiry. However, students may need to be encouraged to engage in these actions.Clustering student data can inform which scaffolds can be delivered to support both self‐directed learning and collaborative inquiry interactions.All students can engage in knowledge‐integration discourse, but some students may need more direct support from teachers to achieve this. [ABSTRACT FROM AUTHOR]

This study examined 57 learners' emotions (i.e., joy, anger, confusion, frustration) as they engaged with scientific content while learning about microbiology with Crystal Island, a game-based learning environment (GBLE). Measures of learners' prior knowledge, in-game text comprehension, facial expressions of emotion, and posttest reading comprehension were collected to examine the relationship between emotions and single- and multiple-text comprehension. Analyses found that both discrete and non-discrete emotions were expressed during reading and answering in-game assessments of single-text comprehension. Learners expressed greater joy during reading and greater expressions of anger, confusion, and frustration during in-game assessments. Further results found that learners who expressed a high number of different emotions throughout reading and completing in-game assessments tended to have lower in-game comprehension scores whereas a higher number of different expressed emotions while completing in-game assessments was associated with greater posttest comprehension. Finally, while increased prior knowledge was associated with higher single- and multiple-text comprehension, there was no interaction between prior knowledge and emotions on multiple-text comprehension. Overall, this study found that (1) learners often express more than one emotion during GBLE activities, (2) emotions expressed while learning with a GBLE shift across different activities, and (3) emotions are related to demonstrated comprehension, but the type of activity influences this relationship. Results from this study provide implications for how emotions can be examined as learners engage in GBLE activities as well as the design of GBLEs to support learners' emotions accounting for different activity demands to increase comprehension of single and multiple texts. [ABSTRACT FROM AUTHOR]

This study examined students' genetics learning in a game-based environment by exploring the connections between the expectancy-value theory of achievement motivation and flow theory. A total of 394 secondary school students were recruited and learned genetics concepts through interacting with a game-based learning environment. We measured their science self-efficacy, science outcome-expectancy beliefs, flow experience, feelings of frustration, and conceptual understanding before and after playing the game, as well as their game satisfaction. Mixed-model ANOVA, correlation tests, and path analysis were run to answer our research questions. Based on the results, we found that the game had a significant impact on students' conceptual understanding of genetics. We also found an acceptable statistical model of the integration between the two theories. Flow experience and in-game performance significantly impacted students' posttest scores. Moreover, science outcome-expectancy belief was found to be a significant predictor of students' flow experiences. In contrast, science self-efficacy and pretest scores were found to be the most significant factors influencing the feeling of frustration during the game. The results have practical implications with regard to the positive role that an adaptive game-based genetics learning environment might play in the science classroom. Findings also underscore the role the teacher should play in establishing productive outcome expectations for students prior to and during gameplay. [ABSTRACT FROM AUTHOR]

Game-based learning environments are designed to provide effective and engaging learning experiences for students. Predictive student models use trace data extracted from students' in-game learning behaviors to unobtrusively generate early assessments of student knowledge and skills, equipping game-based learning environments with the capacity to anticipate student outcomes and proactively deliver adaptive scaffolding or notify instructors. Reflection is a key component of self-regulated learning, and it is critical in effective learning. However, there is currently limited work exploring the utility of reflection for inducing accurate predictive student models. This article presents a predictive student modeling framework that leverages natural language responses to in-game reflection prompts to predict student learning outcomes in a game-based learning environment for middle school microbiology, CRYSTAL ISLAND. With data from a pair of classroom studies involving 118 middle school students, we investigate the accuracy of early prediction models that utilize features extracted from student trace data combined with word embedding-based representations (i.e., GloVe, ELMo) of student reflection responses. We evaluate the accuracy of the predictive models over time using data from incremental segments of each student's interaction with the game-based learning environment, and we compare against models that omit student reflection features. Results reveal that models encoding students' natural language reflections with ELMo word embeddings yield significantly improved accuracy compared to other representations, with the greatest accuracy demonstrated by an ensemble of predictive models. We discuss the implications of these results for the design of game-based learning environments. [ABSTRACT FROM AUTHOR]

Collaborative inquiry learning affords educators a context within which to support understanding of scientific practices, disciplinary core ideas, and crosscutting concepts. One approach to supporting collaborative science inquiry is through problem‐based learning (PBL). However, there are two key challenges in scaffolding collaborative inquiry learning in technology rich environments. First, it is unclear how we might understand the impact of scaffolds that address multiple functions (e.g., to support inquiry and argumentation). Second, scaffolds take different forms, further complicating how to coordinate the forms and functions of scaffolds to support effective collaborative inquiry. To address these issues, we identify two functions that needed to be scaffolded, the PBL inquiry cycle and accountable talk. We then designed predefined hard scaffolds and just‐in‐time soft scaffolds that target the regulation of collaborative inquiry processes and accountable talk. Drawing on a mixed method approach, we examine how middle school students from a rural school engaged with Crystal Island: EcoJourneys for two weeks (N=45). Findings indicate that hard scaffolds targeting the PBL inquiry process and soft scaffolds that targeted accountable talk fostered engagement in these processes. Although the one‐to‐one mapping between form and function generated positive results, additional soft scaffolds were also needed for effective engagement in collaborative inquiry and that these soft scaffolds were often contingent on hard scaffolds. Our findings have implications for how we might design the form of scaffolds across multiple functions in game‐based learning environments. [ABSTRACT FROM AUTHOR]

A distinctive feature of game‐based learning environments is their capacity to create learning experiences that are both effective and engaging. Recent advances in sensor‐based technologies such as facial expression analysis and gaze tracking have introduced the opportunity to leverage multimodal data streams for learning analytics. Learning analytics informed by multimodal data captured during students' interactions with game‐based learning environments hold significant promise for developing a deeper understanding of game‐based learning, designing game‐based learning environments to detect maladaptive behaviors and informing adaptive scaffolding to support individualized learning. This paper introduces a multimodal learning analytics approach that incorporates student gameplay, eye tracking and facial expression data to predict student posttest performance and interest after interacting with a game‐based learning environment, Crystal Island. We investigated the degree to which separate and combined modalities (ie, gameplay, facial expressions of emotions and eye gaze) captured from students (n = 65) were predictive of student posttest performance and interest after interacting with Crystal Island. Results indicate that when predicting student posttest performance and interest, models utilizing multimodal data either perform equally well or outperform models utilizing unimodal data. We discuss the synergistic effects of combining modalities for predicting both student interest and posttest performance. The findings suggest that multimodal learning analytics can accurately predict students' posttest performance and interest during game‐based learning and hold significant potential for guiding real‐time adaptive scaffolding. [ABSTRACT FROM AUTHOR]

The goal of this study was to examine college students' (n = 61) contextualized emotions during in-game actions while playing Crystal Island, a game-based learning environment where students are tasked with solving the mystery of what illness impacted all island inhabitants. We examined emotions during in-game actions: during book reading, after scanning food items for the transmission source, and after submitting a final diagnosis. We dichotomized each activity's feedback into a positive or negative outcome: a relevant or irrelevant book for solving the mystery, testing food items that generate a positive or negative result, or submitting a correct or incorrect final diagnosis. Results revealed that expressing joy while reading a relevant book and expressing confusion after a positive scan significantly positively predicted overall game score, which we used as a proxy for problem-solving performance. Implications include understanding different levels of emotions students express during learning with all advanced learning technologies. [ABSTRACT FROM AUTHOR]

Back in the 1990s, we started work on pedagogical agents -- a novel paradigm for interactive learning. Pedagogical agents are autonomous characters that inhabit learning environments to engage with learners in rich, face-toface interactions. Building on this work, in 2000, together with our colleague Jeff Rickel, we published an article on pedagogical agents (Johnson, Rickel, and Lester 2000) that surveyed and discussed the potential of this new paradigm. We made the case that pedagogical agents that interact with learners in natural, lifelike ways can help learning environments achieve improved learning outcomes. This article has been widely cited, and was a winner of the 2017 IFAAMAS Award for Influential Papers in Autonomous Agents and Multiagent Systems. On the occasion of receiving the IFAAMAS award, and after 20 years of work on pedagogical agents, we take another look at the future of the field. We start by revisiting the predictions we made in 2000 for pedagogical agents, and examine which predictions panned out. Then, informed by what we have learned since then, we take another look at emerging trends and reexamine the future of pedagogical agents. Advances in natural language dialogue, affective computing, machine learning, virtual environments, and robotics are making possible even more lifelike and effective pedagogical agents, with potentially profound effects on the way people learn. [ABSTRACT FROM AUTHOR]

The aim of this study was to explore howNarrative Theatre,a narrative-centered digital learning environment, supported the writing processes of 3 struggling adolescent male writers. We utilized a multicase study approach to capture 3 sixth-grade participants’ experiences with the digital learning environment before, during, and after writing. The case studies provided detailed portraits of the writers as well as insights into their digital writing processes related to student interest, student ability, and value for writing. The across-case analysis revealed 3 themes (i.e., choice, scaffolding, and self-efficacy) that illustrated how the digital learning environment contributed to the students’ writing experiences. Future research and development will focus on the addition of text animation for student products and the degree to which this feature further contributes to engagement and proficiency with struggling writers. [ABSTRACT FROM PUBLISHER]

Intelligent game-based learning environments integrate commercial game technologies with Al methods from intelligent tutoring systems and intelligent narrative technologies. This article introduces the Crystal Island intelligent game-based learning environment, which has been under development in the authors' laboratory for the past seven years. After presenting Crystal Island, the principal technical problems of intelligent game-based learning environments are discussed: narrative-centered tutorial planning, student affect recognition, student knowledge modeling, and student goal recognition. Solutions to these problems are illustrated with research conducted with the Crystal Island learning environment. [ABSTRACT FROM AUTHOR]

Self-regulated learning behaviors such as goal setting and monitoring have been found to be crucial to students’ success in computer-based learning environments. Consequently, understanding students’ self-regulated learning behavior has been the subject of increasing attention. Unfortunately, monitoring these behaviors in real-time has proven challenging. This paper presents an initial investigation into self-regulated learning in a game-based learning environment. Evidence of goal setting and monitoring behaviors is examined through students’ text-based responses to update their ‘status’ in an in-game social network. Students are then classified into SRL-use categories. This article describes the methodology used to classify students and discusses analyses demonstrating the learning and gameplay behaviors across students in different SRL-use categories. Finally, machine learning models capable of predicting these classes early in students’ interaction are presented. [ABSTRACT FROM AUTHOR]

Understanding emotions in technology-based learning environments (TBLEs) has become a paramount goal across different research communities, but to date, these have operated in relative isolation. Based on control-value theory (Pekrun, 2006), we reviewed 186 studies examining emotions in TBLEs that were published between 1965 and 2018. We extracted effect sizes quantifying relations between emotions (enjoyment, curiosity/interest, anxiety, anger/frustration, confusion, boredom) and their antecedents (control-value appraisals, prior knowledge, gender, TBLE characteristics) and outcomes (engagement, learning strategies, achievement). Mean effects largely supported hypotheses (e.g., positive relations between enjoyment and appraisals, achievement, and cognitive support) and remained relatively stable across moderators. These findings imply that levels of emotions differ across TBLEs, but that their functional relations with appraisals and learning are equivalent across environments. Implications for research and designing emotionally sound TBLEs are discussed. • We reviewed 186 studies on emotions in technology-based learning. • Correlations were consistent with hypotheses derived from control-value theory. • Enjoyment correlated positively with control, cognitive support, and achievement. • Anxiety correlated negatively with these variables as well as learning strategies. • Effects were largely unaffected by moderators such as type of learning environment. [ABSTRACT FROM AUTHOR]