Search

Studies of educational games often treat them as "black boxes" (Black and Wiliam in Phi Delta Kappan 80: 139-48, 1998; Buckley et al. in Int J LearnTechnol 5:166-190, 2010; Buckley et al. in J Sci Educ Technol 13: 23-41, 2010) and measure their effectiveness by exposing a treatment group of students to the game and comparing their performance on an external assessment to that of a control group taught the same material by some other method. This precludes the possibility of monitoring, evaluating, and reacting to the actions of individual students as they progress through the game. To do that, however, one must know what to look for because superficial measures of success are unlikely to identify unproductive behaviors such as "gaming the system." (Baker in Philipp Comput J, 2011; Downs et al. in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, USA, 2010) The research reported here advances the ultimate goal of creating educational games that can provide real time, meaningful feedback on the progress of their users, enabling teachers or the game itself to intervene in a timely manner. We present the results of an in-depth analysis of students' actions in "Geniventure," an interactive digital game designed to teach genetics to middle and high school students. "Geniventure" offers a sequence of challenges of increasing difficulty and records students' actions as they progress. We analyzed the resulting log files, taking into account not only whether a student achieved a certain goal, but also the quality of the student's performance on each attempt. Using this information, we quantified students' performance and correlated it to their learning gain as estimated by scores on identical multiple-choice tests administered before and after exposure to "Geniventure." This analysis was performed in classes taught by teachers who had participated in professional development as part of a research project. A two-tailed paired-sample t-test of mean pre-test and post-test scores in these classes indicates a significant positive difference with a large effect size. Multivariate regression analysis of log data finds no correlation between students' post-test scores and their performance on "practice" challenges that invite experimentation, but a highly significant positive correlation with performance on "assessment" challenges, presented immediately following the practice challenges, that required students to invoke relevant mental models. We repeated this analysis with similar results using a second group of classes led by teachers who implemented "Geniventure" on their own after the conclusion of, and with no support from, the research project.

The COVID-19 pandemic produced a dramatic nationwide shift in K-12 education from in-person classroom learning to remote online learning. This shift left teachers and parents facing the challenge of finding engaging online resources to motivate students to become deeply involved in science learning. The pandemic also left educators and researchers, whose work focuses on providing students with experiential learning opportunities in the sciences, with the challenge of adapting to virtual and remote models to continue engaging students in STEM learning activities. In this article we describe: 1) the Health Quest project, which centers on the development of technology-rich learning resources to promote middle grade students' interest in health science careers, with a focus on girls and underrepresented racial and ethnic minorities; and 2) how the project has responded to the challenges presented by the COVID-19 pandemic. In Health Quest, through engaging narrative-based learning scenarios, students work with virtual characters to experience health science careers from multiple perspectives. Although originally envisioned for in-person classroom learning, we discuss how the team is adapting the Health Quest Career Adventure Game to remote learning, including highlighting the role science plays in addressing public health outbreaks. We describe new gameplay features that have been added to support career modeling and how we have adapted the core technology underpinning Health Quest to support broad dissemination to meet the project's broader goal of increasing adolescents' interest in and self-efficacy for pursuing health science careers. We conclude with a discussion of how our evaluation strategies have changed from in-person focus groups and testing to an online data collection model and lessons learned.

Pedagogical agents have been designed to support the significant challenges that learners face when self-regulating in advanced learning environments. Evidence suggests differences in learners' prior skills and abilities, in conjunction with excessive didactic support, can cause overreliance on these external aids, which in turn prevents deeper learning, and pedagogical agents can provide tailored scaffolding to accommodate learners' individual needs. However, there is less evidence about the impact of abstract scaffolding, such as the sharing of non-verbal metacognitive information via a pedagogical agent's facial expressions, on self-regulated learning. To assess factors in the passing of non-verbal metacognitive information via pedagogical agents in a multimedia learning environment, we used growth modeling with self-reports, eye-tracking, and log-file data collected from fifty (n = 50) undergraduates at a large North American university as they learned about human body systems while using MetaTutor-IVH, a multimedia learning environment with a pedagogical agent. We controlled for participant characteristics (perceived utility of emotions for self- and other-centered positive and negative emotions) and characteristics of the metacognitive monitoring information provided by a pedagogical agent (expression type and expression congruency) to assess factors in non-verbally communicating metacognitive information. Results suggest that learners attend to pedagogical agents less over time, but this rate of change is weaker when an agent is providing an expression that is congruent with the ground truth of the environment. Further, only the perceived information utility of other-centered negative emotions has a significant effect on this duration, suggesting learners are driven to consult pedagogical agents to avoid embarrassment or shame. We discuss design implications of these findings for technology-based learning environments.

With accelerating advances in artificial intelligence, it is clear that introducing K-12 students to AI is essential for preparation to interact with and potentially develop AI technologies. To succeed as the workers, creators, and innovators of the future, we argue students should encounter core concepts of AI as early as elementary school. However, building a curriculum that introduces AI content to K-12 students presents significant challenges, such as connecting to prior knowledge, developing curricula that are meaningful for students, and creating content that teachers feel confident to teach. To lay the groundwork for elementary AI education, we investigated the everyday experiences and ideas of students in grades 4 and 5 (ages 9 to 11) about AI to inform possible entry points for learning. This yielded themes around student conceptions, examples, and ethics of AI. For each theme, we juxtapose the student ideas with the teachers' reflections on those ideas as frames of reference to consider in co-designing curricular approaches.

This design case chronicles the efforts of an interdisciplinary team of researchers as they collaborated with middle grades science teachers and students to build and refine an epidemic disease curriculum module. The initial five-day design was delivered in five science classrooms at three nearby schools where researcher classroom observations and teacher feedback drove iterative refinements of the module's materials. The final design of this module consisted of four instructional days of modeling and simulation activities that integrate computational thinking practices, computer science concepts, and life sciences content. The paper aims to illustrate the design motivations to address contextual constraints such as tight curricular schedules and varied levels of exposure to programming for both teachers and students. The instructional materials presented in this design case were the result of a three-year long research-practice partnership with science teachers at nearby middle schools.

Reflection is critical for adolescents' problem solving and learning in game-based learning environments (GBLEs). Yet challenges exist in the literature because most studies lack a theoretical perspective and clear operational definition to inform how and when reflection should be scaffolded during game-based learning. In this paper, we address these issues by studying the quantity and quality of 120 adolescents' written reflections and their relation to their learning and problem solving with Crystal Island, a GBLE. Specifically, we: (1) define reflection and how it relates to skill and knowledge acquisition; (2) review studies examining reflection and its relation to problem solving and learning with emerging technologies; and (3) provide direction for building reflection prompts into GBLEs that are aligned with the learning goals built into the learning session (e.g., learn about microbiology versus successfully solve a problem) to maximize adolescents' reflection, learning, and performance. Overall, our findings emphasize how important it is to examine not only the quantity of reflection but also the depth of written reflection as it relates to specific learning goals. We discuss the implications of using game-learning analytics to guide instructional decision making in the classroom.

Game-based learning environments hold significant promise for facilitating learning experiences that are both effective and engaging. To support individualised learning and support proactive scaffolding when students are struggling, game-based learning environments should be able to accurately predict student knowledge at early points in students' gameplay. Student knowledge is traditionally assessed prior to and after each student interacts with the learning environment with conventional methods, such as multiple choice content knowledge assessments. While previous student modelling approaches have leveraged machine learning to automatically infer students' knowledge, there is limited work that incorporates the fine-grained content from each question in these types of tests into student models that predict student performance at early junctures in gameplay episodes. This work investigates a predictive student modelling approach that leverages the natural language text of the post-gameplay content knowledge questions and the text of the possible answer choices for early prediction of fine-grained individual student performance in game-based learning environments. With data from a study involving 66 undergraduate students from a large public university interacting with a game-based learning environment for microbiology, Crystal Island, we investigate the accuracy and early prediction capacity of student models that use a combination of gameplay features extracted from student log files as well as distributed representations of post-test content assessment questions. The results demonstrate that by incorporating knowledge about assessment questions, early prediction models are able to outperform competing baselines that only use student game trace data with no question-related information. Furthermore, this approach achieves high generalisation, including predicting the performance of students on unseen questions.

The increasing interest in computer science (CS) and CS-integrated STEM teaching and learning has created a need for assessment instruments that can be used to evaluate the efficacy of innovative instructional approaches to K-12 CS education. However, there is a lack of validated assessment tools aligned to core CS concepts for younger students. This paper reports on the development and validation of a CS concept assessment for middle grades (ages 11-13) students. A total of 27 multiple-choice items were developed, guided by focal knowledge, skills and abilities associated with the concepts of variables, loops, conditionals, and algorithms. These items were administered to 457 middle grades students. The items were presented in form of block-based programming code and administered in a week-long computational modeling intervention. A combination of classical test theory and item response theory approaches were used to validate the assessment. Based on results, it was found that only 24 items are considered valid and reliable items to measure CS conceptual understanding. The results also suggested that the assessment can be used as a pre and post-test to investigate students' learning gains. This work fills an important gap by providing a key resource for researchers and practitioners interested in assessing middle grades student CS conceptual understanding.

This exploratory paper highlights how problem-based learning (PBL) provided the pedagogical framework used to design and interpret learning analytics from "Crystal Island: EcoJourneys," a collaborative game-based learning environment centred on supporting science inquiry. In "Crystal Island: EcoJourneys," students work in teams of four, investigate the problem individually and then utilize a brainstorming board, an in-game PBL whiteboard that structured the collaborative inquiry process. The paper addresses a central question: how can PBL support the interpretation of the observed patterns in individual actions and collaborative interactions in the collaborative game-based learning environment? Drawing on a mixed method approach, we first analyzed students' pre- and post-test results to determine if there were learning gains. We then used principal component analysis (PCA) to describe the patterns in game interaction data and clustered students based on the PCA. Based on the pre- and post-test results and PCA clusters, we used interaction analysis to understand how collaborative interactions unfolded across selected groups. Results showed that students learned the targeted content after engaging with the game-based learning environment. Clusters based on the PCA revealed four main ways of engaging in the game-based learning environment: students engaged in low to moderate self-directed actions with: (1) high and (2) moderate collaborative sense-making actions; (3) low self-directed with low collaborative sense-making actions; and (4) high self-directed actions with low collaborative sense-making actions. Qualitative interaction analysis revealed that a key difference among four groups in each cluster was the nature of verbal student discourse: students in the low to moderate self-directed and high collaborative sense-making cluster actively initiated discussions and integrated information they learned to the problem, whereas students in the other clusters required more support. These findings have implications for designing adaptive support that responds to students' interactions with in-game activities.

This study examined 57 learners' emotions (i.e., joy, anger, confusion, frustration) as they engaged with scientific content while learning about microbiology with Crystal Island, a game-based learning environment (GBLE). Measures of learners' prior knowledge, in-game text comprehension, facial expressions of emotion, and posttest reading comprehension were collected to examine the relationship between emotions and single- and multiple-text comprehension. Analyses found that both discrete and non-discrete emotions were expressed during reading and answering in-game assessments of single-text comprehension. Learners expressed greater joy during reading and greater expressions of anger, confusion, and frustration during in-game assessments. Further results found that learners who expressed a high number of different emotions throughout reading and completing in-game assessments tended to have lower in-game comprehension scores whereas a higher number of different expressed emotions while completing in-game assessments was associated with greater posttest comprehension. Finally, while increased prior knowledge was associated with higher single- and multiple-text comprehension, there was no interaction between prior knowledge and emotions on multiple-text comprehension. Overall, this study found that (1) learners often express more than one emotion during GBLE activities, (2) emotions expressed while learning with a GBLE shift across different activities, and (3) emotions are related to demonstrated comprehension, but the type of activity influences this relationship. Results from this study provide implications for how emotions can be examined as learners engage in GBLE activities as well as the design of GBLEs to support learners' emotions accounting for different activity demands to increase comprehension of single and multiple texts.

This paper reports findings from the efforts of a university-based research team as they worked with middle school educators within formal school structures to infuse computer science principles and computational thinking practices. Despite the need to integrate these skills within regular classroom practices to allow all students the opportunity to learn these essential 21st Century skills, prior practice has been to offer these learning experiences outside of mainstream curricula where only a subset of students has access. We have sought to leverage elements of the research-practice partnership framework to achieve our project objectives of integrating computer science and computational thinking within middle science classrooms. Utilizing a qualitative approach to inquiry, we present narratives from three case schools, report on themes across work sites, and share recommendations to guide other practitioners and researchers who are looking to engage in technology-related initiatives to impact the lives of middle grades students.

PurposeAccidents and unintentional injuries account for the greatest number of adolescent deaths, often involving use of alcohol and other substances. This article describes the iterative design and development of Interactive Narrative System for Patient-Individualized Reflective Exploration (INSPIRE), a narrative-centered behavior change environment for adolescents focused on reducing alcohol use. INSPIRE is designed to serve as an extension to clinical preventive care, engaging adolescents in a theoretically grounded intervention for health behavior change by leveraging 3D game engine and interactive narrative technologies.MethodsAdolescents were engaged in all aspects of the iterative, multiyear development process of INSPIRE through over 20 focus groups and iterative pilot testing involving more than 145 adolescents. Qualitative findings from focus groups are reported, as well as quantitative findings from small-scale pilot sessions investigating adolescent engagement with a prototype version of INSPIRE using a combination of questionnaire and interaction trace log data.ResultsAdolescents reported that they found INSPIRE to be engaging, believable, and relevant to their lives. The majority of participants indicated that the narrative's protagonist character was like them (84%) and that the narrative featured virtual characters that they could relate to (79%). In the interactive narrative, the goals most frequently chosen by adolescents were "stay in control" (60%) and "do not get in trouble" (55%).ConclusionsWith a strong theoretical framework (social-cognitive behavior change theory) and technology advances (narrative-centered learning environments), the field is well positioned to design health behavior change systems that can realize significant impacts on behavior change for adolescent preventive health.