A Randomized Controlled Trial of a Technology- and Game-Based Approach to Social-Emotional Learning

Background/Context: Research has shown that students with strong social-emotional skills demonstrate greater motivation to learn, have more positive attitudes toward school, participate more actively in activities, and show better academic performance than students with social-emotional challenges (Gresham et al., 2020; Kostelnik et al. 2015). Despite the social availability of proven social emotional learning (SEL) interventions, many students who stand to benefit from such programs currently do not receive the help they need due to logistical barriers to implementation that limit the scalability of traditional in-person SEL instruction (Agron, et al., 2010; Yoder, et al., 2020). Technology-based interventions provide the opportunity to reduce these barriers to treatment and effectively deliver evidence-based social-emotional skills development content with fidelity (Thomas, 2014). The "Adventures Aboard the S.S. GRIN" ("Adventures") was specifically designed to translate the content and cognitive-behavioral strategies of an established, evidence-based in-person SEL program into an online game-based environment (DeRosier, 2004). Purpose/Objective/Research Question: The purpose of the study is to determine the effectiveness of Adventures, a technology- and game-based SEL intervention, in authentic educational environments for third-grade students. The randomized controlled trial study addresses the following research questions: (1) Is the "Adventures" program treatment more effective than the control for improving students' social-emotional skills? (2) Is the "Adventures" program treatment more effective than the control for improving students' social-emotional skills for students with social-emotional challenges identified at the beginning of the study? (3) Is the "Adventures" program treatment more effective than the control for improving students' social-emotional skills when implemented with higher fidelity? Setting: Thirty-seven elementary schools in four California public school districts participated in the study. The study initially began with in-person school implementation prior to COVID-19 stay-at-home orders (Spring 2020), after which the study adapted to distance learning for the remainder of the 2019-2020 school year and various hybrid models of instruction the following year (2020-2021). Population/Participants/Subjects: The study included 1,645 third-grade students from 88 classrooms in California. Many of the schools are Title I schools (65%). One third of the students were Latinx and 50% qualified for free or reduce-price lunch. Based on teacher rating on an SEL screening instrument, 342 were identified as having social emotional challenges. Intervention/Program/Practice: "Adventures" provides nine weeks of instruction through instructional episodes that are set within an appealing story narrative: the player is a new recruit on the sailing ship who joins the crew and travels around an island, interacting with a host of characters and engaging in social problem solving to address plot conflicts and save friendship on the island. The social problems encountered in the game are true-to-life (e.g., entering group social situations, staying calm in emotionally charged situations, expressing emotions positively, cooperating and compromising with peers) to increase ease of skill transfer to real life. "Adventures" is delivered once per week for 30-45 minutes each. Each episode in "Adventures" focuses on using respect, social planning, responsibility, communication, assumptions, perspective taking, building friendships, cooperation, and emotion regulation. Research Design: The study utilized a multi-site cluster randomized experimental design, which randomly assigned third-grade classrooms to one of two conditions--a treatment condition that used the "Adventures" program described above, or a control condition that used the school's business-as-usual program. Forty-four classrooms (793= students) were randomly assigned to treatment and 44 classrooms (n=852 students) were randomly assigned to control. Data Collection and Analysis: The "Devereux Strengths Assessment-Mini" (Naglieri et al., 2011) is an eight-item behavior rating scale that teachers completed to determine which students were "at-risk" or considered to have social emotional challenges. "Behavioral and Emotional Rating Scale Second Edition" (Epstein, 2004) and "Social Skills and Behavioral Index" (DeRosier, et al., 2011) are teacher rating scales that were collected from teachers on a subsample of students. SELweb (McKown, et al., 2016) and ZooU (DeRosier, et al., 2012) are online performance-based assessments of social-emotional skills. Teachers provided links to the SELweb and ZooU platforms and students navigated through the assessments independently with teacher support and/or supervision. To address whether the "Adventures" program treatment is more effective than the control for improving students' social-emotional skills (RQ1), we used a series of two-level hierarchical liner models, where the dependent variable in each model is a separate SEL outcome. In each model, the effect of treatment on an outcome measure was estimated at the classroom level, controlling for student pretests, baseline student covariates. The stepwise multiple hypothesis testing procedure was applied to account for conducting multiple comparisons (Benjamini & Hochberg, 1995). Using a similar two-level model described above, we analyzed whether the "Adventures" program treatment is more effective than the control for improving students' social-emotional skills when treating students with prior social-emotional challenges (RQ2) and when the program is implemented with higher fidelity (RQ3). Findings/Results: In addition to the successful implementation, the results indicate that the program has a significant and positive impact on students' social-emotional skills. The effect sizes of the treatment impact as measured by ZooU and SELweb are 0.41 and 0.60, respectively (based on Hedges' g), which is medium to large for educational interventions for a sample of this size (Kraft, 2020). The program benefits all students, including students with social-emotional challenges, demonstrating that the program is an appropriated resource for all learners. Conclusions: The combination of increased psychosocial stress and limited access to psychosocial supports places students at particularly heightened risk for developing social-emotional problems and behavioral delays. Scalable digital approaches to evidence-based SEL can meaningfully address the social-emotional needs of many students (Lattie et al., 2019; Wozney et al., 2018). "Adventures" has offered part of a solution to this critical need. This study provides an example of using effective design strategies to empower practical and theoretically sound SEL programs and to create accessible SEL learning experiences. The implementation of "Adventures" in in-school and distance-learning environments contributes to a growing effort to support all students' SEL, especially those experiencing social-emotional challenges who stand to benefit the most from such interventions.