Search

This is an extended conference proceedings report [Long Paper] based on a shorter contributed report at the same conference. A variety of computerized learning platforms exist. In mathematics, most include sets of problems to complete. Feedback to users ranges from a single word like "Correct!" to offers of hints and partially- to fully-worked examples. Behind-the-scenes design of such systems also varies--from static dictionaries of problems to responsive programming that adapts assignments to users' demonstrated skills within the computerized environment. This report presents background on digital learning contexts and early results of a mixed-methods study that included a cluster randomized controlled trial design. The study was in community college algebra classes where the intervention was a particular type of web-based activity and testing system. [This paper was published in: Weinberg, A., Rasmussen, C., Rabin, J., Wawro, M., and Brown, S. (Eds.), "Proceedings of the 20th Annual Conference on Research in Undergraduate Mathematics Education," p638-645. San Diego, CA (2017).]

A web-based activity and testing system (WATS) has features such as adaptive problem sets, videos, and data-driven tools for instructors to monitor and scaffold student learning. Central to WATS adoption and use are questions about the implementation process: What constitutes "good" implementation and how far from "good" is "good enough"? This proceedings paper reports on and illustrates work to provide structure for such examination. The context is a study about implementation that is part of a state-wide randomized controlled trial examining student learning in community college algebra when a particular WATS suite of tools is used. Discussion questions for conference participants dug into the distinctions among intended, enacted, and achieved curriculum and the processes surrounding these as well as the challenges and opportunities in researching fidelity of implementation in the community college context, particularly the role of instructional practice as a contextual component of the research. [This paper was published in the "Proceedings of the 19th Annual Conference on Research in Undergraduate Mathematics Education," T. Fukawa-Connelly, N. Infante, K. Keene, and M. Zandieh (Eds), 2016.]

As online learning platforms become more ubiquitous throughout various curricula, there is a growing need to evaluate the effectiveness of these platforms and the different methods used to structure online education and tutoring. Towards this endeavor, some platforms have performed randomized controlled experiments to compare different user experiences, curriculum structures, and tutoring strategies in order to ensure the effectiveness of their platform and personalize the education of the students using it. These experiments are typically analyzed on an individual basis in order to reveal insights on a specific aspect of students' online educational experience. In this work, the data from 50,752 instances of 30,408 students participating in 50 different experiments conducted at scale within the online learning platform ASSISTments were aggregated and analyzed for consistent trends across experiments. By combining common experimental conditions and normalizing the dependent measures between experiments, this work has identified multiple statistically significant insights on the impact of various skill mastery requirements, strategies for personalization, and methods for tutoring in an online setting. This work can help direct further experimentation and inform the design and improvement of new and existing online learning platforms. The anonymized data compiled for this work are hosted by the Open Science Foundation and can be found at https://osf.io/59shv/. [This paper was published in: "Proceedings of the 15th International Conference on Educational Data Mining," edited by A. Mitrovic and N. Bosch, International Educational Data Mining Society, 2022, pp. 27-38.]

This paper presents evidence confirming the validity of the RD design in the Reading Recovery study by examining the ability of the RD design to replicate the 1st grade results observed in the original i3 RCT focused on short-term impacts. Over 1,800 schools participated in the RD study over all four cohort years. The RD design used cutoff-based assignment established by pre-intervention test scores on the Observation Survey of Early Literacy (OS; Clay, 2005). In order to examine the ability of the RD design to replicate the results observed in the RCT, we estimated impacts for both i3 and non-i3 implementing the RD design, and we compare these results to i3 RCT results across four cohorts. Using multilevel statistical models, the performance of students above and below the cutoff score was compared, with students nested within each participating school. The hierarchical linear model (HLM) used to estimate impacts included the centered pretest assignment variable as a covariate at the student level, a parameter for the discontinuity associated with assignment to RR, a random effect for overall school performance (i.e., a random school intercept), a random effect for the pretest slope (i.e., a random school slope), and a random effect for the impact of RR (i.e., a random treatment effect across schools). In accordance with What Works Clearinghouse (WWC) standards for RD studies (WWC, 2017; Schochet et al., 2010), model fit and potential misspecification was assessed graphically via scatterplots and spline curves and also by testing for an interaction between pretest scores and the treatment assignment variable. Assumptions of linearity in the RD analyses were further assessed by testing polynomial parameters and by imposing various restrictions on the bandwidth around the cutscore. More specifically, analyses were restricted to include only students whose pretest scores fell within ±1.0 or ±0.5 standard deviations of the cutscore. Lastly, robustness of the RD was assessed by arbitrarily shifting the cutscore up or down by 0.50 standard deviations to confirm the absence of a discontinuity where none would be expected. Intent-to-Treat (ITT) effect sizes were calculated by dividing raw impact estimates by the mid-year first-grade population standard deviation of OS scores. Standardized effect sizes based on the RD sample were compared to standardized effect sizes from the previously published RCT study involving i3 schools. The complier average causal effect at the cut-point (CACEC) was created utilizing the formula from Bloom (2009, p. 12, equation 12) for those students whose forcing variable score was within 0.25 standard deviations of the cutscore. Results showed that RD estimates for short-term impacts on OS scores in i3 schools ranged from 0.65 to 0.78 SDs, and from 0.81 to 0.84 SDs in non-i3 schools. These RD estimates are remarkably similar to the RCT estimates of short-term impacts in 1st grade and were highly consistent under numerous model robustness checks (WWC, 2017). The similarity of short-term impact estimates between the RCT and RD designs confirms the validity of the RD design used in this study of Reading Recovery.

As online learning platforms become more ubiquitous throughout various curricula, there is a growing need to evaluate the effectiveness of these platforms and the different methods used to structure online education and tutoring. Towards this endeavor, some platforms have performed randomized controlled experiments to compare different user experiences, curriculum structures, and tutoring strategies in order to ensure the effectiveness of their platform and personalize the education of the students using it. These experiments are typically analyzed on an individual basis in order to reveal insights on a specific aspect of students' online educational experience. In this work, the data from 50,752 instances of 30,408 students participating in 50 different experiments conducted at scale within the online learning platform ASSISTments were aggregated and analyzed for consistent trends across experiments. By combining common experimental conditions and normalizing the dependent measures between experiments, this work has identified multiple statistically significant insights on the impact of various skill mastery requirements, strategies for personalization, and methods for tutoring in an online setting. This work can help direct further experimentation and inform the design and improvement of new and existing online learning platforms. The anonymized data compiled for this work are hosted by the Open Science Foundation and can be found at https://osf.io/59shv/. [For the full proceedings, see ED623995. For the corresponding grantee submission, see ED623511.]

This paper reports a pilot study to determine the potential impact of an integrated STEM curriculum on Pre-K teachers' engineering content knowledge, self-efficacy and teaching practice. Using a randomized control trial design, researchers examined the impact of the curriculum in 17 Pre-K classrooms (8 intervention classrooms, 9 control classrooms) in central Massachusetts. Questionnaires measuring STEM and engineering content knowledge, self-efficacy and teaching practice were administered to participating Pre-K teachers (N = 42; 21 intervention, 21 control) in Fall 2017 and again in Spring 2018. Baseline analysis showed no significant differences in engineering and STEM content knowledge, pedagogical content knowledge, or teaching self-efficacy at the start of the pilot study between intervention and comparison classroom teachers. Fidelity of implementation was measured using an observation instrument developed by the project team based on a published implementation science framework. We hypothesized that teachers who implement the integrated STEM curriculum will have significantly higher engineering pedagogical content knowledge and self-efficacy than teachers in the comparison group. As well the teachers who implement the integrated STEM curriculum will show significant gains in their engineering pedagogical content knowledge and self-efficacy in teaching engineering and STEM as a result of their participation. [This paper was published in: "2018 IEEE Integrated STEM Conference (ISEC)" p234-237. ISBN 978-1-5386-3309-0. Institute of Electrical and Electronics Engineers.]

Randomized controlled trials in education are necessary to keep pace with the evidence-based practices demanded by schools and the nation. In this study, kindergarten students in a school district in Indiana were randomized on the class level: The experimental condition consisted of students utilizing a computer-adaptive reading program for 15 minutes per day, 5 days per week over the course of the 2017-2018 school year; the control condition consisted of students receiving traditional, teacher-directed literacy instruction for the same amount of time. Thus, overall exposure to literacy instruction was the same for both groups. At the end of the school year, students randomized to the experimental condition outperformed their control counterparts on two different literacy assessments, indicating that early literacy instruction in the format of computer-adaptive reading programs may be beneficial to kindergarten students. Experimental students also outperformed their control counterparts across demographics, demonstrating a benefit from the computer-adaptive program for all students. [This paper was published in: "Smart Education and e-Learning 2019. Smart Innovation, Systems and Technologies," edited by V. L. Uskov et al., Springer Nature, 2019, pp. 243-252.]

A variety of computerized interactive learning platforms exist. Most include instructional supports in the form of problem sets. Feedback to users ranges from a single word like "Correct!" to offers of hints and partially to fully worked examples. Behind-the-scenes design of such systems varies as well --from static dictionaries of problems to "intelligent" and responsive programming that adapts assignments to users' demonstrated skills, timing, and an array of other learning theory-informed data collection within the computerized environment. This short paper presents background on digital learning contexts and describes the lively conversation with attendees at the conference session. The topics were the research design and early results of a cluster-randomized controlled trial study in community college elementary algebra classes where the intervention was a particular type of web-based activity and testing system. [This paper was published in the "Proceedings of the 19th Annual Conference on Research in Undergraduate Mathematics Education," T. Fukawa-Connelly, N. Infante, K. Keene, and M. Zandieh (Eds), 2016.]

One of the objectives of this paper is to investigate the pedagogical impact of both the mobile testing system PeLe (Norway, HiST) and the enquiry-based learning approach on language skills development in the context of mobile-assisted learning. The research aims to work out a methodological framework of PeLe implementation into the language classroom through formative assessment, immediate feedback and interactive post-test activities. The framework was developed and pilot tested in a joint research project (MobiLL) by English as a foreign language (EFL) teachers at Lomonosov Moscow State University (LMSU, Russia) and University College HiST (Norway) during two semesters of the 2013-2014 academic year. Students enrolled in a preparatory English course at LMSU were randomly assigned to 2 experimental groups and 2 control groups. Students in the experimental groups took a series of PeLe supported grammar and vocabulary tests as volunteers using handheld devices. The control groups were tested by the traditional testing method--pen and paper. The analysis based on quantitative research data demonstrated that PeLe supported language classes resulted in language skill gains. Qualitative data analysis highlighted the positive effect of mobile formative assessment and of posttest activities on learner motivation and collaboration skills. This study suggests that the use of technology was effective in engaging students in enquiry-based tasks, to produce more output in the target language. [For full proceedings, see ED564162.]

Personalized learning considers that the causal effects of a studied learning intervention may differ for the individual student. Making the inference about causal effects of studies interventions is a central problem. In this paper we propose the Residual Counterfactual Networks (RCN) for answering counterfactual inference questions, such as "Would this particular student benefit more from the video hint or the text hint when the student cannot solve a problem?". The model learns a balancing representation of students by minimizing the distance between the distributions of the control and the treated populations, and then uses a residual block to estimate the individual treatment effect based on the representation of the student. We run experiments on semi-simulated datasets and real-world educational online experiment datasets to evaluate the efficacy of our model. The results show that our model matches or outperforms the state-of-the-art. [For the full proceedings, see ED596512.]

A variety of computerized interactive learning platforms exist. Most include instructional supports in the form of problem sets. Feedback to users ranges from a single word like "Correct!" to offers of hints and partially- to fully-worked examples. Behind-the-scenes design of systems varies as well--from static dictionaries of problems to "intelligent" and responsive programming that adapts assignments to users' demonstrated skills within the computerized environment. This report presents background on digital learning contexts and early results of a cluster-randomized controlled trial study in community college elementary algebra classes where the intervention was a particular type of web-based activity and testing system. [This paper was published in: Weinberg, A., Rasmussen, C., Rabin, J., Wawro, M., and Brown, S. (Eds.), "Proceedings of the 20th Annual Conference on Research in Undergraduate Mathematics Education," p638-645. San Diego, CA (2017).]

Much research has been done on the development of an intelligent tutoring system (ITS), and small empirical studies have demonstrated the effectiveness of ITS at promoting student learning. However, large-scale implementation of ITS in school settings has not been researched thoroughly. In this paper, we describe an ongoing randomized controlled trial (RCT) to evaluate the efficacy of a web-based tutoring system--the ASSISTments--as support for homework. The program is used in 46 middle schools in the state of Maine, to provide immediate feedback to students, and to provide reports to teachers to support homework review and instruction adaptation. We describe the challenges for the RCT, approaches used to understand implementation of the system, and findings on how the system is being used. [This paper was published in: Proceedings of ITS 2014, Honolulu, HI, USA, June 5-9, 2014. p561-566.]

Using an RD design provides statistically robust estimates while allowing researchers a different causal estimation tool to be used in educational environments where an RCT may not be feasible. Results from External Evaluation of the i3 Scale-Up of Reading Recovery show that impact estimates were remarkably similar between a randomized control trial (RCT) and a regression discontinuity (RD) design. This paper discusses RD design advantages including congruence with school practices for assigning interventions, alleviation of ethical concerns, flexible and adaptable design methodology, and the ability to monitor the effectiveness of various types of interventions in the classroom. This presents opportunities for schools, districts, and states to be more proactive in evaluating the causal impacts of interventions for low-achieving students.

While explanations may help people learn by providing information about why an answer is correct, many problems on online platforms lack high-quality explanations. This paper presents AXIS (Adaptive eXplanation Improvement System), a system for obtaining explanations. AXIS asks learners to generate, revise, and evaluate explanations as they solve a problem, and then uses machine learning to dynamically determine which explanation to present to a future learner, based on previous learners' collective input. Results from a case study deployment and a randomized experiment demonstrate that AXIS elicits and identifies explanations that learners find helpful. Providing explanations from AXIS also objectively enhanced learning, when compared to the default practice where learners solved problems and received answers without explanations. The rated quality and learning benefit of AXIS explanations did not differ from explanations generated by an experienced instructor.

The focus of the learning analytics community bridges the gap between controlled educational research and data mining. Online learning platforms can be used to conduct randomized controlled trials to assist in the development of interventions that increase learning gains; datasets from such research can act as a treasure trove for inquisitive data miners. The present work employs a data mining approach on randomized controlled trial data from ASSISTments, a popular online learning platform, to assess the benefits of incorporating additional student performance data when attempting to differentiate between two user groups. Through a resampling technique, we show that partial credit, defined as an algorithmic combination of binary correctness, hint usage, and attempt count, can benefit assessment and group differentiation. Partial credit reduces sample sizes required to reliably differentiate between groups that are known to differ by 58%, and reduces sample sizes required to reliably differentiate between less distinct groups by 9%. [This paper was published in: "LAK '16: Proceedings of the Sixth International Conference on Learning Analytics & Knowledge," (pp. 454-458). New York, NY: Association for Computing Machinery.]

In this paper, we present a dataset consisting of data generated from 22 previously and currently running randomized controlled experiments inside the ASSISTments online learning platform. This dataset provides data mining opportunities for researchers to analyze ASSISTments data in a convenient format across multiple experiments at the same time. The data preprocessing steps are explained in detail to inform researchers about how this dataset was generated. A list of column descriptions is provided to define the columns in the dataset and a set of summary statistics are presented to briefly describe the dataset.

Disparities in academic science performance remain for diverse learners, including students with learning disabilities and English learners. Educational technology offers the potential to support diverse learners by taking into account their specific needs and unique strengths. This paper reports on third-year results of a 3-year randomized controlled trial investigating the impact of using an evidence-based online science unit with eighth-grade students in inclusive classrooms. Results indicate that the unit significantly improved content-specific knowledge in physical science for general education students, students with learning disabilities, and English learners. This study provides support for utilizing educational technology to improve science learning outcomes for diverse learners, helping to make public education more equitable for all.

The Math for All (MFA) qualitative research team approached the randomized control trial (RCT) mixed methods investigation from a "pragmatic" perspective (Cresswell and Plano-Clark, 2011), that is, the belief that multiple realities can exist and that neither a qualitative nor quantitative approach is valued over the other. Further, the perspective presented in this paper reflects the belief that qualitative data, used in conjunction with quantitative data, create a more complete picture of the overall intervention: RCT quantitative data measure impact while qualitative data capture critical explanatory information about the impact. Thus, collaboration among researchers from "both sides of the aisle" is important for enriching the overall analysis. Specifically, qualitative researchers working in the context of RCT studies are able to identify qualitative research themes that inform the overall project and broaden the conceptual understanding of how and why the project works as it does. In this project, qualitative researchers are responsible for bringing a sense of the participants' meanings, an interpretive inquiry perspective, and an inductive data analysis, three of several research characteristics that Creswell notes as unique to qualitative research (2007). While a quantitative study can determine causal attribution, the addition of qualitative methods allows for a deeper analysis of why and how these causal relationships exist. Researchers in this study are seeking convergence through triangulation (Greene, Caracelli, & Graham, 1989), providing greater support for conclusions about the implementation and information that can guide current and future implementations. In the MFA study, qualitative researchers conducted in-depth interviews with teachers and principals in the treatment and control schools. Thus, the RCT mixed method research design increases the depth and the complexity of the study by introducing the perspective of the individuals directly involved in the program. The interviews give "voice" to the teachers and principals in the Math for All project and provide context information to enhance the understanding of the program's implementation and impact. Basically, the qualitative data play three critical roles in the MFA study: the interviews provide formative feedback to guide ongoing program implementation; they identify issues that might contribute to the identification of modifying variables and thus enhance the program's Theory of Change model; and they triangulate the quantitative data collected by the other researchers. The MFA qualitative data explore the teachers' perspectives on the professional development they have been receiving and the changes they may have subsequently made in their classrooms. For example, teachers described the impact of the MFA program on their understanding of how children learn mathematics and the influence this understanding had on their classroom practices. Open-ended questions allowed the researchers to explore how individual teachers were responding to the professional development, including constraints teachers might have experienced in implementing the MFA elements. By gathering explanatory data, researchers can provide schools and research communities with greater insight into critical elements for successful implementation.

In this work, we describe a new statistical method to improve the detection of treatment effects in interventions. We call our method TAME (Trained Across Multiple Experiments). TAME takes advantage of multiple experiments with similar designs to create a single model. We use this model to predict the outcome of the dependent variable in unseen experiments. We use the predictive accuracy of the model on the conditions of the experiment to determine if the treatment had a statistically significant effect. We validated the effectiveness of our model using a large-scale simulation study, where we showed that our model can detect treatment effects with 10% more statistical power than an ANOVA in certain settings. We also applied our model to real data collected from the ASSISTments online learning platform and showed that the treatment effects detected by our model were comparable to the effects detected by the ANOVA. [For the full proceedings, see ED596512.]

This study investigated the impact of a dynamic geometry (DG)-centered teacher professional development program on high school geometry teachers' content knowledge and their students' geometry learning. 64 geometry teachers were randomly assigned to an experimental (DG) group and a control group. Both groups received appropriate and relevant professional development. Classroom observation data and the teachers' responses to the implementation questionnaires revealed that most teachers in the DG group were faithful to the DG instructional approach. Teachers in the DG group scored higher on a conjecturing-proving test than did teachers in the control group. The students of teachers in the DG group scored significantly higher than the students of teachers in the control group on a geometry achievement test. [For the complete proceedings, see ED583989.]

In this testimony by Alexander Mayer, Deputy Director, Postsecondary Education, Manpower Demonstration Research Corporation (MDRC), Before the New Jersey State Assembly Higher Education Committee, the following three points are emphasized: (1) Developmental education is a major challenge facing low-income college students in New Jersey and across the country; (2) Students face a variety of barriers, but there is a lot of innovation in postsecondary institutions and growing evidence about what works to overcome those barriers; and (3) The strongest programs are comprehensive programs that combine opportunity and obligation to address multiple student barriers.

Although used as a common pedagogical tool in K-12 education, online graphic narrative ("comics") software has not generally been incorporated into advanced professional or technical education. This contribution reports preliminary data from a study on the use of cloud-based graphics software Pixton.com to teach basic medical ethics concepts and the human dimension of medical ethics dilemmas in a premedical program at an American medical college in the State of Qatar. Qualitative and quantitative end-of-semester data (open-ended and 7-point Likert scale questions) were collected from randomized control and study groups from 22 total students in a 4-week medical ethics module, half of whom wrote an argumentative essay on a contemporary medical ethics dilemma and the other half who created a 10-15 panel graphic narrative consisting of a real-life doctor-patient-family encounter involving a contemporary ethical issue. [For the full proceedings, see ED562095.]

Instructional design principles for visuals in student materials have been developed through findings based on student-level measures. However, teacher viewpoints may be a rich source of information to better understand how visuals can be optimized for student learning. This study's purpose is to examine teacher viewpoints on visuals. In a randomized, controlled field trial, teachers received mathematics curriculum with either the original visuals (control) or visuals revised based on instructional design principles (treatment). Teachers responded to survey questions regarding their viewpoints on issues related to instructional design principles applied to visuals (N = 22 treatment; N = 21 control). Both treatment and control teachers viewed aspects of certain instructional design-based principles favorably, but viewed issues related to math-irrelevant visuals differently. The following are appended: (1) Figure A1: Example of revisions based on the multimedia principle; (2) Figure A2: Example of revision based on the signaling principle (color coding); (3) Figure A3: Example of revision based on the contiguity principle (labeling); (4) Figure A4: Example of revision based on the contiguity principle (close physical proximity); (5) Figure A5: Example of revisions based on the coherence principle (removal of decorative visuals); (6) Figure A6: Example of revision based on the coherence principle (removal of a representational visual); and (7) Figure A7: Example of representational visual included in the revised curriculum to assist with vocabulary from the lesson context.

Policymakers are increasingly interested in the extent to which experimental results generalize from a sample to a population of inference. When the sample is not randomly selected, propensity score methods are used to reweight the sample. Subclassification by propensity score is commonly used in which the population is partitioned into strata using the propensity score distribution. However, subclassification methods are limited whenever the sample size is small or when there are problems with distributional overlap in the propensity scores. This study extends small area estimation methods to the problem of "sparse" strata under subclassification. The performance of two small area estimators is compared with two design-based estimators in a simulation study based on a completed cluster randomized trial.

Recent studies show cluster randomized trials may be well powered to detect mediation or indirect effects in multilevel settings. However, literature has rarely provided guidance on designing cluster-randomized trials aim to assess indirect effects. In this study, we developed closed-form expression to estimate the variance of and the statistical power to detect both individual and contextual indirect effects in cluster-randomized trials (i.e., 2-1-1 mediation). We then investigated the optimal sample allocation under a fixed budget such that researcher can minimize the resources needed for the study while maintaining adequate power. To facilitate end-user calculations, we have also developed freely available software that implements these formulas and the resampling-based approaches.

We present main effects of treatment and moderating effects of student background on science and engineering outcomes for grades 3-5 children participating in an engineering intervention. Data is from a large-scale randomized-controlled trial of two matched pairs (intervention and comparison) of engineering units. The intervention curriculum utilizes methods of project-based learning with support to apply science concepts to engineering design, while the comparison curriculum relies primarily on direct instruction. Both groups were taught the same science content prior to treatment. Intervention students in one of two curriculum units showed greater improvement on science content outcomes, as compared to control. The white/minority science achievement gap shrank after the intervention. Results suggest that the project-based intervention improves science achievement, particularly for minorities.

Using a low inference observation instrument, we investigated the effect of an ongoing, intensive, and structured virtual professional development [VPD] on fidelity of implementation across conditions in a randomized control trial validation study implemented in seven Texas school districts. The observations were collected three time in the beginning, middle and the end of school year. The results indicated that VPD had statistically significant impact on teachers' fidelity of implementation. Experimental teachers overperformed teachers in control condition in student involvement, level questioning, applying ESL strategies, providing affective and cognitive feedback, and more English material to encourage ELLs participated in classroom activities.

This study examined the effectiveness of two methods of increasing student learning from posterior probability lessons: diagrams and questioning while reading. Undergraduate students (N = 245) read a lesson in one of three diagram conditions and one of three questioning-while-reading conditions (embedded questions, elaboration interrogations, and control in which they read twice). Diagrams helped students in the control group accurately solve posterior probability problems on the posttest, but only if the information presented in the diagram was also presented in the text. Contrary to expectation, students in the control condition accurately solved more posterior probability problems on the posttest than did students in the questioning conditions. Poor answer quality for the embedded questions and elaborative interrogations is a possible explanation for the unexpected results. [For the complete proceedings, see ED584443.]

This study examined the effectiveness of using a Facebook group to increase preservice teachers' knowledge when one was used as a forum to share, answer, and discuss content-related questions in a technology course required for all students seeking teacher licensure. Further, it examined the students' prior use of Facebook groups, how the treatment group used the group, and their perspectives of the use of Facebook as an educational tool. The results revealed no significant gain in achievement. Almost all participants had prior experience using a Facebook group, and the primary purposes of these groups were for organizing events, communication within organizations, communication within classes, and lending support to memorials/dedications. Although participation in the group was required and linked to a grade, most of the participation was characterized as very low level (i.e., "liking"), with only half supplying the answers to questions and about one fifth making comments. Their perspectives on whether Facebook can be used for educational purposes were lukewarm, yet they indicated significant change in their perception that Facebook assignments were an invasion of privacy. While they perceived the idea to be good, they viewed the expectations for participation as too lenient, thus causing lack of in-depth participation. [For the full proceedings, see ED562107.]

Research on labour market disparities between men and women has long since identified horizontal segregation in the educational system as one of the main factors driving the gender pay gap. One of the factors identified in the literature believed to be at the heart of horizontal gender segregation is the lack of representation and visibility of women in STEM fields. In this paper, we investigate the effectiveness of a program named "STEM in Genere", developed within the Gender Equality Plan 2022-2024 of the University of Brescia (Italy), which is aimed at contrasting the underrepresentation and the stereotypical representation of women in science among primary and lower secondary school students. Students of participant classes will meet an educator for a total of 2 hours, in which counter-stereotypical thinking of women in science will be stimulated via learning games. Teachers of participant classes will follow in parallel a dedicated workshop about nonstereotypical science teaching. The effects of the program on students' gender stereotypes in science and study/job aspirations will be evaluated via a randomised controlled trial (RCT) conducted in primary and secondary schools of the Brescia province, an affluent province located in the northern part of Italy. In particular, we will randomise classes within participating schools to have a good balance between the statistical power of the experiment and potential contamination threats. The contribution of this evaluation is threefold. First, we will collect first-hand data on students' views and aspirations in a country - Italy - in which the population traditionally holds quite conservative views about gender roles. Second, we will provide robust, experimental evidence on the effectiveness of the program "STEM in Genere", which is potentially scalable nationwide. Lastly, we will contribute to the methodological debate about question framing in survey methodology, by randomly varying the formulation of the questions in the surveys. Policy implications for educational and learning environments will be discussed. [ABSTRACT FROM AUTHOR]

Our objective in this Evidence-Based Practice (EBP) paper is to describe an innovative approach and the impact of that approach on undergraduates engaged in engineering and computer science research experience involving robotics swarms. While critiques of a narrow reading of EBP approaches rightly assert inapplicability when applied as a mandate to educational practitioners [1], our goal here is to contribute to the evidence base supporting our educational practitionerdriven pedagogy thereby expanding the definition and applicability of EBP in education. More recent approaches to EBP in disciplines aligned with the learning sciences have made it clear that a broad range of evidentiary warrant (i.e., beyond quantitative measures in randomized control trials), integration of cognitive learning theory with human developmental theory, and both practitioner and learner input are critical to developing an evidence base appropriate to guide educational practice [2], [3]. We seek to help undergraduate students develop into independent researchers through challenging them with real-world problems utilizing human-swarm interactions as both research context and pedagogical model for how to engage students in undergraduate research. We describe our program and the sequential mixed methods evaluation informing quality improvement and summative evidence of impact on participants. Our undergraduate researchers explore innovative ways of interacting with collectively emergent systems behavior without directly controlling individual building blocks of swarms, and we utilize our understanding of human-swarm interaction as a model for organizing their research experience. Students gain first-hand knowledge of the nonlinearity and robustness of robot swarm behaviors through observations and interactions during hands-on experiments. We give students ownership of engineering problems formulated as challenges their "swarm" must solve. With this active learning approach, students' creativities are stimulated, and they become more confident, comfortable, and competent in solving complex robotics problems [4], [5]. We utilized a sequential mixed methods evaluation design for triangulation, complementarity, and development [6], [7]. This allowed us to leverage analysis of collected data to inform development of later data sources (i.e., development), as well as to examine the degree of overlap (i.e., triangulation) and independent contribution (i.e., complementarity) across quantitative and qualitative data sources. Data sources included a pre-post survey and focus groups during week 4 and week 8 of our 10-week program. These data sources allowed quantitative pre-post change analysis to be triangulated with participant self-assessment of impact and complemented with participant description of activities and perspectives that facilitated such impact as well as suggestions for improvement. Participants described positive impacts of the autonomy-supportive structure and authentic nature of the program in all areas and most strongly in Research Skills development. Participant recommendations centered on better communication, increased direct support from mentors, and focus on leadership, careers/graduate school, and scientist identity development. We recommend consideration of literature on cognitive apprenticeship in communities of practice organized around research groups [8] to inform projects such as this. [ABSTRACT FROM AUTHOR]

STEMadium is a serious mobile game designed to improve middle school age youth’s skills in science and math by using the science and math inherent in baseball combined with a compelling narrative. This paper summarizes the development and evaluation of STEMadium. This includes the formative research which utilized play testing, interviews, and focus groups, with a particular focus on what would engage girls and minority youth. The results of that formative research drove the development of the game, including the inclusion of a strong narrative with a female protagonist who helps bring the team to victory, an array of team members, each with a unique backstory, who need to work on their STEM (Science, Technology, Engineering, and Math) skills to improve their baseball skills, and the inclusion of exploration and customization elements. The focus during development was to create an engaging science of baseball focused game for youth who may not necessarily be interested in sports or sports games while ensuring the pedagogy and presentation of content was teacher approved and mapped to math and science standards. The completed game was subjected to a three-arm randomized controlled trial (RCT) in which the game was tested against a control condition, and a condition in which the game was combined with virtual class instruction on math and science topics. Results from the RCT show that (1) the game performed as well as the game plus instruction condition; and (2) compared to the control condition, both conditions improved math and science knowledge, reduced the decrease in math skills during non-school periods (“summer slide”), and worked equally well across genders. The youth also reported liking the game, with the majority noting that it was better or much better than most educational games. The results provide support for the use of mobile gaming to improve STEM skills in middle school aged youth. As wellthe development may provide a roadmap for using gaming to engage youth not typically interested in STEM. [ABSTRACT FROM AUTHOR]

Sample size is an important indicator of the power of randomized controlled trials (RCTs). In this paper, we designed a total sample size extractor using a combination of syntactic and machine learning methods, and evaluated it on 300 Covid-19 abstracts (Covid-Set) and 100 generic RCT abstracts (General-Set). To improve the performance, we applied transfer learning from a large public corpus of annotated abstracts. We achieved an average F1 score of 0.73 on the Covid-Set testing set, and 0.60 on the General-Set using exact matches. The F1 scores for loose matches on both datasets were over 0.74. Compared with the state-of-the-art tool, our extractor reports total sample sizes directly and improved F1 scores by at least 4% without transfer learning. We demonstrated that transfer learning improved the sample size extraction accuracy and minimized human labor on annotations. [ABSTRACT FROM AUTHOR]

Medication errors are a leading cause of preventable harm in hospitals. Electronic medication systems (EMS) have shown success in reducing the risk of prescribing errors, but considerable less evidence is available about whether these systems support a reduction in medication administration errors in paediatrics. Using a stepped wedge cluster randomized controlled trial we investigated changes in medication administration error rates following the introduction of an EMS in a paediatric referral hospital in Sydney, Australia. Direct observations of 284 nurses as they prepared and administered 4555 medication doses was undertaken and observational data compared against patient records to identify administration errors. We found no significant change in administration errors post EMS (adjusted Odds Ratio [aOR] 1.09; 95% CI 0.89-1.32) and no change in rates of potentially serious administration errors (aOR 1.18; 95%CI 0.9-1.56), or those resulting in actual harm (aOR 0.92; 95%CI 0.34-2.46). Errors in administration of medications by some routes increased post EMS. In the first 70 days of EMS use medication administration error rates were largely unchanged. [ABSTRACT FROM AUTHOR]

Within the past decade, a number of studies have analysed and investigated the aspects and potential applications of games-based learning. Educationalists and industrialists continue to explore it as it is perceived as a potentially beneficial supplementary learning mechanism, teaching and training approach and perfectly matches the determinants of intrinsic motivation. Previous studies indicate that games can promote learning and engage learning experiences compared with traditional methods, while one of the primary arguments against games-based learning focuses on a dearth of empirical evidence in particular a lack of Randomised Controlled Trials (RCTs) and longitudinal studies. The lack of empirical evidence in supporting the efficacy of games-based learning can be challenged as unsubstantiated optimism. This paper provides empirical evidence by reporting the results of a survey conducted at tertiary education institutions in China. The survey is conducted on the basis of a number of previous studies on the motivation of games-based learning. The survey reports on the evidence from the Chinese institutions where 346 undergraduate and postgraduate students completed the survey in March 2018 over a three-week period. Participants had positive attitude towards playing computer games for learning. The top motivations of playing computer game were: enjoyment, competition and relationships whereas the top motivation of playing games in education were: relaxation, leisure and pleasure. Respondents perceived that computer games could potentially provide the following skills: collaboration, problem-solving and analysis. The paper contributes to empirical evidence in games-based learning through a comprehensive quantitative analysis of data including gender, education, game platform and play types of game comparisons. The results were compared to similar studies conducted at different levels of education in different countries and provide a synthesis. The results will also provide insight into game design associated with the motivations for using GBL at tertiary education level. [ABSTRACT FROM AUTHOR]

This paper brings a historical perspective to understand the recent dissemination of randomized controlled trials (RCTs) as the new "gold standard" method to assess the success (and failure) of poverty alleviation efforts. Although the buzz around RCT evaluations dates from the 2000s, we show that the experimental approach has a much longer pedigree in the field of development economics. Indeed, what we are witnessing now is a second wave of RCTs in international development, while a first wave took place in the 1970s and has ended by the early 1980s. In this paper, we contrast the first and second waves as cases of "reiterated problemsolving" (Haydu 1998) to analyze the practical, legal and methodological obstacles that made RCTs ultimately unviable in the 1970s, and thus highlight the transformations and strategies that made possible their resurgence in the 2000s. Our argument is that the contemporary expansion of RCTs can only be understood in a context of the privatization of foreign aid and the multiple new dynamics associated with it, which allowed randomistas to effectively change what it means to evaluate development policy. We rely on a rich set of archival data, policy documents and academic papers to show how differences in the network of actors, modes of presentation, scale of interventions and funding opportunities between the first and second waves explain the relative impracticality and consequent discontinuation of RCTs in the 1970s, in contrast to their rapid expansion now. [ABSTRACT FROM AUTHOR]

Genetic ancestry testing is a controversial industry, with scholars questioning the tests legitimacy as well as their potential to increase racial essentialism. Yet despite these controversies, genetic ancestry testing continues to grow in popularity in the United States as a way to either confirm or uncover ethnoracial ancestry. While research is beginning to examine people's interpretations of their ancestry test results, little consideration has been given to realworld outcomes of taking these tests, like their impact on changes in the ethnoracial make-up of social networks. This paper asks whether (1) taking the test, and (2) receiving results of non- European background impacts ethnoracial network diversity for a sample of White Americans. We address these questions using an original randomized controlled trial of White, non-Hispanic Americans, half of whom received genetic ancestry tests. Unexpected findings show that testtakers network diversity shrinks post-test, and especially for those with more diverse networks to begin with. We further explore these patterns, drawing on 57 qualitative follow-up interviews with study participants, and present three potential theoretical explanations for the finding. We discuss how future research can adjudicate between these explanations. [ABSTRACT FROM AUTHOR]

Perspective-taking is a core theory of mind process, entailing the ability to understand and predict others' mental states. While the literature suggests that priming a perspective-taking stance increases empathic concern and intergroup cohesion and reduces implicit negative attitudes such as racial bias and prejudice, no study up to date has attempted to foster perspective-taking implicitly, by training visual perspective-taking in a computerized task. In the current paper, we propose a computerized, gamified intervention for visual perspective-taking in adolescents and young adults. The intervention is an adapted version of extant visual perspective-taking assessment protocols and is envisioned to train participants to selectively attend to the visual perspective of an animated avatar, instead of their own. By training an allocentric bias, the intervention aims to increase empathic and theory of mind abilities and to foster interpersonal communication and cohesion. An additionally hypothesized benefit of the intervention is a decrease in self-focused attention, a marker of social anxiety disorder. As such, the intervention is suited to address two different, but inter-related needs: optimizing social functioning and managing social anxiety symptomatology. Introducing users to gamification elements, such as earning badges, advancing through increasing difficulty levels, receiving feedback and leaderboards highlighting achievements, is intended to increase the intervention's efficacy and to promote adherence. Design considerations for testing the intervention in an experimental trial will additionally be proposed, with the intervention contrasted against a sham control condition that will equally train egocentric and allocentric biases in visual perspective-taking. Theoretical and practical implications for educational settings will be discussed. [ABSTRACT FROM AUTHOR]

PICO (Population/problem, Intervention, Comparison, and Outcome) is widely adopted for formulating clinical questions to retrieve evidence from the literature. It plays a crucial role in Evidence-Based Medicine (EBM). This paper contributes a scalable deep learning method to extract PICO statements from RCT articles. It was trained on a small set of richly annotated PubMed abstracts using an LSTM-CRF model. By initializing our model with pretrained parameters from a large related corpus, we improved the model performance significantly with a minimal feature set. Our method has advantages in minimizing the need for laborious feature handcrafting and in avoiding the need for large shared annotated data by reusing related corpora in pretraining with a deep neural network. [ABSTRACT FROM AUTHOR]

Over the years, obesity prevention has been a broadly studied subject by both academicians and practitioners. It is one of the most serious public health issue as it can cause numerous chronic health and psychosocial problems. Research is needed to suggest a population-based strategy for obesity prevention. In the academic environment, the importance of obesity prevention has triggered various problem solving approaches. A good obesity prevention model, should comprehend and cater all complex and dynamics issues. Hence, the main purpose of this paper is to discuss the qualitative and quantitative approaches on obesity prevention study and to provide an extensive literature review on various recent modelling techniques for obesity prevention. Based on these literatures, the comparison of both quantitative and qualitative approahes are highlighted and the justification on the used of system dynamics technique to solve the population of obesity is discussed. Lastly, a potential framework solution based on system dynamics modelling is proposed. [ABSTRACT FROM AUTHOR]

Recent research reveals that other more discrete factors, like "Knowledge-based work (KBW)" have been identified as a potential factor that induces overeating and/or obesity in workers, mostly in women. That might be the explanation for epidemic obesity in person stress-exposed in work-place for a long time. Based on auto-regulation theory, the goal of this paper was to explore the relations between psychological characteristics, mental workrelated efforts, and food intake, a cause which is capable to explain the very high incidence and prevalence of epidemic obesity in this kind of population (occupational category). The results are in agreement with the current literature. However, the controlled motivation variable shows no association with bodily satisfaction even though this relationship is well documented in women. Additional researches are needed to total understanding of complex interactions, the reasons underlying this association partially known between KBW, psychological characteristics, and ad libitum food intake. [ABSTRACT FROM AUTHOR]

There has been a consensus amongst serious games advocates about the effectiveness of serious games in improving learning. This provides the basis for the assumption that its role and value in education is undeniable. The interest in using serious games in education has been gaining attention and momentum over the last decade or so as many believe that digital games can be a powerful tool to assist the teaching and learning process due to its highly engaging, motivating and interactive nature. This alone provides many advantages over traditional education methods. While there is no dispute that serious games in education is generally a valid tool in the approach to teaching, the question remains; are all of them genuinely effective as an educational tool in comparison with traditional education methods and what would be a methodology for obtaining that crucial data. As researchers and educators, we cannot just rely on test validities to gauge the effectiveness of a game-based application as this merely measures that product as an indicator of performance within the application but rather an experimental validity study as a criterion of its effectiveness in order to validate the causal relation between the serious games educational application and the actual process learning. The very fact that most serious games educational applications are not validated on the premise that it makes a difference on the outcome of education as compared to other non-digital means creates an assumption that is may not be as effective as postulated. The aim of this paper is to explore the need and effectives methods for a standardized form of educational serious games validation. [ABSTRACT FROM AUTHOR]

Platforms critically rely on rating systems to learn the quality of market participants. In practice, however, these ratings are often highly inflated, and therefore not very informative. In this paper, we investigate whether the platform can obtain less inflated ratings by altering the meaning and relative importance of the levels in the rating system. We then seek a principled approach to make these choices in the design of the rating system. First, we analyze the results of a randomized controlled trial on an online labor market in which an additional question was added to the feedback form. Between treatment conditions, we vary the question phrasing and answer choices; in particular, the treatment conditions include several positive-skewed verbal rating scales with descriptive phrases or adjectives providing specific interpretation for each rating level. The online labor market test reveals that current inflationary norms can in fact be countered by re-anchoring the meaning of the levels of the rating system. In particular, the positive-skewed verbal rating scales yield rating distributions that significantly reduce rating inflation and are much more informative about seller quality. Second, we develop a model-based framework to compare and select among rating system designs. We apply this framework to the data obtained from the online labor market test, demonstrating that our model-based framework for scale design and optimization can identify the most informative rating system and substantially improve the quality of information obtained over baseline designs. Overall, our study illustrates that rating systems that are informative in practice can be designed, and demonstrates how to design them in a principled manner. [ABSTRACT FROM AUTHOR]

Pressure and coercion in adolescence relationship can lead to adverse psychological and physical health outcomes. Eliminating coercive sexual experiences for young people is one of the aims of delivering good quality Relationships and Sex Education (RSE). Engaging young people on sensitive subjects can however be challenging, and using interactive technology such as games may provide a more conducive platform for educators and young people within a classroom setting. This paper describes the on-going development and evaluation of a serious game (SG) to raise awareness on experiences of sexual coercion. This involves an iterative process that draws on stakeholder engagement and the theory and evidence base for what works, to support health improvement intervention planning. SG developers took the game concept plan and transformed it into an interactive game show, led by a teacher or facilitator to engage students in game play and discussion around the issue of sexual coercion. The final product known as PR:EPARe (Positive Relationships: Eliminating Coercion and Pressure in Adolescent Relationships) is currently the subject of a cluster Randomised Controlled Trial (RCT) in local schools. Early data analysis shows that the learners are engaged with the game with potential contribution to psychological preparedness to deal with sexual coercion. This work represents the first attempt to use of Serious Gaming for RSE delivery. [ABSTRACT FROM AUTHOR]

This paper evaluates the influence of housing, services, and individual characteristics on housing loss among formerly homeless mentally ill persons who participated in a five-site (4-city) study in the U.S. Housing and service availability were manipulated within randomized experimental designs and substance abuse and other covariates were measured with a common protocol. Findings indicate that housing availability was the primary predictor of subsequent ability to avoid homelessness, while enhanced services reduced the risk of homelessness if housing was also available. Substance abuse increased the risk of housing loss in some projects, but specific findings differed between projects and with respect to time spent in shelters and on the streets. We identify implications for research on homeless persons with mental illness that spans different national and local contexts and involves diverse ethnic groups. ..PAT.-Unpublished Manuscript [ABSTRACT FROM AUTHOR]

Although vast amounts of Real-World Evidence are available from Electronic Health Records, there is little information on current trends in Real-World Data analysis, thus hindering informed decision-making by healthcare professionals. Real World Evidence includes administrative claims, disease or product registries, patient-generated health data, and other digital health technologies. In this study, we review current and relevant developments, as well as guidelines across different countries, to summarize recent advancements and extract valuable insights from the existing literature on Real-World Evidence. [ABSTRACT FROM AUTHOR]

To establish the feasibility of embedding an RCT into EMR in the ICU, we evaluated the route of phosphate replacement. The EMR screened 207 patients who met the inclusion criteria from 20 April 2022 to 30 June 2022. 162 patients were randomised and 145 patients allocated to treatment. Our study showed that it was feasible to embed screening, randomisation, and treatment allocation for an RCT within an EMR in the ICU. [ABSTRACT FROM AUTHOR]

If science is the new religion then the proponents of randomised control trials as the gold standard methodology for evaluation research in criminology are the new evangelists. Their zealous pursuit of this goal in the UK is not just about capturing the hearts and minds of the Home Office Research Development and Statistics Directorate but their money. Naturally this has fuelled a heated debate among British criminologists about whether evaluative methodologies should be considered as a hierarchy or a typology and whether the big bucks should be placed on horses chosen for courses or placed in the RCT Gold Cup. The politics of this debate are fascinating particularly given the fall out from the experience of the research evaluations of the Crime Reduction Programme that ran in England and Wales from 1999 to 2002 and involved an unprecedented research budget. This paper explores the politics of evaluation research and evidence based policy and practice in the UK and tries to assess whether this is a hurdles or a flat race. ..PAT.-Unpublished Manuscript [ABSTRACT FROM AUTHOR]

Randomised Controlled Trials (RCTs) are relatively common in the US and have been used on a number of occasions to evaluate criminal justice interventions and provide government officials with strong evidence on which to base their policy decisions. In UK criminology however, very few RCTs have been carried out in the past twenty-five years. Now, within the research programme of the National Offender Management Service (RDS NOMS), there is a drive to use RCTs again to provide robust evidence to NOMS policy makers on 'what works' to reduce re-offending. RDS NOMS are working to develop the use of RCTs in a number of areas. This is part of a mixed-method evaluation strategy which also includes qualitative research to explore key issues and economic analysis to ensure interventions are cost-effective. This paper discusses the challenges facing NOMS researchers as they attempt to encourage the use of RCTs, where appropriate, in the UK criminal justice system. It draws on findings from two RCT feasibility studies which were conducted in 2005/6, lessons from ongoing studies in the UK involving RCTs, support for the development of policy trials, and work in progress to design future RCTs. ..PAT.-Unpublished Manuscript [ABSTRACT FROM AUTHOR]