Search

Preregistration and registered reports are two promising open science practices for increasing transparency in the scientific process. In particular, they create transparency around one of the most consequential distinctions in research design: the data analytics decisions made before data collection and post-hoc decisions made afterwards. Preregistration involves publishing a time-stamped record of a study design before data collection or analysis. Registered reports are a publishing approach that facilitates the evaluation of research without regard for the direction or magnitude of findings. In this paper, I evaluate opportunities and challenges for these open science methods, offer initial guidelines for their use, explore relevant tensions around new practices, and illustrate examples from educational psychology and social science. This paper was accepted for publication in Educational Psychologist volume 56, issue 2; scheduled for April 2021, as a part of a special issue titled, “Educational psychology in the open science era.”This preprint has been peer reviewed, but not copy edited by the journal and may differ from the final published version. The DOI of the final published version is: [insert preprint DOI number]. Once the article is published online, it will be available at the following permanent link: [insert doi link]

For decades, technology advocates have claimed that we are on the cusp of a complete transformation in education. But, as Justin Reich explains, such transformations have not yet come to pass. Even during the COVID-19 pandemic, teachers did not use technology to significantly alter their teaching. Instead, technology enabled them to maintain many of their classroom routines (using learning management systems or video conferencing) or supplement their usual instruction (using gamified apps). Teachers did, however, tinker with their methods throughout the pandemic, gradually improving over time. Reich suggests that this tinkering framework is a more realistic way to think about ed tech’s potential to support teaching and learning.

Interactive simulations allow preservice teachers to connect education theory and pedagogy in scaffolded environments. We created digital simulations with scenarios from in-person simulatio...

The late 2000s and 2010s saw the full arc of a dramatic hype cycle in learning at scale, where charismatic technologists made bold and ultimately unfounded predictions about how technologies would disrupt schooling systems. Looking toward the 2020s, a more productive approach to learning at scale is the tinkerer's stance, one that emphasizes incremental improvements on the long history of learning at scale. This article offers two organizational constructs for navigating and building on that history. Classifying learning-at-scale technologies into three genres-instructor-guided, algorithm-guided, and peer-guided approaches-helps identify how emerging technologies build on prior efforts and throws into relief that which is genuinely new. Four as-yet intractable dilemmas-the curse of the familiar, the edtech Matthew effect, the trap of routine assessment, and the toxic power of data and experiments-offer a set of grand challenges that learning-at-scale tinkerers will need to tackle in order to see more dramatic improvements in school systems.

Digital clinical simulations (DCSs) are a promising tool for professional learning on diversity, equity, and inclusion (DEI) issues across a variety of fields. Although digital clinical simulations can be integrated into large-scale learning environments, less is known about how to design these types of simulations so they can scale effectively. We describe the results of two studies of a digital clinical simulation tool called Jeremy's Journal. In Study 1, we implemented this simulation in an in-person workshop with a human facilitator. We found that participants described their learning experiences positively and reported changes in attitudes. In Study 2, we used the simulation within an online course but replaced the human facilitator with an asynchronous, text-based adaptation of the facilitation script. Although learners in Study 2 described the experience in the simulation positively, we did not observe changes in attitudes. We discuss the implications of these findings for the design of DCSs at scale.

Many higher education institutions have begun offering opportunities to earn credit for in-person courses through massive open online courses (MOOCs). This mixed-methods study examines the experiences of students participating in one of the first iterations of this trend: a blended professional master's degree program that admitted students based on performance in MOOC-based online courses. We found that the blended master's program attracted a cohort of highly educated mid-career professionals from developed countries who were looking for more flexible alternatives to traditional graduate programs. Success in the online courses was correlated with higher levels of prior formal education and effective use of learning strategies. Students who enrolled in the blended graduate program reported being academically prepared for their coursework and had higher GPAs (3.86, p

Significance Low persistence in educational programs is a major obstacle to social mobility. Scientists have proposed many scalable interventions to support students learning online. In one of the largest international field experiments in education, we iteratively tested established behavioral science interventions and found small benefits depending on individual and contextual characteristics. Forecasting intervention efficacy using state-of-the-art methods yields limited improvements. Online education provides unprecedented access to learning opportunities, as evidenced by its role during the 2020 coronavirus pandemic, but adequately supporting diverse students will require more than a light-touch intervention. Our findings encourage funding agencies and researchers conducting large-scale field trials to consider dynamic investigations to uncover and design for contextual heterogeneity to complement static investigations of overall effects., Online education is rapidly expanding in response to rising demand for higher and continuing education, but many online students struggle to achieve their educational goals. Several behavioral science interventions have shown promise in raising student persistence and completion rates in a handful of courses, but evidence of their effectiveness across diverse educational contexts is limited. In this study, we test a set of established interventions over 2.5 y, with one-quarter million students, from nearly every country, across 247 online courses offered by Harvard, the Massachusetts Institute of Technology, and Stanford. We hypothesized that the interventions would produce medium-to-large effects as in prior studies, but this is not supported by our results. Instead, using an iterative scientific process of cyclically preregistering new hypotheses in between waves of data collection, we identified individual, contextual, and temporal conditions under which the interventions benefit students. Self-regulation interventions raised student engagement in the first few weeks but not final completion rates. Value-relevance interventions raised completion rates in developing countries to close the global achievement gap, but only in courses with a global gap. We found minimal evidence that state-of-the-art machine learning methods can forecast the occurrence of a global gap or learn effective individualized intervention policies. Scaling behavioral science interventions across various online learning contexts can reduce their average effectiveness by an order-of-magnitude. However, iterative scientific investigations can uncover what works where for whom.

Registered reports engender a review process in which reviewers evaluate the merits of the research question rather than the magnitude or direction of the results. In this process, authors first submit introduction, methods, and prespecified analyses sections of an article. Next, an initial peer-review process assesses the merits of the research question, in the absence of knowing the results. Worthy articles are granted an “in-principle acceptance.” The authors, then conduct their study, analyze their data, write up the remainder of their article, and resubmit it. The in-principle acceptance means that if the authors adhere to the research design as originally submitted, then the article will be accepted. Building from seven of the earliest registered reports in education research, our introduction describes the benefits, opportunities, and challenges of registered reports as a means to increasing transparency in research and reducing the number of illusory results.

Massive Open Online Courses (MOOCs) have opened new educational possibilities for learners around the world. Numerous providers have emerged, which usually have different targets (geographical, topics or language), but most of the research and spotlight has been concentrated on the global providers and studies with limited generalizability. In this work we apply a multi-platform approach generating a joint and comparable analysis with data from millions of learners and more than ten MOOC providers that have partnered to conduct this study. This allows us to generate learning analytics trends at a macro level across various MOOC providers towards understanding which MOOC trends are globally universal and which of them are context-dependent. The analysis reports preliminary results on the differences and similarities of trends based on the country of origin, level of education, gender and age of their learners across global and regional MOOC providers. This study exemplifies the potential of macro learning analytics in MOOCs to understand the ecosystem and inform the whole community, while calling for more large scale studies in learning analytics through partnerships among researchers and institutions.

The relationship between pricing and learning behavior is an important topic in research on massive open online courses (MOOCs). We report on two case studies where cohorts of learners were offered coupons for free certificates to explore how price reductions might influence behavior in MOOC-based online learning settings. In Case Study 1, we compare participation and certification rates between courses with and without free-certificate coupons. In the courses with a free-certificate track, participants signed up for the verified-certificate track at higher rates, and completion rates among verified students were higher than in the paid-certificate track courses. In Case Study 2, we compare learner behavior within the same courses by whether they received access to a free-certificate track. Access to free certificates was associated with lower certification rates, but overall, certification rates remained high, particularly among those who viewed the courses. These findings suggest that some incentives, other than simply the cost of paying for a verified-certificate track, may motivate learners to complete MOOCs.

© 2019 Elsevier Ltd Recent studies of massive open online courses (MOOCs) have focused on global providers such as edX, Coursera, and FutureLearn, with less attention to local initiatives that target regional learners. In this study we combine data from the main edX platform and one regional MOOC provider, Edraak in Jordan, to explore differences in learners’ behavior and preferences. We find that regional provider Edraak attracts younger learners, more females and those with lower levels of education compared to global providers. Edraak learners value local courses because they cater to their interests and learning needs. We document our multi-platform learning analytics procedure, where we establish a common data format and script that enables an “apples-to-apples” comparison without exchanging data — a common privacy and data security concern. These findings suggest the potential of this methodological approach to study and learn from regional MOOC providers, particularly around the questions of equity and access in the global MOOC ecosystem.

Massive Open Online Courses (MOOCs) came into the educational ecosystem attracting the attention of the public media, businesses, teachers, and learners from all over the world. The original courses were completely open and free, targeting the worldwide population. However, current MOOC providers have pivoted towards more private directions, and we often find that MOOC materials are completely closed within their hosting platforms and cannot be retrieved from them by their learners. This diminishes the potential of MOOCs by making content available to a small proportion of learners and severely limits the reusability of the educational resources. In this paper, we present a process that we call &lsquo, unMOOCing&rsquo, in which we transform the resources of a MOOC into OERs. We taught a MOOC on Open Education in the UNED Abierta platform, and we &lsquo, unMOOCed&rsquo, all of its educational resources, making them available to download by the learners that are taking the course. The results of the unMOOCing were very encouraging: the possibility of downloading the course resources was the most highly rated component of the course. Additionally, the two unMOOCed materials that were considered as most useful (presentations and contents in a PDF) were downloaded by 90% of the learners. Now that the majority of MOOC providers are moving towards a more closed educational approach, we believe that this paper sends a powerful message for bringing back the original MOOC concept of &lsquo, Openness&rsquo, with the unMOOCing process, thus contributing to the wider dissemination and democratization of education across the globe.

Over two iterations of a Massive Open Online Course (MOOC) for school leaders, Launching Innovation in Schools, we developed and tested design elements to support the transfer of online learning into offline action. Effective professional learning is job-embedded: learners should employ news skills and knowledge at work as part of their learning experience. This MOOC aimed to get participants to plan and actually launch new change efforts, and a subset of our most engaged participants were able and willing to do so during the course. Required assessments spurred student actions, along with instructor calls to action and modeling and exemplars provided by course elements. We found that participants led change initiatives, held stakeholder meetings, collected new data about their contexts, and shared and used course materials collaboratively. Collecting data about participant learning and behavior outside the MOOC environment is essential for researchers and designers looking to create effective online environments for professional learning.

La creciente utilización de sistemas de mediación digital en la mayoría de espacios educativos —ya sean presenciales o no, formales o abiertos, y tanto en el nivel de educación básica como en situaciones de aprendizaje a lo largo de la vida— está acelerando el avance de la analítica del aprendizaje y haciendo que el uso de la información digital sea una práctica común en el campo de la educación. Las herramientas educativas digitales facilitan la interacción entre estudiantes, profesores y recursos de aprendizaje, y generan de manera continua un notable volumen de datos que pueden analizarse aplicando una variedad de metodologías. Esto ha hecho que aumenten exponencialmente las investigaciones que toman como referencia la información que procede de la actividad de los estudiantes en esos espacios digitales. Partiendo de esas evidencias, este número especial muestra un conjunto de estudios en el campo del aprendizaje digital y la investigación educativa basada en datos, que enriquecen el conocimiento sobre los procesos de aprendizaje y la gestión de la enseñanza en espacios mediados digitalmente.

We present Discourse, a tool for coding and annotating MOOC discussion forum data. Despite the centrality of discussion forums to learning in online courses, few tools are available for analyzing these discussions in a context-aware way. Discourse scaffolds the process of coding forum data by enabling multiple coders to work with large amounts of forum data. Our demonstration will enable attendees to experience, explore, and critique key features of the app.

Among all of the learners in Massive Open Online Courses (MOOCs) who intend to complete a course, the majority fail to do so. This intention-action gap is found in many domains of human experience, and research in similar goal pursuit domains suggests that plan-making is a cheap and effective nudge to encourage follow-through. In a natural field experiment in three HarvardX courses, some students received open-ended planning prompts at the beginning of a course. These prompts increased course completion by 29%, and payment for certificates by 40%. This effect was largest for students enrolled in traditional schools. Furthermore, the contents of students' plans could predict which students were least likely to succeed - in particular, students whose plans focused on specific times were unlikely to complete the course. Our results suggest that planning prompts can help learners adopted productive frames of mind at the outset of a learning goal that encourage and forecast student success.

Scientific progress is built on research that is reliable, accurate, and verifiable. The methods and evidentiary reasoning that underlie scientific claims must be available for scrutiny. Like other fields, the education sciences suffer from problems such as failure to replicate, validity and generalization issues, publication bias, and high costs of access to publications—all of which are symptoms of a nontransparent approach to research. Each aspect of the scientific cycle—research design, data collection, analysis, and publication—can and should be made more transparent and accessible. Open Education Science is a set of practices designed to increase the transparency of evidentiary reasoning and access to scientific research in a domain characterized by diverse disciplinary traditions and a commitment to impact in policy and practice. Transparency and accessibility are functional imperatives that come with many benefits for the individual researcher, scientific community, and society at large—Open Education Science is the way forward.

Student intention and motivation are among the strongest predictors of persistence and completion in Massive Open Online Courses (MOOCs), but these factors are typically measured through fixed-response items that constrain student expression. We use natural language processing techniques to evaluate whether text analysis of open responses questions about motivation and utility value can offer additional capacity to predict persistence and completion over and above information obtained from fixed-response items. Compared to simple benchmarks based on demographics, we find that a machine learning prediction model can learn from unstructured text to predict which students will complete an online course. We show that the model performs well out-of-sample, compared to a standard array of demographics. These results demonstrate the potential for natural language processing to contribute to predicting student success in MOOCs and other forms of open online learning.

Advocates for free massive open online courses (MOOCs) have heralded them as vehicles for democratizing education and bridging divides within and across countries ( 1 ). More than 25 million people enrolled in MOOCs between 2012 and 2015, including 39% from less-developed countries (LDCs) ( 2 ). But the educated and affluent in all countries enroll in and complete MOOCs at relatively higher rates ( 3 , 4 ). Judged by completion rates, MOOCs do not spread benefits equitably across global regions. Rather, they reflect prevailing educational disparities between nations (see the first chart) ( 5 ). Although the global achievement gap could be caused by barriers in LDCs, such as less broadband Internet access, formal education, and English proficiency, we explore another potential but underappreciated cause. Members of LDCs may suffer from the cognitive burden of wrestling with feeling unwelcome while trying to learn and, therefore, underperform. This can be exacerbated by social identity threat, which is the fear of being seen as less capable because of one's group ( 6 ). We discuss field experiments with interventions that targeted social identity threat and caused substantial improvements in MOOC persistence and completion rates among learners in LDCs, eliminating the global achievement gap.

Big data from massive open online courses (MOOCs) have enabled researchers to examine learning processes at almost infinite levels of granularity. Yet, such data sets do not track every important element in the learning process. Many strategies that MOOC learners use to overcome learning challenges are not captured in clickstream and log data. In this study, we interviewed 92 MOOC learners to better understand their worlds, investigate possible mechanisms of student attrition, and extend conversations about the use of big data in education. Findings reveal three important domains of the experience of MOOC students that are absent from MOOC tracking logs: the practices at learners’ workstations, learners’ activities online but off-platform, and the wider social context of their lives beyond the MOOC. These findings enrich our understanding of learner agency in MOOCs, clarify the spaces in-between recorded tracking log events, and challenge the view that MOOC learners are disembodied autodidacts.

International audience; MOOCs are becoming more and more integrated in the higher education landscape of learning, with many institutions now pushing their students towards MOOC as part of their curriculum. But what does it mean for other MOOC learners? Are these students socializing the same way when they have an easier possibility to interact with classmates offline? Is the fact that they do not personally choose to enroll in a MOOC also having an effect? In this paper, we compare university-enrolled students to other MOOC participants and in particular other self-enrolled students, to examine how and why they socialize on and around the MOOC. Using data from two French MOOCs in project management, we show that university-enrolled students are less attracted by forums and seem to interact less than others when the workload increases , which could lead to misleading conclusions when analyzing data. We therefore encourage MOOC researchers to be particularly mindful of this new trend when performing social network analyses.