Search

This research was triggered by the identified need in literature for large-scale studies about the kinds of designs that teachers create for mobile learning (m-learning). These studies require analyses of large datasets of learning designs. The common approach followed by researchers when analyzing designs has been to manually classify them following high-level pedagogically guided coding strategies, which demands extensive work. Therefore, the first goal of this paper is to explore the use of supervised machine learning (SML) to automatically classify the textual content of m-learning designs using pedagogically relevant classifications, such as the cognitive level demanded by students to carry out specific designed tasks, the phases of inquiry learning represented in the designs, or the role that the situated environment has in the designs. Because not all SML models are transparent, but researchers often need to understand their behaviour, the second goal of this paper is to consider the trade-off between models’ performance and interpretability in the context of design analytics for m-learning. To achieve these goals, we compiled a dataset of designs deployed using two tools, Avastusrada and Smartzoos. With this dataset, we trained and compared different models and feature extraction techniques. We further optimized and compared the best performing and most interpretable algorithms (EstBERT and Logistic Regression) to consider the second goal with an illustrative case. We found that SML can reliably classify designs with accuracy > 0.86 and Cohen’s kappa > 0.69.

Research indicates that data-informed practice helps teachers change their teaching and promotes teacher professional development (TPD). Although educational data are often collected from digital s...

Recent advances in machine learning and natural language processing have the potential to transform human activity in many domains. The field of learning analytics has applied these techniques successfully to many areas of education but has not been able to permeate others, such as doctoral education. Indeed, doctoral education remains an under-researched area with widespread problems (high dropout rates, low mental well-being) and lacks technological support beyond very specialized tasks. The inherent uniqueness of the doctoral journey may help explain the lack of generalized solutions (technological or otherwise) to these challenges. We propose a novel approach to apply the aforementioned advances in computation to support doctoral education. Single-case learning analytics defines a process in which doctoral students, researchers, and computational elements collaborate to extract insights about a single (doctoral) learner's experience and learning process (e.g., contextual cues, behaviors and trends related to the doctoral student’s sense of progress). The feasibility and added value of this approach are demonstrated using an authentic dataset collected by nine doctoral students over a period of at least two months. The insights from this feasibility study also serve to spark a research agenda for future technological support of doctoral education, which is aligned with recent calls for more human-centered approaches to designing and implementing learning analytics technologies.

Monitoring and guiding multiple groups of students in face-to-face collaborative work is a demanding task which could possibly be alleviated with the use of a technological assistant in the form of learning analytics. However, it is still unclear whether teachers would indeed trust, understand, and use such analytics in their classroom practice and how they would interact with such an assistant. The present research aimed to find out what the perception of in-service secondary school teachers is when provided with a dashboard based on audio and digital trace data when monitoring a collaborative learning activity. In a vignette study, we presented twenty-one in-service teachers with videos from an authentic collaborative activity, together with visualizations of simple collaboration analytics of those activities. The teachers perceived the dashboards as providers of useful information for their everyday work. In addition to assisting in monitoring collaboration, the involved teachers imagined using it for picking out students in need, getting information about the individual contribution of each collaborator, or even as a basis for assessment. Our results highlight the need for guiding dashboards as only providing new information to teachers did not compel them to intervene and additionally, a guiding dashboard could possibly help less experienced teachers with data-informed assessment.

Producción Científica, Multimodal Learning Analytics (MMLA) systems, understood as those that exploit multimodal evidence of learning to better model a learning situation, have not yet spread widely in educational practice. Their inherent technical complexity, and the lack of educational stakeholder involvement in their design, are among the hypothesized reasons for the slow uptake of this emergent field. To aid in the process of stakeholder communication and systematization leading to the specification of MMLA systems, this paper proposes a Multimodal Data Value Chain (M-DVC). This conceptual tool, derived from both the field of Big Data and the needs of MMLA scenarios, has been evaluated in terms of its usefulness for stakeholders, in three authentic case studies of MMLA systems currently under development. The results of our mixed-methods evaluation highlight the usefulness of the M-DVC to elicit unspoken assumptions or unclear data processing steps in the initial stages of development. The evaluation also revealed limitations of the M-DVC in terms of the technical terminology employed, and the need for more detailed contextual information to be included. These limitations also prompt potential improvements for the M-DVC, on the path towards clearer specification and communication within the multi-disciplinary teams needed to build educationally-meaningful MMLA solutions., Junta de Castilla y León (Project VA257P18), Ministerio de Ciencia, Innovación y Universidades (Project TIN2017-85179-C3-2-R), CEITER (grant agreements no. 669074)

Aim/Purpose: This paper explores an intervention approach (in the form of workshops) focusing on doctoral progress, to address the problems of low emotional well-being experienced by many doctoral candidates. Background: Doctoral education suffers from two severe overlapping problems: high dropout rates and widespread low emotional well-being (e.g., depression or anxiety symptoms). Yet, there are few interventional approaches specifically designed to address them in the doctoral student population. Among structural, psychosocial, and demographic factors influencing these problems, the self-perception of progress has emerged recently as a crucial motivational factor in doctoral persistence. Methodology: This paper reports on an iterative design-based research study of workshop interventions to foster such perception of progress in doctoral students’ everyday practice. We gathered mixed data over four iterations, with a total of 82 doctoral students from multiple disciplines in Spain and Estonia. Contribution: An approach to preventive interventions that combines research-backed education about mental health and productivity, peer sharing and discussion of experiences, and indicators of progress, as well as self-tracking, analysis, and reflection upon everyday evidence of their own progress. The paper provides initial evidence of the effectiveness of the proposed interventions, across two institutions in two different countries. Further, our data confirms emergent research on the relationships among progress, emotional well-being, and dropout ideation in two new contexts. Finally, the paper also distills design knowledge about doctoral interventions that focus on progress, relevant for doctoral trainers, institutions, and researchers. Findings: Our quantitative and qualitative results confirm previous findings on the relationships among progress, burnout, and dropout ideation. Our iterative evaluation of the workshops also revealed a large positive effect in students’ positive psychological capital after the workshops (Cohen’s d=0.83). Our quantitative and qualitative analyses also started teasing out individual factors in the variance of these benefits. Recommendations for Practitioners: Intervention design guidelines for doctoral trainers include: focusing on actionable productivity and mental health practices, the use of activities targeting perception biases and taboos, or the use of active practices and real (anonymous) data from the participants to make progress visible and encourage reflection. Recommendation for Researchers: The construct of progress, its components, and its relationships with both emotional well-being and doctoral dropout need to be more deeply studied, using multiple methods of data collection, especially from more frequent, ecologically valid data sources (e.g., diaries). Impact on Society: The proposed interventions (and focusing on doctoral progress more generally) hold promise to address the current emotional well-being and dropout challenges facing hundreds of thousands of doctoral students worldwide, ultimately helping increase the research and innovation potential of society as a whole. Future Research: More rigorous evaluative studies of the proposed approach need to be conducted, with larger samples and in other countries/contexts. Aside from the proposed one-shot training events, complementary longitudinal interventions focusing on supporting everyday progress and reflection throughout the doctoral process should be trialed.

Producción Científica, Multimodal Learning Analytics (MMLA) researchers are progressively employing machine learning (ML) techniques to develop predictive models to improve learning and teaching practices. These predictive models are often evaluated for their generalizability using methods from the ML domain, which do not take into account MMLA’s educational nature. Furthermore, there is a lack of systematization in model evaluation in MMLA, which is also reflected in the heterogeneous reporting of the evaluation results. To overcome these issues, this paper proposes an evaluation framework to assess and report the generalizability of ML models in MMLA (EFAR-MMLA). To illustrate the usefulness of EFAR-MMLA, we present a case study with two datasets, each with audio and log data collected from a classroom during a collaborative learning session. In this case study, regression models are developed for collaboration quality and its sub-dimensions, and their generalizability is evaluated and reported. The framework helped us to systematically detect and report that the models achieved better performance when evaluated using hold-out or cross-validation but quickly degraded when evaluated across different student groups and learning contexts. The framework helps to open up a “wicked problem” in MMLA research that remains fuzzy (i.e., the generalizability of ML models), which is critical to both accumulating knowledge in the research community and demonstrating the practical relevance of these techniques., Fondo Europeo de Desarrollo Regional - Agencia Nacional de Investigación (grants TIN2017-85179-C3-2-R and TIN2014-53199-C3-2-R), Fondo Europeo de Desarrollo Regional - Junta de Castilla y León (grant VA257P18), Comisión Europea (grant 588438-EPP-1- 2017-1-EL-EPPKA2-KA)

The usage of educational technologies does not necessarily imply the adoption of the pedagogical approaches they are designed to support. Existing works analysing learning design practices often focus on the usage metrics of the authoring platform, the authoring process or structural aspects of the designs themselves. While such usage metrics are useful to understand technology adoption, to understand pedagogical adoption we need to take into account the content of the designs created by practitioners as well. For example, in the case of inquiry-based learning, such content-related aspects include whether the learning designs scaffold the inquiry, promote engagement and collaboration. This paper proposes a concrete content-oriented design analysis approach for inquiry-based learning, which can be applied to digitally-authored inquiry designs. To illustrate its application and usefulness, we have applied this framework to learning designs created within Go-Lab (an initiative to promote inquiry learning in primary and secondary school). More concretely, we manually analyzed 44 learning designs published by Estonian practitioners using content analysis. Despite the small scale of the illustrative case study, the results from the content analysis show the potential of our analytical approach to inform teacher training and the development of authoring tools.

Certain educational contexts like lifelong learning have been comparatively understudied, due to the uniqueness of such learning processes, which make it difficult to ascertain generalizable models or average intervention effects. The ability to collect large amounts of data from a single learner longitudinally, throughout their lifelong trajectory, and the use of learning analytics (LA), presents novel opportunities in this regard. However, quantitative data and models may not be enough to deeply understand such unique learning processes and their context, without the help of qualitative data and ethnographic methods of collecting and analyzing it. This paper presents an approach to understanding lifelong learning that combines both qualitative and quantitative data (which we have termed single-case learning analytics). We illustrate how these two kinds of longitudinal data can be combined and the role of Epistemic Network Analysis (ENA) in this approach, through a case study in which daily quantitative and qualitative data were gathered about a lifelong learner’s social-emotional learning (SEL), for about nine months. Our results show how qualitative data (and ENA outputs) can be used to not only close the interpretive loop but also help interpret and improve the accuracy of quantitative models of a single learner’s process and contextual influences. Along with ongoing advances in automated coding, this approach opens the door to user-facing, ENA-enhanced LA to support personalized learning over extended periods of time.

Multimodal Learning Analytics (MMLA) aims to support evidence-based educational practices by collecting, processing, analyzing and sense-making of multimodal evidence of learning. MMLA is not widespread yet and there are few tailored MMLA solutions to meet the requirements of a specific learning scenario. This PhD project investigates the main challenges behind the limited development of MMLA solutions and proposes an MMLA infrastructure as the main contribution. The proposed infrastructure includes three components - a data value chain, a data model and a software architecture. The overall project follows the design-based research methodology where multiple iterations are involved to refine the contributions.

Producción Científica, The estimation of collaboration quality using manual observation and coding is a tedious and difficult task. Researchers have proposed the automation of this process by estimation into few categories (e.g., high vs. low collaboration). However, such categorical estimation lacks in depth and actionability, which can be critical for practitioners. We present a case study that evaluates the feasibility of quantifying collaboration quality and its multiple sub-dimensions (e.g., collaboration flow) in an authentic classroom setting. We collected multimodal data (audio and logs) from two groups collaborating face-to-face and in a collaborative writing task. The paper describes our exploration of different machine learning models and compares their performance with that of human coders, in the task of estimating collaboration quality along a continuum. Our results show that it is feasible to quantitatively estimate collaboration quality and its sub-dimensions, even from simple features of audio and log data, using machine learning. These findings open possibilities for in-depth automated quantification of collaboration quality, and the use of more advanced features and algorithms to get their performance closer to that of human coders., European Union via the European Regional Development Fund and in the context of CEITER and Next-Lab (Horizon 2020 Research and Innovation Programme, grant agreements no. 669074 and 731685), Junta de Castilla y León (Project VA257P18), Ministerio de Ciencia, Innovación y Universidades (Project TIN2017-85179-C3-2-R)

Orchestration load is the effort a teacher spends in coordinating multiple activities and learning processes. It has been proposed as a construct to evaluate the usability of learning technologies at the classroom level, in the same way that cognitive load is used as a measure of usability at the individual level. However, so far this notion has remained abstract. In order to ground orchestration load in empirical evidence and study it in a more systematic and detailed manner, we propose a method to quantify it, based on physiological data (concretely, mobile eye-tracking measures), along with human-coded behavioral data. This paper presents the results of applying this method to four exploratory case studies, where four teachers orchestrated technology-enhanced face-to-face lessons with primary, secondary school, and university students. The data from these studies provide a first validation of this method in different conditions, and illustrate how it can be used to understand the effect of different classroom factors on orchestration load. From these studies, we also extract empirical insights about classroom orchestration using technology.

In learning situations that do not occur exclusively online, the analysis of multimodal evidence can help multiple stakeholders to better understand the learning process and the environment where it occurs. However, Multimodal Learning Analytics (MMLA) solutions are often not directly applicable outside the specific data gathering setup and conditions they were developed for. This paper focuses specifically on authentic situations where MMLA solutions are used by multiple stakeholders (e.g., teachers and researchers). In this paper, we propose an architecture to process multimodal evidence of learning taking into account the situation’s contextual information. Our adapter-based architecture supports the preparation, organisation, and fusion of multimodal evidence, and is designed to be reusable in different learning situations. Moreover, to structure and organise such contextual information, a data model is proposed. Finally, to evaluate the architecture and the data model, we apply them to four authentic learning situations where multimodal learning data was collected collaboratively by teachers and researchers.

Multimodal Learning Analytics (MMLA) has sparked researcher interest in investigating learning in real-world settings by capturing learning traces from multiple sources of data. Though multimodal data offers a more holistic picture of learning, its inherent complexity makes it difficult to understand and interpret. This paper illustrates the use of dimensionality reduction (DR) to find a simple representation of multimodal learning data collected from co-located collaboration in authentic settings. We employed multiple DR methods and used triangulation to interpret their result which in turn provided a more simplistic representation. Additionally, we also show how unexpected events in authentic settings (e.g., missing data) can affect the analysis results.

Despite the recent surge of interest in learning analytics (LA), their adoption in everyday classroom practice is still slow. Knowledge gaps and lack of inter-stakeholder communication (particularly with educational practitioners) have been posited as critical factors for previous LA adoption failures. Yet, what issues should researchers, practitioners and other actors communicate about, when considering the adoption of an LA innovation in a particular context? We reviewed and synthesised existing literature on four focus areas related to LA, their adoption, implications for practice, and more general factors that have emerged as crucial when studying everyday classroom adoption of technologies (i.e., classroom orchestration). This synthesis resulted in two conversational frameworks and an inter-stakeholder communication tool. These can be used to guide and support conversations and decision-making about the adoption of LA innovations. We illustrate their usefulness with examples of use in ongoing LA adoption processes in Australia, Spain and Estonia.

The collection of evidence from authentic classroom practice is important both for teacher professional development (TPD) and educational research. However, most current learning analytics (LA) fails to capture the physical occurrences of the classroom, and do not always address individual teachers' needs. More customizable forms of classroom data collection (e.g., through video-recordings or by human observers) are time-consuming and expensive to implement. Navigating this tradeoff between the personalization of data collection and the strict time/effort constraints of classroom practice is still an unsolved challenge for designers of LA and teaching analytics (TA) systems to be used in face-to-face classrooms. In this paper, we extract lessons learnt from a design-based research process that explores this tradeoff, towards the development of teacher-driven, personalizable data collection tools. Through a survey of 15 expert teachers and paper and software prototype testing with a total of 14 teachers in simulated and authentic settings, we gathered information about teachers' preferences for teacher-driven data collection in classrooms, and derived design insights (e.g., a cold-start problem and the need for multiple layers in the personalization), which can be useful for the design of TA/LA tools that collect personalized data from everyday classrooms.

The pedagogical modelling of everyday classroom practice is an interesting kind of evidence, both for educational research and teachers' own professional development. This paper explores the usage of wearable sensors and machine learning techniques to automatically extract orchestration graphs (teaching activities and their social plane over time), on a dataset of 12 classroom sessions enacted by two different teachers in different classroom settings. The dataset included mobile eye-tracking as well as audiovisual and accelerometry data from sensors worn by the teacher. We evaluated both time-independent and time-aware models, achieving median F1 scores of about 0.7-0.8 on leave-one-session-out k-fold cross-validation. Although these results show the feasibility of this approach, they also highlight the need for larger datasets, recorded in a wider variety of classroom settings, to provide automated tagging of classroom practice that can be used in everyday practice across multiple teachers.

In blended learning scenarios, evidence needs to be gathered from digital and physical spaces to obtain a more complete view of the teaching and learning processes. However, these scenarios are highly heterogeneous, and the varying data sources available in each particular context can condition the accuracy, relevance, interpretability and actionability of the Learning Analytics (LA) solutions, affecting also the user’s sense of agency and trust in such solutions. To aid stakeholders in making use of learning analytics, we propose a process to involve teachers in customizing multimodal LA (MMLA) solutions, adapting them to their particular blended learning situation (e.g., identifying relevant data sources and metrics). Since measuring the added value of adopting an LA solution is not straightforward, we also propose a concrete method for doing so. The results obtained from two case studies in authentic, blended computer-supported collaborative learning settings show an improvement in the sensitivity and F1 scores of the customized MMLA solution. Aside from these quantitative improvements, participant teachers reported both an increment in the effort involved, but also increased relevance, understanding and actionability of the results., Ministerio de Economía, Industria y Competitividad (Project TIN2014-53199-C3-2-R), Junta de Castilla y León (programa de apoyo a proyectos de investigación – Ref. VA277U14 and VA082U16)

Producción Científica, Learning design tools aim at supporting practitioners in their task of creating more innovative and effective computer-supported learning situations. Despite there being a myriad of proposed tools, their use presents challenges that recent studies link with practitioners' varied pedagogical approaches and context restrictions, as well as with barriers to practical application derived from the fact that most tools only cover limited functionality and do not support cooperation between practitioners. In this paper we investigate whether it is possible to provide a flexible community system that supports multiple learning design tasks. We propose an Integrated Learning Design Environment (ILDE), which is a networked system integrating collaboration functions, design editors and middleware that enables deployment of the designed learning situations into Virtual Learning Environments. We describe the iterative user-centered process adopted in the design of ILDE as well as its architecture. The architecture is implemented to show its feasibility and that it is capable of providing the targeted functionality. We also present the results of its use in training workshops with 148 practitioners from five different institutions in vocational training, higher and adult education. Some of the learning designs were deployed in VLEs and enacted with students in real learning situations.

Multimodal Learning Analytics (MMLA) captures, integrates and analyzes learning traces from different sources in order to obtain a more holistic understanding of the learning process, wherever it happens. MMLA leverages the increasingly widespread availability of diverse sensors, high-frequency data collection technologies and sophisticated machine learning and artificial intelligence techniques. The aim of this workshop is twofold: first, to expose participants to, and develop, different multimodal datasets that reflect how MMLA can bring new insights and opportunities to investigate complex learning processes and environments; second, to collaboratively identify a set of grand challenges for further MMLA research, built upon the foundations of previous workshops on the topic.

Producción Científica, echnology offers rich opportunities for learning across different physical and virtual spaces. However, most of current across-spaces proposals are either highly teacher-centered, inflexible in the students’ self-management of learning artifacts during the enactment, or allow the teacher little/no control of such students’ management of artifacts. Moreover, these proposals tend to be disconnected from the practices and tools that are usual in the classroom. How can we achieve a middle ground between keeping the teacher in control of across-spaces situations and, at the same time, providing students with a degree of flexibility to manage learning artifacts? Aiming to address such challenge we propose the notion of learning bucket, and the Bucket-Server, a system implementing such notion. A learning bucket is a container of learning artifacts which are generated and/or accessed across-spaces by the students during the enactment, according to constraints configured by teachers at design time. The responsive evaluation conducted, based on a feature analysis and a pilot study with experts, suggests that learning buckets can help evolve from teacher- to student-centered approaches, while maintaining the teacher in control of students’ actions. The evaluation also indicates that the Bucket-Server surpasses the support provided by alternative proposals to across-spaces learning., Ministerio de Economía, Industria y Competitividad (Project TIN2014- 53199-C3-2- R), Junta de Castilla y León (programa de apoyo a proyectos de investigación – Ref. VA277U14)

Although not unheard of, there is a scarcity of intermediate-level concepts (not as generalizable as theories, but with an applicability wider than a single technology or intervention) in technology-enhanced learning (TEL) research. In this paper we propose ‘strong TEL concepts’, as intermediate-level bodies of design knowledge that are both grounded in research evidence from multiple technologies and contexts, and have clear theoretical connections. We describe the main features of this kind of concepts, along with a practical method for developing them as valuable research contributions. We also propose ‘purposeful disengagement’ as an example of strong TEL concept, to ignite the dialogue in our community about the necessity and benefits of this kind of knowledge to support both TEL design and theory advancement.

Producción Científica, Across a broad range of design professions, there has been extensive research on design practices and considerable progress in creating new computer-based systems that support design work. Our research is focused on educational/instructional design for students' learning. In this sub-field, progress has been more limited. In particular, neither research nor systems development have paid much attention to the fact that design is becoming a more collaborative endeavor. This paper reports the latest research outcomes from R&D in the Educational Design Studio (EDS), a facility developed iteratively over four years to support and understand collaborative, real-time, co-present design work. The EDS serves to (i) enhance our scientific understanding of design processes and design cognition and (ii) provide insights into how designers' work can be improved through appropriate technological support. In the study presented here, we introduced a complex, multi-user, digital design tool into the existing ecology of tools and resources available in the EDS. We analysed the activity of four pairs of ‘teacher-designers’ during a design task. We identified different behaviors - in reconfiguring the task, the working methods and toolset usage. Our data provide new insights about the affordances of different digital and analogue design surfaces used in the Studio.

Education is experiencing a paradigm shift towards blended learning models in technology-enhanced learning (TEL). Despite the potential benefits of blended learning, it also entails additional complexity in terms of monitoring, awareness and reflection, as learning happens across different spaces and modalities. In recent years, literature on learning analytics (LA) and educational data mining (EDM) has gained momentum and started to address the issue. To provide a clear picture of the current state of the research on the topic and to outline open research gaps, this paper presents a systematic literature review of the state-of-the-art of research in LA and EDM on monitoring, awareness and reflection in blended TEL scenarios. The search included six main academic databases in TEL that were enriched with the proceedings of the workshop on 'Awareness and Reflection in TEL' (ARTEL), resulting in 1089 papers out of which 40 papers were included in the final analysis.