Search

Background: A practical test that measures the information and communication technology (ICT) skills students need for effectively using ICT in primary education has yet to be developed (Oh et al., 2021). This paper reports on the development, validation, and reliability of a test measuring primary school students' ICT skills required for effectively using ICT (the ECC‐ICT test). Objectives: Based on existing literature, three ICT use domains were identified for effectively using ICT: Effective, collaborative, and creativeuse of ICT. For these three domains, 24 corresponding teaching objectives were identified from a widely used digital literacy framework. Thirty‐four test items cover these teaching objectives in an online test. Methods: A mixed‐method approach was used for the ECC‐ICT test. Four pilot rounds (n=25) implemented qualitative interviews for cognitive validity and refining the test items, followed by a qualitative usability study(n=6). Confirmatory factor analysis and ANOVA provided quantitative insight into the large‐scale test administration(n=575). Results and Conclusions: Composite reliability of our conceptual 3‐factor confirmatory model showed that the test reliably measured primary school effective use of ICT (ω = 0.82), collaborative use of ICT (ω = 0.80) and creative use of ICT (ω = 0.64). Convergent validity (ranging from 0.41 to 0.46) was acceptable. Internal consistency (ranging from 0.84 to 0.91) and discriminant validity (HTMT values below 0.90) are good. ANOVA results show that mean test scores are higher for students in higher grade levels (p < 0.001). The post hoc Bonferroni results show that most grade‐by‐grade comparisons are significant (p < 0.001). Lay Description: What is already known about this topic: Information and communication technology (ICT) skills are finding their way into national curriculums as teachers implement more digital learning environments.Teachers currently do not have a measurement instrument for the ICT skills primary education students need to learn in a personalised way. What this paper adds: This paper gives teachers a test that reliably measures the effective, collaborative, and creative ICT skills required for primary school students to learn using ICT.To our knowledge, this test is the first of its kind.Al materials to reproduce the test are available digitally for teachers. The implications of study findings for practitioners: For policy, this test can be used to check if a student is ready to use ICT in a self‐regulated setting.With this test, teachers can assess and support students' growth on the effective, collaborative, and creative ICT skills required for primary school students using ICT. [ABSTRACT FROM AUTHOR]

Personalized learning, a topic that has garnered significant attention in education, is known for its potential to cater to student's unique needs and improve educational outcomes. However, most large-scale longitudinal studies on personalized learning have primarily focused on middle school students and above (age ≥11). This study, in contrast, delves into the uncharted territory of how personalized learning affects younger students (ages 7–12), a domain largely overlooked by large-scale studies. To understand the effect of PL on young learners' academic performance, metacognitive awareness, and motivation. Multidisciplinary design teams embedded personalized learning in eight participating elementary schools, resulting in personalized learning interventions tailored to each school in four subjects. The effects were measured over three years among 588 students and 82 teachers and analyzed using a Bayesian Gaussian regression with random intercept models and nested groups. We found significant evidence that the personalized learning interventions fostered academic performance in two of the four subjects: math and spelling. Regarding spelling, we found that the schools in which metacognitive skills were explicitly trained improved their students' spelling performance significantly compared to other schools. We found significant evidence suggesting that student ICT skills improved metacognitive awareness, intrinsic motivation, and math performance. We also found significant evidence that teachers' ICT skills support student metacognitive awareness. However, we could not confirm the theorized effect of personalized learning on metacognitive awareness or students' intrinsic motivation. Our study provides evidence-based recommendations for implementing personalized learning interventions in elementary schools, particularly for math and spelling. Finally, improving ICT skills among students and teachers benefits students in math and in their metacognitive skills. • Personalized learning boosted academic performance in math and spelling. • School explicitly training in metacognition improved spelling performance. • Student digital literacy improved metacognition evaluation and intrinsic motivation. • Teacher digital literacy supported student metacognition knowledge. [ABSTRACT FROM AUTHOR]

Successful implementation of effective acquisition strategies (e.g., example study, problem-solving) could help improve mathematics performance. However, it is not yet fully understood when each acquisition strategy should be used, despite the practical value of this knowledge for mathematics textbook authors, teachers, and students. Building upon two recent perspectives on when example study and problem-solving are beneficial, we proposed that the optimal acquisition strategy could depend on both task complexity and retention interval (i.e., time between the final practice opportunity and the test). We conducted a multi-classroom experiment to test this proposition. 166 typically-developing Dutch fifth-grade students participated (M age = 11.14 years; 42.2% boys). We used a 2 (Task Complexity: simple vs. complex) x 2 (Acquisition Strategy: example study vs. problem-solving) x 2 (Retention Interval: 5 min vs. 1 week) between-subjects design with problem-solving performance as dependent variable. There was no evidence for the hypothesised three-way interaction effect of task complexity, acquisition strategy, and retention interval. However, there was evidence for the hypothesised two-way interaction effect of acquisition strategy and retention interval, independent of task complexity. More specifically, after 5 min, there was no statistically significant performance difference between students studying worked examples and those solving practice problems, but after 1 week, students solving practice problems outperformed those studying worked examples. Our findings imply that, after initial acquisition, problem-solving leads to better long-term problem-solving performance than example study. This holds true even for a relatively complex task and with limited instruction. • The effect of acquisition strategy on performance depended on retention interval. • After initial acquisition, problem-solving led to better long-term performance. • This held true even for a relatively complex task and with limited instruction. [ABSTRACT FROM AUTHOR]

Learners in the process of developing complex skills need a rich mental model of what such skills entail. Textual analytics rubrics (TR) are a widely used instrument to support formative assessment of complex skills, supporting feedback, reflection, and thus mental model development of complex skills. However, the textual nature of a rubric limits the possibility to deliver contextual and dynamic (time‐and behaviour‐oriented) information. In the Viewbrics online tool, we developed a version supporting the delivery of contextual and dynamic information by adding video‐modelling examples with embedded self‐explanation prompts to textual analytics rubrics. We called this combination as video‐enhanced rubrics (VERs). Our current study investigates whether the Viewbrics online tool supports complex skills development and whether either textual‐ or video‐enhanced rubrics best support complex skills' mastery. The study was a three‐group (VERS n = 49, TR n = 54, control n = 50) within‐subjects design. Learners' performance of complex skills was measured through expert, peer, and self‐assessment using the Viewbrics online tool. A multilevel regression analysis shows learners in the TR and VERS conditions consistently outperforming the control condition to varying degrees across skills. However, no differences have been found between the two experimental conditions. Positive results across different complex skills indicate the Viewbrics online tool can be used to support the development of a wide range of complex skills in secondary education. Lay Description: Known about the subject matter: The acquisition of cross‐curricular complex skills such is crucial for pupils in secondary education.It is important that students and teachers develop a concrete and consistent picture of (aspects of) these so‐called 21st century skills in order to streamline both their acquisition.Textual rubrics may still have room for improvement, as many aspects of desired behavior can hardly be put into words.Many schools in secondary education are struggling with how to teach and assess cross‐curricular skills. What their paper adds to this: So, we designed a online (online) formative assessment methodology, with textual rubrics and the video‐enhanced rubrics.And measured complex skill performance in two secondary, pre‐university education schools for 24 weeks and compared with existing educational practice.Both experimental groups perform significantly better in the three complex skills compared to the control group.However, the video‐enhanced group did not outperform the textual rubric group. The implications of study findings for practitioners: Supporting the formative assessment of complex skills is a recommendable activity for practitioners, that will likely bear fruit.However, incorporating video modeling examples into an online assessment methodology does not result in improvement in complex skill mastery. [ABSTRACT FROM AUTHOR]

Learners can experience difficulties to form a mental model of a complex skill and what its mastery implicates in terms of actual behavior. Text-based analytic rubrics can support training and formative assessment of complex skills. However, they lack (1) contextual information needed to convey real-world attributes (2) sensor-motoric information (such as gesturing in the complex skill of presenting) (3) procedural information needed to support the automation of (sub)skills. Text-based analytic rubrics deficiencies for skill-mastery can be remedied by adding video-modeling examples with embedded self-explanation prompts, turning text-based analytic rubrics (TR) into so-called 'video-enhanced analytic rubrics' (VERS). The current study contrasts two experimental conditions (TR, n = 54; VERS, n = 49) for fostering the growth of learners' mental model of complex skills. This was done using the formative assessment supporting Viewbrics online tool and one control condition with a secondary school's current approach for skills-development (without rubrics nor Viewbrics-formative assessment (n = 50)). We measured learners' mental models at three times during their project work by assessing the richness of their constructed concept maps. A multilevel regression analysis showed that learners in the VERS condition form richer mental models when compared to the control condition for most complex skills. • We designed an online formative assessment methodology, with textual rubrics and the video-enhanced rubrics. • 1377 open concept maps captured the mental model development of 3 complex skills. • The video-enhanced rubric condition enriched mental models for 2 out of 3 complex skills. [ABSTRACT FROM AUTHOR]

For learners, it can be difficult to imagine how to perform a complex skill based on textual information from solely a text-based analytic rubric. Rubrics lack (1) the contextual information needed to convey real-world attributes (2) the dynamic information (such as gesturing in the complex skill of presenting) (3) the procedural information required to support the automation of constituent skills. We propose to address the text-based rubric's deficiencies by adding video-modeling examples, self-explanation prompts, an intertwined educational- and instructional narrative, natural segmentation, and a non-verbal script. With the resulting Video Enhanced Rubric, we aim to improve the formative assessment of complex skills by fostering learner's mental model development, feedback quality, and complex skill mastery. Designing multimedia to support the formative assessment of complex skills using can cause dilemmas for instructional designers. For example, is learner control needed to foster intrinsic motivation or is it deemed to create extraneous cognitive load? Is it wise to use a video modeling example of peer-aged learners when the model does not display perfect performance? We found seven dilemmas around proven complex skill development, motivational design, and multimedia design guidelines. This paper presents a theoretical contribution to instructional design by introducing a framework to address dilemmas regarding such design dilemmas. As a practical contribution, we support educational researchers and practitioners by presenting six practical guidelines for designing a Video Enhanced Rubric. A use case of the Viewbrics-project provides insight into the practical application of the framework within the context of Dutch pre-university education. [ABSTRACT FROM AUTHOR]

Acquiring complex oral presentation skills is cognitively demanding for students and demands intensive teacher guidance. The aim of this study was twofold: (a) to identify and apply design guidelines in developing an effective formative assessment method for oral presentation skills during classroom practice, and (b) to develop and compare two analytic rubric formats as part of that assessment method. Participants were first-year secondary school students in the Netherlands (n = 158) that acquired oral presentation skills with the support of either a formative assessment method with analytic rubrics offered through a dedicated online tool (experimental groups), or a method using more conventional (rating scales) rubrics (control group). One experimental group was provided text-based and the other was provided video-enhanced rubrics. No prior research is known about analytic video-enhanced rubrics, but, based on research on complex skill development and multimedia learning, we expected this format to best capture the (non-verbal aspects of) oral presentation performance. Significant positive differences on oral presentation performance were found between the experimental groups and the control group. However, no significant differences were found between both experimental groups. This study shows that a well-designed formative assessment method, using analytic rubric formats, outperforms formative assessment using more conventional rubric formats. It also shows that higher costs of developing video-enhanced analytic rubrics cannot be justified by significant more performance gains. Future studies should address the generalizability of such formative assessment methods for other contexts, and for complex skills other than oral presentation, and should lead to more profound understanding of video-enhanced rubrics. [ABSTRACT FROM AUTHOR]