Search

In recent years, the scientific community has called for improvements in the credibility, robustness and reproducibility of research, characterized by increased interest and promotion of open and transparent research practices. While progress has been positive, there is a lack of consideration about how this approach can be embedded into undergraduate and postgraduate research training. Specifically, a critical overview of the literature which investigates how integrating open and reproducible science may influence student outcomes is needed. In this paper, we provide the first critical review of literature surrounding the integration of open and reproducible scholarship into teaching and learning and its associated outcomes in students. Our review highlighted how embedding open and reproducible scholarship appears to be associated with (i) students' scientific literacies (i.e. students’ understanding of open research, consumption of science and the development of transferable skills); (ii) student engagement (i.e. motivation and engagement with learning, collaboration and engagement in open research) and (iii) students' attitudes towards science (i.e. trust in science and confidence in research findings). However, our review also identified a need for more robust and rigorous methods within pedagogical research, including more interventional and experimental evaluations of teaching practice. We discuss implications for teaching and learning scholarship.

The emergence of large-scale replication projects yielding successful rates substantially lower than expected caused the behavioural, cognitive, and social sciences to experience a so-called ‘replication crisis’. In this Perspective, we reframe this ‘crisis’ through the lens of a credibility revolution, focusing on positive structural, procedural and community-driven changes. Second, we outline a path to expand ongoing advances and improvements. The credibility revolution has been an impetus to several substantive changes which will have a positive, long-term impact on our research environment.

Study preregistration has become increasingly popular in psychology, but its effectiveness in restricting potentially biasing researcher degrees of freedom remains unclear. We used an extensive protocol to assess the strictness of preregistrations and the consistency between preregistration and publications of 300 preregistered psychology studies. We found that preregistrations often lack methodological details and that undisclosed deviations from preregistered plans are frequent. Combining the strictness and consistency results highlights that biases due to researcher degrees of freedom are prevalent and likely in many preregistered studies. More comprehensive registration templates typically yielded stricter and hence better preregistrations. We did not find that effectiveness of preregistrations differed over time or between original and replication studies. Furthermore, we found that operationalizations of variables were generally more effectively preregistered than other study parts. Inconsistencies between preregistrations and published studies were mainly encountered for data collection procedures, statistical models, and exclusion criteria. Our results indicate that, to unlock the full potential of preregistration, researchers in psychology should aim to write stricter preregistrations, adhere to these preregistrations more faithfully, and more transparently report any deviations from the preregistrations. This could be facilitated by training and education to improve preregistration skills, as well as the development of more comprehensive templates.

In recent years, the scientific community has called for improvements in the credibility, robustness and reproducibility of research, characterized by increased interest and promotion of open and transparent research practices. While progress has been positive, there is a lack of consideration about how this approach can be embedded into undergraduate and postgraduate research training. Specifically, a critical overview of the literature which investigates how integrating open and reproducible science may influence student outcomes is needed. In this paper, we provide the first critical review of literature surrounding the integration of open and reproducible scholarship into teaching and learning and its associated outcomes in students. Our review highlighted how embedding open and reproducible scholarship appears to be associated with (i) students' scientific literacies (i.e. students’ understanding of open research, consumption of science and the development of transferable skills); (ii) student engagement (i.e. motivation and engagement with learning, collaboration and engagement in open research) and (iii) students' attitudes towards science (i.e. trust in science and confidence in research findings). However, our review also identified a need for more robust and rigorous methods within pedagogical research, including more interventional and experimental evaluations of teaching practice. We discuss implications for teaching and learning scholarship.

The ability to learn and generalize abstract algebraic rules (Marcus et al., 1999) has often been demonstrated in infants and is currently assessed in the large-scale multi-lab ManyBabies3 (MB3) study (Visser et al., accepted pending data collection). To study individual differences in infant development, it is essential to know how stably a certain ability can be measured over multiple measurement time points. Research on the reliability of infant looking time measures to date has been sparse and has found little or no evidence of test-retest reliability. Marimon & Höhle (2022) found relatively stable individual looking times across 3 test sessions with the same infants, but no correlations across sessions with regard to the manipulations of interest. Crista et al. (2016) reported good test-retest reliability for some but not all experiments in a larger battery of speech perception tasks with infants. Results from a previous ManyBabies study testing infant preference for infant-directed speech (Shreiner et al., 2023) found low test-retest reliability. Therefore, the aim of this study is to assess the test-retest reliability of looking-time measures of abstract rule learning in the context of the MB3 research project. (Note on contributions: Author list will be finalized after data collection at the time of finalizing the manuscript to be submitted. Authors of the final manuscript are expected to contribute to data collection and/or to provide substantive input in multiple stages of project preparation, data analysis, writing, and preparation of the final manuscript.)

A ‘sleep debt’ is widely reported among adolescents. Addressing this is of societal importance given that sleep deprivation has been linked to a range of cognitive and affective disturbances. Initial evidence suggests that modest gains in total sleep time over a week or less can improve attention and mood in adolescents; however, extant studies have methodological caveats and findings have been inconsistent. The present proof-of-concept study tests a protocol for extending sleep duration in teens. We also explored whether this has benefits for daytime sleepiness, mood, attention and word learning. Thirty-nine 13-15-year-olds who were sleeping less than the recommended 9 hours per night completed one week of baseline sleep monitoring before being randomly allocated to a control (business as usual) group or a four-night sleep extension group where individuals’ bedtimes were gradually brought forward. The extension protocol successfully increased total sleep time by ~50 minutes on average but did not alter sleep efficiency or time spent in different sleep stages. Decreases in subjective sleepiness were associated with increases in sleep duration. There were no observed benefits for mood or cognitive abilities. These data suggest that adolescent sleep can be modified over a short time span, and call for further research into how to optimize sleep to bolster cognitive and affective performance in adolescence.

The present meta-analysis aims to extend the findings obtained by Hamrick et al. (2018) by analysing the relationship between procedural learning and language and literacy skills, across more language and literacy components, in both populations with and without language and literacy impairments

The Serial Reaction Time task is one of the most commonly used tasks to measure procedural learning. Despite its frequent use both in children and adults, this task has been found to show sub-optimal test-retest reliability when compared to clinical measures, whose reliability coefficients are usually above .80 (Portney & Watkins, 2015). This pattern is especially noticeable in children (West et al., 2019). Since test-retest reliability refers to the ability of the test to yield similar results for repeated assessments of the participants behaviour, ensuring the stability of the measuring instrument is crucial as neither researchers nor clinicians can draw conclusions from inconsistent performances. In cases where the reliability is very poor, as has been found in the SRT task when used in children (West et al., 2018; 2019), inferences based on this task will be mostly unreliable (Wagenmakers & Gronau, 2017). Thus, efforts have been put forward to determine the best practices to address the stability issues of the Serial Reaction Time task. Study 1 (N = 103) investigated whether the similarity between sequences used during Session 1 and Session 2 would influence the amount and/or stability of procedural learning. The results demonstrated clear evidence of procedural learning effects in both sessions, as well as a positive relationship with similarity: more similar sequences in Session 2 showed larger learning effects than less similar sequences. In terms of reliability, although split-half reliability was acceptable (r > .55), test-retest reliability was poor, regardless of similarity (r < .25). Given the low levels of test retest reliability, Study 2 went on to replicate the design used by West et al (2019) which reports the highest levels of stability to our knowledge, Study 2 (N = 47, age range 17-34; mean = 20.09, SD = 2.09) went on to replicate the design used by West et al. (2019) as it had produced the highest levels of stability. Additionally, this study aimed to test whether the test-retest of the SRTT would increase over multiple sessions (Palmer et al., 2018), and whether the task would be sensitive to individual differences in language and literacy (as predicted by the Procedural Deficit Hypothesis (Ullman, 2004)), and sustained attention (in accordance with West et al. (2019)). As before, participants consistently showed evidence of procedural learning in all sessions, with practice effects decreasing from session 2 to 3. Split-half reliability was higher than test-retest reliability and stability of the SRT task was higher between sessions 2 and 3 than between sessions 1 and 2 (test-retest r = .60 and r = .43, respectively). Even though the test-retest reliability was considerably higher than for Study 1, particularly for the last sessions, it was still lower than the stability coefficient obtained by West et al. (2019; r = .66) after just two sessions, especially when using the same procedural learning measures, i.e. difference scores. The lower stability observed in Study 2 relative to West et al. (2019) may potentially be explained by two different design features: the age of the participants and the presence/absence of an interstimulus interval. Firstly, West et al. (2019) analysed the performance of older participants (18 - 61 years, mean = 25.33 years; SD = 10.33 years) than those who participated in Study 2 (17 - 34 years, mean = 20.09 years, SD = 2.09 years). This could have contributed to increasing the stability of the SRT task as test-retest reliability has been found to increase with age in intelligence measures (Schuerger & Witt, 1989). Secondly, West et al. (2019) included a 250 ms interstimulus interval (ISI) between trials which was absent in our previous experiment in an attempt to reduce explicit awareness (Destrebecqz & Cleeremans, 2001; 2003). The inclusion of an ISI, however, could have contributed to the higher test-retest reliability by inducing better quality representations (Cleeremans & Sarrazin, 2007; Gaillard, Destrebecqz, Michiels, & Cleeremans, 2009), with explicit awareness possibly emerging as a consequence of the gradual increases in strength and stability of the representations (Cleeremans, 2011; Timmermans et al., 2012). Thus, to determine whether these factor(s) contributed to the lower stability in our previous experiment, this study will compare the performance of participants in the SRT task in two separate groups, one group will perform the SRT task with an ISI of 250 ms, as in West et al. (2019), and the other will perform the task without an ISI, as in experiment 2. To further explore the effect of age on the stability of procedural learning, we will recruit participants in the age range of those included in West et al. (2019), with groups being age-matched to avoid age related differences between groups. Thus, it will be possible to compare the test-retest reliability of the two groups with ISI and no-ISI; as well as verify if there are age-related changes in test-retest reliability by dividing groups in younger and older adults. Another purpose of the present study is to assess the reliability of an online version of the SRT task as a reliable online measure may hold practical advantages for research and clinical purposes. Not only would it allow for mass testing of populations in naturalistic settings, but would also have the potential to increase reach to a more diverse array of participants and make it more likely that the sample will be representative of the population, relative to testing undergraduate populations (Raz et al., 2014). Finally, as predicted, Study 2 replicated the positive and significant correlation between procedural learning and attention observed by West et al. (2019). The relationship was particularly strong in sessions 1 (r = .44) and 2 (r = .49), with a decrease in session 3 (r = .25). This pattern could be potentially explained by Thomas et al. (2004) findings which showed less reliance on attentional resources once the sequence becomes more predictable since, even though distinct sequences are used per session, their level of similarity is still high. Thus, comparing the relationship between procedural learning and attention in both groups may provide a better understanding of the role of attention in the SRT task, as the predictability of the sequence would be expected to be higher in the group with ISI (Destrebecqz and Cleeremans, 2001; 2003).

Data from the project: Relationship between Dyslexia and DLD in European Portuguese school-aged children; https://doi.org/10.1080/13803395.2020.1870101

The Serial Reaction Time task, one of the most widely used tasks to index procedural memory, has been increasingly employed in individual differences research examining the role of procedural memory in language and other cognitive abilities. Yet, despite consistently producing robust procedural learning effects at the group level (i.e., faster responses to sequenced/probable trials versus random/improbable trials), these effects have recently been found to have poor reliability. In this meta-analysis (N datasets = 7), comprising 719 participants (M = 20.81, SD = 7.13), we confirm this “reliability paradox”. The overall retest reliability of the robust procedural learning effect elicited by the SRTT was found to be well below acceptable psychometric standards (r < .40). However, split-half reliability within a session is better, with an overall estimate of .68. There were no significant effects of sampling (participants’ age), methodology (number of trials, sequence type, inclusion of an interstimulus interval, version of the SRTT) and analytical decisions (whether all trials were included when computing the procedural learning scores; using difference scores, ratio scores, or random slopes as an index of learning). Future research will be better equipped to examine the influence of these factors on the reliability of the SRTT once reporting of the psychometric properties of experimental tasks becomes the norm instead of the exception. Thus, despite producing robust effects at the group-level, until we have a better understanding of the factors that improve the reliability of this task and/or have an improved understanding of how best to quantify retest reliability, using the SRTT for individual differences research should be done with caution.

The ability to extract patterns from sensory input across time and space is thought to underlie the development and acquisition of language and literacy skills, particularly the subdomains marked by the learning of probabilistic knowledge. Thus, impairments to the procedural learning mechanisms are hypothesised by the Procedural Deficit Hypothesis to underlie neurodevelopmental disorders such as dyslexia and developmental language disorder (DLD). In the present meta-analysis, comprising 2396 participants from 39 independent studies, the relationship between language, literacy and procedural memory on the Serial Reaction Time task (SRTT) was assessed across children and adults with typical development (TD), dyslexia and DLD. Based on the Procedural/Declarative model a positive relationship was expected between procedural memory and language and literacy measures for the typically developing group; however, no such relationship was observed. This was also the case for the disordered groups (ps < .05). Also counter to the Procedural Deficit Hypothesis, the magnitude of the relationship between procedural learning and grammar and phonology did not differ between TD and DLD (ps < .05), nor between the TD and dyslexic group on reading, spelling and phonology (ps < .05). Whilst lending little support to the Procedural/Declarative model and the Procedural Deficit Hypothesis, we consider that these results may be the consequence of poor psychometric properties of the SRTT as a measure of procedural learning.