Search

There is a great deal of research about item response theory (IRT) conducted by simulations. Item and ability parameters are estimated with varying numbers of replications under different test conditions. However, it is not clear what the appropriate number of replications should be. The aim of the current study is to develop guidelines for the adequate number of replications in conducting Monte Carlo simulation studies involving unidimensional IRT models. For this aim, 192 simulation conditions which included four sample sizes, two test lengths, eight replication numbers, and unidimensional IRT models were generated. Accuracy and precision of item and ability parameter estimations and model fit values were evaluated by considering the number of replications. In this context, for the item and ability parameters; mean error, root mean square error, standard error of estimates, and for model fit; M[subscript 2], RMSEA[subscript 2], and Type I error rates were considered. The number of replications did not seem to influence the model fit, it was decisive in Type I error inflation and error prediction accuracy for all IRT models. It was concluded that to get more accurate results, the number of replications should be at least 625 in terms of accuracy of the Type I error rate estimation for all IRT models. Also, 156 replications and above can be recommended. Item parameter biases were examined, and the largest bias values were obtained from the 3PL model. It can be concluded that the increase in the number of parameters estimated by the model resulted in more biased estimates.

The instrument that assesses teachers' competence on AR methodology is limited. Thus, it is one of the issues concerning evaluating the effectiveness of a professional development program on designing AR projects. It is difficult to determine how much and what teachers have learned in a course or training. Thus, this cross-sectional study aimed to evaluate further the validity and reliability of the Teacher's Competence in Action Research Questionnaire, a seven-factor instrument previously proposed by Cortes, Pineda, and Geverola (2020). This self-report scale was not subjected to confirmatory factor analysis, had less sample size, and had homogenous participants. In the present study, 450 participants, both pre- and in-service teachers and from different teaching specializations, answered the survey. The data were analyzed using the confirmatory factor analysis method through the Maximum Likelihood approach. Four model fit indices recorded satisfactory results (CFI = 0.890; TLI = 0.884; RMSEA = 0.072; SRMR = 0.039), thus, supporting the seven-factor scale. The standardized factor loading, composite reliability, average variance extracted, and Cronbach's alpha coefficient/s of the entire scale and within subscales also provide evidence of the convergent validity and reliability of the scale. There may be an issue in the discriminant validity of the scale, but the conceptual distinctions of each factor as supported by theoretical foundation and arguments provide a principal reason for retaining all the items and factors.

Distance learning has become a popular phenomenon across the world during the COVID-19 pandemic. This led to answer copying behavior among individuals. The cut point of the Kullback-Leibler Divergence (KL) method, one of the copy detecting methods, was calculated using the Youden Index, Cost-Benefit, and Min Score p-value approaches. Using the cut point obtained, individuals were classified as a copier or not, and the KL method was examined for cases where the determination power of the KL method was 1000, and 3000 sample size, 40 test length, copiers' rate was 0.05 and 0.15, and copying percentage was 0.1, 0.3 and 0.6. As a result, when the cut point was obtained with the Min Score p-value approach, one of the cutting methods approaches, it was seen that the power of the KL index to detect copier was high under all conditions. Similarly, under all conditions, it was observed that the second method, in which the detection power of the KL method was high, was the Youden Index approach. When the sample size and the copiers' rate increased, it was observed that the power of the KL method decreased when the cut point with the cost-benefit approach was used.

Weighted least squares (WLS), weighted least squares mean-and-variance-adjusted (WLSMV), unweighted least squares mean-and-variance-adjusted (ULSMV), maximum likelihood (ML), robust maximum likelihood (MLR) and Bayesian estimation methods were compared in mixed item response type data via Monte Carlo simulation. The percentage of polytomous items, distribution of polytomous items, categories of polytomous items, average factor loading, sample size and test length conditions were manipulated. ULSMV and WLSMV were found to be the more accurate methods under all simulation conditions. All methods except WLS had acceptable relative bias and relative standard error bias. No method gives accurate results with small sample sizes and low factor loading, however, the ULSMV method can be recommended to researchers because it gives more appropriate results in all conditions.

The aim of this study was to conduct a systematic literature review on the use of serious games in science education between 2016 and 2020 years. A total of 39 articles were included from Science Citation Index-Expanded (SCI-Expanded), Social Science Citation Index (SSCI), Arts & Humanities Citation Index (A&HCI) and Emerging Sources Citation Index (ESCI) listed in the Web of Science database. The results showed that experiential science was the most emphasized subject area in the articles. The results also indicated that learning/academic achievement was the common examined topic in the articles. For the research designs, more than half of the articles implemented quantitative research design. Considering the data collection tools, achievement tests were the most used data collection tool. Regarding the use of sampling methods, convenience sampling was preferred in more than half of the articles. Secondary and primary school students were the most used sampling groups. The sample size of 31-100-person group was commonly chosen in the articles. Considering the data analysis methods, a majority of articles implemented descriptive analyses methods. It was also found that computer game was the most used game type. Regarding the game genres, adventure was the most preferred one in serious games.

This study investigated the role of virtual reality (VR) in computer science (CS) education over the last 10 years by conducting a bibliometric and content analysis of articles related to the use of VR in CS education. A total of 971 articles published in peer-reviewed journals and conferences were collected from Web of Science and Scopus databases to conduct the bibliometric analysis. Furthermore, content analysis was conducted on 39 articles that met the inclusion criteria. This study demonstrates that VR research for CS education was faring well around 2011 but witnessed low production output between the years 2013 and 2016. However, scholars have increased their contribution in this field recently, starting from the year 2017. This study also revealed prolific scholars contributing to the field. It provides insightful information regarding research hotspots in VR that have emerged recently, which can be further explored to enhance CS education. In addition, the quantitative method remains the most preferred research method, while the questionnaire was the most used data collection technique. Moreover, descriptive analysis was primarily used in studies on VR in CS education. The study concludes that even though scholars are leveraging VR to advance CS education, more effort needs to be made by stakeholders across countries and institutions. In addition, a more rigorous methodological approach needs to be employed in future studies to provide more evidence-based research output. Our future study would investigate the pedagogy, content, and context of studies on VR in CS education.

The study aimed to identify the impact of game-based teaching practices available in different curricula on academic achievement. The study adopted meta-analysis method via grouping similar studies about that subject, theme under the heading of certain criteria, and combining the quantitative data obtained. For this purpose, 412 studies conducted between the years of 2000-2020 were examined and 54 experimental studies on the game-based teaching practices were included in the study. The data of 54 experimental studies were analyzed via the Comprehensive Meta-Analysis Program (CMA) and MetaWin programs. The study results revealed that the game-based teaching practices applied in different curricula affect the students' academic achievement positively compared to traditional practices. The effect sizes of game-based curricula were identified to significantly differ across education levels, treatment duration, sample sizes and publication types.

The study aimed to examine the effects of secondary school students' gender variable and the variables that could be moderators on their attitudes towards physical education and sports classes. The study adopted the meta-analysis method, which is one of the methods of synthesizing research results, constituted the research model. Within the scope of the study, researches on secondary school students' attitudes towards physical education and sports classes were examined, and the findings obtained constituted the main data source of this study. As a result of the meta-analysis, it was revealed that there was no publication bias. According to the fixed effects model, the average effect size value was d = 0.083 in favor of male students, and 0.081 in the random effects model. As a result of the moderator analysis, it was determined that the researcher (s) was not a moderator for the calculated average effect size of the variables such as gender, research region, research type, class level, research year and sample number. In short, the gender variable had almost no effect on secondary school students' attitudes towards physical education and sports lessons.

This study aims to examine the effectiveness of Computer-Supported Collaborative Learning (CSCL) on academic achievement. The study was conducted using the meta-analysis method. In the present study, a total of 40 studies that were carried out between 2010 and 2020 and met the inclusion criteria were subjected to meta-analysis. In the present study, the values of the effect size and combined effect size of each study included in the meta-analysis were calculated using Comprehensive Meta-Analysis (CMA) software. The sample of the study consists of 3474 participants. The results of the study revealed that the studies were usually conducted at the university stage with a medium sample size, and published as articles. It was also revealed that most of the studies were carried out in the field of sciences and social sciences. Considering the intervention durations, it was observed that the studies were particularly carried out within a period between 1 and 4 weeks (37.8%). According to the results of the analysis, the average effect size was calculated as 0.523. Considering the results, it can be stated that CSCL has a positive and moderate effect on academic achievement. Also, the results of the moderator analysis revealed that the effect of CSCL on academic achievement did not change by the learning stage, domain subject, and the sample size but it changed by the intervention duration.

Collaborative inhibition (reduced recall in collaborative vs. nominal groups) is a robust phenomenon. However, it is possible that not everyone is as susceptible to collaborative inhibition, such as those higher in working memory capacity (WMC). In the current study, we examined the relationship between WMC and collaborative inhibition. Participants completed three shortened span tasks (automated operation span, automated reading span, symmetry span). They then viewed categorized word lists individually and then recalled the word lists alone or with a partner (Test 1), followed by an individual recall (Test 2). For correct recall, collaborative inhibition was greater among lower WMC individuals, and they showed no post collaborative benefits. Only higher WMC individuals benefited from prior collaboration. For false recall, higher WMC individuals had less false recall on Tests 1 and 2, and collaboration reduced errors on Test 1 for both lower- and higher WMC individuals. There were no lasting effects of collaboration on Test 2 errors. Furthermore, partner WMC appeared to influence recall, although this tentative finding is based on a smaller sample size. Specifically, on Test 2, participants had less false recall when their partner was higher in WMC and greater correct recall when both they and their partner were higher in WMC. We conclude that collaboration is relatively more harmful for lower WMC individuals and more beneficial for higher WMC individuals. These results inform theories of collaborative inhibition by identifying attentional control and WMC as mechanisms that moderate the magnitude of the effect.

Intersectional approaches have become increasingly important for explaining educational inequalities because they help to improve our understanding of how individual experiences are shaped by simultaneous membership in multiple social categories that are associated with interconnected systems of power, privilege, and oppression. For years, there has been a call in psychological and educational research for quantitative approaches that can account for the intersection of multiple social categories. The present paper introduces the Multilevel Analysis of Individual Heterogeneity and Discriminatory Accuracy (MAIHDA) approach, a novel intersectional approach from epidemiology, to study educational inequalities. The MAIHDA approach uses a multilevel model as the statistical framework to define intersectional strata that represent individuals' membership in multiple social categories. By partitioning the variance within and between intersectional strata, the MAIHDA approach allows identifying intersectional effects at the strata level as well as obtaining information on the discriminatory accuracy of these strata for predicting individual educational outcomes. Compared to conventional quantitative intersectional approaches, MAIHDA analyses have several advantages, including better scalability for higher dimensions, model parsimony, and precision-weighted estimates of strata with small sample sizes. We provide a systematic review of its past application and illustrate its use by analyzing inequalities in reading achievement across 40 unique intersectional strata (combining the social categories of gender, immigrant background, parental education, and parental occupational status) using data from 15-year-old students in Germany (N = 5451). We conclude that the MAIHDA approach is a valuable intersectional tool to study inequalities in educational contexts.

Analysis of the differential treatment effects across targeted subgroups and contexts is a critical objective in many evaluations because it delineates for whom and under what conditions particular programs, therapies or treatments are effective. Unfortunately, it is unclear how to plan efficient and effective evaluations that include these moderated effects when the design includes partial nesting (i.e., disparate grouping structures across treatment conditions). In this study, we develop statistical power formulas to identify requisite sample sizes and guide the planning of evaluations probing moderation under two-level partially nested designs. The results suggest that the power to detect moderation effects in partially nested designs is substantially influenced by sample size, moderation effect size, and moderator variance structure (i.e., varies within groups only or within and between groups). We implement the power formulas in the R-Shiny application PowerUpRShiny and demonstrate their use to plan evaluations.

In cluster randomized evaluations, a treatment or intervention is randomly assigned to a set of clusters each with constituent individual units of observations (e.g., student units that attend schools, which are assigned to treatment). One consideration of these designs is how many units are needed per cluster to achieve adequate statistical power. Typically, researchers state that "about 30 units per cluster" is the most that will yield benefit towards statistical precision. To avoid rules of thumb not grounded in statistical theory and practical considerations, and instead provide guidance for this question, the ratio of the minimum detectable effect size (MDES) to the larger MDES with one less unit per cluster is related to the key parameters of the cluster randomized design. Formulas for this subsequent difference effect size ratio (SDESR) at a given number of units are provided, as are formulas for finding the number of units for an assumed SDESR. In general, the point of diminishing returns occurs with smaller numbers of units for larger values of the intraclass correlation. [For the corresponding grantee submission, see ED625939.]

We examined the performance of coefficient alpha and its potential competitors (ordinal alpha, omega total, Revelle's omega total [omega RT], omega hierarchical [omega h], greatest lower bound [GLB], and coefficient "H") with continuous and discrete data having different types of non-normality. Results showed the estimation bias was acceptable for continuous data with varying degrees of non-normality when the scales were strong (high loadings). This bias, however, became quite large with moderate strength scales and increased with increasing non-normality. For Likert-type scales, other than omega h, most indices were acceptable with non-normal data having at least four points, and more points were better. For different exponential distributed data, omega RT and GLB were robust, whereas the bias of other indices for binomial-beta distribution was generally large. An examination of an authentic large-scale international survey suggested that its items were at worst moderately non-normal; hence, non-normality was not a big concern. We recommend (a) the demand for continuous and normally distributed data for alpha may not be necessary for less severely non-normal data; (b) for severely non-normal data, we should have at least four scale points, and more points are better; and (c) there is no single golden standard for all data types, other issues such as scale loading, model structure, or scale length are also important.

To detect differential item functioning (DIF), Rasch trees search for optimal split-points in covariates and identify subgroups of respondents in a data-driven way. To determine whether and in which covariate a split should be performed, Rasch trees use statistical significance tests. Consequently, Rasch trees are more likely to label small DIF effects as significant in larger samples. This leads to larger trees, which split the sample into more subgroups. What would be more desirable is an approach that is driven more by effect size rather than sample size. In order to achieve this, we suggest to implement an additional stopping criterion: the popular Educational Testing Service (ETS) classification scheme based on the Mantel-Haenszel odds ratio. This criterion helps us to evaluate whether a split in a Rasch tree is based on a substantial or an ignorable difference in item parameters, and it allows the Rasch tree to stop growing when DIF between the identified subgroups is small. Furthermore, it supports identifying DIF items and quantifying DIF effect sizes in each split. Based on simulation results, we conclude that the Mantel-Haenszel effect size further reduces unnecessary splits in Rasch trees under the null hypothesis, or when the sample size is large but DIF effects are negligible. To make the stopping criterion easy-to-use for applied researchers, we have implemented the procedure in the statistical software R. Finally, we discuss how DIF effects between different nodes in a Rasch tree can be interpreted and emphasize the importance of purification strategies for the Mantel-Haenszel procedure on tree stopping and DIF item classification.

Multilevel structural equation models (MSEMs) are well suited for educational research because they accommodate complex systems involving latent variables in multilevel settings. Estimation using Croon's bias-corrected factor score (BCFS) path estimation has recently been extended to MSEMs and demonstrated promise with limited sample sizes. This makes it well suited for planned educational research which often involves sample sizes constrained by logistical and financial factors. However, the performance of BCFS estimation with MSEMs has yet to be thoroughly explored under common but difficult conditions including in the presence of non-normal indicators and model misspecifications. We conducted two simulation studies to evaluate the accuracy and efficiency of the estimator under these conditions. Results suggest that BCFS estimation of MSEMs is often more dependable, more efficient, and less biased than other estimation approaches when sample sizes are limited or model misspecifications are present but is more susceptible to indicator non-normality. These results support, supplement, and elucidate previous literature describing the effective performance of BCFS estimation encouraging its utilization as an alternative or supplemental estimator for MSEMs.

Cluster randomized control trials often incorporate a longitudinal component where, for example, students are followed over time and student outcomes are measured repeatedly. Besides examining how intervention effects induce changes in outcomes, researchers are sometimes also interested in exploring whether intervention effects on outcomes are modified by moderator variables at the individual (e.g., gender, race/ethnicity) and/or the cluster level (e.g., school urbanicity) over time. This study provides methods for statistical power analysis of moderator effects in two- and three-level longitudinal cluster randomized designs. Power computations take into account clustering effects, the number of measurement occasions, the impact of sample sizes at different levels, covariates effects, and the variance of the moderator variable. Illustrative examples are offered to demonstrate the applicability of the methods.

In meta-analyses of rare events, it can be challenging to obtain a reliable estimate of the pooled effect, in particular when the meta-analysis is based on a small number of studies. Recent simulation studies have shown that the beta-binomial model is a promising candidate in this situation, but have thus far only investigated its performance in a frequentist framework. In this study, we aim to make the beta-binomial model for meta-analysis of rare events amenable to Bayesian inference by proposing prior distributions for the effect parameter and investigating the models' robustness to different specifications of priors for the scale parameter. To evaluate the performance of Bayesian beta-binomial models with different priors, we conducted a simulation study with two different data generating models in which we varied the size of the pooled effect, the degree of heterogeneity, the baseline probability, and the sample size. Our results show that while some caution must be exercised when using the Bayesian beta-binomial in meta-analyses with extremely sparse data, the use of a weakly informative prior for the effect parameter is beneficial in terms of mean bias, mean squared error, and coverage. For the scale parameter, half-normal and exponential distributions are identified as candidate priors in meta-analysis of rare events using the Bayesian beta-binomial model.

Morphological awareness has been assessed longitudinally for monolinguals and bilinguals to trace the developmental trend. Researchers have found the important role it plays in literacy development including vocabulary growth and reading development. Conclusions about the important role morphological awareness play in literacy development are dependent upon valid methods. Unfortunately, some morphological awareness measurement issues have persisted in the literature. This article addressed some issues of morphological awareness assessment that persisted in the literature, which would cause inaccurate findings for studies. Twenty-six longitudinal studies that have assessed morphological awareness in Chinese and English at multiple time points have been reviewed, to investigate and address issues of measuring morphological awareness longitudinally. Four major issues were identified in the current article including the issue of high attrition and small sample size, using the same measure for multiple time points, inappropriate difficulty levels and types of the measurement used, and limitation of using one measurement. It provided several implications for future studies which could measure morphological awareness longitudinally.

A Monte Carlo simulation study was conducted to examine the performance of [alpha], [lambda]2, [lambda][subscript 4], [lambda][subscript 2], [omega][subscript T], GLB[subscript MRFA], and GLB[subscript Algebraic] coefficients. Population reliability, distribution shape, sample size, test length, and number of response categories were varied simultaneously. The results indicate that a and [lambda][subscript 2] perform the worst overall. However, the performance of a is improved if the population reliability is high. [lambda][subscript 4] is relatively unbiased but the most imprecise. [mu][subscript 2] and [omega][subscript T] perform relatively well under most conditions. GLB[subscript Algebraic] outperforms other coefficients under many conditions. GLB[subscript MRFA] is useful under few conditions if the population reliability is high. The results corroborate previous suggestions that large samples, longer tests, higher number of response categories, and normally distributed results can make reliability estimates more dependable. Some insights on the interaction of these factors are provided. We discuss the findings compared to previous research. The complete R code used for the simulation is provided in the online supplement.

Introduction: Psychological science is undergoing a period of change and transformation. Statement of the Problem: The crisis in confidence over psychological science has led to an emphasis on larger and larger sample sizes, sustaining an unfortunate neglect of single-subject research designs in undergraduate education. Literature Review: We identified several excellent articles advocating for the benefits of single-subject and small "N" designs over group-based research designs, yet single-case designs are seldom taught at undergraduate level. Teaching Implications: Teachers of psychology are provided with resources for implementing training in single-case research designs at undergraduate level, enabling students to draw objective conclusions in an "N"-of-1 research report. We do this using an example from a recently developed module on wellbeing science. Conclusion: Embedding an underused methodological approach for determining objective change in single individuals into undergraduate psychology curricula will help to develop practical skills applicable to many roles in the discipline of psychology, the healthcare sector and the quantified-self community.

Many generalization studies in education are typically based on a sample of 30-70 schools while the inference population is at least twenty times larger. This small sample to population size ratio limits the precision of design-based estimators of the population average treatment effect. Prior work has shown the potential of small area estimation methods to improve generalizations from small samples, specifically within the subclassification framework. However, small area estimation methods are model-based so that the validity of the estimates depends on the model assumptions. In this study, we explore a type of robust small area estimator and assess its performance in settings when core model assumptions are violated. We use a simulation study to compare the robust estimator with a small area estimator that is commonly used in practice and identify the conditions, if any, under which the robust estimator provides improvement. We illustrate the methods using an empirical example and discuss the implications for generalization studies with small samples.

While several studies have investigated the various effects of open educational resources (OER) and open educational practices (OEP), few have focused on its connection to learning achievement. The related scientific literature is divided about the effects of OER and OEP with regards to their contribution to learning achievement. To address this tension, a meta-analysis and research synthesis of 25 studies (N = 119,840 participants) was conducted to quantitatively investigate the effects of OER and OEP on students' learning achievement. The analysis included course subject, level of education, intervention duration, sample size, geographical distribution, and research design as moderating variables of the obtained effects. The findings revealed that OER and OEP have a significant yet negligible (g = 0.07, p < 0.001) effect. Additionally, the analysis found that the obtained effect can be moderated by several variables, including course subject, level of education and geographical distribution. The study findings can help various stakeholders (e.g., educators, instructional designers or policy makers) in understanding what might hinder OER and OEP effect on learning achievement, hence accommodating better learning outcomes and more effective interventions.

Articles published in 22 counseling journals from 2010-2019 were compared on 26 author and article characteristics. Women now constitute majorities of authorship and counseling journals are publishing larger proportions of research articles, 63% of which used quantitative approaches and about 11% of which studied counseling interventions.

The mass participation in higher education has led to greater spending by governments and students which has increased the focus on graduate outcomes. In England, the Office for Students (OfS) is planning to take regulatory action, using the Proceed metric, against universities and their courses which do not have 60% of students with positive outcomes within 15 months of graduation. This study uses simulations to explore how effectively the Proceed metric can (a) identify the true population level of positive outcomes, (b) explore the precision of those estimates, and (c) accurately classify courses below the 60% threshold. The simulation varied: level of positive outcomes within the population (20-90%), sample size (40-1000), and the percentage of the population sampled (30-90%). The bias (difference between sample and population estimate), coverage probability (proportion of true population estimates within the intervals), and precision of confidence intervals (average range of confidence intervals) were calculated. Three main findings were (a) the Proceed metric is accurate in terms of bias and coverage probability, (b) the estimates lack precision, especially with small sample sizes, and (c) the imprecision will impact the Proceed metric ability to correctly classify courses below the 60% threshold. Governments seeking to collect graduate survey data and use it to regulate universities should make every effort to maximise sample size and should resist the temptation to regulate at a micro-level (i.e. courses) since classification will likely be inaccurate, especially for those just below the threshold.

Propensity score matching and weighting methods are often used in observational effectiveness studies to reduce imbalance between treated and untreated groups on a set of potential confounders. However, much of the prior methodological literature on matching and weighting has yet to examine performance for scenarios with a majority of treated units, as is often encountered with programs and interventions that have been widely disseminated or "scaled-up." Using a series of Monte Carlo simulations, we compare the performance of k:1 matching with replacement and weighting methods with respect to covariate balance, bias, and mean squared error. Results indicate that the accuracy of all methods declined as treatment prevalence increased. While weighting produced the largest reduction in covariate imbalance, 1:1 matching with replacement provided the most unbiased treatment effect estimates. An applied example using empirical school-level data is provided to further illustrate the application and interpretation of these methods to a real-world scale-up effort. We conclude by considering the implications of propensity score methods for observational effectiveness studies with a particular focus on educational research. [For the corresponding grantee submission, see ED622050.]

Latent transition analysis is an informative statistical tool for depicting heterogeneity in learning as latent profiles. We present a Monte Carlo simulation study to guide researchers in selecting fit indices for identifying the correct number of profiles. We simulated data representing profiles of learners within a typical pre- post- follow up-design with continuous indicators, varying sample size (N from 50 to 1,000), attrition rate (none/10% per wave), and profile separation (entropy; from 0.73 to 0.87). Results indicate that the most commonly used fit index, the Bayesian information criterion ("BIC"), and the consistent Akaike information criterion ("CAIC") consistently underestimate the real number of profiles. A combination of the "AIC" or the "AIC3" with the adjusted Bayesian Information Criterion ("aBIC") provides the most precise choice for selecting the number of profiles and is accurate with sample sizes of at least N = 200. The "AIC3" excels starting from N = 500. Results were mostly robust toward differing numbers of time points, profiles, indicator variables, and alternative profiles. We provide an online tool for computing these fit indices and discuss implications for research.

When higher education students are assessed multiple times, teachers need to consider how these assessments can be combined into a single pass or fail decision. A common question that arises is whether students should be allowed to take a resit. Previous research has found little to no clear learning benefits of resits and therefore suggested they might not be advantangeous as they are costly for both students and institutions. However, we conducted a simulation study that shows such a conclusion to be presumptuous. Absence of evidence is not evidence of absence; our results illustrate that if a resit effect were to exist, the analysis used in these studies (i.e. regression discontinuity analysis; RDA) lacked the power to detect such an effect. Power of RDA was only sufficient under extremely implausible conditions (i.e. large sample, large effect size, high correlation between examinations). To adequately compare the effect of assessment policies, researchers are recommended to use other methods than RDA.

Rasch common-item equating is often used in high-stakes testing to maintain equivalent passing standards across test administrations. If unaddressed, item parameter drift poses a major threat to the accuracy of Rasch common-item equating. We compared the performance of well-established and newly developed drift detection methods in small and large sample sizes, varying the proportion of test items used as anchor (common) items and the proportion of drifted anchors. In the simulated-data study, the most accurate equating was obtained in large-sample conditions with a small-moderate number of drifted anchors using the mINFIT/mOUTFIT methods. However, when any drift was present in small-sample conditions and when a large number of drifted anchors were present in large-sample conditions, all methods performed ineffectively. In the operational-data study, percent-correct standards and failure rates varied across the methods in the large-sample exam but not in the small-sample exam. Different recommendations for high- and low-volume testing programs are provided.

Technology-mediated learning has become a valuable tool, especially in the English language classroom. Nowadays, mobile learning is considered a significant educational tool for promoting education among learners. Several recent studies have attempted to examine different aspects of mobile learning. The size of the experimental group, and the duration and suitability of the assessment methods are important aspects in designing experimental studies in the context of mobile English language learning. Yet, very few studies have paid attention to these aspects. Therefore, this study aimed to investigate the utilisation of state-of-the-art mobile technology in English language learning, especially in terms of the size of the experimental groups, and the duration and suitability of the assessment methods. Following a systematic review protocol, 151 articles published between 2010 and 2017 were selected for analysis. Furthermore, this study considered, separately, three factors addressed in the reviewed studies: language acquisition problems, types of participants, and targeted English skills. The findings showed that the "group size" and "study duration" significantly differed among the studies based on the above factors. Questionnaires and tests were found to be the most common methods of "assessment" for each factor, whether used separately or in combination. The results of this review can assist in the formulation of experimental designs to mitigate the challenges and facilitate the effective use of mobile learning in English language learning.

Online calibration is a key technology for item calibration in computerized adaptive testing (CAT) and has been widely used in various forms of CAT, including unidimensional CAT, multidimensional CAT (MCAT), CAT with polytomously scored items, and cognitive diagnostic CAT. However, as multidimensional and polytomous assessment data become more common, only a few published reports focus on online calibration in MCAT with polytomously scored items (P-MCAT). Therefore, standing on the shoulders of the existing online calibration methods/designs, this study proposes four new P-MCAT online calibration methods and two new P-MCAT online calibration designs and conducts two simulation studies to evaluate their performance under varying conditions (i.e., different calibration sample sizes and correlations between dimensions). Results show that all of the newly proposed methods can accurately recover item parameters, and the adaptive designs outperform the random design in most cases. In the end, this paper provides practical guidance based on simulation results.

The purpose of this study was to compare calibration and linking methods for placing pretest item parameter estimates on the item pool scale in a 1-3 computerized multistage adaptive testing design in terms of item parameter recovery. Two models were used: embedded-section, in which pretest items were administered within a separate module, and embedded-items, in which pretest items were distributed across operational modules. The calibration methods were separate calibration with linking (SC) and fixed calibration (FC) with three parallel approaches under each (FC-1 and SC-1; FC-2 and SC-2; and FC-3 and SC-3). The FC-1 and SC-1 used only operational items in the routing module to link pretest items. The FC-2 and SC-2 also used only operational items in the routing module for linking, but in addition, the operational items in second stage modules were freely estimated. The FC-3 and SC-3 used operational items in all modules to link pretest items. The third calibration approach (i.e., FC-3 and SC-3) yielded the best results. For all three approaches, SC outperformed FC in all study conditions which were module length, sample size and examinee distributions.

We compared coefficient alpha with five alternatives (omega total, omega RT, omega h, GLB, and coefficient H) in two simulation studies. Results showed for unidimensional scales, (a) all indices except omega h performed similarly well for most conditions; (b) alpha is still good; (c) GLB and coefficient H overestimated reliability with small samples and short scales, and (d) sensitivity to scale quality reduced with longer scales. For contaminated scales, (a) all indices except omega h were reasonably unbiased with non-severe contamination; (b) alpha, omega total, and GLB were more sensitive in picking up contamination with shorter scales, whereas omega RT and omega h were not; and (c) coefficient H could not pick up contaminated items among high-quality items. For applied researchers, (a) supplementary information of scale characteristics helps choose the appropriate index; (b) comparing different scales with one golden standard is inappropriate; (c) omega h should not be used alone.

Native American students appear to be overrepresented in the special education population of PreK-12 schools. In 2021, the National Center for Education Statistics (NCES) reported that a higher percentage of American Indian/Alaska Native children were served by the Individuals with Disabilities Education Act (IDEA) than children in any other racial/ethnic group. In this study, the researchers surveyed and interviewed teachers and administrators in schools with high Native American student populations. They analyzed the survey results and anecdotal observations and examined the relationships among the IDEA Special Education Indicator Reports, the provision of special education services, and Native American values and contextual factors. According to this State's Special Education Report Cards, almost without exception, schools with a high American Indian enrollment had much higher Child Count percentages. For example, it appeared that the State average for Child Count is 14.67%. However, in several schools with high Native American enrollment, the Child Count percentages were 20.75, 21.2, 21.17, 25.07, and 27.14.

The piecewise latent growth models (PWLGMs) can be used to study changes in the growth trajectory of an outcome due to an event or condition, such as exposure to an intervention. When there are multiple outcomes of interest, a researcher may choose to fit a series of PWLGMs or a single parallel-process PWLGM. A comparison of these models is provided with an illustrative example using data from the Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 (ECLS-K:2011), and a Monte Carlo simulation study. Conditions manipulated included sample size, location of the transition point, the number of time points, and covariate effect size. The results showed that the power to test parameter estimates with both models depended on the two-way interaction of sample size and covariate effect, and the three-way interaction of sample size by the number of time points by transition point location. Parameter coverage also depended on the three-way interaction of sample size by the number of time points by the transition point location. Recommendations for the use of the PWLGM are provided.

Background: Randomised trials have been on the rise in social policy over the last decade and a half, particularly in areas working with young people and vulnerable adults. Informed consent is an important principle for ethics committees governing research conducted by universities. Aims and objectives: We consider the arguments for and against opt-in consent by parents, and opt-out assent, when it comes to trials taking place, particularly in schools. Methods: We review what is known about this from a methodological standpoint. Findings: We find that extant evidence suggests that requiring opt-in consent, rather than assent, to participation, risks reducing the ethical standards of trials by minimising participation; and by potentially risking disclosure of sensitive information about a child's life to their parents. Moreover, there are important equity considerations, with more vulnerable groups likely to be excluded from research findings under an opt-in framework. Discussion and conclusion: We conclude that the ethical argument for assent rather than consent is compelling under some circumstances, and should be considered on a case-by-case basis. Precautions must always be taken to safeguard participants.

Virtual experiments are widely used in the field of education for their low cost, strong interaction, hyperspace and safety advantages. While some researchers have analysed the impact of virtual technologies such as AR or VR on teaching effectiveness, few researchers have comprehensively explored the impact of virtual experiments on students' learning effectiveness. In view of this, this study uses meta-analysis method to systematically review 62 experimental and quasi-experimental studies on the educational applications of virtual experiments published in international journals to systematically examine the impact of virtual experiments on students' learning effectiveness. The results of the study find that the impact of virtual experiments on learning effectiveness is influenced by five moderating variables (subjects, academic level, sample size, experimental period, and virtual forms). Based on this, this study makes recommendations in three areas: the design, conduct and evaluation of virtual experiments.

Screening items for parameter drift helps protect against serious validity threats and ensure score comparability when equating forms. Although many high-stakes credentialing examinations operate with small sample sizes, few studies have investigated methods to detect drift in small sample equating. This study demonstrates that several newly researched drift detection strategies can improve equating accuracy under certain conditions with small samples where some anchor items display item parameter drift. Results showed that the recently proposed methods "mINFIT" and "mOUTFIT" as well as the more conventional Robust-z helped mitigate the adverse effects of drifting anchor items in conditions with higher drift levels or with more than 75 examinees. In contrast, the Logit Difference approach excessively removed invariant anchor items. The discussion provides recommendations on how practitioners working with small samples can use the results to make more informed decisions regarding item parameter drift.

Do centre-based early education programmes prepare children for starting school? We wanted to find out if attending a centre-based early education programme was better than: (a) not attending any programme; (b) receiving a standard pre-school programme; and (c) if there were any unwanted effects. We found 32 studies involving 16,899 children aged 3 to 7 years old. Four studies compared centre-based early education interventions with no treatment. Twenty-two trials compared an enriched school curriculum to treatment as usual. The largest study was in 3,726 children and the smallest study was in 59 children. Studies were conducted around the world, mostly in the USA (28 studies). Studies, on average, lasted for 12 months; the longest trial lasted 6 years. Centre-based early education programmes for improving school readiness probably make little to no difference to children receiving either no programme or a standard curriculum. This systematic review found very low, low and moderate certainty evidence that centre-based school readiness interventions or enriched school readiness interventions make little to no difference in school readiness outcomes. Interventions may convey marginal benefits for children who do not attend preschool or who receive a standard preschool generic curriculum on specific academic areas such as mathematics, reading and vocabulary but any gains were not sustained over the long-term. There were a lack of studies measuring school readiness, health development and physical development. No studies assessed adverse effects.

Cluster randomized trials (CRTs) are commonly used to evaluate educational interventions, particularly their effectiveness. Recently there has been greater emphasis on using these trials to explore cost-effectiveness. However, methods for establishing the power of cluster randomized cost-effectiveness trials (CRCETs) are limited. This study develops power computation formulas and statistical software to help researchers plan two- and three-level CRCETs. We illustrate the application of our formulas and software for the designs of CRCETs and discuss the influence of sample size, nesting effects, covariates, and the covariance between cost and effectiveness measures on the statistical power of cost-effectiveness estimates.

Purpose: Based on the general formula, which depends on the length and difficulty of the test, the number of respondents and the number of ability levels, this study aims to provide a closed formula for the adaptive tests with medium difficulty (probability of solution is p = 1/2) to determine the accuracy of the parameters for each item and in the case of calibrated items, determine the required test length given number of respondents. Design/methodology/approach: Empirical results have been obtained on computerized or multistage adaptive implementation. Simulation studies and classroom/experimental results show that adaptive tests can measure test subjects' ability to the same quality over half the test length compared to linear versions. Due to the complexity of the problem, the authors discuss a closed mathematical formula: the relationship between the length of the tests, the difficulty of solving the items, the number of respondents and the levels of ability. Findings: The authors present a closed formula that provides a lower bound for the minimum test length in the case of adaptive tests. The authors also present example calculations using the formula, based on the assessment framework of some student assessments to show the similarity between the theoretical calculations and the empirical results. Originality/value: With this formula, we can form a connection between theoretical and simulation results.

We propose a new item response theory growth model with item-specific learning parameters, or ISLP, and two variations of this model. In the ISLP model, either items or blocks of items have their own learning parameters. This model may be used to improve the efficiency of learning in a formative assessment. We show ways that the ISLP model's learning parameters can be estimated in simulation using Markov chain Monte Carlo (MCMC), demonstrate a way that the model could be used in the context of adaptive item selection to increase the rate of learning, and estimate the learning parameters in an empirical data analysis using the ISLP. In the simulation studies, the one-parameter logistic model was used as the measurement model to generate random response data with various test lengths and sample sizes. Ability growth was modeled with a few variations of the ISLP model, and it was verified that the parameters were accurately recovered. Secondly, we generated data using the linear logistic test model with known Q-matrix structure for the item difficulties. Using a two-step procedure gave very comparable results for the estimation of the learning parameters even when item difficulties were unknown. The potential benefit of using an adaptive selection method in conjunction with the ISLP model was shown by comparing total improvement in the examinees' ability parameter to two other methods of item selection that do not utilize this growth model. If the ISLP holds, adaptive item selection consistently led to larger improvements over the other methods. A real data application of the ISLP was given to illustrate its use in a spatial reasoning study designed to promote learning. In this study, interventions were given after each block of ten items to increase ability. Learning parameters were estimated using MCMC.

There has been increasing interest in applying integrative data analysis (IDA) to analyze data across multiple studies to increase sample size and statistical power. Measures of a construct are frequently not consistent across studies. This article provides a tutorial on the complex decisions that occur when conducting harmonization of measures for an IDA, including item selection, response coding, and modeling decisions. We analyzed caregivers' self-reported data from the ADHD Teen Integrative Data Analysis Longitudinal (ADHD TIDAL) dataset; data from 621 of 854 caregivers were available. We used moderated nonlinear factor analysis (MNLFA) to harmonize items reflecting depressive symptoms. Items were drawn from the Symptom Checklist 90-Revised, the Patient Health Questionnaire--9, and the World Health Organization Quality of Life questionnaire. Conducting IDA often requires more programming skills (e.g., Mplus), statistical knowledge (e.g., IRT framework), and complex decision-making processes than single-study analyses and meta-analyses. Through this paper, we described how we evaluated item characteristics, determined differences across studies, and created a single harmonized factor score that can be used to analyze data across all four studies. We also presented our questions, challenges, and decision-making processes; for example, we explained the thought process and course of actions when models did not converge. This tutorial provides a resource to support prevention scientists to generate harmonized variables accounting for sample and study differences.

This simulation study investigated to what extent departures from construct similarity as well as differences in the difficulty and targeting of scales impact the score transformation when scales are equated by means of concurrent calibration using the partial credit model with a common person design. Practical implications of the simulation results are discussed with a focus on scale equating in health-related research settings. The study simulated data for two scales, varying the number of items and the sample sizes. The factor correlation between scales was used to operationalize construct similarity. Targeting of the scales was operationalized through increasing departure from equal difficulty and by varying the dispersion of the item and person parameters in each scale. The results show that low similarity between scales goes along with lower transformation precision. In cases with equal levels of similarity, precision improves in settings where the range of the item parameters is encompassing the person parameters range. With decreasing similarity, score transformation precision benefits more from good targeting. Difficulty shifts up to two logits somewhat increased the estimation bias but without affecting the transformation precision. The observed robustness against difficulty shifts supports the advantage of applying a true-score equating methods over identity equating, which was used as a naive baseline method for comparison. Finally, larger sample size did not improve the transformation precision in this study, longer scales improved only marginally the quality of the equating. The insights from the simulation study are used in a real-data example.