Search

Popular statistical software provides the Bayesian information criterion (BIC) for multi‐level models or linear mixed models. However, it has been observed that the combination of statistical literature and software documentation has led to discrepancies in the formulas of the BIC and uncertainties as to the proper use of the BIC in selecting a multi‐level model with respect to level‐specific fixed and random effects. These discrepancies and uncertainties result from different specifications of sample size in the BIC's penalty term for multi‐level models. In this study, we derive the BIC's penalty term for level‐specific fixed‐ and random‐effect selection in a two‐level nested design. In this new version of BIC, called BICE1, this penalty term is decomposed into two parts if the random‐effect variance–covariance matrix has full rank: (a) a term with the log of average sample size per cluster and (b) the total number of parameters times the log of the total number of clusters. Furthermore, we derive the new version of BIC, called BICE2, in the presence of redundant random effects. We show that the derived formulae, BICE1 and BICE2, adhere to empirical values via numerical demonstration and that BICE (E indicating either E1 or E2) is the best global selection criterion, as it performs at least as well as BIC with the total sample size and BIC with the number of clusters across various multi‐level conditions through a simulation study. In addition, the use of BICE1 is illustrated with a textbook example dataset. [ABSTRACT FROM AUTHOR]

Variability in treatment effects is common in intervention studies using cluster randomized controlled trial (C-RCT) designs. Such variability is often examined in multilevel modeling (MLM) to understand how treatment effects (TRT) differ based on the level of a covariate (COV), called TRT × COV. In detecting TRT × COV effects using MLM, relationships between covariates and outcomes are assumed to vary across clusters linearly. However, this linearity assumption may not hold in all applications and an incorrect assumption may lead to biased statistical inference about TRT × COV effects. In this study, we present generalized additive mixed model (GAMM) specifications in which cluster-specific functional relationships between covariates and outcomes can be modeled using by-variable smooth functions. In addition, the implementation for GAMM specifications is explained using the mgcv R package (Wood, 2021). The usefulness of the GAMM specifications is illustrated using intervention data from a C-RCT. Results of simulation studies showed that parameters and by-variable smooth functions were recovered well in various multilevel designs and the misspecification of the relationship between covariates and outcomes led to biased estimates of TRT × COV effects. Furthermore, this study evaluated the extent to which the GAMM can be treated as an alternative model to MLM in the presence of a linear relationship. [ABSTRACT FROM AUTHOR]

A cluster randomized controlled trial (C‐RCT) is common in educational intervention studies. Multilevel modelling (MLM) is a dominant analytic method to evaluate treatment effects in a C‐RCT. In most MLM applications intended to detect an interaction effect, a single interaction effect (called a conflated effect) is considered instead of level‐specific interaction effects in a multilevel design (called unconflated multilevel interaction effects), and the linear interaction effect is modelled. In this paper we present a generalized additive mixed model (GAMM) that allows an unconflated multilevel interaction to be estimated without assuming a prespecified form of the interaction. R code is provided to estimate the model parameters using maximum likelihood estimation and to visualize the nonlinear treatment‐by‐covariate interaction. The usefulness of the model is illustrated using instructional intervention data from a C‐RCT. Results of simulation studies showed that the GAMM outperformed an alternative approach to recover an unconflated logistic multilevel interaction. In addition, the parameter recovery of the GAMM was relatively satisfactory in multilevel designs found in educational intervention studies, except when the number of clusters, cluster sizes, and intraclass correlations were small. When modelling a linear multilevel treatment‐by‐covariate interaction in the presence of a nonlinear effect, biased estimates (such as overestimated standard errors and overestimated random effect variances) and incorrect predictions of the unconflated multilevel interaction were found. [ABSTRACT FROM AUTHOR]

Multilevel data structures are often found in multiple substantive research areas, and multilevel models (MLMs) have been widely used to allow for such multilevel data structures. One important step when applying MLM is the selection of an optimal set of random effects to account for variability and heteroscedasticity in multilevel data. Literature reviews on current practices in applying MLM showed that diagnostic plots are only rarely used for model selection and for model checking. In this study, possible random effects and a generic description of the random effects were provided to guide researchers to select necessary random effects. In addition, based on extensive literature reviews, level-specific diagnostic plots were presented using various kinds of level-specific residuals, and diagnostic measures and statistical tests were suggested to select a set of random effects. Existing and newly proposed methods were illustrated using two data sets: a cross-sectional data set and a longitudinal data set. Along with the illustration, we discuss the methods and provide guidelines to select necessary random effects in model-building steps. R code was provided for the analyses. [ABSTRACT FROM AUTHOR]

This article presents a multilevel longitudinal nested logit model for analyzing correct response and error types in multilevel longitudinal intervention data collected under a pretest–posttest, cluster randomized trial design. The use of the model is illustrated with a real data analysis, including a model comparison study regarding model complexity and cluster bias. Two substantive research questions regarding the intervention effect on correct response probability and error patterns are investigated using the proposed model. The recovery of item parameters for the proposed model using two sample size conditions is examined via a simulation study. The accuracy of the parameter estimates is comparable with those found in previous studies for the same family of models, except for the intercept parameters of correct responses. Finally, the impact of ignoring cluster membership in the model on the parameter estimation is also studied by fitting a single-level model to multilevel data. Ignoring cluster membership in the model adversely affects the estimation of intercept parameters in correct and error responses. [ABSTRACT FROM AUTHOR]

In a pre-test-post-test cluster randomized trial, one of the methods commonly used to detect an intervention effect involves controlling pre-test scores and other related covariates while estimating an intervention effect at post-test. In many applications in education, the total post-test and pre-test scores, ignoring measurement error, are used as response variable and covariate, respectively, to estimate the intervention effect. However, these test scores are frequently subject to measurement error, and statistical inferences based on the model ignoring measurement error can yield a biased estimate of the intervention effect. When multiple domains exist in test data, it is sometimes more informative to detect the intervention effect for each domain than for the entire test. This paper presents applications of the multilevel multidimensional item response model with measurement error adjustments in a response variable and a covariate to estimate the intervention effect for each domain. [ABSTRACT FROM AUTHOR]

This paper presents an explanatory multidimensional multilevel random item response model and its application to reading data with multilevel item structure. The model includes multilevel random item parameters that allow consideration of variability in item parameters at both item and item group levels. Item-level random item parameters were included to model unexplained variance remaining when item related covariates were used to explain variation in item difficulties. Item group-level random item parameters were included to model dependency in item responses among items having the same item stem. Using the model, this study examined the dimensionality of a person's word knowledge, termed lexical representation, and how aspects of morphological knowledge contributed to lexical representations for different persons, items, and item groups. [ABSTRACT FROM AUTHOR]