Search

The probability of superiority (PS) has been recommended as a simple-to-interpret effect size for comparing two independent samples-there are several methods for computing the PS for this particular study design. However, educational and psychological interventions increasingly occur in clustered data contexts; and a review of the literature returned only one method for computing the PS in such contexts. In this paper, we propose a method for estimating the PS in clustered data contexts. Specifically, the proposal addresses study designs that compare two groups and group membership is determined at the cluster level. A cluster may be: (i) a group of cases with each case measured once, or (ii) a single case with each case measured multiple times, resulting in longitudinal data. The proposal relies on nonparametric point estimates of the PS coupled with cluster-robust variance estimation, such that the proposed approach should remain adequate regardless of the distribution of the response data. Using Monte Carlo simulation, we show the approach to be unbiased for continuous and binary outcomes, while maintaining adequate frequentist properties. Moreover, our proposal performs better than the single extant method we found in the literature. The proposal is simple to implement in commonplace statistical software and we provide accompanying R code. Hence, it is our hope that the method we present helps applied researchers better estimate group differences when comparing two groups and group membership is determined at the cluster level.

In this paper, we present a model for comparing groups on scale score outcomes. The model has a number of features that make it desirable for analysis of scale scores. The model is based on ordinal regression, hence it is able to capture the shape of the data even when the data are highly discrete, or display marked ceiling or floor effects. Additionally, the model incorporates hierarchical modelling to create accurate summaries of the differences in the scale scores across groups. Statistically, the model assumes the data are ordinal, and hierarchically estimates the entire distribution of each group using factor smooths. Substantively, the model is capable of: estimating location-based, dispersion-based and ordinal descriptives estimates for each group; estimating the uncertainty about these estimates; and performing pairwise comparisons of the different estimates. The estimation method is Bayesian, however, we have created a GUI-based application that users may install on their computer to run the model, reducing the barrier to applying the method. The application takes in the raw data and user input, runs the model, and returns multiple model-based graphical summaries of patterns in the data, as well as tables for more precise reporting. We also share code that allows users extend the model to additional research contexts.

Multilevel modeling techniques have gained traction among experimental psychologists for their ability to account for dependencies in nested data structures. Increasingly, these techniques are extended to the analysis of binary data (e.g., choices, accuracy). Despite their popularity, these logistic multilevel models are often underutilized when researchers focus solely on fixed effects and ignore important heterogeneity that exists within and between participants, the random effects. In this tutorial, we review four techniques for estimating and quantifying residual- and cluster-level variance in logistic multilevel models in an accessible manner using real data. First, we introduce logistic multilevel modeling, including the interpretation of fixed and random effects. Second, we review the challenges associated with the estimation and interpretation of within- and between-participant variation in logistic multilevel models, particularly computing the intraclass correlation coefficient (ICC), which is usually a first, simple step in a linear MLM. Third, we demonstrate four existing methods of quantifying within- and between-participant variation in logistic multilevel models and discuss their relative advantages and disadvantages. Fourth, we present bootstrapping methods to make statistical inference about these variance estimates. To facilitate reuse, we developed R code to implement the discussed techniques, which is provided throughout the text and as supplemental materials.

The goal of this paper is to frame structural equation models (SEMs) as Bayesian multilevel regression models. Framing SEMs as Bayesian regression models provides an alternative approach to understanding SEMs that can improve model transparency and enhance innovation during modeling. For demonstration, we analyze six indicators of living standards data from 101 countries. The data are proportions and we develop confirmatory factor analysis as regression while accommodating the fact that the data are proportions. We also provide extensive guidance on prior specification, which is relevant for estimating complex regression models such as these. Finally, we run through regression equations for SEMs beyond the scope of the demonstration.