Search

Social desirability bias (SDB) has been a major concern in educational and psychological assessments when measuring latent variables because it has the potential to introduce measurement error and bias in assessments. Person-fit indices can detect bias in the form of misfitted response vectors. The objective of this study was to compare the performance of 14 person-fit indices to identify SDB in simulated responses. The area under the curve (AUC) of receiver operating characteristic (ROC) curve analysis was computed to evaluate the predictive power of these statistics. The findings showed that the agreement statistic (A) outperformed all other person-fit indices, while the disagreement statistic (D), dependability statistic (E), and the number of Guttman errors (G) also demonstrated high AUCs to detect SDB. Recommendations for practitioners to use these fit indices are provided.

Despite the proliferation of video-based instruction and its benefits--such as promoting student autonomy and self-paced learning--the complexities of online teaching remain a challenge. To be effective, educators require extensive training in digital teaching methodologies. As such, there's a pressing need to examine and comprehend the intricacies of instructors' communication patterns within this context. This research addresses the pressing need to understand pedagogical discourse in online video lectures in Algebra classes by employing computational linguistic tools and natural language processing (NLP). Using transcripts from 125 Algebra 1 video lectures--comprising 4962 instances of pedagogical discourse--from five instructors at Math Nation, a virtual math learning environment, we analyzed the conveyance of linguistic, attitudinal, and emotional nuances. With the aid of 26 Coh-Metrix and SÉANCE features, we classified educators' language choices, achieving an accuracy of 86.7%. Furthermore, variations in language choices, as signified by discourse markers, were examined through a K-means clustering approach. The resulting 17 clusters were grouped into interpersonal, structural, and cognitive pedagogic functions. Through this exploration, we demonstrate the promising potential of NLP in efficiently deciphering pedagogical communication patterns in video lectures. These insights open a new avenue for research, aimed at assessing the efficacy of digital instruction by scrutinizing pedagogical discourse characteristics in computer-based learning environments.

Objective: This study aims to: (1) examine gender differences for weight conscious drinking among college students accounting for the broader phenomenon (e.g. including the Alcohol Effects dimension); and (2) longitudinally examine the effect of weight conscious drinking behaviors on body mass index (BMI). Participants: United States freshmen students from eight participating universities (N= 1,149). Methods: Structural equation modeling was used to model the effect of gender on weight conscious drinking dimensions at 7-month follow-up. Results: Findings suggest a significant effect of gender on Alcohol Effects ([beta] = -0.15, SE = 0.05, p = 0.005) at 7-month follow-up among college freshmen. Weight conscious drinking dimensions predicted no significant change in BMI at 7-month follow-up among college freshmen. Conclusion: Findings contribute to weight conscious drinking theory and provide campus weight conscious drinking prevention initiatives with evidence to tailor their programming to address female tendencies to engage in compensatory strategies to enhance the psychoactive effects of alcohol.

The piecewise latent growth models (PWLGMs) can be used to study changes in the growth trajectory of an outcome due to an event or condition, such as exposure to an intervention. When there are multiple outcomes of interest, a researcher may choose to fit a series of PWLGMs or a single parallel-process PWLGM. A comparison of these models is provided with an illustrative example using data from the Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 (ECLS-K:2011), and a Monte Carlo simulation study. Conditions manipulated included sample size, location of the transition point, the number of time points, and covariate effect size. The results showed that the power to test parameter estimates with both models depended on the two-way interaction of sample size and covariate effect, and the three-way interaction of sample size by the number of time points by transition point location. Parameter coverage also depended on the three-way interaction of sample size by the number of time points by the transition point location. Recommendations for the use of the PWLGM are provided.

Academic discourse communities and learning circles are characterized by collaboration, sharing commonalities in terms of social interactions and language. The discourse of these communities is composed of jargon, common terminologies, and similarities in how they construe and communicate meaning. This study examines the extent to which discourse reveals "shared language" among its participants that can promote inclusion or affinity. Shared language is characterized in terms of linguistic features and lexical, syntactical, and semantic similarities. We leverage a multi-method approach, including (1) feature engineering using state-of-the-art natural language processing techniques to select the most appropriate features, (2) the bag-of-words classification model to predict linguistic similarity, (3) explainable AI using the local interpretable model-agnostic explanations to explain the model, and (4) a two-step cluster analysis to extract innate groupings between linguistic similarity and emotion. We found that linguistic similarity within and between the threaded discussions was significantly varied, revealing the dynamic and unconstrained nature of the discourse. Further, word choice moderately predicted linguistic similarity between posts within threaded discussions (accuracy = 0.73; F1-score = 0.67), revealing that discourse participants' lexical choices effectively discriminate between posts in terms of similarity. Lastly, cluster analysis reveals profiles that are distinctly characterized in terms of linguistic similarity, trust, and affect. Our findings demonstrate the potential role of linguistic similarity in supporting social cohesion and affinity within online discourse communities.

The unstructured multiple-attempt (MA) item response data in virtual learning environments (VLEs) are often from student-selected assessment data sets, which include missing data, single-attempt responses, multiple-attempt responses, and unknown growth ability across attempts, leading to a complex and complicated scenario for using this kind of data set as a whole in the practice of educational measurement. It is critical that methods be available for measuring ability from VLE data to improve VLE systems, monitor student progress in instructional settings, and conduct educational research. The purpose of this study is to explore the ability recovery of the multidimensional sequential 2-PL IRT model in unstructured MA data from VLEs. We conduct a simulation study to evaluate the effects of the magnitude of ability growth and the proportion of students who make two attempts, as well as the moderated effects of sample size, test length, and missingness, on the bias and root mean square error of ability estimates. Results show that the model poses promise for evaluating ability in unstructured VLE data, but that some data conditions can result in biased ability estimates.

Data sets from large-scale longitudinal surveys involving young children and families have become available for secondary analysis by researchers in a variety of fields. Researchers in early intervention have conducted secondary analyses of such data sets to explore relationships between nonmalleable and malleable factors and child outcomes, and to address issues of measurement. Survey data have been used to a lesser extent to examine plausible causal relationships between variables, perhaps due to the increased likelihood of selection bias that results with nonexperimental data. In this article, we use National Early Intervention Longitudinal Study data to demonstrate the use of inverse probability of treatment weighting, a quasi-experimental methodology based on propensity scores that can be used to reduce selection bias and examine plausible causal relationships. We discuss the advantages and disadvantages of this approach, and implications for its use in early intervention research.

Artificial neural networks (NN) can help researchers estimate propensity scores for quasi-experimental estimation of treatment effects because they can automatically detect complex interactions involving many covariates. However, NN is difficult to implement due to the complexity of choosing an algorithm for various treatment levels and monitoring model performance. This research aims to develop a tutorial to facilitate the use of NN to derive causal inferences. The tutorial provides social scientists with a gentle overview of machine learning terminology and best practices for training, validating, and testing NN to estimate propensity scores. The veracity of NN is demonstrated in this study using data on 5,770 teachers from the Beginner Teacher Longitudinal Study. Propensity score analysis was used to estimate the effects of assigning mentors to new teachers on the probability of continuing in the teaching profession. The results show that NN provided a better covariate balance between treatment versions than multinomial logistic regression and generalized boosted modeling. The study's findings align with previous research showing NN's advantages over conventional propensity score estimation methods.

The current study examines both student self-regulated learning (SRL) and teacher orchestration in a virtual learning environment (VLE), with respect to student achievement. The study used SRL indicators derived from the log data on how students used the VLE system, survey data on how teachers made use of the VLE for Algebra instruction, as well as formative and summative Algebra assessment scores. The sample included 6,174 students being taught by 93 teachers in 49 schools in the 2018-2019 academic year. Multilevel structural equation modeling (SEM) analysis revealed the relationship between SRL indicators and student achievement. We found a positive relationship between student SRL and student achievements. Specifically, three SRL indicators, "Watched Recommended Videos", "Watched Videos" and "Answered Quizzes after Watching a Video", were significantly associated with student achievement. We did not observe direct associations between teacher orchestration and student achievement, but we found an indirect association between teacher orchestration and student achievement via one SRL indicator, "Watched Recommended Videos". The results show that a positive relationship between EOC scores and student reviewing incorrect questions increased as the use of instructional videos by the teacher increased. This study supports the critical role of SRL and indicates that teachers should have flexibility in adapting learning activities for online learning.

This study compares automated methods to develop short forms of psychometric scales. Obtaining a short form that has both adequate internal structure and strong validity with respect to relationships with other variables is difficult with traditional methods of short-form development. Metaheuristic algorithms can select items for short forms while optimizing on several validity criteria, such as adequate model fit, composite reliability, and relationship to external variables. Using a Monte Carlo simulation study, this study compared existing implementations of the ant colony optimization, Tabu search, and genetic algorithm to select short forms of scales, as well as a new implementation of the simulated annealing algorithm. Selection of short forms of scales with unidimensional, multidimensional, and bifactor structure were evaluated, with and without model misspecification and/or an external variable. The results showed that when the confirmatory factor analysis model of the full form of the scale was correctly specified or had only minor misspecification, the four algorithms produced short forms with good psychometric qualities that maintained the desired factor structure of the full scale. Major model misspecification resulted in worse performance for all algorithms, but including an external variable only had minor effects on results. The simulated annealing algorithm showed the best overall performance as well as robustness to model misspecification, while the genetic algorithm produced short forms with worse fit than the other algorithms under conditions with model misspecification.

Objective: To examine the effect of weight-conscious drinking and compensatory behavior temporality on binge drinking frequency of college freshmen. Participants: Freshmen (n = 1149) from eight US universities, Fall 2015. Methods: Participants completed the Compensatory Eating Behaviors in Response to Alcohol Consumption Scale and Alcohol Use Disorders Identification Test--Consumption. Structural equation modeling was used to model the effect of weight-conscious drinking constructs on binge drinking frequency. Results: Bulimia, Dietary Restraint and Exercise, Restriction, proactive Alcohol Effects, during Alcohol Effects, and proactive Dietary Restraint and Exercise factors significantly predicted binge drinking frequency. Conclusion: Weight-conscious drinking among this cohort of college students comprises temporal factors significantly associated with binge drinking frequency. Relationships between Bulimia, Dietary Restraint and Exercise, and Restriction compensatory behaviors and binge drinking should be considered in interventions to address binge drinking among college students.

Social desirability bias (SDB) is a common threat to the validity of conclusions from responses to a scale or survey. There is a wide range of person-fit statistics in the literature that can be employed to detect SDB. In addition, machine learning classifiers, such as logistic regression and random forest, have the potential to distinguish between biased and unbiased responses. This study proposes a new application of these classifiers to detect SDB by considering several person-fit indices as features or predictors in the machine learning methods. The results of a Monte Carlo simulation study showed that for a single feature, applying person-fit indices directly and logistic regression led to similar classification results. However, the random forest classifier improved the classification of biased and unbiased responses substantially. Classification was improved in both logistic regression and random forest by considering multiple features simultaneously. Moreover, cross-validation indicated stable area under the curves (AUCs) across machine learning classifiers. A didactical illustration of applying random forest to detect SDB is presented. [ABSTRACT FROM AUTHOR]

This Monte Carlo simulation study compares methods to estimate the effects of programs with multiple versions when assignment of individuals to program version is not random. These methods use generalized propensity scores, which are predicted probabilities of receiving a particular level of the treatment conditional on covariates, to remove selection bias. The results indicate that inverse probability of treatment weighting (IPTW) removes the most bias, followed by optimal full matching (OFM), and marginal mean weighting through stratification (MMWTS). The study also compared standard error estimation with Taylor series linearization, bootstrapping and the jackknife across propensity score methods. With IPTW, these standard error estimation methods performed adequately, but standard errors estimates were biased in most conditions with OFM and MMWTS.

The global COVID-19 health pandemic caused major interruptions to educational assessment systems, partially due to shifts to remote learning environments, entering the post-COVID educational world into one that is more open to heterogeneity in instructional and assessment modes for secondary students. In addition, in 2020, educational inequities were brought to the forefront of social conscience. The purpose of this study is to empirically explore how contextual (i.e., school-level) race and economic factors may relate to and explain measurement challenges that can arise during shifts to remote learning. We fit a series of multilevel models to explore school-level factors in assessment data alongside psychometric problems of differential item functioning and person fit in classroom assessment measurement models. Our results demonstrate ways in which our project's classroom assessments were impacted by shifts to remote learning, emphasizing the importance of researchers and practitioners evaluating such concerns when seeking validity evidence for interpretation of classroom assessment data. [ABSTRACT FROM AUTHOR]

AbstractPropensity score analysis (PSA) is a crucial tool for researchers in mitigating selection bias arising from multiple covariates in quasi-experimental studies. Nevertheless, the impact of low-reliability covariates on PSA necessitates careful consideration. This study employs Monte Carlo simulation to assess five methods to adjust propensity scores for unreliability of covariates. The findings reveal that the latent variable model incorporating inclusive factor scores (PSIF) results in the smallest relative bias of treatment effect estimates. Notably, only PSIF consistently provides unbiased treatment effect estimates across all conditions. Furthermore, the study underscores the potential for a misleading covariate balance evaluation when dealing with unreliable covariates, given that treatment effect estimates may be biased even when the covariate balance is perceived as adequate. [ABSTRACT FROM AUTHOR]

In the current study, we compare propensity score (PS) matching methods for data with a cross-classified structure, where each individual is clustered within more than one group, but the groups are not hierarchically organized. Through a Monte Carlo simulation study, we compared sequential cluster matching (SCM), preferential within cluster matching (PWCM), greedy matching (GM), and optimal full matching (OFM), using propensity scores from four different models. The results indicated that the four matching methods performed well when PSs were estimated with logistic regression containing both level-1 and level-2 covariates. When the level-2 covariates were omitted in the logistic regression PS model, matching methods resulted in biased treatment effect estimates. However, omission of level-2 covariates did not result in biased estimates when the PS model was a logistic cross-classified random effects model (CCREM). SCM and PWCM outperformed GM and OFM with a logistic CCREM that included level-1 and level-2 covariates. [ABSTRACT FROM AUTHOR]

Propensity score analyses (PSA) of continuous treatments often operationalize the treatment as a multi-indicator composite, and its composite reliability is unreported. Latent variables or factor scores accounting for this unreliability are seldom used as alternatives to composites. This study examines the effects of the unreliability of indicators of a latent treatment in PSA using the generalized propensity score (GPS). A Monte Carlo simulation study was conducted varying composite reliability, continuous treatment representation, variability of factor loadings, sample size, and number of treatment indicators to assess whether Average Treatment Effect (ATE) estimates differed in their relative bias, Root Mean Squared Error, and coverage rates. Results indicate that low composite reliability leads to underestimation of the ATE of latent continuous treatments, while the number of treatment indicators and variability of factor loadings show little effect on ATE estimates, after controlling for overall composite reliability. The results also show that, in correctly specified GPS models, the effects of low composite reliability can be somewhat ameliorated by using factor scores that were estimated including covariates. An illustrative example is provided using survey data to estimate the effect of teacher adoption of a workbook related to a virtual learning environment in the classroom. [ABSTRACT FROM AUTHOR]

Cognitive diagnosis models are diagnostic models used to classify respondents into homogenous groups based on multiple categorical latent variables representing the measured cognitive attributes. This study aims to present longitudinal models for cognitive diagnosis modeling, which can be applied to repeated measurements in order to monitor attribute stability of individuals and to account for respondent dependence. Models based on combining latent transition analysis modeling and the DINA and DINO cognitive diagnosis models were developed and then evaluated through a Monte Carlo simulation study. The study results indicate that the proposed models provide adequate convergence and correct classification rates.

We investigated methods of including covariates in two-level models for cluster randomized trials to increase power to detect the treatment effect. We compared multilevel models that included either an observed cluster mean or a latent cluster mean as a covariate, as well as the effect of including Level 1 deviation scores in the model. A Monte Carlo simulation study was performed manipulating effect sizes, cluster sizes, number of clusters, intraclass correlation of the outcome, patterns of missing data, and the squared correlations between Level 1 and Level 2 covariates and the outcome. We found no substantial difference between models with observed means or latent means with respect to convergence, Type I error rates, coverage, and bias. However, coverage could fall outside of acceptable limits if a latent mean is included as a covariate when cluster sizes are small. In terms of statistical power, models with observed means performed similarly to models with latent means, but better when cluster sizes were small. A demonstration is provided using data from a study of the Tools for Getting Along intervention.

Neural networks are a contending data mining procedure to estimate propensity scores due to its robustness to non-normal residual distributions, ability to detect complex nonlinear relationships between treatments and confounding variables, nonessential model specification, and compatibility to train based on observed events. In this study, we develop artificial neural network architectures to estimate propensity scores for categorical treatments. For comparison, we estimated propensity scores with more popular techniques: logistic regression, multinomial logistic regression, and generalized boosted logistic regression using regression trees (GBM). Previous studies found lower prediction error of GBM compared with alternative methods and demonstrated that it does not require model specification yet mentions several cases of overfitting. We used Monte Carlo simulations manipulating sample coefficients, model specifications, and fixed sample sizes to compare the generalization error of trained machine-learning algorithms to never-before-seen data. Neural networks resulted in higher correlations between true propensity scores and estimated propensity scores. Also, other performance measures, such as cross-entropy values, suggest that artificial neural networks may be more accurate than more popular methods to estimate propensity scores. [ABSTRACT FROM AUTHOR]

Because random assignment is not possible in observational studies, estimates of treatment effects might be biased due to selection on observable and unobservable variables. To strengthen causal inference in longitudinal observational studies of multiple treatments, we present 4 latent growth models for propensity score matched groups, and evaluate their performance with a Monte Carlo simulation study. We found that the 4 models performed similarly with respect to model fit, bias of parameter estimates, Type I error, and power to test the treatment effect. To demonstrate a multigroup latent growth model with dummy treatment indicators, we estimated the effect of students changing schools during elementary school years on their reading and mathematics achievement, using data from the Early Childhood Longitudinal Study Kindergarten Cohort. (Contains 4 tables and 1 figure.)