Search

In data collected from virtual learning environments (VLEs), item response theory (IRT) models can be used to guide the ongoing measurement of student ability. However, such applications of IRT rely on unbiased item parameter estimates associated with test items in the VLE. Without formal piloting of the items, one can expect a large amount of nonignorable missing data in the VLE log file data, and this is expected to negatively affect IRT item parameter estimation accuracy, which then negatively affects any future ability estimates utilized in the VLE. In the psychometric literature, methods for handling missing data have been studied mostly around conditions in which the data and the amount of missing data are not as large as those that come from VLEs. In this article, we introduce a semisupervised learning method to deal with a large proportion of missingness contained in VLE data from which one needs to obtain unbiased item parameter estimates. First, we explored the factors relating to the missing data. Then we implemented a semisupervised learning method under the two-parameter logistic IRT model to estimate the latent abilities of students. Last, we applied two adjustment methods designed to reduce bias in item parameter estimates. The proposed framework showed its potential for obtaining unbiased item parameter estimates that can then be fixed in the VLE in order to obtain ongoing ability estimates for operational purposes.

Social desirability bias (SDB) has been a major concern in educational and psychological assessments when measuring latent variables because it has the potential to introduce measurement error and bias in assessments. Person-fit indices can detect bias in the form of misfitted response vectors. The objective of this study was to compare the performance of 14 person-fit indices to identify SDB in simulated responses. The area under the curve (AUC) of receiver operating characteristic (ROC) curve analysis was computed to evaluate the predictive power of these statistics. The findings showed that the agreement statistic (A) outperformed all other person-fit indices, while the disagreement statistic (D), dependability statistic (E), and the number of Guttman errors (G) also demonstrated high AUCs to detect SDB. Recommendations for practitioners to use these fit indices are provided.

The piecewise latent growth models (PWLGMs) can be used to study changes in the growth trajectory of an outcome due to an event or condition, such as exposure to an intervention. When there are multiple outcomes of interest, a researcher may choose to fit a series of PWLGMs or a single parallel-process PWLGM. A comparison of these models is provided with an illustrative example using data from the Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 (ECLS-K:2011), and a Monte Carlo simulation study. Conditions manipulated included sample size, location of the transition point, the number of time points, and covariate effect size. The results showed that the power to test parameter estimates with both models depended on the two-way interaction of sample size and covariate effect, and the three-way interaction of sample size by the number of time points by transition point location. Parameter coverage also depended on the three-way interaction of sample size by the number of time points by the transition point location. Recommendations for the use of the PWLGM are provided.

The unstructured multiple-attempt (MA) item response data in virtual learning environments (VLEs) are often from student-selected assessment data sets, which include missing data, single-attempt responses, multiple-attempt responses, and unknown growth ability across attempts, leading to a complex and complicated scenario for using this kind of data set as a whole in the practice of educational measurement. It is critical that methods be available for measuring ability from VLE data to improve VLE systems, monitor student progress in instructional settings, and conduct educational research. The purpose of this study is to explore the ability recovery of the multidimensional sequential 2-PL IRT model in unstructured MA data from VLEs. We conduct a simulation study to evaluate the effects of the magnitude of ability growth and the proportion of students who make two attempts, as well as the moderated effects of sample size, test length, and missingness, on the bias and root mean square error of ability estimates. Results show that the model poses promise for evaluating ability in unstructured VLE data, but that some data conditions can result in biased ability estimates.

In reading intervention research, implementation fidelity is assumed to be positively related to student outcomes, but the methods used to measure fidelity are often treated as an afterthought. Fidelity has been conceptualized and measured in many different ways, suggesting a lack of construct validity. One aspect of construct validity is the fidelity index of a measure. This methodological case study examined how different decisions in fidelity indices influence relative rank ordering of individuals on the construct of interest and influence our perception of the relation between the construct and intervention outcomes. Data for this study came from a large state-funded project to implement multi-tiered systems of support for early reading instruction. Analyses were conducted to determine whether the different fidelity indices are stable in relative rank ordering participants and if fidelity indices of dosage and adherence data influence researcher decisions on model building within a multilevel modeling framework. Results indicated that the fidelity indices resulted in different relations to outcomes with the most commonly used fidelity indices for both dosage and adherence being the worst performing. The choice of index to use should receive considerable thought during the design phase of an intervention study.

The field of educational measurement places validity and fairness as central concepts of assessment quality. Prior research has proposed embedding fairness arguments within argument-based validity processes, particularly when fairness is conceived as comparability in assessment properties across groups. However, we argue that a more flexible approach to fairness arguments that occurs outside of and complementary to validity arguments is required to address many of the views on fairness that a set of assessment stakeholders may hold. Accordingly, we focus this manuscript on two contributions: (a) introducing the argument-based fairness approach to complement argument-based validity for both traditional and artificial intelligence (AI)-enhanced assessments and (b) applying it in an illustrative AI assessment of perceived hireability in automated video interviews used to prescreen job candidates. We conclude with recommendations for further advancing argument-based fairness approaches. [For the corresponding grantee submission, see ED620206.]

According to the "Standards for Educational and Psychological Testing" (2014), one aspect of test fairness concerns examinees having comparable opportunities to learn prior to taking tests. Meanwhile, many researchers are developing platforms enhanced by artificial intelligence (AI) that can personalize curriculum to individual student needs. This leads to a larger overarching question: When personalized learning leads to students having differential exposure to curriculum throughout the K-12 school year, how might this affect test fairness with respect to summative, end-of-year high-stakes tests? As a first step, we traced the differences in content exposure associated with personalized learning and more traditional learning paths. To better understand the implications of differences in content coverage, we conducted a simulation study to evaluate the degree to which curriculum exposure varied across students in a particular AI-enhanced learning platform for Algebra instruction with high-school students. Results indicate that AI-enhanced personalized learning may pose threats to test fairness as opportunity-to-learn on K-12 summative high-stakes tests. We discuss the implications given different perspectives of the role of testing in education

The current study examines both student self-regulated learning (SRL) and teacher orchestration in a virtual learning environment (VLE), with respect to student achievement. The study used SRL indicators derived from the log data on how students used the VLE system, survey data on how teachers made use of the VLE for Algebra instruction, as well as formative and summative Algebra assessment scores. The sample included 6,174 students being taught by 93 teachers in 49 schools in the 2018-2019 academic year. Multilevel structural equation modeling (SEM) analysis revealed the relationship between SRL indicators and student achievement. We found a positive relationship between student SRL and student achievements. Specifically, three SRL indicators, "Watched Recommended Videos", "Watched Videos" and "Answered Quizzes after Watching a Video", were significantly associated with student achievement. We did not observe direct associations between teacher orchestration and student achievement, but we found an indirect association between teacher orchestration and student achievement via one SRL indicator, "Watched Recommended Videos". The results show that a positive relationship between EOC scores and student reviewing incorrect questions increased as the use of instructional videos by the teacher increased. This study supports the critical role of SRL and indicates that teachers should have flexibility in adapting learning activities for online learning.

Based on the achievement goal theory, this experimental study explored the influence of predictive and descriptive learning analytics dashboards on graduate students' motivation and statistics anxiety in an online graduate-level statistics course. Participants were randomly assigned into one of three groups: (1) predictive dashboard; (2) descriptive dashboard; or (3) control (i.e., no dashboard). Measures of motivation and statistical anxiety were collected in the beginning and the end of the semester via the Motivated Strategies for Learning Questionnaire and Statistical Anxiety Rating Scale. Individual semi-structured interviews were used to understand learners' perceptions of the course and whether the use of the dashboards influenced the meaning of their learning experiences. Results indicate that, compared to the control group, the predictive dashboard significantly reduced learners' interpretation anxiety and had an effect on intrinsic goal orientation that depended on learners' lower or higher initial levels of intrinsic goal orientation. In comparison to the control group, both predictive and descriptive dashboards reduced worth of anxiety (negative attitudes towards statistics) for learners who started the course with higher levels of worth anxiety. Thematic analysis revealed that learners who adopted a more performance-avoidance goal orientation approach demonstrated higher levels of anxiety regardless of the dashboard used.

School-wide positive behavior interventions and supports (SWPBIS) have been widely implemented in more than 25,000 schools. Although extant literature provides experimental evidence supporting the positive impacts of SWPBIS on decreased office discipline referrals and suspensions, those effects for diverse students have not been fully investigated. Therefore, this study aims to examine the effects of SWPBIS on eight different school-level discipline outcomes for all students and disaggregated by student demographics (i.e., gender, race/ethnicity, disability status). Using a post hoc quasi-experimental design and propensity score matching, we matched 112 schools implementing SWPBIS with fidelity with 112 schools in which teachers and administrators never received SWPBIS training. Poisson regression models revealed significant reductions in the use of school discipline outcomes for all students and for student subgroups. Discussion and implications of results are provided.

The purpose of this study was to demonstrate that applied researchers need methodologists, as evidenced by citation records. Results include (a) that methodologists are authors on more than one half of the top-cited recent empirical articles in selected journals, (b) that a bibliometric analysis of educational research articles recorded in the Social Science Citation Index displayed methodological work as the dominant area of scholarship, (c) that even a small fraction of PsycINFO American Psychological Association (APA) citation records of well-known methodological studies demonstrates that the work of methodologists is used by researchers across a wide range of disciplines, and (d) that books written by methodologists in education or a closely related discipline constitute a sizeable proportion of the top 100 cited books in Google Scholar.

Research Findings: Preschool social-emotional education has become an increasingly important area of research and practice in mainland China. The social development domain has been recognized as an independent preschool curricular domain since 2001. Little is known, however, about the specific practices that preschool teachers in China are using to promote children's social-emotional competence. An investigator-developed and validated measure, the "Social-Emotional Teaching Practices Questionnaire-Chinese" (SETP-C), was used to gather data about social-emotional practices from a sample of 1,599 Chinese teachers in 120 preschools. Chinese preschool teachers reported they were implementing many social-emotional practices included on the SETP-C, but were less likely to use and less confident in implementing practices that address the needs of children with persistent challenging behaviors. Teacher's role in the classroom, having teaching certification, years of teaching experience, use of a social-emotional curriculum, child-to-teacher ratio, inclusion of children with disabilities, and age group of children had statistically significant and noteworthy associations with teachers' reported frequency of use and confidence in implementing social-emotional practices. Practice or Policy: Findings from this study support the multidimensionality of preschool social-emotional practices and suggest the need for further attention to teacher, classroom, and program-wide variables related to teachers' practice implementation in support of young children's social-emotional competence.

The semi-generalized partial credit model (Semi-GPCM) has been proposed as a unidimensional modeling method for handling not applicable scale responses and neutral scale responses, and it has been suggested that the model may be of use in handling missing data in scale items. The purpose of this study is to evaluate the ability of the unidimensional Semi-GPCM to aid in the recovery of person parameters from item response data in the presence of item-level missingness, and to compare the performance of the model with two other proposed methods for handling such missingness: a multidimensional modeling approach for missingness and full information maximum likelihood estimation. The results indicate that the Semi-GPCM performs acceptably in an absolute sense when less than 30% of the item data is missing but does not outperform the other two methods under any particular conditions. We conclude with a discussion about when practitioners may or may not want to use the Semi-GPCM to recover person parameters from item response data with missingness.

The purpose of this study is to evaluate whether a recently developed semiordered model can be used to explore the functioning of neutral response options in rating scale data. Huggins-Manley, Algina, and Zhou developed a class of unidimensional models for semiordered data within scale items (i.e., items with both ordered response categories and an additional nominal response category) and found promising results when applying them to scale data with Not Applicable response categories. In this study, we extended the application of the semi-partial credit model (PCM) to evaluate whether the semi-PCM can be used to calibrate potentially unordered neutral responses in rating scale data, and if so, how the approach compares with alternate methods of dealing with the neutral response option. Findings indicate that the semi-PCM can (a) assist practitioners in evaluating the ordered or unordered nature of neutral responses and (b) provide a viable alternative for [theta] estimation in the presence of an unordered neutral category. The process used in this study also provides a methodological framework for researchers and practitioners to use when dealing with neutral responses in their own data.

Social desirability bias (SDB) is a common threat to the validity of conclusions from responses to a scale or survey. There is a wide range of person-fit statistics in the literature that can be employed to detect SDB. In addition, machine learning classifiers, such as logistic regression and random forest, have the potential to distinguish between biased and unbiased responses. This study proposes a new application of these classifiers to detect SDB by considering several person-fit indices as features or predictors in the machine learning methods. The results of a Monte Carlo simulation study showed that for a single feature, applying person-fit indices directly and logistic regression led to similar classification results. However, the random forest classifier improved the classification of biased and unbiased responses substantially. Classification was improved in both logistic regression and random forest by considering multiple features simultaneously. Moreover, cross-validation indicated stable area under the curves (AUCs) across machine learning classifiers. A didactical illustration of applying random forest to detect SDB is presented. [ABSTRACT FROM AUTHOR]

Numerous studies have been undertaken to design, develop, and provide validity evidence for using instruments to measure students' attitudes toward STEM (Science, Technology, Engineering, and Mathematics). This study presents validity evidence of scores produced from the S-STEM measurement tool and used to evaluate changes in attitudes during an educational intervention in a middle school robotics learning environment. All data were collected from middle school students who were involved in a school district-wide effort for integrating educational robotics into the classroom. Findings from this study provided not only internal structure validity evidence, but also criterion-related validity evidence of the proposed S-STEM tool use. In addition, measurement invariance results revealed that items in the S-STEM had equivalence in statistical properties of measurement across groups (e.g., grade level). The study provides further evidence that S-STEM survey is a powerful and useful tool to evaluate student attitude changes during STEM educational programs; offers suggestions for its future implementation; and presents other inspiring ideas for future STEM instrument development.

Routing examinees to modules based on their ability level is a very important aspect in computerized adaptive multistage testing. However, the presence of missing responses may complicate estimation of examinee ability, which may result in misrouting of individuals. Therefore, missing responses should be handled carefully. This study investigated multiple missing data methods in computerized adaptive multistage testing, including two imputation techniques, the use of full information maximum likelihood and the use of scoring missing data as incorrect. These methods were examined under the missing completely at random, missing at random, and missing not at random frameworks, as well as other testing conditions. Comparisons were made to baseline conditions where no missing data were present. The results showed that imputation and the full information maximum likelihood methods outperformed incorrect scoring methods in terms of average bias, average root mean square error, and correlation between estimated and true thetas.

This study aimed to assess the accuracy of the empirical item characteristic curve (EICC) preequating method given the presence of test speededness. The simulation design of this study considered the proportion of speededness, speededness point, speededness rate, proportion of missing on speeded items, sample size, and test length. After crossing all of the manipulated factors and then normalizing the evaluation criteria (bias and root mean square difference [RMSD]) with regard to test length, the results revealed that (1) when test speededness was present, conversions from the EICC preequating method tended to be positively distorted; (2) no practically meaningful moderation effect associated with sample size was found on the relationship between test speededness and the accuracy of EICC preequating; and (3) the location of the speededness point was the driving factor in terms of its impact on the accuracy of EICC preequating. Implications and suggestions were discussed.

The global COVID-19 health pandemic caused major interruptions to educational assessment systems, partially due to shifts to remote learning environments, entering the post-COVID educational world into one that is more open to heterogeneity in instructional and assessment modes for secondary students. In addition, in 2020, educational inequities were brought to the forefront of social conscience. The purpose of this study is to empirically explore how contextual (i.e., school-level) race and economic factors may relate to and explain measurement challenges that can arise during shifts to remote learning. We fit a series of multilevel models to explore school-level factors in assessment data alongside psychometric problems of differential item functioning and person fit in classroom assessment measurement models. Our results demonstrate ways in which our project's classroom assessments were impacted by shifts to remote learning, emphasizing the importance of researchers and practitioners evaluating such concerns when seeking validity evidence for interpretation of classroom assessment data. [ABSTRACT FROM AUTHOR]

Developing a diagnostic tool within the diagnostic measurement framework is the optimal approach to obtain multidimensional and classification-based feedback on examinees. However, end users may seek to obtain diagnostic feedback from existing item responses to assessments that have been designed under either the classical test theory or item response theory frameworks. Retrofitting diagnostic classification models to existing assessments designed under other psychometric frameworks could be a plausible approach to obtain more actionable scores or understand more about the constructs themselves. This study (a) discusses the possibility and problems of retrofitting, (b) proposes a step-by-step retrofitting framework, and (c) explores the information one can gain from retrofitting through an empirical application example. While retrofitting may not always be an ideal approach to diagnostic measurement, this article aims to invite discussions through presenting the possibility, challenges, process, and product of retrofitting.

Score equity assessment (SEA) refers to an examination of population invariance of equating across two or more subpopulations of test examinees. Previous SEA studies have shown that score equity may be present for examinees scoring at particular test score ranges but absent for examinees scoring at other score ranges. No studies to date have performed research for the purpose of understanding why score equity can be inconsistent across the score range of some tests. The purpose of this study is to explore a source of uneven subpopulation score equity across the score range of a test. It is hypothesized that the difficulty of anchor items displaying differential item functioning (DIF) is directly related to the score location at which issues of score inequity are observed. The simulation study supports the hypothesis that the difficulty of DIF items has a systematic impact on the uneven nature of conditional score equity.

This study was conducted to collect validity evidence to support the use of an observation instrument to evaluate the performance of special education teachers (SETs) of students with significant disabilities (SWSD). In the study, a purposive sample of 49 SETs of SWSD, who were appropriately credentialed and experienced, evaluated the content of the Teacher Rating and Assessment Instrument for Teachers of Students with Significant Disabilities (TRAIT-SD) to determine each item's importance for teaching SWSD. To do this, the participants were presented with the 37 items on the current version of the TRAIT-SD and asked to rate the importance of each using a 5-point Likert-type scale that ranged from very important to not important. They were also requested to provide narrative statements about the items and to recommend any relevant changes to the items, or recommend any new items they deemed to be missing from the current instrument. The results of the study showed that all participants, regardless of degree level, experience, or geographical location, rated all items between important and very important, resulting in all items being retained on the instrument. Additional comments resulted in a small number of items being reworded for clarity, and one new item being recommended for a subsequent version of the instrument. [ABSTRACT FROM AUTHOR]

There is an increasing demand for assessments that can provide more fine-grained information about examinees. In response to the demand, diagnostic measurement provides students with feedback on their strengths and weaknesses on specific skills by classifying them into mastery or nonmastery attribute categories. These attributes often form a hierarchical structure because student learning and development is a sequential process where many skills build on others. However, it remains to be seen if we can use information from the attribute structure and work that into the design of the diagnostic tests. The purpose of this study is to introduce three approaches of Q-matrix design and investigate their impact on classification results under different attribute structures. Results indicate that the adjacent approach provides higher accuracy in a shorter test length when compared with other Q-matrix design approaches. This study provides researchers and practitioners guidance on how to design the Q-matrix in diagnostic tests, which are in high demand from educators.

The purpose of this article is to explore validity evidence and appropriate uses of the revised Technology Uses and Perceptions Survey (TUPS) designed to measure in-service teacher perspectives about technology integration in K-12 schools and classrooms. The revised TUPS measures 10 domains, including Access and Support; Preparation of Technology Use; Perceptions of Professional Development; Perceptions of Technology Use; Confidence and Comfort Using Technology; Technology Integration; Teacher Use of Technology; Student Use of Technology; Perceived Technology Skills; and Technology Usefulness. We first provide a review of the literature supporting the design of the revised TUPS. We collected data from N = 1,376 teachers from one medium-sized school district in the state of Florida and conducted a variety of psychometric analyses. We performed internal structure analysis, correlation analysis, and factor analysis with these data. The results demonstrate that data collected from the TUPS are best used as descriptive, granular information about reported behaviors and perceptions related to technology, rather than treated as a series of 10 scales. These findings have implications for the appropriate uses of the TUPS.

This study defines subpopulation item parameter drift (SIPD) as a change in item parameters over time that is dependent on subpopulations of examinees, and hypothesizes that the presence of SIPD in anchor items is associated with bias and/or lack of invariance in three psychometric outcomes. Results show that SIPD in anchor items is associated with a lack of invariance in dimensionality structure of an anchor test, a lack of invariance in scaling coefficients across subpopulations, and a lack of invariance in ability estimates. It is demonstrated that these effects go beyond what can be understood from item parameter drift or differential item functioning.

Cognitive diagnosis models (CDMs) estimate student ability profiles using latent attributes. Model fit to the data needs to be ascertained in order to determine whether inferences from CDMs are valid. This study investigated the usefulness of some popular model fit statistics to detect CDM fit including relative fit indices (AIC, BIC, and CAIC), and absolute fit indices (RMSEA2, "ABS"("fcor") and "MAX"("?[superscript 2][subscript jj']")). These fit indices were assessed under different CDM settings with respect to Q-matrix misspecification and CDM misspecification. Results showed that relative fit indices selected the correct DINA model most of the times and selected the correct G-DINA model well across most conditions. Absolute fit indices rejected the true DINA model if the Q-matrix was misspecified in any way. Absolute fit indices rejected the true G-DINA model whenever the Q-matrix was under-specified. RMSEA2 could be artificially low when the Q-matrix was over-specified.

The TPACK (technological pedagogical content knowledge) framework (Mishra & Koehler, 2006) has gained tremendous momentum from within the educational technology community. Specifically, much discourse has focused on how to measure this multidimensional construct to further define the contours of the framework and potentially make some meaningful predictions. Some have proposed observation scales while other have proposed self-report measures to gauge the phenomenon. The Survey of Pre-service Teachers' Knowledge of Teaching and Technology instrument is one popular tool designed to measure TPACK (Schmidt et al., 2009) specifically from preservice teachers in teacher education programs. This study extends the measurement framework by providing a confirmatory factor analysis of the theoretical model proposed by Schmidt et al. (2009) on a sample of 227 preservice teachers from four public institutions of higher education in the southeastern United States. The data did not fit the theoretical 10-factor model implied by Schmidt et al. (2009), thus, an exploratory factor analysis was conducted to determine the optimal structure of the measurement tool for these data. This resulted in a nine-factor model, and there were measurement issues for several of the constructs. Additionally, the article provides evidence of external validity by correlating the instrument scores with other known technology constructs.

Using data collected from two multiyear teacher professional development projects employing randomized control trials, this study describes the development and validation of a paper-based test of elementary teachers' science content knowledge (SCK). Evidence of construct validity is presented, including evidence on internal structural features using Rasch measurement models. Results from 183 treatment group and 176 control group teachers from Project 1 demonstrated that the SCK test had acceptable person reliability at baseline; at later time points the test was easy for the teachers and person reliability was below acceptable. Results from Project 1 informed changes made to the test for use in Project 2, including an increase in the difficulty level and the development of two equated forms. Results from the 148 treatment and 139 control teachers from Project 2 demonstrated that the test had acceptable reliability across two time points and was a better match to teachers' SCK.

The Q-matrix is commonly used in diagnostic classification models and has recently been incorporated into the multidimensional item response theory (MIRT) models to add information about the relationship between items and dimensions of the latent trait. The reformulation of the MIRT models with Q-matrix (MIRT-Q) has presented to improve the precision of the parameters of these models and to provide a simple and intuitive method for users to define the item-trait relationship. This paper aims to explore the incorporation of the Q-matrix in the formulation of MIRT models for polytomous item responses. Specifically, we introduce the incorporation of the Q-matrix into two of the polytomous MIRT models most known and used: the multidimensional graded response (MGR) model, hereinafter called MGR-Q, and the multidimensional generalized partial credit (MGPC) model, hereinafter called MGPC-Q. We provide readers the code of the MGR-Q and MGPC-Q models in Stan, a Bayesian estimation software, and we conduct a simulation study in order to evaluate the parameter recovery of the estimation method. To illustrate the use of both models in practice, we fit them to an operational dataset from 2400 individuals on 13 items and demonstrate the estimation of MGR-Q and MGPC-Q using the Stan program. [ABSTRACT FROM AUTHOR]