Search

We used multilevel analysis to examine the predictive validity of scores from the Framework for Teaching (FFT), the observation measure used most often to evaluate teachers' instruction. We investigated how well 81 kindergarten teachers' FFT scores for eight reading and eight mathematics lessons observed throughout the year predicted students' year-end achievement and motivation in reading and mathematics, controlling for students' sex, ethnicity, and achievement entering kindergarten. Standardized reading and mathematics achievement were each predicted by FFT scores; however, they accounted for very little of the overall variance in students' achievement: 2.5% for reading and 1.3% for mathematics. Neither students' end-of-year criterion-referenced achievement nor motivation were predicted by FFT scores.

We evaluated the score stability of the Framework for Teaching (FFT), a prominent observation instrument used for teacher evaluation. Three raters each scored 200 reading and mathematics lessons taught by 20 kindergarten teachers. Using Generalizability theory analyses, we decomposed the FFT's Classroom Environment, Instruction, and Total scores into potential sources of variation (teachers, lessons, raters, and their interactions). The scores' variances attributable to differences among teachers were 71% and 76% for Classroom Environment, 49% and 37% for Instruction, and 69% and 66% for the Total score, for reading and mathematics, respectively. Reliability estimates (G) ranged from 0.92 to 0.96 for Classroom Environment and Total scores; they were 0.87 and 0.79 for reading and mathematics Instruction. Decision studies indicated that two raters, each scoring three reading lessons or four mathematics lessons, are necessary to achieve sufficiently reliable Total scores. For Instruction scores, three raters each scoring seven readings lessons are needed; more than four raters each scoring eight lessons are needed for mathematics.

This study advances the cross-cultural temperament literature by comparing temperament ratings of toddlers from 14 nations. Multilevel modeling (MLM) procedures were utilized to regress negative emotionality (NE) and component subscales on Hofstede's cultural value dimensions while controlling for age and gender. More individualistic values were associated with lower NE, and component discomfort, fear, motor activity, perceptual sensitivity, and soothability scales. The discomfort subscale was negatively associated with power distance and positively associated with masculine cultural values. Higher ratings of shyness were related to a more long-term cultural orientation. Results illustrate the feasibility of an MLM approach to cross-cultural research and provide a new perspective on the intersection of culture and temperament development. Limitations and future implications are discussed.

A lack of familiarity with the lexical, grammatical, and discursive features associated with academic language presents comprehension challenges for many adolescent readers, especially those with still-developing English language skills. Although many studies focused on reading comprehension in content areas, the prevalent use of standardized tests combining narrative and expository readings across academic domains undermines our understandings of the specialized language and skills required for comprehension in each particular discipline. In this study, we examined contributions of domain-general and domain-specific language skills needed for reading in science, a linguistically and cognitively demanding subject area. The sample included 241 Grade 8 students, 64 of whom were emergent English speakers (current ELs). Proficient English speakers included 118 former ELs. Our statistical modeling results explained 80% of the variance in students' science reading comprehension and science vocabulary scores. Mediated effects of morphosyntactic awareness and a direct effect of science vocabulary on reading comprehension of physical sciences texts were supported. Support is also offered for an appropriate measurement specification for the construct of morphosyntactic awareness and the appropriateness of using definition (breadth) and picture (depth) match tasks for assessing science vocabulary. Less variance was explained in students' morphosyntactic awareness, a variable that needs further examination.

The Academic Intrinsic Motivation Scale (AIMS) measures key components of student intrinsic motivation (IM). We investigate score validity and reliability of a downward extension of the AIMS developed for students in the high school context using a sample of students from the Pacific Northwest region of the United States. Through classical test theory, this study documents the scale's (1) item difficulty and discrimination, (2) underlying factor structure and score reliability via confirmatory factor analysis and omega coefficients, and (3) relationship with the theoretically related variables student retention, achievement, and teamwork ability. Findings support scoring, generalizability, and extrapolation inferences for the downward extension of the AIMS in a high school sample. Implications for additional psychometric validity evidence are discussed.

The use and interpretation of educational and psychological test scores are paramount to individual outcomes and opportunities. Methods for detecting differential item functioning (DIF) are imperative for item analysis when developing and revising assessments, particularly as it pertains to fairness across populations, languages, and cultures. We discuss the practical application of third generation of DIF, which proposes an ecological framework for explaining why DIF occurs using sociological, community, and contextual variables as possible explanations of item responding. We examined school district clusters for DIF through a model-based approach via multilevel logistic regression random-intercept and random-coefficient models to detect racial/ethnic DIF on a youth risk and needs assessment. We demonstrate how the use of district-level administrative data as estimates of student's contextual environments shows promise to understand ecological sources of DIF and DIF variance. The implication of the work progresses DIF research, enhances the approach to understanding item responses, and provides a more accurate understanding of youth development through an ecological framework. Recommendations for practice and future studies are discussed.

Truancy can have a detrimental impact on student outcomes. Risk assessments are used to identify behavioral and emotional problems associated with school truancy. Although imperative for informing decisions about student's welfare, risk assessments generally lack substantial validity evidence. Specifically, supporting score interpretation across cultural groups through measurement invariance (MI) is needed for such assessments. This study examined MI, specifically factor invariance (FI), of the Washington Assessment of the Risks and Needs of Students (WARNS) across African American, Latinx, and Caucasian students using confirmatory factor analysis with traditional FI criteria and the equivalence testing (ET) procedure. Results from traditional criteria suggested that the factor structure is similar across groups. The ET procedure demonstrated minor model misspecifications.

The present study aims to gain a greater understanding of the manner in which culture may impact parenting and, thus, child development by examining the relationship between cultural values, socialization goals (SGs), and parental ethnotheories (PEs). Specifically, this study examined links between cultural value dimensions (i.e., individualism/collectivism, power distance, masculinity/femininity, uncertainty avoidance, indulgence/restraint, and long-term/short-term orientation; Hofstede et al., 2010) and autonomous as well as relational SGs and PEs. We examined data collected from mothers of toddlers (N = 865) between 17 and 40 months of age (M = 26.88 months, SD = 5.65 months; 52% boys) from 14 nations represented in the Joint Effort Toddler Temperament Consortium. We hypothesized that: (a) Cultural values consistent with independent cultural ideals would be positively associated with SGs and PEs representative of greater autonomy and independence, and (b) Cultural values consistent with interdependent ideals would be positively associated with SGs and PEs representative of greater interrelatedness. Multilevel modeling was used to regress parental psychology on Hofstede's cultural values. Support for these hypotheses was somewhat mixed; higher ratings of culture-level indulgence were associated with higher autonomous PEs, as well as with higher relational and autonomous SGs. Furthermore, higher ratings of culture-level masculinity were associated with lower relational PEs and with lower autonomous SGs. The results suggest differences in the effects for cultural values associated with parenting versus cultural values associated with child outcomes and highlight considerations related to dichotomous cultural frameworks. The findings help explain both individual- and country-level variations in aspects of parental psychology. [ABSTRACT FROM AUTHOR]

Estimating the number of latent factors underlying a set of dichotomous items is a major challenge in social and behavioral research. Mokken scale analysis (MSA) and exploratory graph analysis (EGA) are approaches for partitioning measures consisting of dichotomous items. In this study we perform simulation-based comparisons of two EGA methods (EGA with graphical least absolute shrinkage and selector operator; EGAtmfg with triangulated maximally filtered graph algorithm), two MSA methods (AISP: automated item selection procedure; GA: genetic algorithm), and two widely used factor analytic techniques (parallel analysis with principal component analysis (PApc) and parallel analysis with principal axis factoring (PApaf)) for partitioning dichotomous items. Performance of the six methods differed significantly according to the data structure. AISP and PApc had highest accuracy and lowest bias for unidimensional structures. Moreover, AISP demonstrated the lowest rate of misclassification of items. Regarding multidimensional structures, EGA with GLASSO estimation and PApaf yielded highest accuracy and lowest bias, followed by EGAtmfg. In addition, both EGA techniques exhibited the lowest rate of misclassification of items to factors. In summary, EGA and EGAtmfg showed comparable performance to the highly accurate traditional method, parallel analysis. These findings offer guidance on selecting methods for dimensionality analysis with dichotomous indicators to optimize accuracy in factor identification. [ABSTRACT FROM AUTHOR]

The Washington Assessment of Risk and Needs of Students (WARNS) is a brief self-report measure designed for schools, courts, and youth service providers to identify student behaviors and contexts related to school truancy. Empirical support for WARNS item invariance between ethnic groups is lacking. This study examined differential item functioning (DIF) to ensure that items on the WARNS function similarly across groups, especially for groups where truancy rates are highest. The item response theory graded response model was used to examine DIF between Caucasian, African American, and Latinx students. DIF was identified in six items across WARNS domains. The DIF amount and magnitude likely will not influence decisions based on total scores. Implications for practice and suggestions for an ecological framework to explain the DIF results are discussed.

The State of Washington requires school districts to file court petitions on students with excessive unexcused absences. The "Washington Assessment of Risks and Needs of Students" (WARNS), a self-report screening instrument developed for use by high school and juvenile court personnel in such situations, purports to measure six facets of risks and needs of youth relevant to improving school and life outcomes. Many psychometric analyses have been documented either in the technical manual or through peer-reviewed publications. We review this body of evidence in the context of a central claim about score interpretation/use and the inferences that underlie that claim. Such evidence is strong for inferences related to the target domain, scoring, generalization, and extrapolation. Evidence for an implication inference, however, is pending. We propose a validation trajectory partnership for the WARNS to build evidence through a collaborative research program.

The usefulness of item response theory (IRT) models depends, in large part, on the accuracy of item and person parameter estimates. For the standard 3 parameter logistic model, for example, these parameters include the item parameters of difficulty, discrimination, and pseudo-chance, as well as the person ability parameter. Several factors impact traditional marginal maximum likelihood (ML) estimation of IRT model parameters, including sample size, with smaller samples generally being associated with lower parameter estimation accuracy, and inflated standard errors for the estimates. Given this deleterious impact of small samples on IRT model performance, use of these techniques with low-incidence populations, where it might prove to be particularly useful, estimation becomes difficult, especially with more complex models. Recently, a Pairwise estimation method for Rasch model parameters has been suggested for use with missing data, and may also hold promise for parameter estimation with small samples. This simulation study compared item difficulty parameter estimation accuracy of ML with the Pairwise approach to ascertain the benefits of this latter method. The results support the use of the Pairwise method with small samples, particularly for obtaining item location estimates.

Highlights Pre- and post-tests, administered to 2,141 emergency first responder participants, showed an average improvement in test scores from 67% to 75%, highlighting the efficacy of the training. Interviews conducted within 3 years post-training revealed high participant satisfaction, with over 25% reporting adoption of key strategies discussed in the training by their fire/rescue service. Areas of concern were identified, including the lack of understanding related to certain hazards, such as free-flowing grain, which may put first responders at risk of secondary victimization. Purdue University’s Agricultural Safety and Health Program has provided leadership for nearly 40 years in the documentation of fatalities and injuries associated with agricultural confined spaces, especially those relating to grain storage, handling, and transport. Findings have been used to develop evidence-based resources to assist in the prevention and mitigation of these incidents, including the design of in-service training resources for emergency rescue and medical personnel responding to entrapments or engulfment in agricultural confined spaces. To enhance the efficacy and consistency of these training resources, a list of core competencies was developed with companion test questions by a panel of experts to validate the baseline understanding and knowledge gain of training participants. The test questions were pilot tested as pre- and post-tests and incorporated into a curriculum developed under a U.S. Department of Labor Susan Harwood Training Grant. The twenty-question pre- and post-tests were administered to 2,141 registered emergency first responder participants in training conducted primarily in Indiana. Participation was voluntary, providing 671 usable matched pre- and post-tests. On average, test scores improved from 67% to 75%. A question-by-question review highlighted areas of common knowledge as well as at least one topic in which the potential for confusion was increased by the instructional content. In addition, participants were interviewed within 3 years to assess the impact of the training received. Interviewees indicated a high level of satisfaction with the training, and over 25% indicated that their fire/rescue service adopted at least one of the seven key strategies discussed in the training. One key concern observed in training was the lack of understanding related to certain hazards, such as the nature of free-flowing grain, that may put first responders at risk of becoming secondary victims during rescue and extrication efforts. A need was identified for continued improvement of emergency first responder training through the incorporation of recent research findings on confined space rescue, greater attention to the prevention of secondary injuries, and more consistent instructor preparation in order to increase the probability of successful outcomes from incidents involving grain storage, handling, and transport. [ABSTRACT FROM AUTHOR]

Objective: Modified Mini-Mental State Examination (3MSE) is often used to screen for dementia, but little is known about psychometric validity in American Indians. Methods: We recruited 818 American Indians aged 65–95 for 3MSE examinations in 2010–2013; 403 returned for a repeat examination in 2017–2019. Analyses included standard psychometrics inferences for interpretation, generalizability, and extrapolation: factor analysis; internal consistency-reliability; test-retest score stability; multiple indicator multiple cause structural equation models. Results: This cohort was mean age 73, majority female, mean 12 years education, and majority bilingual. The 4-factor and 2nd-order models fit best, with subfactors for orientation and visuo-construction (OVC), language and executive functioning (LEF), psychomotor and working memory (PMWM), verbal and episodic memory (VEM). Factor structure was supported for both research and clinical interpretation, and factor loadings were moderate to high. Scores were generally consistent over mean 7 years. Younger participants performed better in overall scores, but not in individual factors. Males performed better on OVC and LEF, females better on PMWM. Those with more education performed better on LEF and worse on OVC; the converse was true for bilinguals. All differences were significant, but small. Conclusion: These findings support use of 3MSE for individual interpretation in clinic and research among American Indians, with moderate consistency, stability, reliability over time. Observed extrapolations across age, sex, education, and bilingual groups suggest some important contextual differences may exist. [ABSTRACT FROM AUTHOR]

Proper understanding of scaling and large-scale hydrologic processes is often not explicitly incorporated in the teaching curriculum. This makes it difficult for students to connect the effect of small scale processes and properties (like soil texture and structure, aggregation, shrinkage, and cracking) on large scale hydrologic responses (like watershed runoff). An instructional module that introduces the concept of process-based scaling, as the framework to connect hydrologic processes and properties at different scales, was developed and evaluated. This paper examines how incorporating the concept of scaling into student curriculum impacts students' learning and decision making capabilities. It presents the evaluation of this module in an undergraduate environmental and natural resources engineering course. Evaluation results supported the hypothesis that introducing the concept of scaling and its application (using computer models) into undergraduate engineering courses enhanced students' learning and decision making skills. Students' levels of confidence in their knowledge of hydrologic systems also increased after the introduction of the scaling concept and following computer modeling exercises.

This study modeled the time-dependent nature of pre-academic skills development for a multi-lingual sample of 382 preschoolers attending Head Start. A two-slope latent variable model accurately described the trajectories, which included five assessment points obtained throughout the school year for both pre-literacy and pre-numeracy variables. Classroom quality and child demographic variables including age, gender, native language (Spanish or English), and disability status served as predictors of growth. Results revealed a reversal effect such that steeper growth from October to March was offset, although not completely, by less growth from March to May. Similar interaction effects were observed for predictor variables and their interaction terms across the two slopes. IEP status predicted less growth, while Spanish-language status predicted increased growth, despite the fact that both were associated with low initial achievement. The complexity of the relationship between classroom quality, native language, disability status, and pre-academic growth is discussed.

To meet recent accountability mandates, school districts are implementing assessment frameworks to document teachers' effectiveness. Observational assessments play a key role in this process, albeit without compelling evidence of their psychometric rigor. Using a sample of kindergarten teachers, we employed Generalizability theory to investigate (across teachers, raters, and lessons) the stability of scores obtained with two different observation measures: The CLASS K-3 and the FFT. We conducted a series of Decision studies to document (for both measures' constituent domains) the number of lessons per teacher and raters per lesson that would justify the use of observation scores for high stakes decisions. Acceptable, stable scores for individual-level decisions about teachers may generally require more raters and lessons than is typically used in practice (1-2 raters and fewer than 3 lessons). The considerable variability of observation-based scores raises concerns about either measure's appropriateness for making individual or group decisions about teachers' effectiveness.

The written certification exam that accompanies the Gearing Up for Safety-Agricultural Production Safety Training for Youth curriculum was designed to partially meet the testing requirements of the Agricultural Hazardous Occupations Order (AgHOs) Certification Training Program. This curriculum and accompanying assessment tools are available for national implementation. Psychometric properties of the exam were examined based on a sample of high-school agricultural education students, who were consistent with the population of youth for which the AgHOs were designed to protect from certain workplace hazards. The analyses included evaluation of score reliability (i.e., internal consistency) and the theoretical structure of the exam via confirmatory factor analysis. Reliability estimates were satisfactory. Confirmatory factor analysis revealed that the preferred theoretical model had adequate fit. Implications and future directions are discussed. (Contains 2 tables and 1 figure.)

The Controlled Oral Word Association (COWA) test is used to assess phonemic fluency and executive function. Formal validation of test scores is important for accurate cognitive evaluation. However, there is a dearth of psychometric validation among American Indian adults. Given high burden of dementia risk and key contextual factors associated with cognitive assessments, this represents a critical oversight. In a large, longitudinal population-based cohort study of adult American Indians, we examined several validity inferences for COWA, including scoring, generalization, and extrapolation inferences, by investigation of factor structure, internal consistency, test–retest reliability, and differential test functioning. We found adequate unidimensional model fit, with high factor loadings. Internal consistency reliability and test–retest reliability were 0.88 and 0.77, respectively, for the full group. COWA scores were lowest among the oldest, lowest education, bilingual speakers; group effects for sex and bilingual status were small; age effect was medium; and education effect was largest. However, Wide Range Achievement Test (WRAT) score effect was stronger than education effect, suggesting better contextualization may be needed. These results support interpretation of total COWA score, including across sex, age, or language use strata. [ABSTRACT FROM AUTHOR]

Recent research has demonstrated an intersectional approach to the study of differential item functioning (DIF). This approach expands DIF to account for the interactions between what have traditionally been treated as separate grouping variables. In this paper, we compare traditional and intersectional DIF analyses using data from a state testing program (nearly 20,000 students in grade 11, math, science, English language arts). We extend previous research on intersectional DIF by employing field test data (embedded within operational forms) and by comparing methods that were adjusted for an increase in Type I error (Mantel-Haenszel and logistic regression). Intersectional analysis flagged more items for DIF compared with traditional methods, even when controlling for the increased number of statistical tests. We discuss implications for state testing programs and consider how intersectionality can be applied in future DIF research. [ABSTRACT FROM AUTHOR]

Differential item functioning (DIF) assessment is key in score validation. When DIF is present scores may not accurately reflect the construct of interest for some groups of examinees, leading to incorrect conclusions from the scores. Given rising immigration, and the increased reliance of educational policymakers on cross-national assessments such as Programme for International Student Assessment, Trends in International Mathematics and Science Study, and Progress in International Reading Literacy Study (PIRLS), DIF with regard to native language is of particular interest in this context. However, given differences in language and cultures, assuming similar cross-national DIF may lead to mistaken assumptions about the impact of immigration status, and native language on test performance. The purpose of this study was to use model-based recursive partitioning (MBRP) to investigate uniform DIF in PIRLS items across European nations. Results demonstrated that DIF based on mother's language was present for several items on a PIRLS assessment, but that the patterns of DIF were not the same across all nations.

Being attentive to and aware of one's experiences in the present moment with qualities of acceptance and openness reflects the state of mindfulness. Positive associations exist between state mindfulness and state autonomous motivation for everyday activities. Though this suggests that state mindfulness links with adaptive motivational experiences, no suitable measure of state mindfulness exists that would facilitate the examination of these relationships in a physical activity context. Thus, we revised the State Mindfulness Scale (Tanay & Bernstein, 2013) and provided score validity evidence for the measure in a physical activity context. A bi-factor model reflecting mindfulness of the mind and body as specific factors and a general mindfulness factor was supported. Validity evidence, such as positive relationships with intrinsic motivation, and a negative relationship with body surveillance support score use. The revised scale can facilitate investigations of the role of mindfulness in physical activity settings.

The Trait Meta-Mood Scale (TMMS) is an emotional intelligence (EI) assessment originally developed for the U.S. population. This scale measures three EI factors--attention, clarity, and repair--to evaluate how an individual perceives one's own EI skills. Although the TMMS has been adapted for use in several languages and cultures, the structure of the TMMS requires continuous examination across cultures. Specifically, there is a need for stronger validity evidence using confirmatory analyses. This study evaluates the factor structure of the TMMS-Spanish version, known as the TMMS-24, in a sample of students from northern Mexico. Data from high school and college students were used to examine the factor structure of the scale via confirmatory factor analysis. Results support the basis for future cross-cultural research conducted with Hispanic populations within Mexico with the TMMS-24.

SIBTEST is a differential item functioning (DIF) detection method that is accurate and effective with small samples, in the presence of group mean differences, and for assessment of both uniform and nonuniform DIF. The presence of multilevel data with DIF detection has received increased attention. Ignoring such structure can inflate Type I error. This simulation study examines the performance of newly developed multilevel adaptations of SIBTEST in the presence of multilevel data. Data were simulated in a multilevel framework and both uniform and nonuniform DIF were assessed. Study results demonstrated that naïve SIBTEST and Crossing SIBTEST, ignoring the multilevel data structure, yield inflated Type I error rates, while certain multilevel extensions provided better error and accuracy control.

The integration of engineering concepts and practices into elementary science education has become an emerging concern for science educators and practitioners, alike. Moreover, how children, specifically preadolescents (grades 1-5), engage in engineering design-based learning activities may help science educators and researchers learn more about children's earliest identification with engineering. The purpose of this study was to examine the extent to which engineering identity differed among preadolescents across gender and grade, when exposing students to engineering design-based science learning activities. Five hundred fifty preadolescent participants completed the Engineering Identity Development Scale (EIDS), a recently developed measure with validity evidence that characterizes children's conceptions of engineering and potential career aspirations. Data analyses of variance among four factors (i.e., gender, grade, and group) indicated that elementary school students who engaged in the engineering design-based science learning activities demonstrated greater improvements on the EIDS subscales compared to those in the comparison group. Specifically, students in the lower grade levels showed substantial increases, while students in the higher grade levels showed decreases. Girls, regardless of grade level and participation in the engineering learning activities, showed higher scores in the academic subscale compared to boys. These findings suggest that the integration of engineering practices in the science classroom as early as grade one shows potential in fostering and sustaining student interest, participation, and self-concept in engineering and science.

Social-emotional (SE) skills in the early developmental years of children influence outcomes in psychological, behavioral, and learning domains. The adult ratings of a child's SE skills can be influenced by sex stereotypes. These rating differences could lead to differential conclusions about developmental progress or risk. To ensure that differences between boys and girls in SE skills are accurate, validity evidence should support that differences are not based on such issues as stereotypes influencing ratings. Differential item functioning (DIF) analysis allows for the assessment of group differences in item responses while controlling for ability. This study utilized a new multilevel Mantel-Haenszel (MMH) DIF procedure to examine sex differences in item responses for examiner ratings of children's SE skills on the Brigance Inventory of Early Development III SE scale. Of 50 items examined, 4 were identified as large DIF items. The scores do not appear to be influenced by item-level rating distortions based on sex stereotypes.