Search

This study describes the measurement properties of two curated surveys, consisting of previously developed scales, assessing student perceptions of their social-emotional competencies (SEC) and the school culture/climate (CC). The surveys were administered through the Transforming Education/NewSchools Venture Fund partnership that worked with the NewSchools Invent Cohort, a network of 23 schools across 11 states, to expand the definition of student success. The analytic sample for this study includes more than 3,000 students in grades 4 through 12 at 18 schools surveyed in the 2016-17 school year. Using classical test theory as our conceptual and analytic lens, we conducted tests of internal consistency, item discrimination, omit rates, differential item functioning, and exploratory and confirmatory factor analysis. Our findings provide initial evidence of the validity of the survey scores for measuring student perceptions of their SEC and the school CC in the NewSchools Invent schools, suggesting that the surveys are suitable for school and cohort-level decision-making related to students' social emotional competencies and experiences within the school. [Written with Education Analytics.]

Higher levels of civic and community engagement in higher education are positively associated with students' academic performance and they also build upon citizenship skills such as informed voting. Yet, while these are worthy and important outcomes of higher education, students from disadvantaged backgrounds can have more difficulty navigating civic engagement. Focusing on students at thirteen major public universities in the United States, and utilizing survey data generated by the Student Experience in the Research University (SERU) Consortium, this study suggest social perspective-taking has a significant positive effect on all students' probability of voting and participating in community service. Students who were asked to identify challenge/solutions to social problems and reflect/act on community issues in the classroom were also more likely to participate in community service. Additionally, Hispanic students and students from lower-income households are significantly less likely to vote and perform community service--findings corroborated by previous research. Females are more likely to vote and participate in community service--findings also corroborated by prior research. Hispanic students and students from low-income backgrounds were more likely to participate in voting and community service if they had increased rates of social perspective-taking. These results suggest that increased opportunities to connect with classmates in the classroom and develop perspective-taking skills (e.g. appreciating the world from someone else's perspective, acknowledging personal differences, interacting with someone with views that are different from your own, and discussing and navigating controversial issues) are potentially quite powerful ways to foster civic engagement among students who traditionally have the lowest civic engagement rates compared to their peers.

Identifying and understanding the foundational skills children need to participate effectively in formal schooling is an important objective for research in early childhood education. One component of school readiness is cognitive self-regulation (CSR). The question this study addresses is how to assess CSR with prekindergarten-aged children in a way that taps into the learning-related cognitive engagement behaviors teachers observe in the classroom that are predictive of later academic achievement. A number of candidate measures applicable to prekindergarten age children have been generated for research on attention, effortful control, executive function, and related constructs. A diverse set of twelve candidate measures that can be easily administered in school settings was selected from these domains and applied to a sample of prekindergarten children. These measures were then examined for construct validity, developmental change, convergent validity with teacher ratings of CSR, and predictive validity for subsequent academic achievement and achievement gain. Six measures performed well by these criteria: Peg Tapping, Head-Toes-Knees-Shoulders (HTKS), the Kansas Reflection-Impulsivity Scale for Preschoolers (KRISP), Dimensional Change Card Sort (DCCS), Copy Design, and Backwards Digit Span. Cross-validation with a new sample of children confirmed the validity of these measures, estimated their test-retest reliability, identified the best performing individual measures, and demonstrated that a composite score combining results from all these measures performed better than any single measure. The Scoring Scheme for the Child Measures of LRCSR is appended.

Large-scale surveys such as the Programme for International Student Assessment (PISA), the Teaching and Learning International Survey (TALIS), and the Programme for the International Assessment of Adult Competences (PIAAC) use advanced statistical models to estimate scores of latent traits from multiple observed responses. The comparison of such estimated scores across different groups of respondents is valid to the extent that the same set of estimated parameters holds in each group surveyed. This issue of invariance of parameter estimates is addressed in model fit indices which gauge the likelihood that one set of parameters can be used across all groups. Therefore, the problem of scale invariance across groups of respondents can typically be framed as the question of how well a single model fits the responses of all groups. However, the procedures used to evaluate the fit of these models pose a series of theoretical and practical problems. The most commonly applied procedures to establish invariance of cognitive and non-cognitive scales across countries in large-scale surveys are developed within the framework of confirmatory factor analysis and item response theory. The criteria that are commonly applied to evaluate the fit of such models, such as the decrement of the Comparative Fit Index in confirmatory factor analysis, work normally well in the comparison of a small number of countries or groups, but can perform poorly in large-scale surveys featuring a large number of countries. More specifically, the common criteria often result in the non-rejection of metric invariance; however, the step from metric invariance (i.e. identical factor loadings across countries) to scalar invariance (i.e. identical intercepts, in addition to identical factor loadings) appears to set overly restrictive standards for scalar invariance (i.e. identical intercepts). This report sets out to identify and apply novel procedures to evaluate model fit across a large number of groups, or novel scaling models that are more likely to pass common model fit criteria. Using both real and simulated data, the following procedures are described and applied: multigroup confirmatory factor analysis, followed by alignment analysis of the same data set; Bayesian approximate measurement invariance; Bayesian measurement invariance testing in Item-Response Theory (IRT) models; and multigroup and multilevel latent class analysis. These approaches have the potential to resolve recurrent fit problems in invariance testing. Though promising, more work with these new approaches is needed to establish their suitability in large-scale surveys. The last chapter reports the conclusions from a conference in which these approaches were discussed, along with traditional approaches, in order to provide recommendations for how to address invariance issues in OECD education surveys.

The sizable gender gap in college enrolment, especially among African Americans, constitutes a puzzling empirical regularity that may have serious consequences on marriage markets, male labor force participation and the diversity of college campuses. For instance, only 35.7 percent of all African American undergraduate students were men in 2004. Reduced form results show that, while family background covariates cannot account for the observed gap, proxy measures for non-cognitive skills are crucial to explain it. Moreover, a sequential model of educational attainment indicates that males have actually higher preferences for education than females after controlling for latent factors (i.e. cognitive and non-cognitive skills). The model also shows that cognitive skills strongly affect the decision to move from one school level to the next, especially after finishing high school, but cannot account for disparities between genders. On the contrary, the substantial differences in the distribution of non-cognitive skills between males and females make these abilities critical to explain the gender gap in educational attainment across and within races. Appendices present 12 tables and 1 figure.

Pre-apprenticeship programs have generated interest recently from government, employers and other stakeholders in the training system as one means of improving apprenticeship completion rates and thereby ameliorating skill shortages. However, there has not yet been any research which establishes that pre-apprenticeship programs actually increase apprentice satisfaction and completion rates. This report uses data from the 2010 National Centre for Vocational Education Research (NCVER) Apprentice and Trainee Destination Survey and finds that there is no universal benefit attached to undertaking a pre-apprenticeship. Instead, the impact of pre-apprenticeships varies with occupation and prior education level. Key messages from this study are: (1) Pre-apprenticeships lead to only a modest increase in satisfaction with job-related aspects of apprenticeships (but not off-the-job training aspects); (2) Pre-apprenticeships increase the likelihood of completion for apprentices in the construction, food and electro-technology trades and those with a Year 10 or Year 12 level of education; (3) Pre-apprenticeships reduce the likelihood of completing an apprenticeship for hairdressers and apprentices in the automotive and engineering trades and for those people who already have a certificate III or higher qualification. This suggests that the design of pre-apprenticeships is important; and (4) In general, apprentices who have undertaken a pre-apprenticeship are less likely to discontinue their training because they did not like the type of work or training, but this does not translate into a higher likelihood of completion. Appendices include: (1) Apprentice and Trainee Destination Survey; (2) Tables; (3) Factor analysis--satisfaction; (4) Likelihood of completing an apprenticeship or traineeship; and (5) Reason for not completing an apprenticeship or traineeship. (Contains 18 tables, 2 figures and 6 footnotes.)

This paper draws on mathematics achievement data from Milwaukee middle school students to illustrate the value-added approach to estimating school performance. Value-added indicators differ from traditional attainment indicators such as average test scores in that they are designed to capture the contribution of schools to growth in student achievement. The indicators are thus based on longitudinal as opposed to cross-sectional student data. The empirical results of this study indicate that the (value-added) productivity of middle schools in Milwaukee varied substantially across schools over the period 1999 to 2000. Some middle schools generated almost twice as much achievement growth as the average middle school in Milwaukee. Others produced almost no growth in mathematics achievement from seventh to eighth grade. One very interesting finding is that most of the middle schools that served students with low prior achievement did a better than average job of producing growth in mathematics achievement. The overall statistical reliability of the school performance estimates was very high (89.4%). Finally, it was found that value-added indicators of school performance (based on longitudinal data) differed substantially from average mathematics attainment. The paper explains that attainment indicators such as an average test score are, in general, highly flawed as measures of school performance. (Contains 4 figures, 2 tables, and 10 footnotes.)

This study of social capital employs data from the Longitudinal Study of Adult Learning (LSAL), funded by the National Center for the Study of Adult Learning and Literacy (NCSALL) and conducted at Portland State University to increase understanding of the learning process of adults with limited formal education. It provides longitudinal data on how people continue learning in multiple contexts--through formal adult education programs and in their daily lives. This paper is presented in two sections. The first is a theoretical exploration of the intersections of critical literacy, adult learning, and social capital theories. The intent of this discussion is to introduce key concepts from multiple disciplines and develop a synthetic model of social capital influences on lifelong learning (SCILL) in which adults construct their social environment and the discourses that influence their learning strategies. The second section empirically tests a piece of the SCILL Model related to the structural and discursive dimensions of social capital, using data from the first wave of the LSAL. Multiple measures of social capital are detailed. The author argues that in addition to the structural dimension of social capital, local discourse generated through interaction is an interpretive dimension of social capital. The discussion ties the empirical and theoretical elements, offering implications for the study of adult education participation in adult education for research on social capital. Appendices include: (1) Longitudinal Study of Adult Literacy Study Design; (2) Instrument Items from LSAL Wave 1; (3) Confirmatory Factor Analysis of Lifelong Learning; (4) Tables of Descriptive Statistics and Univariate Analyses of Indicators; (5) Items Parameters in Final Models; and (6) Model Calibration. (Contains 18 tables, 3 figures, and 6 footnotes.)

The purpose of this essay is to explore both what is known and what needs to be learned about the information value of performance items "when they are used in large scale assessments." Within the context of the National Assessment of Educational Progress (NAEP), there is substantial motivation for answering these questions. Over the past decade, in order to adequately portray the breadth and depth of important curriculum standards, NAEP designers have invested substantial time and energy in creating extended constructed-response items and enormous financial resources in scoring these items. While these items are popular with curriculum experts within various content areas (Linn, Glaser, & Bohrnstedt, 1997), it is not clear whether they possess the marginal utility required to justify their cost; that is, they may not provide new information above and beyond that which is provided by a more standard mix of multiple-choice and short items with known measurement characteristics and much more economical scoring protocols. Even worse, there is some reason to believe, based on research in non-NAEP settings, that extended constructed-response items may provide negative returns in terms of the overall goal of accurate measurement of performance on some broad domain such as reading, mathematics, or science (see Forsyth, Hambleton, Linn, Mislevy, and Yen, 1996). To investigate this claim, the authors traversed the "information value" terrain along as many paths as they could find--combing the measurement literature to determine the various ways in which scholars have conceptualized and operationalized the "information value" construct, reviewing research conducted within those approaches, consulting essays that emphasize the importance of strong conceptual grounding in content frameworks, studying the broadest available construal of the construct of validity (e.g., Messick, 1989), and, finally, and most unsatisifyingly, attempting to determine, at a conceptual and philosophical level, what the assessment community means when they talk about the information provided by assessments. This essay is organized into four sections. First, the authors consider the construct of information value in its broadest philosophical sense and then describe the classical ways of operationalizing it. Second, they review the available literature within each of these operational traditions (IRT, factor analysis, correlational studies). Third, they consider some alternative versions of information value, based more on cognitive, conceptual, and pragmatic considerations. Finally, they outline a series of studies that they believe ought to be supported by the National Center for Educational Statistics (NCES) in order to answer the question of how NAEP can be modified to increase its "information value."

The purpose of this study was to provide an analysis and evaluation of composite variables in the National Education Longitudinal Study of 1988 (NELS:88) and School and Staffing Survey (SASS) in a way that provides guidance to the staff of the National Center for Education Statistics (NCES) in the more effective use of survey resources. The study provides ways of understanding the contributions made by individual survey items through an appraisal of the contributions they make to composite measures of which they are a part. It also shows how composite variables may provide more reliable measures of the concepts of interest than do individual survey items, and how they permit the more effective summarization and communication of survey results. The composite variables examined include: (1) self-concept and locus of control; (2) socioeconomic status; and (3) school climate. Additional variables designated by NCES and variables found to be of interest during the course of the study were also analyzed. The guiding question was whether a particular variable contributed to or detracted from the efficacy of a composite variable. The strategy followed in the study used several factor analyses followed by calculation of reliability and validity estimates on a given composite variable. While the field costs of surveys have escalated, the computer has made analysis comparatively quick and cheap, making it cost effective to conduct a variety of analyses of survey items and their composites. A summary of the analyses is included. (SLD)

Many multiple aptitude test batteries, including the Air Force Officer Qualifying Test (AFOQT), used for assigning or classifying individuals to jobs or for occupational counseling have subtests covering a broad range of content such as science, mathematics, reading, vocabulary, perceptual, mechanical, or technical knowledge. This content reflects a belief that performance in different jobs is best predicted by subtests whose content appears to be closely related to jobs. It has been demonstrated that the subtests of a multiple aptitude test battery all measure, in large part, an examinee's general learning ability, often called psychometric g, in addition to the specific abilities implied by the differing contents of the subtests. This study investigated methods for estimating psychometric g from the Air Force Officer Qualifying Test Form O. The methods used and compared were unrotated principal components, unrotated principal factors and variants of hierarchical factor analysis. Subjects were 2,984 applicants to Air Force commissioning programs. Results indicated that the methods produced estimates of g which were equal except for scale with a range of correlations from .980 to .999. This relationship was all predictable from a theorem presented by S. S. Wilks. (Contains 16 references and 12 tables.) (Author)

A statistical comparison was made of two test batteries, the Armed Services Vocational Aptitude Battery (ASVAB) and the General Aptitude Test Battery (GATB), using a sample of 406 subjects (98 civilian high school students and 308 military recruit examinees). The sample was predominantly white and male. A first analyses described the sample and its performance on the subtests and composites of the GATB and the Department of Defense Student Testing Program composites of the ASVAB. A second analysis investigated the extent to which the ASVAB can predict GATB subtests and composites, and vice versa. The third analysis was a canonical correlation of the subtests of the two batteries. The fourth analyses consisted of principal components factor analyses of the batteries separately and combined. Results show that the batteries do not overlap enough to be considered equivalent or interchangeable, but that they do share a large amount of variance. Such shared variance is to be expected in batteries that have been developed for occupational selection or guidance. Nineteen tables present analysis results. (Author/SLD)

Using the 2012 PIAAC data, our analysis confirms that there are significantly higher earnings and employment returns to "both" increasing levels of formally recognised education, and to increasing levels of numeracy, literacy and information and communication technologies (ICT) skills proficiencies controlling for the level of education. Unsurprisingly, the labour market returns to changes in formally recognised levels of education in general exceed the labour market returns associated with increasing levels of skills proficiency. In the case of literacy and numeracy proficiencies, improved literacy and numeracy skills narrow the labour market outcomes gap between individuals with different levels of formally recognised education, but do not close it completely. The analysis demonstrates more substantial returns to ICT skills. Furthermore, possession of higher levels of ICT skills and lower levels of formally recognised qualification are often associated with higher returns compared to individuals with higher levels of formally recognised education but lower ICT proficiency levels. In other words, ICT skills proficiencies often entirely compensate for lower formally recognised qualifications in the labour market. Contains the following annexes: (1) Country Literacy and Numeracy Profiles; and (2) Demographic Control Variables.

We study the relationship between school organizational contexts, teacher turnover, and student achievement in New York City (NYC) middle schools. Using factor analysis, we construct measures of four distinct dimensions of school contexts captured on the annual NYC School Survey. We identify credible estimates by isolating variation in organizational contexts within schools over time. We find that improvements in school leadership, academic expectations, teacher relationships, and school safety are all independently associated with corresponding reductions in teacher turnover. Increases in school safety and academic expectations for students also correspond to increases in student achievement. These results are robust to a range of potential threats to validity, suggesting that our findings are likely driven by an underlying causal relationship. The following is appended: Raw Factor Loadings for School Context Dimensions.