Search

Three mode comparability studies were conducted on the following Saturday national ACT test dates: October 26, 2019, December 14, 2019, and February 8, 2020. The primary goal of these studies was to evaluate whether ACT scores exhibited mode effects between paper and online testing that would necessitate statistical adjustments to the online scores to make them comparable to paper scores. All participants took the full ACT multiple-choice test (English, math, reading, and science), and participants in the February study had the option to take the writing test. The studies each employed an experimental, randomly equivalent groups design. All study participants received college-reportable scores, so their motivation to perform well on the ACT was expected to be like typical examinees during a national testing event. Findings reveal that students who test online tend to perform slightly better than students who test on paper, especially on the English, reading, and writing tests. ACT will equate scores across modes as needed to ensure that ACT scores can be treated as interchangeable regardless of testing mode, and ACT will continue to monitor comparability between paper and online testing and investigate reasons for observed mode effects on the ACT test.

This study describes the measurement properties of two curated surveys, consisting of previously developed scales, assessing student perceptions of their social-emotional competencies (SEC) and the school culture/climate (CC). The surveys were administered through the Transforming Education/NewSchools Venture Fund partnership that worked with the NewSchools Invent Cohort, a network of 23 schools across 11 states, to expand the definition of student success. The analytic sample for this study includes more than 3,000 students in grades 4 through 12 at 18 schools surveyed in the 2016-17 school year. Using classical test theory as our conceptual and analytic lens, we conducted tests of internal consistency, item discrimination, omit rates, differential item functioning, and exploratory and confirmatory factor analysis. Our findings provide initial evidence of the validity of the survey scores for measuring student perceptions of their SEC and the school CC in the NewSchools Invent schools, suggesting that the surveys are suitable for school and cohort-level decision-making related to students' social emotional competencies and experiences within the school. [Written with Education Analytics.]

Higher levels of civic and community engagement in higher education are positively associated with students' academic performance and they also build upon citizenship skills such as informed voting. Yet, while these are worthy and important outcomes of higher education, students from disadvantaged backgrounds can have more difficulty navigating civic engagement. Focusing on students at thirteen major public universities in the United States, and utilizing survey data generated by the Student Experience in the Research University (SERU) Consortium, this study suggest social perspective-taking has a significant positive effect on all students' probability of voting and participating in community service. Students who were asked to identify challenge/solutions to social problems and reflect/act on community issues in the classroom were also more likely to participate in community service. Additionally, Hispanic students and students from lower-income households are significantly less likely to vote and perform community service--findings corroborated by previous research. Females are more likely to vote and participate in community service--findings also corroborated by prior research. Hispanic students and students from low-income backgrounds were more likely to participate in voting and community service if they had increased rates of social perspective-taking. These results suggest that increased opportunities to connect with classmates in the classroom and develop perspective-taking skills (e.g. appreciating the world from someone else's perspective, acknowledging personal differences, interacting with someone with views that are different from your own, and discussing and navigating controversial issues) are potentially quite powerful ways to foster civic engagement among students who traditionally have the lowest civic engagement rates compared to their peers.

In preparation for online administration of the ACT® test, ACT conducted studies to examine the comparability of scores between online and paper administrations, including a timing study in fall 2013, a mode comparability study in spring 2014, and a second mode comparability study in spring 2015. This report presents major findings from these studies, focusing on the mode comparability studies.

Identifying and understanding the foundational skills children need to participate effectively in formal schooling is an important objective for research in early childhood education. One component of school readiness is cognitive self-regulation (CSR). The question this study addresses is how to assess CSR with prekindergarten-aged children in a way that taps into the learning-related cognitive engagement behaviors teachers observe in the classroom that are predictive of later academic achievement. A number of candidate measures applicable to prekindergarten age children have been generated for research on attention, effortful control, executive function, and related constructs. A diverse set of twelve candidate measures that can be easily administered in school settings was selected from these domains and applied to a sample of prekindergarten children. These measures were then examined for construct validity, developmental change, convergent validity with teacher ratings of CSR, and predictive validity for subsequent academic achievement and achievement gain. Six measures performed well by these criteria: Peg Tapping, Head-Toes-Knees-Shoulders (HTKS), the Kansas Reflection-Impulsivity Scale for Preschoolers (KRISP), Dimensional Change Card Sort (DCCS), Copy Design, and Backwards Digit Span. Cross-validation with a new sample of children confirmed the validity of these measures, estimated their test-retest reliability, identified the best performing individual measures, and demonstrated that a composite score combining results from all these measures performed better than any single measure. The Scoring Scheme for the Child Measures of LRCSR is appended.

Large-scale surveys such as the Programme for International Student Assessment (PISA), the Teaching and Learning International Survey (TALIS), and the Programme for the International Assessment of Adult Competences (PIAAC) use advanced statistical models to estimate scores of latent traits from multiple observed responses. The comparison of such estimated scores across different groups of respondents is valid to the extent that the same set of estimated parameters holds in each group surveyed. This issue of invariance of parameter estimates is addressed in model fit indices which gauge the likelihood that one set of parameters can be used across all groups. Therefore, the problem of scale invariance across groups of respondents can typically be framed as the question of how well a single model fits the responses of all groups. However, the procedures used to evaluate the fit of these models pose a series of theoretical and practical problems. The most commonly applied procedures to establish invariance of cognitive and non-cognitive scales across countries in large-scale surveys are developed within the framework of confirmatory factor analysis and item response theory. The criteria that are commonly applied to evaluate the fit of such models, such as the decrement of the Comparative Fit Index in confirmatory factor analysis, work normally well in the comparison of a small number of countries or groups, but can perform poorly in large-scale surveys featuring a large number of countries. More specifically, the common criteria often result in the non-rejection of metric invariance; however, the step from metric invariance (i.e. identical factor loadings across countries) to scalar invariance (i.e. identical intercepts, in addition to identical factor loadings) appears to set overly restrictive standards for scalar invariance (i.e. identical intercepts). This report sets out to identify and apply novel procedures to evaluate model fit across a large number of groups, or novel scaling models that are more likely to pass common model fit criteria. Using both real and simulated data, the following procedures are described and applied: multigroup confirmatory factor analysis, followed by alignment analysis of the same data set; Bayesian approximate measurement invariance; Bayesian measurement invariance testing in Item-Response Theory (IRT) models; and multigroup and multilevel latent class analysis. These approaches have the potential to resolve recurrent fit problems in invariance testing. Though promising, more work with these new approaches is needed to establish their suitability in large-scale surveys. The last chapter reports the conclusions from a conference in which these approaches were discussed, along with traditional approaches, in order to provide recommendations for how to address invariance issues in OECD education surveys.

The sizable gender gap in college enrolment, especially among African Americans, constitutes a puzzling empirical regularity that may have serious consequences on marriage markets, male labor force participation and the diversity of college campuses. For instance, only 35.7 percent of all African American undergraduate students were men in 2004. Reduced form results show that, while family background covariates cannot account for the observed gap, proxy measures for non-cognitive skills are crucial to explain it. Moreover, a sequential model of educational attainment indicates that males have actually higher preferences for education than females after controlling for latent factors (i.e. cognitive and non-cognitive skills). The model also shows that cognitive skills strongly affect the decision to move from one school level to the next, especially after finishing high school, but cannot account for disparities between genders. On the contrary, the substantial differences in the distribution of non-cognitive skills between males and females make these abilities critical to explain the gender gap in educational attainment across and within races. Appendices present 12 tables and 1 figure.

Pre-apprenticeship programs have generated interest recently from government, employers and other stakeholders in the training system as one means of improving apprenticeship completion rates and thereby ameliorating skill shortages. However, there has not yet been any research which establishes that pre-apprenticeship programs actually increase apprentice satisfaction and completion rates. This report uses data from the 2010 National Centre for Vocational Education Research (NCVER) Apprentice and Trainee Destination Survey and finds that there is no universal benefit attached to undertaking a pre-apprenticeship. Instead, the impact of pre-apprenticeships varies with occupation and prior education level. Key messages from this study are: (1) Pre-apprenticeships lead to only a modest increase in satisfaction with job-related aspects of apprenticeships (but not off-the-job training aspects); (2) Pre-apprenticeships increase the likelihood of completion for apprentices in the construction, food and electro-technology trades and those with a Year 10 or Year 12 level of education; (3) Pre-apprenticeships reduce the likelihood of completing an apprenticeship for hairdressers and apprentices in the automotive and engineering trades and for those people who already have a certificate III or higher qualification. This suggests that the design of pre-apprenticeships is important; and (4) In general, apprentices who have undertaken a pre-apprenticeship are less likely to discontinue their training because they did not like the type of work or training, but this does not translate into a higher likelihood of completion. Appendices include: (1) Apprentice and Trainee Destination Survey; (2) Tables; (3) Factor analysis--satisfaction; (4) Likelihood of completing an apprenticeship or traineeship; and (5) Reason for not completing an apprenticeship or traineeship. (Contains 18 tables, 2 figures and 6 footnotes.)

This paper draws on mathematics achievement data from Milwaukee middle school students to illustrate the value-added approach to estimating school performance. Value-added indicators differ from traditional attainment indicators such as average test scores in that they are designed to capture the contribution of schools to growth in student achievement. The indicators are thus based on longitudinal as opposed to cross-sectional student data. The empirical results of this study indicate that the (value-added) productivity of middle schools in Milwaukee varied substantially across schools over the period 1999 to 2000. Some middle schools generated almost twice as much achievement growth as the average middle school in Milwaukee. Others produced almost no growth in mathematics achievement from seventh to eighth grade. One very interesting finding is that most of the middle schools that served students with low prior achievement did a better than average job of producing growth in mathematics achievement. The overall statistical reliability of the school performance estimates was very high (89.4%). Finally, it was found that value-added indicators of school performance (based on longitudinal data) differed substantially from average mathematics attainment. The paper explains that attainment indicators such as an average test score are, in general, highly flawed as measures of school performance. (Contains 4 figures, 2 tables, and 10 footnotes.)

The purpose of this study was to examine the factor structure that underlies students' perceptions of intellectual progress and personal growth and to suggest additional ways of using survey information. The study was based on a sample of 68,650 students who completed the American College Testing College Outcomes between 1999 and 2001. From the Progress scale of 26 items, researchers identified four factors: General Education and Skills; Critical Thinking; Science, Mathematics, and Technology; and Career Preparation. From the Growth scale of 36 items, four other factors were identified: Academic and Whole Person Skills; Social Interaction Skills, Political and Societal Awareness; and Personal and Spiritual Values. These two sets of factors were similar across institution type, especially with regard to items in each factor that had the highest factor loadings. In addition, factor means were computed for each institutional type so that colleges can compare their results with those of similar institutions. (Contains 12 tables and 22 references.) (Author/SLD)

This study of social capital employs data from the Longitudinal Study of Adult Learning (LSAL), funded by the National Center for the Study of Adult Learning and Literacy (NCSALL) and conducted at Portland State University to increase understanding of the learning process of adults with limited formal education. It provides longitudinal data on how people continue learning in multiple contexts--through formal adult education programs and in their daily lives. This paper is presented in two sections. The first is a theoretical exploration of the intersections of critical literacy, adult learning, and social capital theories. The intent of this discussion is to introduce key concepts from multiple disciplines and develop a synthetic model of social capital influences on lifelong learning (SCILL) in which adults construct their social environment and the discourses that influence their learning strategies. The second section empirically tests a piece of the SCILL Model related to the structural and discursive dimensions of social capital, using data from the first wave of the LSAL. Multiple measures of social capital are detailed. The author argues that in addition to the structural dimension of social capital, local discourse generated through interaction is an interpretive dimension of social capital. The discussion ties the empirical and theoretical elements, offering implications for the study of adult education participation in adult education for research on social capital. Appendices include: (1) Longitudinal Study of Adult Literacy Study Design; (2) Instrument Items from LSAL Wave 1; (3) Confirmatory Factor Analysis of Lifelong Learning; (4) Tables of Descriptive Statistics and Univariate Analyses of Indicators; (5) Items Parameters in Final Models; and (6) Model Calibration. (Contains 18 tables, 3 figures, and 6 footnotes.)

This paper demonstrates the application of common statistical methods to evaluate the dimensionality, reliability, generalizability, and potential biasing factors of the student assessment of instruction (SAI) instrument used at Florida Atlantic University. Findings indicated: (1) factor analysis uncovered just two factors, one describing instruction effectiveness and one describing workload/difficulty; (2) reliability of the class-average response depended on the number of students rating the class, raising questions about use of such averages in tenure and promotion decisions; (3) generalizability analyses indicated the SAIs primarily reflect the effectiveness of the instructor rather than the influence of the course; and (4) positive correlations were found between three potential biasing variables--student ratings, both expected and actual grades, class size, and workload/difficulty. (DB)

This paper outlines procedures used to derive variables from data in the National Survey of Postsecondary Faculty; these variables were then used to create measures not expressly included as items in that survey. The derived variables were used to examine faculty satisfaction in two contexts: first, the complexity of satisfaction, and second, the importance of the interaction between faculty professional values and institutional norms. The sample consisted of 480 nonproprietary postsecondary institutions in the United States, granting 2-year or higher degrees. Faculty were selected through stratified random sampling and were then further stratified by program area. Four independent variables were identified: tenure status, academic discipline, scholarly activity, and role concept; the dependent variable was measured by a Likert-like scale of faculty responses concerning adequacy of institutional/departmental funding, professional authority/autonomy, institutional mission/philosophy, administrative leadership, relationships with colleagues, and support services. Factor analysis reduced the original 19 measures to three factors: collegiality, workload, and autonomy. The paper concludes that use of large samples such as those collected by the National Center for Education Statistics will prove to be an important tool for researchers studying issues facing education in general and postsecondary education in particular. (Contain 30 references.)(CH)

The purpose of this essay is to explore both what is known and what needs to be learned about the information value of performance items "when they are used in large scale assessments." Within the context of the National Assessment of Educational Progress (NAEP), there is substantial motivation for answering these questions. Over the past decade, in order to adequately portray the breadth and depth of important curriculum standards, NAEP designers have invested substantial time and energy in creating extended constructed-response items and enormous financial resources in scoring these items. While these items are popular with curriculum experts within various content areas (Linn, Glaser, & Bohrnstedt, 1997), it is not clear whether they possess the marginal utility required to justify their cost; that is, they may not provide new information above and beyond that which is provided by a more standard mix of multiple-choice and short items with known measurement characteristics and much more economical scoring protocols. Even worse, there is some reason to believe, based on research in non-NAEP settings, that extended constructed-response items may provide negative returns in terms of the overall goal of accurate measurement of performance on some broad domain such as reading, mathematics, or science (see Forsyth, Hambleton, Linn, Mislevy, and Yen, 1996). To investigate this claim, the authors traversed the "information value" terrain along as many paths as they could find--combing the measurement literature to determine the various ways in which scholars have conceptualized and operationalized the "information value" construct, reviewing research conducted within those approaches, consulting essays that emphasize the importance of strong conceptual grounding in content frameworks, studying the broadest available construal of the construct of validity (e.g., Messick, 1989), and, finally, and most unsatisifyingly, attempting to determine, at a conceptual and philosophical level, what the assessment community means when they talk about the information provided by assessments. This essay is organized into four sections. First, the authors consider the construct of information value in its broadest philosophical sense and then describe the classical ways of operationalizing it. Second, they review the available literature within each of these operational traditions (IRT, factor analysis, correlational studies). Third, they consider some alternative versions of information value, based more on cognitive, conceptual, and pragmatic considerations. Finally, they outline a series of studies that they believe ought to be supported by the National Center for Educational Statistics (NCES) in order to answer the question of how NAEP can be modified to increase its "information value."

This study describes the use of event history modeling as a tool for understanding student departure from college. Using data from the National Center for Education Statistics High School and Beyond (HS&B)/Sophomore Cohort longitudinal study a detailed history of individual students' college careers was constructed using a regression-like methodology often called "survival analysis" or "hazard modeling". The model used customized event history files that included student files (e.g., transcripts), course files (college courses, year-specific college credits, grade point averages, and cumulative college credits), and institution-specific information (type and selectivity of institution) to create indicators of enrollment status, yearly performance data, time to graduation. Other factors considered by the analysis included financial aid and grade point average. It was concluded that the study demonstrated the potential of using HS&B data to study student outcomes. And it also demonstrates how event history modeling can be applied to longitudinal data to study events that take place over time. A third outcome of the study is the information provided to researchers and policymakers on the factors that affect student departure from college and how these effects change over time. Four figures and 11 data tables are included. (Contains approximately 53 references.) (CH)

This paper presents the results of a study that examined the relationship between parental income and students' college choice process, and identified factors influencing enrollment decisions of students from different income levels. The study found several statistically significant relationships between students' income and the college choice process, with higher income students attributing more importance to the college's surroundings and lower income students focusing more on substantive issues such as opportunities for internships, the academic program, and costs. Among enrolling students, significantly greater numbers of higher income students rated their college of choice positively on academic reputation, quality of the faculty, majors of interest, and perception of academic challenge, while lower income enrolling students rated the college on surroundings, social life, extracurricular activities, and cost. The study recommends improving vibrancy of college social life, developing collaborative programs with others, intensifying efforts to promote the college as prestigious and selective, and building on the college's strengths by increasing contact between prospective students and faculty members. (Contains 14 references.) (CH)

This study examines the undergraduate characteristics and factors in the undergraduate experience that appear to correlate with beneficial classroom experiences. The study was conducted at a large research university with an undergraduate population of about 10,000 students. Data were derived from a survey instrument that is part of the university's on-going assessment program, with 150 items of information collected in four categories: (1) background information (age, class year, sex, ethnicity, employment, admissions status, academic major, financial aid, residence), Scholastic Assessment Test scores, and high school grade point averages; (2) student plans, goals, and reasons for attendance; (3) level of student satisfaction with a variety of campus services and facilities, as well as with the institution's academic, administrative, and social environments; and (4) cognitive and noncognitive experiences, such as classroom experiences, faculty contact, course-taking patterns, and self-reported growth. Multivariate data analysis examined surveys of 740 undergraduates who entered the university as freshmen. The study found that the most beneficial classroom experiences were reported by students who perceived high levels of faculty concern and interaction with students, who reported active engagement in the academic and social structure of the institution, and who observed a campus climate of racial harmony and political tolerance. (Contains 52 references.) (CH)

The impact of the use of data arising from balanced incomplete block (BIB) spiralled designs on the chi-square goodness-of-fit test in factor analysis is considered. Data from BIB designs posses a unique pattern of missing data that can be characterized as missing completely at random (MCAR). Standard approaches to factor analyzing such data rest on forming pairwise available case (PAC) correlation matrices. Developments in statistical theory for missing data show that PAC correlation matrices may not satisfy Wishart distribution assumptions underlying factor analysis, this impacting tests of model fit. A new approach for handling missing data in structural equation modeling advocated by B. Muthen, D. Kaplan, and M. Hollis (1987) is proposed as a possible solution to these problems. The new approach is compared to the standard PAC approach in a Monte Carlo simulation framework. Simulation results show that tests of goodness-of-fit are very sensitive to PAC approaches even when data are MCAR, as is the case for BIB designs. The new approach outperforms the PAC approach for continuous variables and is comparatively much better for dichotomous variables. One table and one figure illustrate the discussion. (SLD)

The purpose of this study was to provide an analysis and evaluation of composite variables in the National Education Longitudinal Study of 1988 (NELS:88) and School and Staffing Survey (SASS) in a way that provides guidance to the staff of the National Center for Education Statistics (NCES) in the more effective use of survey resources. The study provides ways of understanding the contributions made by individual survey items through an appraisal of the contributions they make to composite measures of which they are a part. It also shows how composite variables may provide more reliable measures of the concepts of interest than do individual survey items, and how they permit the more effective summarization and communication of survey results. The composite variables examined include: (1) self-concept and locus of control; (2) socioeconomic status; and (3) school climate. Additional variables designated by NCES and variables found to be of interest during the course of the study were also analyzed. The guiding question was whether a particular variable contributed to or detracted from the efficacy of a composite variable. The strategy followed in the study used several factor analyses followed by calculation of reliability and validity estimates on a given composite variable. While the field costs of surveys have escalated, the computer has made analysis comparatively quick and cheap, making it cost effective to conduct a variety of analyses of survey items and their composites. A summary of the analyses is included. (SLD)

Many multiple aptitude test batteries, including the Air Force Officer Qualifying Test (AFOQT), used for assigning or classifying individuals to jobs or for occupational counseling have subtests covering a broad range of content such as science, mathematics, reading, vocabulary, perceptual, mechanical, or technical knowledge. This content reflects a belief that performance in different jobs is best predicted by subtests whose content appears to be closely related to jobs. It has been demonstrated that the subtests of a multiple aptitude test battery all measure, in large part, an examinee's general learning ability, often called psychometric g, in addition to the specific abilities implied by the differing contents of the subtests. This study investigated methods for estimating psychometric g from the Air Force Officer Qualifying Test Form O. The methods used and compared were unrotated principal components, unrotated principal factors and variants of hierarchical factor analysis. Subjects were 2,984 applicants to Air Force commissioning programs. Results indicated that the methods produced estimates of g which were equal except for scale with a range of correlations from .980 to .999. This relationship was all predictable from a theorem presented by S. S. Wilks. (Contains 16 references and 12 tables.) (Author)

A statistical comparison was made of two test batteries, the Armed Services Vocational Aptitude Battery (ASVAB) and the General Aptitude Test Battery (GATB), using a sample of 406 subjects (98 civilian high school students and 308 military recruit examinees). The sample was predominantly white and male. A first analyses described the sample and its performance on the subtests and composites of the GATB and the Department of Defense Student Testing Program composites of the ASVAB. A second analysis investigated the extent to which the ASVAB can predict GATB subtests and composites, and vice versa. The third analysis was a canonical correlation of the subtests of the two batteries. The fourth analyses consisted of principal components factor analyses of the batteries separately and combined. Results show that the batteries do not overlap enough to be considered equivalent or interchangeable, but that they do share a large amount of variance. Such shared variance is to be expected in batteries that have been developed for occupational selection or guidance. Nineteen tables present analysis results. (Author/SLD)

Using the 2012 PIAAC data, our analysis confirms that there are significantly higher earnings and employment returns to "both" increasing levels of formally recognised education, and to increasing levels of numeracy, literacy and information and communication technologies (ICT) skills proficiencies controlling for the level of education. Unsurprisingly, the labour market returns to changes in formally recognised levels of education in general exceed the labour market returns associated with increasing levels of skills proficiency. In the case of literacy and numeracy proficiencies, improved literacy and numeracy skills narrow the labour market outcomes gap between individuals with different levels of formally recognised education, but do not close it completely. The analysis demonstrates more substantial returns to ICT skills. Furthermore, possession of higher levels of ICT skills and lower levels of formally recognised qualification are often associated with higher returns compared to individuals with higher levels of formally recognised education but lower ICT proficiency levels. In other words, ICT skills proficiencies often entirely compensate for lower formally recognised qualifications in the labour market. Contains the following annexes: (1) Country Literacy and Numeracy Profiles; and (2) Demographic Control Variables.

We study the relationship between school organizational contexts, teacher turnover, and student achievement in New York City (NYC) middle schools. Using factor analysis, we construct measures of four distinct dimensions of school contexts captured on the annual NYC School Survey. We identify credible estimates by isolating variation in organizational contexts within schools over time. We find that improvements in school leadership, academic expectations, teacher relationships, and school safety are all independently associated with corresponding reductions in teacher turnover. Increases in school safety and academic expectations for students also correspond to increases in student achievement. These results are robust to a range of potential threats to validity, suggesting that our findings are likely driven by an underlying causal relationship. The following is appended: Raw Factor Loadings for School Context Dimensions.

Large-sample item response data for the 10-item Paper Folding worksample 622 (N=2,749) and for five new experimental paper folding items (N=2,514) are analyzed with the logistic item response model and with full-information item factor analysis. The main results of the unidimensional analysis are that: (1) item discrimination is heterogeneous, so that a Rasch model cannot be ascertained for the Paper Folding worksample; (2) item difficulty does not increase with presentation order; and (3) there are not enough items in the midrange of the scale. Item factor analysis identifies an additional minor factor attributable to during-the-test learning of specific solutions for items with similar stems. Based on these results, a modified 11-item version of the Paper Folding worksample is proposed. (Six graphs and five tables present the data.) (Author)