Search

This study describes the measurement properties of two curated surveys, consisting of previously developed scales, assessing student perceptions of their social-emotional competencies (SEC) and the school culture/climate (CC). The surveys were administered through the Transforming Education/NewSchools Venture Fund partnership that worked with the NewSchools Invent Cohort, a network of 23 schools across 11 states, to expand the definition of student success. The analytic sample for this study includes more than 3,000 students in grades 4 through 12 at 18 schools surveyed in the 2016-17 school year. Using classical test theory as our conceptual and analytic lens, we conducted tests of internal consistency, item discrimination, omit rates, differential item functioning, and exploratory and confirmatory factor analysis. Our findings provide initial evidence of the validity of the survey scores for measuring student perceptions of their SEC and the school CC in the NewSchools Invent schools, suggesting that the surveys are suitable for school and cohort-level decision-making related to students' social emotional competencies and experiences within the school. [Written with Education Analytics.]

Higher levels of civic and community engagement in higher education are positively associated with students' academic performance and they also build upon citizenship skills such as informed voting. Yet, while these are worthy and important outcomes of higher education, students from disadvantaged backgrounds can have more difficulty navigating civic engagement. Focusing on students at thirteen major public universities in the United States, and utilizing survey data generated by the Student Experience in the Research University (SERU) Consortium, this study suggest social perspective-taking has a significant positive effect on all students' probability of voting and participating in community service. Students who were asked to identify challenge/solutions to social problems and reflect/act on community issues in the classroom were also more likely to participate in community service. Additionally, Hispanic students and students from lower-income households are significantly less likely to vote and perform community service--findings corroborated by previous research. Females are more likely to vote and participate in community service--findings also corroborated by prior research. Hispanic students and students from low-income backgrounds were more likely to participate in voting and community service if they had increased rates of social perspective-taking. These results suggest that increased opportunities to connect with classmates in the classroom and develop perspective-taking skills (e.g. appreciating the world from someone else's perspective, acknowledging personal differences, interacting with someone with views that are different from your own, and discussing and navigating controversial issues) are potentially quite powerful ways to foster civic engagement among students who traditionally have the lowest civic engagement rates compared to their peers.

Identifying and understanding the foundational skills children need to participate effectively in formal schooling is an important objective for research in early childhood education. One component of school readiness is cognitive self-regulation (CSR). The question this study addresses is how to assess CSR with prekindergarten-aged children in a way that taps into the learning-related cognitive engagement behaviors teachers observe in the classroom that are predictive of later academic achievement. A number of candidate measures applicable to prekindergarten age children have been generated for research on attention, effortful control, executive function, and related constructs. A diverse set of twelve candidate measures that can be easily administered in school settings was selected from these domains and applied to a sample of prekindergarten children. These measures were then examined for construct validity, developmental change, convergent validity with teacher ratings of CSR, and predictive validity for subsequent academic achievement and achievement gain. Six measures performed well by these criteria: Peg Tapping, Head-Toes-Knees-Shoulders (HTKS), the Kansas Reflection-Impulsivity Scale for Preschoolers (KRISP), Dimensional Change Card Sort (DCCS), Copy Design, and Backwards Digit Span. Cross-validation with a new sample of children confirmed the validity of these measures, estimated their test-retest reliability, identified the best performing individual measures, and demonstrated that a composite score combining results from all these measures performed better than any single measure. The Scoring Scheme for the Child Measures of LRCSR is appended.

The sizable gender gap in college enrolment, especially among African Americans, constitutes a puzzling empirical regularity that may have serious consequences on marriage markets, male labor force participation and the diversity of college campuses. For instance, only 35.7 percent of all African American undergraduate students were men in 2004. Reduced form results show that, while family background covariates cannot account for the observed gap, proxy measures for non-cognitive skills are crucial to explain it. Moreover, a sequential model of educational attainment indicates that males have actually higher preferences for education than females after controlling for latent factors (i.e. cognitive and non-cognitive skills). The model also shows that cognitive skills strongly affect the decision to move from one school level to the next, especially after finishing high school, but cannot account for disparities between genders. On the contrary, the substantial differences in the distribution of non-cognitive skills between males and females make these abilities critical to explain the gender gap in educational attainment across and within races. Appendices present 12 tables and 1 figure.

Pre-apprenticeship programs have generated interest recently from government, employers and other stakeholders in the training system as one means of improving apprenticeship completion rates and thereby ameliorating skill shortages. However, there has not yet been any research which establishes that pre-apprenticeship programs actually increase apprentice satisfaction and completion rates. This report uses data from the 2010 National Centre for Vocational Education Research (NCVER) Apprentice and Trainee Destination Survey and finds that there is no universal benefit attached to undertaking a pre-apprenticeship. Instead, the impact of pre-apprenticeships varies with occupation and prior education level. Key messages from this study are: (1) Pre-apprenticeships lead to only a modest increase in satisfaction with job-related aspects of apprenticeships (but not off-the-job training aspects); (2) Pre-apprenticeships increase the likelihood of completion for apprentices in the construction, food and electro-technology trades and those with a Year 10 or Year 12 level of education; (3) Pre-apprenticeships reduce the likelihood of completing an apprenticeship for hairdressers and apprentices in the automotive and engineering trades and for those people who already have a certificate III or higher qualification. This suggests that the design of pre-apprenticeships is important; and (4) In general, apprentices who have undertaken a pre-apprenticeship are less likely to discontinue their training because they did not like the type of work or training, but this does not translate into a higher likelihood of completion. Appendices include: (1) Apprentice and Trainee Destination Survey; (2) Tables; (3) Factor analysis--satisfaction; (4) Likelihood of completing an apprenticeship or traineeship; and (5) Reason for not completing an apprenticeship or traineeship. (Contains 18 tables, 2 figures and 6 footnotes.)

This paper draws on mathematics achievement data from Milwaukee middle school students to illustrate the value-added approach to estimating school performance. Value-added indicators differ from traditional attainment indicators such as average test scores in that they are designed to capture the contribution of schools to growth in student achievement. The indicators are thus based on longitudinal as opposed to cross-sectional student data. The empirical results of this study indicate that the (value-added) productivity of middle schools in Milwaukee varied substantially across schools over the period 1999 to 2000. Some middle schools generated almost twice as much achievement growth as the average middle school in Milwaukee. Others produced almost no growth in mathematics achievement from seventh to eighth grade. One very interesting finding is that most of the middle schools that served students with low prior achievement did a better than average job of producing growth in mathematics achievement. The overall statistical reliability of the school performance estimates was very high (89.4%). Finally, it was found that value-added indicators of school performance (based on longitudinal data) differed substantially from average mathematics attainment. The paper explains that attainment indicators such as an average test score are, in general, highly flawed as measures of school performance. (Contains 4 figures, 2 tables, and 10 footnotes.)

This study of social capital employs data from the Longitudinal Study of Adult Learning (LSAL), funded by the National Center for the Study of Adult Learning and Literacy (NCSALL) and conducted at Portland State University to increase understanding of the learning process of adults with limited formal education. It provides longitudinal data on how people continue learning in multiple contexts--through formal adult education programs and in their daily lives. This paper is presented in two sections. The first is a theoretical exploration of the intersections of critical literacy, adult learning, and social capital theories. The intent of this discussion is to introduce key concepts from multiple disciplines and develop a synthetic model of social capital influences on lifelong learning (SCILL) in which adults construct their social environment and the discourses that influence their learning strategies. The second section empirically tests a piece of the SCILL Model related to the structural and discursive dimensions of social capital, using data from the first wave of the LSAL. Multiple measures of social capital are detailed. The author argues that in addition to the structural dimension of social capital, local discourse generated through interaction is an interpretive dimension of social capital. The discussion ties the empirical and theoretical elements, offering implications for the study of adult education participation in adult education for research on social capital. Appendices include: (1) Longitudinal Study of Adult Literacy Study Design; (2) Instrument Items from LSAL Wave 1; (3) Confirmatory Factor Analysis of Lifelong Learning; (4) Tables of Descriptive Statistics and Univariate Analyses of Indicators; (5) Items Parameters in Final Models; and (6) Model Calibration. (Contains 18 tables, 3 figures, and 6 footnotes.)

The purpose of this essay is to explore both what is known and what needs to be learned about the information value of performance items "when they are used in large scale assessments." Within the context of the National Assessment of Educational Progress (NAEP), there is substantial motivation for answering these questions. Over the past decade, in order to adequately portray the breadth and depth of important curriculum standards, NAEP designers have invested substantial time and energy in creating extended constructed-response items and enormous financial resources in scoring these items. While these items are popular with curriculum experts within various content areas (Linn, Glaser, & Bohrnstedt, 1997), it is not clear whether they possess the marginal utility required to justify their cost; that is, they may not provide new information above and beyond that which is provided by a more standard mix of multiple-choice and short items with known measurement characteristics and much more economical scoring protocols. Even worse, there is some reason to believe, based on research in non-NAEP settings, that extended constructed-response items may provide negative returns in terms of the overall goal of accurate measurement of performance on some broad domain such as reading, mathematics, or science (see Forsyth, Hambleton, Linn, Mislevy, and Yen, 1996). To investigate this claim, the authors traversed the "information value" terrain along as many paths as they could find--combing the measurement literature to determine the various ways in which scholars have conceptualized and operationalized the "information value" construct, reviewing research conducted within those approaches, consulting essays that emphasize the importance of strong conceptual grounding in content frameworks, studying the broadest available construal of the construct of validity (e.g., Messick, 1989), and, finally, and most unsatisifyingly, attempting to determine, at a conceptual and philosophical level, what the assessment community means when they talk about the information provided by assessments. This essay is organized into four sections. First, the authors consider the construct of information value in its broadest philosophical sense and then describe the classical ways of operationalizing it. Second, they review the available literature within each of these operational traditions (IRT, factor analysis, correlational studies). Third, they consider some alternative versions of information value, based more on cognitive, conceptual, and pragmatic considerations. Finally, they outline a series of studies that they believe ought to be supported by the National Center for Educational Statistics (NCES) in order to answer the question of how NAEP can be modified to increase its "information value."

The purpose of this study was to provide an analysis and evaluation of composite variables in the National Education Longitudinal Study of 1988 (NELS:88) and School and Staffing Survey (SASS) in a way that provides guidance to the staff of the National Center for Education Statistics (NCES) in the more effective use of survey resources. The study provides ways of understanding the contributions made by individual survey items through an appraisal of the contributions they make to composite measures of which they are a part. It also shows how composite variables may provide more reliable measures of the concepts of interest than do individual survey items, and how they permit the more effective summarization and communication of survey results. The composite variables examined include: (1) self-concept and locus of control; (2) socioeconomic status; and (3) school climate. Additional variables designated by NCES and variables found to be of interest during the course of the study were also analyzed. The guiding question was whether a particular variable contributed to or detracted from the efficacy of a composite variable. The strategy followed in the study used several factor analyses followed by calculation of reliability and validity estimates on a given composite variable. While the field costs of surveys have escalated, the computer has made analysis comparatively quick and cheap, making it cost effective to conduct a variety of analyses of survey items and their composites. A summary of the analyses is included. (SLD)

Many multiple aptitude test batteries, including the Air Force Officer Qualifying Test (AFOQT), used for assigning or classifying individuals to jobs or for occupational counseling have subtests covering a broad range of content such as science, mathematics, reading, vocabulary, perceptual, mechanical, or technical knowledge. This content reflects a belief that performance in different jobs is best predicted by subtests whose content appears to be closely related to jobs. It has been demonstrated that the subtests of a multiple aptitude test battery all measure, in large part, an examinee's general learning ability, often called psychometric g, in addition to the specific abilities implied by the differing contents of the subtests. This study investigated methods for estimating psychometric g from the Air Force Officer Qualifying Test Form O. The methods used and compared were unrotated principal components, unrotated principal factors and variants of hierarchical factor analysis. Subjects were 2,984 applicants to Air Force commissioning programs. Results indicated that the methods produced estimates of g which were equal except for scale with a range of correlations from .980 to .999. This relationship was all predictable from a theorem presented by S. S. Wilks. (Contains 16 references and 12 tables.) (Author)

A statistical comparison was made of two test batteries, the Armed Services Vocational Aptitude Battery (ASVAB) and the General Aptitude Test Battery (GATB), using a sample of 406 subjects (98 civilian high school students and 308 military recruit examinees). The sample was predominantly white and male. A first analyses described the sample and its performance on the subtests and composites of the GATB and the Department of Defense Student Testing Program composites of the ASVAB. A second analysis investigated the extent to which the ASVAB can predict GATB subtests and composites, and vice versa. The third analysis was a canonical correlation of the subtests of the two batteries. The fourth analyses consisted of principal components factor analyses of the batteries separately and combined. Results show that the batteries do not overlap enough to be considered equivalent or interchangeable, but that they do share a large amount of variance. Such shared variance is to be expected in batteries that have been developed for occupational selection or guidance. Nineteen tables present analysis results. (Author/SLD)

Large-sample item response data for the 10-item Paper Folding worksample 622 (N=2,749) and for five new experimental paper folding items (N=2,514) are analyzed with the logistic item response model and with full-information item factor analysis. The main results of the unidimensional analysis are that: (1) item discrimination is heterogeneous, so that a Rasch model cannot be ascertained for the Paper Folding worksample; (2) item difficulty does not increase with presentation order; and (3) there are not enough items in the midrange of the scale. Item factor analysis identifies an additional minor factor attributable to during-the-test learning of specific solutions for items with similar stems. Based on these results, a modified 11-item version of the Paper Folding worksample is proposed. (Six graphs and five tables present the data.) (Author)

Structural integrity of a measure is defined in terms of its replicability, constancy, invariance, and stability. Work completed in the development and validation of the Self Observation Scales (SOS) Primary Level (Stenner and Katzenmeyer, 1973) serves to illustrate one method of establishing structural integrity. The name of each scale of the SOS is a construct: an abbreviated thesis about the nature of the underlying variable which determines the configuration of indicants (questions) comprising the scale. SOS constructs (self acceptance, social maturity, school affiliation, and self security) are analyzed for different student groups. Traditional psychometrics assumes that the same questions measure the same underlying variables in all respondents; the validity of interperson and intergroup comparisons rests on this assumption. However, ample evidence suggests that this assumption is frequently in error. Procedures outlined in this paper provide a method for assessing the validity of this assumption, and a basis for developing reliable and valid instruments. (Author/MV)

This paper is an extended series of appendices to the report "Exploring the T.R.Q.: An Assessment of the Effectiveness of the Teachers' Rating Questionnaire," Report Number 123. It should be most useful to those familiar with multiple regression analysis and factor analysis and wishing a more complex record of the statistical procedures originally employed. This supplement takes a precise look at estimation of reliability coefficients from delayed retest data, indices used in evaluating theoretical distributions and analysis of the correlation matrices of the TRQ questions. (Author)

In 1969, the American Educational Research Association (AERA) empaneled a Task Force on Research Training and charged it with (1) building a relevant knowledge base about training needs and problems and (2) developing procedures to attack these problems. During its two years of study, the Task Force carried out a series of three USOE-supported activities designed to identify specific skills and knowledge required in educational research and evaluation. The first activity included two steps. First, the Task Force drafted lists of skills they thought were necessary in conducting research and evaluation in education. Second, the lists of skills were "reality tested" through interviews with 60 employers and supervisors of research and evaluation personnel. The second activity was not strictly empirical in nature, although it used as input the results yielded by the interviews described above and depended on informal observation for the rest. In the third activity, a task analysis of research and evaluation activities in 13 agencies was conducted. These agencies included universities, research and development centers, private research agencies, regional educational laboratories, and public schools rated by the Task Force members as doing high quality work. The purpose of this paper is to synthesize the results of these three previous Task Force efforts into a single list of competencies frequently needed in educational research and evaluation. (MV)

Trait-Treatment Interaction (TTI), a research method for observing experimental effects of treatments on subjects of different aptitudes and learning characteristics, is suggested as an effective evaluation tool to provide evaluators and educators in compensatory education programs with information about which program is best for different kinds of learners. The premise of TTI research is that different instructional conditions work best when matched with selected learner traits, with "trait" defined as any characteristic of the learner that increases or impairs his probability of success in a given treatment. The methodology of Trait-Treatment Interaction (TTI) as outlined provides an alternative of the dilemma of control groups; and a proposed evaluation design is presented using Campbell's terminology of invited-accepted, invited-rejected, and uninvited. To tap the interaction of learner characteristics with instruction, the invited-accepted are tested and blocked or grouped on relevant learner characteristics, and then, within each block, randomly assigned to a treatment. The goal of this TTI factorial design is to develop alternative instructional programs for compensatory education which produce optimal educational payoff among pupils assigned differently to these programs on the basis of learner characteristics. (CS)

This study explores the use of the adjunct question paradigm in self-instruction and distance education materials. The subjects were 187 high school students who read text passages and answered one high level (implication or inference) or low level (verbatim or paraphrase) question. These questions were either inserted in the text or massed at the end of the passage. Subjects were either allowed to look back at the text while answering the questions (the freedom group, FREE) or were not allowed to look back (the read-read-question group, RRQ; the read-question-read group, RQR). One control group read the passages twice without questions and another neither read the passages nor answered the questions. All subjects were tested on recall of low level and performance on high level posttest items. On high level incidental items, the RQR group performed better than the RRQ group, but the FREE group recalled more low level incidental material than the RQR group. In the two no-lookback groups, those who were asked high level adjunct questions performed better when the questions were massed together, while those who were asked low level adjunct questions performed better when the questions were inserted. In general the lookback group recalled as much, and performed as well, as the two no-lookback groups. For all incidental dependent variables, most treatment groups did not score significantly higher than the READ-TWICE control group. Discussions of the limitations of the study that may have influenced the results and the implications of the findings conclude the report. A reference list of 101 items is appended. (Author/RP)

This study investigated the effects of scorekeeping on student motivation in a computer assisted arithmetic drill and practice game. A 2x2 factorial design was used which incorporated the four treatments formed by time score (present and absent) and rank score (present and absent). Selected on the basis of their previous experience with the procedure of carrying in addition problems, the subjects were 52 second grade students who were randomly assigned to the four treatments. After a brief introduction and demonstration of the rules of the game, each student could elect to play further rounds to a maximum of 15 minutes. Motivation was measured by the number of rounds elected. An analysis of variance calculated for the number of rounds played indicated that there was no significant treatment effect. Discussion of the results and suggestions for further studies conclude the report. (Author/RP)

A new typology for faculty is proposed, based on their expressed priorities toward activities in which research universities engage. Responses of 617 faculty to a survey of 66 statements about university activities were factor analyzed, and scores on eight factors obtained were used in a cluster analysis that produced three clusters of faculty. When coded into disciplinary groupings, the faculty clustered along lines defined by their political description and gender. Items included in the following eight factors and their loadings are identified: (1) concerns about minorities and women; (2) liberal arts, liberal education; (3) nontraditional education; (4) intercollegiate athletics; (5) concerns about quality and research; (6) health care and training; (7) professional education; and (8) concerns about student health. The primary strength of this approach involves the effort to relate faculty groupings to institutional priorities as well as the assumption of heterogeneity rather than homogeneity among faculty members belonging to a specific department. The results are compared to Biglan's (1973) typology; some correspondence is found with his "hard-soft" axis, or applied-pure and life-non-life dimensions, which were used to classify departments. (Author/SW)

This study examined the effects of the type of adjunct questions (use-a-generality, remember-a-generality, and remember-an-instance) and the position of these questions either before the relevant text passage (pre-questions) or after it (post-questions) on the three corresponding performance levels of learning. The subjects were 74 eighth-grade students at a suburban school in Syracuse, New York, and the materials for the learning task consisted of an approximately 450-word passage selected from a social studies textbook for this grade level. Six different treatment groups were used, with each group receiving adjunct questions which varied as to performance level and placement of the questions. The results were analyzed separately for each level of performance as well as for the total test score. None of the effects reached significance for any of the subtests. These results suggest that, in order to optimize learning, the position of the adjunct questions must be taken into account as well as the type of question to be used. An 18-item reference list is provided. (RP)