Search

Recently, the authors have been exploring the use of propensity score methods for developing evidence of program impact. Specifically, they have been developing evidence (after one year of implementation) of the effects of the Math Science Partnership in New York City ("MSPinNYC2") on high school students' achievement--both in terms of course grades and scores on end-ofcourse tests in two key Science, Technology, Engineering and Mathematics (STEM) disciplines: Integrated Algebra and Living Environment. Using an evidence-based approach which relies on propensity score matching, the authors asked if the program in its early stages is making a difference in students' academic achievement and college readiness. The "MSPinNYC2" program restructures early high school STEM courses to include 6-8 Teaching Assistant Scholars (TAS) who, along with the teachers, facilitate in-classroom group work on a daily basis. Early pilot studies suggested the model of peer-enabled restructured classrooms (PERC) increases student achievement and narrows the achievement gap in high school STEM courses. In its inaugural year the "MSPinNYC2" project recruited over 700 students (n = 711) in four New York City public high schools to participate in PERC classes. The students served by the project are not the academically elite (e.g., only about 20-25% were proficient in math and/or English language arts at the end of 8th grade). In the first year (2011-12) the PERC courses were taught by eleven different high school teachers--all trained in the PERC pedagogical model by the Program's staff. Each course served between 25-30 students who were tutored in-class by 6 or 7 TAS. The "MSPinNYC2" program is a multi-year, multi-site STEM intervention, with new cohorts of students entering the program each year, it is essential to develop a sound, rigorous method for evaluating the effectiveness and the scalability of the intervention--one that can be replicated each year. The authors are keenly interested in models that allow them to replicate their results from year-to-year with each new cohort of students. Thus, the purpose of this proposed study is to explore the use of a propensity score model that uses a genetic matching algorithm to evaluate the replicability of limited first-year findings, using as participants a new cohort of students who participated in the second year of the program. Specifically, they examine: (1) the effectiveness of using propensity score methods to evaluate a math/science educational intervention; and (2) the replicability of the findings on a new cohort of students. The current study uses an observational design and is a replication of the initial, preliminary propensity score analysis conducted at the end of the first year of the program. The authors describe the variables used, and will continue to use, during the replication phase of the program evaluation efforts. The results from Phase I of the study are presented, while Phase II is still underway during the production of this paper.

This report presents evidence that links participation in extracurricular activities (ECAs) in high school with higher SAT Reasoning Test™ (SAT®) scores. Using structural equation models (SEMs) with latent means, we analyzed data from a national sample of college-bound high school students. A series of structural equation models--isolating the influence of ECAs on SAT verbal and mathematics scores--were fit simultaneously to eight subgroups (disaggregated by both gender and ethnicity) of high school students. The SEMs analyses suggest: (1) that often observed group differences in SAT scores shrink, and (2) that students' levels of participation in ECAs in high school are related to meaningful gains in SAT scores, once the influences of socioeconomic background and academic achievement are controlled statistically. These analyses suggest that participation in ECAs benefits minority and socioeconomically disadvantaged students as much as, or more than, economically advantaged white students. These findings support the conclusion that supplementary education programs benefit minorities and disadvantaged high school students who are often ill-served by traditional academic curricula.

This paper explores the complex, hierarchical relationship among school characteristics, individual differences in academic achievement, extracurricular activities, and socioeconomic background on performance on the SAT Reasoning Test™ verbal and mathematical sections. Using multilevel structural equation models (SEMs) with latent means, data from a national sample of college-bound high school students were analyzed. A nested series of structural equation models were fit simultaneously to eight subgroups (disaggregated by both gender and ethnicity) of high school students. Analyses suggest that multilevel structural equation models provide a reasonably good fit to the data, that family background influences SAT® scores directly and indirectly, that learning opportunities in and outside the school curriculum are related to SAT performance, and that the characteristics of the schools matter when it comes to performance on the SAT. The paper's main contention is that context matters and that researchers ought to move beyond analyses of individual differences when attempting to understand performance on large-scale standardized tests. An appendix contains 2 tables.

To date the authors have completed 23 studies of knowledge monitoring and its relationship to learning from instruction. Their earlier work, 12 studies in all, is summarized and reported elsewhere (see Tobias and Everson, 1996; Tobias and Everson, 2000). In this paper they continue this line of research and summarize the results of 11 studies that have been conducted over the past three years. The work reported here attempts to address a number of general issues, e.g., the domain specificity of knowledge monitoring, measurement concerns, and the relationship of knowledge monitoring to academic ability.

The importance of learning and study strategies in fostering academic achievement has generated a demand for assessing these behaviors. The Learning and Study Strategies Inventory-High School version (LASSI-HS) is one of the most popular of these assessment devices. This study analyzed the second order latent structure of the inventory, using a sample of 1,645 students, and tested it through a confirmatory factor analytic model. The hypothesized model of the LASSI-HS with three correlated factors and unique factor loadings demonstrated a good fit to the data. The resulting model of the LASSI-HS was subsequently used in a more complex model of academic success with the addition of an indicator of verbal and mathematics achievement as measured by the Preliminary Scholastic Aptitude Test/National Merit Scholarship Qualifying Test (PSAT/NMSQT) results and high school grade point average (GPA). The resulting model also indicated a good fit to the data with the LASSI-HS having a direct effect on GPA as well as a mediating effect through PSAT verbal achievement. The factor structure of the LASSI-HS and model of academic achievement also demonstrated invariance with respect to grade level. (Contains 2 figures, 7 tables, and 18 references.) (Author/SLD)

A knowledge monitoring assessment (KMA) was developed and evaluated. The KMA, which evaluates how well students distinguish between what they know and do not know by comparing their knowledge estimates to test performance, is partially performance based and may be group or computer administered and objectively scored. Participants were 462 academically gifted high school students in a summer enrichment program. Mathematical and verbal analogy KMAs were developed. Results confirm prior findings that the KMA has significant but moderate relationships with scholastic aptitude, and that these relationships are somewhat higher than those that study skills and metacognitive self-report scales have with scholastic aptitude. Results also suggest that the KMA has low relationships with self-report measures of metacognition and study skills, which are highly related to one another. Results do indicate that knowledge monitoring has significant relationship with school grades, and that accurate monitoring is an important variable in school learning. Evidence supports the construct validity of the KMA and its usefulness in the study of metacognition. (Contains 5 tables and 24 references.) (SLD)

This paper discusses learning outcomes resulting from three national curricular reform efforts in mathematics. Focus is placed on the College Board's Equity 2000 program which operates on the principle that all students are given the opportunity to demonstrate high levels of academic achievement in mathematics. Information is provided on the Equity 2000 program and its effectiveness is evaluated. Enrollment and achievement trend data are also presented in discussing the effects of the program. This paper emphasizes the limitations and challenges of measuring Equity 2000's effect on mathematics achievement and concludes that it continues to be effective in raising student achievement in mathematics. (ASK)

Current theories of metacognition suggest that effective control of learning by either metacognitive or self-regulatory processes cannot occur without accurate monitoring of learning. Given this theoretical framework, there are questions of whether knowledge monitoring and self-regulated learning abilities are domain-specific or whether metacognitive processes, in particular knowledge monitoring ability, generalize across academic domains. This study examines that issue by exploring the correlations among measures of metacognitive knowledge, learning, and study strategies, and academic achievement across the domains of verbal ability and mathematics. Using parallel measures of knowledge monitoring in both the verbal and mathematical domains, 120 undergraduates estimated their metacognitive knowledge, reported their confidence in the accuracy of those estimates, and completed a self-report measure of learning and study strategies. Results suggest that metacognitive knowledge is generalizable across both the verbal and mathematical domains. The correlations between the two knowledge monitoring measures and students' confidence estimates were also in the expected directions. Moreover, both knowledge monitoring measures correlated with students' grade point averages. Correlations with subscales of the Learning and Study Strategies Inventory were not significant. Findings are discussed in the framework of current theory in metacognition and conceptions of strategic learning. An appendix shows multiple regression results. (Contains 1 table and 47 references.) (Author/SLD)

With the goal of academic excellence for all students in mind, The College Board's EQUITY 2000 educational reform program was designed to test the hypothesis that enrollment and success in the gatekeeper courses of algebra and geometry will bridge the gap of achievement and college-going rates between minority/nonminority students and economically advantaged/disadvantaged students. As such, the project's goal is not only to bring about change in student achievement levels but also to influence educational policies and practices with regard to tracking, professional development, guidance and counseling, and educational leadership within and across the school districts that participated in the pilot phase of the effort. This report provides a summary of what the evaluation efforts suggest thus far about the project's overall effectiveness at the six pilot sites. Given the length of the program's pilot phase and the long-term outcomes it seeks (i.e., closing the gap in college-going rates), it is not yet possible to document the full effects of EQUITY 2000. Thus, the conclusions reached in this report are based on the reviews of the project's initial set of indicators such as growth in enrollment and achievement in algebra and geometry; changes in teaching practices in mathematics classrooms; as well as shifts in attitudes and perceptions of faculty, staff, and students about learning at higher levels and subsequent success in college. This report aims to inform and assist a variety of stakeholders including the College Board's leadership, the program's executive director, and others as they plan and coordinate efforts for the dissemination of EQUITY 2000. Appended to the document are enrollment comparisons for algebra and geometry, 1991-1997. (ASK)

This study demonstrates the utility of a HYBRID psychometric model, which incorporates both item response theoretic and latent class models, for detecting test speededness. The model isolates where in a sequence of test items examinee response patterns shift from one providing reasonable estimates of ability to those best characterized by a random response pattern. The study applied the HYBRID model to three distinct data sets: (1) simulated data representing the performance of 3,000 examinees on a 70-item test; (2) data from a statewide field test (n=5,997) of a 40-item reading comprehension test with a fixed time limit; and (3) data from a study of urban university students (n=752) who took a similar 40-item reading comprehension test under varying time limits. The HYBRID model successfully identified "switch points" in examinees' item response patterns in all three data sets. This paper discusses applications of the model used to detect speededness and to provide adjusted estimates of item parameters and examinee abilities. (Contains 4 tables, 4 figures, and 38 references.) (Author)

This paper explores the feasibility of neural computing methods such as artificial neural networks (ANNs) and abductory induction mechanisms (AIM) for use in educational measurement. ANNs and AIMS methods are contrasted with more traditional statistical techniques, such as multiple regression and discriminant function analyses, for making classification or placement decisions in schools and colleges. Classification rates obtained with multiple regression and discriminant analysis were compared with ANN (back propagation) and AIM methods across a number of plausible models of algebra proficiency that included measures of arithmetic ability, high school achievement, test anxiety, and gender. Analyses were conducted on a sample of 290 male and 310 female college freshmen for the entire sample and for each gender. At each stage 10 randomly selected subsets were used to train and test the neural computing methods. In general, ANN and AIM methods outperformed the more traditional methods. Results suggest that neural computing methods may lead to higher rates of classification accuracy, particularly when underlying models are nonlinear. Included are four tables, and one figure. (Contains 17 references.) (Author/SLD)

A brief sketch is presented of the assessment strategies of the College Board in two unique programs, EQUITY 2000 and Pacesetter, both works in progress that attempt to meet the challenges of the new era of equity and excellence in education. EQUITY 2000, launched in 1990, is a multi-faceted approach to closing the gap between college-going rates of minority and non-minority students currently being implemented in six urban school districts. It is an intervention that emphasizes all aspects of a child's education, focusing on mathematics at the middle school and high school levels, and attempting to eliminate tracking in mathematics courses as a way to encourage other significant curriculum reform. Current EQUITY 2000 plans call for using a standardized test to measure achievement. In Pacesetter, the College Board will develop both formative and summative performance-based assessments for particular courses. Pacesetter also aims to eliminate tracking in school systems. Pacesetter courses will include mathematics, English, world history, science, and Spanish. Professional development will be an essential part of the Pacesetter process. Both initiatives are designed to help create an educational system in which the vast majority of students are exposed to quality education that only the top third have been exposed to up to now. (Contains 23 references.) (SLD)

College students' self-reported test anxiety levels in English, mathematics, physical science, and social science were compared to develop empirical support for the claim that students, in general, are more anxious about tests in rigorous academic subjects than in the humanities and to understand the curriculum-related sources of anxiety. It was hypothesized that students' perceptions of a subject's difficulty are correlated positively with their levels of test anxiety in that subject. It was assumed that students would report greater test anxiety when they believed mastery of a subject demanded precise answers on tests rather than a general understanding of the course content. First-year college students (N=196) (131 males and 65 females) were assigned randomly to groups using a 4 x 3 factorial design (4 levels of subject matter by 3 levels of test demand instruction). A revised version of the Worry-Emotionality Scale and a five-item scale describing their opinions and attitudes of the difficulty of the four academic curricular areas were used. Test anxiety scores and perceptions of subject matter difficulty correlated, independently of the particular subject and the test demands. Analyses of covariance indicate that physical science elicited the highest levels of self-reported evaluative anxiety, after controlling for perceptions of difficulty and test demands. Effects for test demand instructions were not significant. Results are discussed concerning the importance of subject matter as an intervening variable in test anxiety research generally and the significance of the role of test anxiety in impeding science achievement. (Contains 22 references.) (RLC)

This study explored the relationship between test anxiety and metacognitive word knowledge on performance on a reading comprehension test. One hundred and seventeen college students completed four paper and pencil measures: (1) a self-report measure of test anxiety; (2) an archival test of reading ability; (3) a metacognitive word knowledge task; and (4) a standardized measure of reading comprehension. A series of multiple regression analyses suggest that the cognitive component of test anxiety (worry) exerted a negative influence on students' performance on the metacognitive word knowledge task, independent of overall reading ability. Analyses of performance on three reading comprehension subscales suggests that when reading ability is controlled statistically, students' level of anxious worrying and their metacognitive word knowledge influenced performance on all three reading comprehension subscales. However, on the most demanding reading comprehension subscale (understanding the writers' assumptions, opinions and tone) metacognitive word knowledge interacted with worry, such that when anxious worrying was low increases in metacognitive word knowledge were associated with higher performance and, conversely, when anxious worrying was high increases in metacognitive word knowledge were associated with lower levels of performance. Findings are discussed within the framework of a cognitive capacity formulation of test anxiety. Six tables and one figure are included. One appendix contains a word list. (Contains 44 references.) (Author)

Reviews psychometric approaches to problems of modeling student achievement (the student model) within intelligent tutoring systems (ITS). A number of cognitively oriented psychometric approaches, including latent-trait models, statistical pattern recognition methods, and causal probabilistic networks are described and discussed within the current ITS framework. Contains 62 references. (Author/JKP)

This review employs a conceptual framework that distinguishes methods of detecting measurement bias based on either observed or unobserved conditional invariance models. The primary interest is in group-level measurement bias. Methods for bias detection in both continuous and ordered-categorical measures are reviewed, as are methods for testlets and multiple measures. (SLD)

Prior research suggests that the traditionally difficult academic subject matter, such as mathematics or the natural sciences, may evoke greater levels of evaluative (test) anxiety than curricular areas with easier content. These higher levels of anxiety may contribute to poor performance in mathematics and science courses. College freshmen's self-reported test anxieties were explored in four traditional academic curricular areas: (1) English; (2) mathematics; (3) science; and (4) social studies. It was hypothesized that students would report greater anxiety when they thought that mastery of a subject required precise answers rather than a general understanding of the subject matter. A total of 214 students (139 males and 75 females) attending a pre-freshman summer program at a major urban university were randomly assigned to groups in a 4 x 3 factorial design of subject matter by demand instructions. The Worry-Emotionality Scale was adapted to elicit students' anxiety concerning future college examinations in the four academic subjects. A five-item Likert-type scale was developed to determine students' perceptions about the difficulty of the academic subjects. Analysis of covariance results show that science elicited the highest levels of self-reported evaluative anxiety, after controlling for perceptions of subject matter difficulty. The effects for demand instructions were not significant. The implications of the results for both the role of test anxiety in science courses and the importance of subject matter in test anxiety research are considered. Three tables are included; the two scales are appended. (SLD)

The relationship between a fear of automation and teachers' attitudes toward a variety of instructional media is explored by comparing the attitude scores of teacher-education and computer science majors. Eighteen terms describing the instructional media were presented to both groups. Nine of the instructional devices were considered traditional, while the remaining nine were associated with automated instructional technology. The subjects, 78 undergraduate students, rated each term using six bipolar scales from the semantic differential. A repeated measures ANOVA lends further support to the hypothesis that a fear of automation is an important variable modifying teachers' attitudes toward instructional media. (Author)

The results of an interim evaluation of the Prefreshman Summer Basic Skills Program at the City University of New York are presented. The program provided instruction to prefreshmen needing remediation in one or more of the basic skills of reading, writing, and mathematics. In an effort the provide effective remedial instruction and to facilitate students' transition to college, the program featured small-group instruction, peer tutoring, and counseling. Results include the following: prefreshman basic skills instruction delivered over the summer is effective; retention rates showed more than 90% of the 1,414 participants completed the program and subsequently enrolled as freshmen; 46% of English-as-a-Second-Language (ESL) students progressed in reading and 61% in writing; and students began their freshman year with a sense of familiarity and firsthand knowledge of the college and its services. Six recommendations include: develop ESL instructional components; use instruction in computer skills and study strategies to augment basic skills instruction; and pay more attention to support services available over the summer. Contains 4 references. (SM)

The results of the City University of New York's 1988 Prefreshman Summer Program are presented, noting that the program provides small group instruction, peer tutoring, and counseling in an effort to offer both effective instruction in one or more of the basic skills of reading, writing, and mathematics and to facilitate students' transition from high school to college. Information is included on program growth, the students, achievement (reading, writing, and mathematics), and transition to college. Highlights include the following: for the fourth consecutive year, program enrollments have dramatically increased; enrollment of English-as-a-Second-Language students was up 70% in 1988; some colleges offered flexible programs to provide services to a wider audience; and many students acquired computer skills and study strategies that better enabled them to manage the demands of college. Some of the 14 recommendations are: instruction in computer skills should augment basic skills instruction whenever possible; colleges should expand efforts to provide well-structured training programs for tutors; and as student numbers increase, campuses should see their commitment to the summer program as a year-round activity. Two appendices offer comments from the coordinators and program highlights. Contains 6 references. (SM)