Search

This report examines how class size and course motivation are related to evaluation results, its purpose being to provide a comparison of rating differences on the Instructor and Course Evaluation System (ICES). ICES offers faculty a computer-based catalog of over 1,000 survey items. The article is based on a study that sought to extend the ICES research by using an unbalanced nested ANOVA to analyze the effects of five factors: course motivation, course level, class size, academic discipline, and individual class differences on (1) student ratings of instructors; and (2) student ratings of overall course quality. The results of the ANOVA showed that course motivation, course level, class size, discipline, and individual class do influence student ratings of both teaching effectiveness and course quality. Different combinations of course motivation, level, and discipline displayed different patterns of student ratings. The significant two-way interactions (in which one of the factors alters the effect of the other) were course motivation and discipline, course level and class size, and course level and discipline. Generally, ratings of elective courses were higher than those of mixed courses, and ratings of mixed courses were higher than those of required courses. Higher level classes had higher ratings than lower level ones. Contains 27 references, 10 figures, and 9 tables. (RJM)

The relationship between differential item functioning (DIF)and test item context effects was studied in an investigation of whether the patterns of gender DIF in parcels of items are influenced by changes in item position, as seen in two forms of a test. A second aim was to determine whether performance of male and female test takers is differentially affected by variation in item position. Items were studied collectively to detect differential bundle functioning (DBF) within content areas of the Midwestern Mathematics Placement Exam, a test for college freshmen. Data from 5 test administrations and 2 test forms for over 5,000 students were used. The simultaneous item bias statistic and the bias estimator, beta, were calculated with the SIBTEST computer program. There were changes in the amount of gender DIF present when the SIBTEST results for forms 1 and 2 were compared. Results from the DBF analyses suggest that analytic geometry items were differentially easier for women on form 2. The idea of DBF is a useful addition to the study of DIF in that this approach provides added power for detecting patterns of differences. An appendix describes test item attributes.(Contains 5 tables and 27 references.) (SLD)

The use of scientific calculators on standardized mathematics tests is becoming more common, and how their use affects the standardized testing process continues to be a topic of study. This study extends previous work by examining test item characteristics and equity issues, asking whether items designed to be "calculator neutral" function as intended, and whether such items function the same for males and females. Data came from trials of the Midwestern Mathematics Placement Examination for precalculus and calculus classes for over 1,000 college students. A logistic regression analysis was conducted to detect differential item functioning (DIF) or nonuniform DIF. Findings suggest that calculator neutral items can be constructed, and that the items that were constructed to be calculator neutral were largely free from gender DIF. Results also showed that the logistic regression approach was a useful addition to studying DIF, although larger sample sizes would have been highly preferable. Using the rule-space approach as a basis for the content analyses was also quite effective. An appendix contains two sample problems and a description of the mathematical challenge items. (Contains 4 tables, 4 figures, and 17 references.) (SLD)

The implementation of a collaborative approach to evaluating family literacy programs was studied using a conceptual framework and applied to 36 family literacy programs from a midwestern state. Evaluation participants learned how results could be used to develop curriculum in addition to reporting to funding agencies. (SLD)

Results for 3,244 female and 3,033 male junior high school students from the Second International Mathematics Study show that applied items in algebra, geometry, and computation were easier for males but arithmetic items were differentially easier for females. Implications of these findings for assessment and instruction are discussed. (SLD)

Whether specification of focal group membership based on ethnicity as well as educational opportunity affects differential item functioning (DIF) of test items was studied. Subjects were 4,342 freshmen admitted to a large midwestern university who completed the 40-item Advanced Mathematics Placement Examination in the spring of 1991. There were 3,896 White, 338 African American, and 108 Hispanic American examinees. All of the Hispanic Americans and 166 of the African Americans were from high schools judged to be educationally and economically disadvantaged. The Mantel-Haenszel (MH) procedure was used to examine DIF. Although there were few verbal items, the mean MH DIF delta difference did change substantially in relation to focal group specification. There were also changes in item classification depending on the specification of focal group membership. The question of whether sample sizes for focal group measurement were adequate in this investigation is discussed. There are five tables of study data and one illustrative plot. A 23-item list of references is included. (SLD)

In an investigation of item bias, the stability of Mantel-Haenszel (MH) estimates across different samples of test takers and different sample sizes and the robustness of the MH procedure with respect to item context effects were investigated. Data from the Second International Mathematics Study (1985) were analyzed. The data consisted of responses to 40 core items on the mathematics tests and one of four 35-item rotated mathematics tests for a core sample of 670 black and 5,015 white eighth graders in the United States. Most analyses were performed with a sample of just over 100 blacks and slightly over 1,000 whites. Correlations between different samples of test takers were low, suggesting that relatively larger sample sizes were necessary for stable estimates. Correlational analyses also suggest that the MH procedure was robust to item context effects. Four of the 40 core items were identified as functioning differently for black and white test takers. The MH procedure is useful in detecting differential item functioning; however, further investigation is needed to determine the sample size necessary for stable estimates. Seven tables present study results. A 21-item list of references is included. (SLD)

A study was made of cross-cultural patterns of achievement motivation in relationship to the mathematics achievement of Japanese and American boys and girls approximately 16 years of age. Sample sizes were 9,582 for the United States subjects (specifically, from Illinois) and 1,700 for the participants from Japan. Data came from performance on the High School Mathematics Test, which was administered during the spring term of the 1981-82 testing cycle. The test consisted of 60 items covering a range of mathematical topics, including algebra, geometry, modern math, data interpretation, and probability. A background questionnaire including two measures of achievement motivation (success/failure attributions and test anxiety) was completed by all students during the testing session. Results of correlational analyses revealed a number of associations between (1) gender, test comfort, and attribution variables; and (2) mathematics achievement performance in both within- and between-country comparisons. Specifically discussed are between- and within-country differences in the relationship between gender and mathematics performance, the relationship between test comfort and mathematics achievement performance, and between-country differences in relationships of attribution variables to mathematics achievement performance. It was concluded that current measures of achievement attribution are inadequate to assess motivational patterns in Japan, and that the much higher test scores of the Japanese students do not justify new policy recommendations, given the basic differences between the two countries.