Search

This study investigated the relationships between when a test is administered and the amount of test-taking effort exhibited by examinees. Three time-related variables were investigated: the time of year the test was administered, the day of the week the test event occurred, and the time of day that the test event occurred. Mean effort did not appear to vary across either time of year nor day of week. There was, however, a clear time of day effect, with mean effort decreasing throughout the day. In addition, mean effort was found to decreases across grade, and male examinees were found to exhibit lower mean effort than female examinees. (Contains 10 tables, 5 figures and 4 footnotes.)

At the heart of the No Child Left Behind Act (NCLB) is the call for all students to be "proficient" in reading and mathematics by 2014. Yet the law expects each state to define proficiency as it sees fit and design its own tests. This study investigated three research questions related to this policy: (1) How consistent are various states' expectations for proficiency in reading and mathematics? In other words, is it harder to pass some states' tests than others? (2) Is there evidence that states' expectations for proficiency have changed since NCLB's enactment? If so, have they become more or less difficult to meet? In other words, is it getting easier or harder to pass state tests? (3) How closely are proficiency standards calibrated across grades? Are the standards for earlier grades equivalent in difficulty to those for later grades (taking into account obvious grade-linked differences in subject content and children's development)? In other words, is a state's bar for achievement set straight, sloping, or uneven? This study used data from schools whose pupils participated both in state testing and in assessment by the Northwest Evaluation Association (NWEA) to estimate proficiency cut scores (the level students need to reach in order to pass the test for NCLB purposes) for assessments in twenty-six states. Among the findings: (1) State tests vary greatly in their difficulty; (2) Most state tests have not changed in difficulty in recent years; (3) Improvements in passing rates on state tests can largely be explained by declines in the difficulty of those tests; (4) Mathematics tests are consistently more difficult to pass than reading tests; and (5) Eighth-grade tests are consistently and dramatically more difficult to pass than those in earlier grades (even after taking into account obvious differences in subject-matter complexity and children's academic development). As a result, students may be performing worse in reading, and worse in elementary school, than is readily apparent by looking at passing rates on state tests. The report presents both national and state findings. The following are appended: (1) Methodology; (2) Summary of Concurrent Validity Studies; (3) Tables A3.1-mathematics and A3.2-reading summarize key information about each of the state alignment studies, showing the year and school term in which the study was conducted, the grades evaluated, and the average number of students in each grade included; (4) Estimated State-Test Proficiency Cut Scores in Reading using MAP (in Percentile Ranks); (5) Estimated State-Test Proficiency Cut Scores in Mathematics using MAP (in Percentile Ranks); (6) Changes in Proficiency Cut Score Estimates and Reported Proficiency Rates on State Assessments--Reading; (7) Changes in Proficiency Cut Score Estimates and Reported Proficiency Rates on State Assessments--Mathematics; and (8) How Consistent Are the Results from This Study and the NCES Mapping 2005 State Proficiency Standards Study? [A foreword by Chester E. Finn, Jr. and Michael J. Petrilli is also included. This report represents a collaboration of the Thomas B. Fordham Institute and the Northwest Evaluation Association.]

This report represents the first in a yearly series of studies investigating the impact of the No Child Left Behind act on the achievement of students in the United States. This series will use achievement information from a broad sample of students and schools to provide evidence about the changes that have occurred since the law passed. This study examines how well the law is beginning to meet its promise in its first years of implementation. It investigates: (1) how much student achievement status has changed since the law was implemented; (2) whether and to what extent student achievement growth has changed since the law was implemented; and (3) the impact of the law on the achievement status and growth of students by ethnic group. It also addresses with the following research questions: (1) Are students' achievement scores higher than they were when NCLB first went into effect? (2) Is student achievement growth higher than it was when NCLB first went into effect? (3) Are achievement gaps among ethnic groups shrinking under NCLB? and (4) Given current rates of change in achievement, are schools likely to meet the requirements of NCLB? To investigate these questions, the study uses a large sample of student achievement scores, selected from a broad cross section of school districts throughout the United States. Supplemental tabular data is presented in three appendixes. (Contains 26 tables, 19 graphs, and 5 figures.)

Conventional mastery tests designed to make optimal mastery classifications were compared with fixed-length and variable-length adaptive mastery tests in terms of validity of decisions with respect to an external criterion measure. Comparisons between the testing procedures were made across five content areas in an introductory biology course from tests administered to over 400 volunteer students. The criterion measure used was the student's final standing in the course based on course examinations and laboratory grades. Results indicated that the adaptive test resulted in mastery classifications that were more consistent with final class standing than those obtained from the conventional rest.

Two procedures for making mastery decisions with variable length tests and a conventional mastery testing procedure were compared in Monte Carlo simulation. The simulation varied the characteristics of the item pool used for testing and the maximum test length allowed. The procedures were compared in terms of the mean test length needed to make a decision, the validity of the decisions made by each procedure, and the types of classification errors made by each procedure. Both of the variable test length procedures were found to result in important reductions in mean test length from the conventional test length. The Sequential Probability Ratio Test (SPRT) procedure resulted in greater test length reductions, on the average, than the Adaptive Mastery Testing (AMT) procedure. However, the AMT procedure resulted both in more valid mastery decisions and in more balanced error rates than the SPRT procedure under all conditions. In addition, the AMT procedure produced the best combination of test length and validity. (Author)

In an attempt to increase the efficiency of mastery testing while maintaining a high level of confidence for each mastery decision, the theory and technology of item characteristics curve (ICC) response theory and adaptive testing were applied to the problem of judging individuals' competencies against a prespecified mastery level to determine whether each individual is a 'master' or a 'nonmaster' of a specified content domain. Items from two conventionally administered classroom mastery tests administered in a military training environment were calibrated using the unidimensional three-parameter logistic ICC model. Then, using response data originally obtained from the conventional administration of the tests, a computerized adaptive mastery testing (AMT) strategy was applied in a real-data simulation. Results obtained from the AMT procedure were compared to results obtained from the traditional mastery testing paradigm in terms of the reduction in mean test length, information characteristics, and the correspondence between decisions made by the two procedures for three different mastery levels and for each of the two tests. The AMT procedure reduced the average test length 30% to 81% over all circumstances examined (with modal test length reductions of up to 92%), while reaching the same decision as the conventional procedure for 96% of the trainees. Additional advantages and possible applications of AMT procedures in certain classroom situations are noted and discussed.

Item characteristic curve (ICC) theory has potential for solving some of the problems inherent in the pretest-test and test-posttest paradigms for measuring change in achievement levels. However, if achievement tests given at different points in the course of instruction tap different achievement dimensions, the use of ICC approaches and/or change scores from these tests is not desirable. This problem is investigated in two studies designed to determine whether or not achievement tests administered at different times during a sequence of instruction actually measure the same achievement dimensions. Results raised questions about the utility of the pretest-test paradigm for measuring change in achievement levels, since a comparison of a ICC parameter estimates indicated that a change in the dimensionality of achievement had occurred within the short (4-week) period of instruction. This change was also observed using a factor analytic comparison. Use of the test-posttest paradigm to measure retention was supported, since a regression comparison of students acheivement level estimates did not indicate any significant change in the achievement metric up to 1 month after the peak of instruction. The significance of this result for the use of adaptive testing technology in measuring achievement is described.

Results for all data sets showed a high degree of similarity among parameter value estimates for the one- and two-parameter data, with slight decreases in correlations as information on the discrimination parameter was used in scoring. In general, however, the adaptive test data were less likely to result in convergence failures than were the conventional test data. The data also illustrated how each of the three scoring methods tend to utilize ICC parameter information in arriving at parameter value estimates and the relationships of these estimates to a number-correct scoring philosophy. Advantages and disadvantages of each of the scoring methods are discussed. It is suggested that future research examine the relative validities of scoring methods and model combinations.

The applicability of item characteristic curve (ICC) theory to a multiple-choice test item pool used to measure achievement is described. The rationale for attempting to use ICC theory in an achievement framework is summarized, and the adequacy for adaptive testing of a classroom achievement test item pool in a college biology class is studied. Using criteria usually applied to ability measurement item pools, the item difficulties and discriminations in this achievement test pool were found to be similar to those used in adaptive testing pools for ability testing. Studies of the dimensionality of the pool indicate that it is primarily unidimensional. Analysis of the item parameters of items administered to two different samples reveals the possibility of a deviation from invariance in the discrimination parameter but a high degree of invariance for the difficulty parameter. The pool as a whole, as well as two subpools, is judged to be adequate for use in adaptive testing. It is also concluded that the ICC model is not inappropriate for application to typical college classroom achievement tests similar to the one studied. (Author)