Search

Sound testing practices and the high-quality information that can result are helpful to those who have oversight, responsibility, or interest in American education. To the extent that tests provide high-quality information, they form the basis for making accurate judgments about individual students. It is equally true, however, that factors which attenuate the validity of tests or degrade the usefulness of the information they yield represent threats to sound decision making. This chapter addresses both student cheating and educator cheating, and recommended strategies for preventing cheating. (Contains 38 references and 1 table.) (GCP)

To investigate the potential effects of the "Fourth Grade Reading Guarantee," an Ohio policy that mandates that students will not be promoted to grade 5 unless they reach a given score on the Ohio Proficiency Test (OPT) in reading at grade 4, a series of studies was performed in the 1998-1999 and 1999-2000 school years. The extent of agreement among educators' judgments regarding students' academic preparation for work in the fifth grade, the relationships between these judgments and OPT scores, and the relationships between educators' judgments and policies intended to enhance educational quality were studied. Surveys in spring 1999 completed by teachers and principals yielded information on 6,065 (useable records), and 5,611 student records were obtained for the following school year, early fifth grade for these students. Teachers and principals tended to demonstrate a high degree of agreement about whether students were sufficiently prepared to enter fourth grade, but the operational definitions of adequate preparation differed in different school districts. Districts in which students generally perform less well had lower conceptualizations of competence. Educators' judgments were generally in line with students' actual pass/fail rates on the OPT. A fairly substantial discrepancy existed between the proportion of students that educators classified as not reading well enough to be academically successful in fifth grade and the proportion of students retained in grade, with the percentage of students judged underprepared more than 15 times greater than the percentage of students actually retained. Some areas of further research are outlined. (SLD)

Issues involved in linking the National Assessment of Educational Progress (NAEP) to the proposed Voluntary National Tests (VNTs) are discussed. Linkage is used to refer to procedures intended to permit scores from two different tests that are designed to measure the same variable to be expressed on the same scale. There are substantial differences between NAEP and the VNT that present serious challenges to linking the VNT to NAEP. The single greatest consideration in evaluating the potential for score scales to be linked is that of construct equivalence. Allied with the notion of construct equivalence are questions of the purposes of the assessments, administration conditions, and the stakes, visibility, and motivation. The greatest impact on overall construct equivalence is the extent to which content covered on the proposed VNTs can be viewed as consistent with that covered by the respective NAEP tests. Achievement levels, reporting methods, interpretations, and audiences must also be considered. The technical problems are serious enough, and the weight of policy considerations and uncertainty about how a VNT will affect NAEP are also worth contemplating. There are policy issues that should be addressed before considering linking methods. (Contains 1 table and 11 references.) (SLD)

The contrasting groups method is one of many possible methods for setting passing scores. The most commonly used method is probably that developed by W. H. Angoff (1971), but it has been suggested that the Angoff method may not be appropriate for many standard setting applications in education. The contrasting groups method is explored as an alternative for educational research. To implement the contrasting groups method, experts are asked to make a dichotomous judgment about examinees, usually in the form of master/nonmaster, competent/not competent, certify/deny, and so on. All judged examines then take a test covering the content area of the domain of interest. This process results in two distributions of test scores, one for the group judged masters and one for the group judged nonmasters. These two distributions can be examined and used to derive a cutting score for the examination which is then applied to examinees who take the test for whom expert judgments of mastery are not available. In this study the contrasting groups graphing procedure was used in conjunction with various combinations of population and standard setting characteristics to examine the conditions under which it most reliably captures a known standard. A Monte Carlo approach was used to simulate and analyze populations with differing distributional forms, different percentages of master and nonmasters, various sample sizes, differing sampling strategies, and varying judge error rates. Overall, findings produced suggestions that the contrasting groups graphing procedure can be applied confidently to estimate a "true" cutting score in a variety of applications. Best sampling practices are discussed, and limitations of the expert criterion judgments are reviewed. (Contains 1 table, 8 figures, and 23 references.) (SLD)

The gain, or difference, score is defined as the difference between the posttest score and the pretest score for an individual. Gain scores appear to be a natural measure of growth for education and the social sciences, but they contain two sources of measurement error, error in either the pretest or posttest scores, and cannot be considered perfectly reliable. This study assessed the effect that different combinations of pretest and posttest reliability coefficients, pretest and posttest standard deviations, and the correlation between the pretest and posttest scores have on: (1) raw gain score reliability coefficients; (2) residual gain score coefficients; (3) estimated true score reliability coefficients; and (4) the correlation between pretest scores and raw gain scores. Simulations indicated that the reliability coefficients of the pretest and posttest scores determined to a large extent whether raw gain score, residual gain score, and estimated true gain scores were reliable. The study demonstrated that, for standardized tests of educational research such as those commonly encountered in elementary and secondary schools, gain scores can be useful indicators of progress if the user is knowledgeable about the proper type of gain score to use and its correct interpretation. (Contains 6 tables and 40 references.) (SLD)

A group-process approach to standard setting was compared to an independent approach for a medical specialty certification examination. Both approaches used the Angoff (1971) standard-setting method. In the group-process method, reviewers discussed items and their ratings during the rating process; in the independent condition, reviewers provided their ratings in isolation. The effects of having previous exposure to the group-process condition or the independent condition, the effects of knowing other reviewers initial ratings, and the cost effectiveness of the procedures were studied. Participants were 10 subject matter specialists, 5 in each condition. Reviewers in the independent condition made original ratings and then submitted a second rating after they were notified of ratings provided by other reviewers (the "with-information" condition). The results demonstrated fairly large, although nonsignificant, differences in results obtained by group and independent reviewers using the same standard-setting method on identical test content. Although the differences were not statistically significant, a substantial effect on pass-fail decisions was noted. A reviewer's "with information" rating could be fairly well predicted by knowledge of the reviewer's original rating and knowledge of the group mean. Both independent conditions were more economically feasible for the small-panel situation in that they appeared to require a smaller time commitment from participants. (Contains 11 tables and 11 references.) (SLD)

A sample of 143 midwestern elementary and secondary school teachers from a variety of practice settings responded to a survey and provided comments regarding their assessment and practices. The study collected background (demographic) information on the teachers and information on some assessment-related practices, including: (1) the frequency with which teachers assign routine class assignments; (2) the types of marks used to report student performance; (3) the frequency and grading of major assignments and tests; (4) the source of classroom tests; (5) the kinds of marks used; (6) the methods used to combine marks; (7) the meaning of grades; (8) teachers' knowledge and perceptions regarding district grading policies; and (9) teachers' awareness of the grading policies of their peers. It was found that assessment practices vary widely and unpredictably. Few relationships were observed between teachers' assessment and grading practices and personal or background characteristics such as practice level, years of experience, gender, or familiarity with district policies. Teachers generally claimed to use a variety of factors in assigning grades, and a majority of teachers surveyed indicated that they were unaware of both their districts' policies and their colleagues' practices. Conclusions, recommendations, and implications of these findings are discussed. (Contains 33 references and 4 tables.) (Author)

The concept of due process provides an analogy for the process of standard setting that emphasizes many of the procedural and substantive elements of the process over technical and statistical concerns. Surely such concerns can and should continue to be addressed. However, a sound rationale for standard setting does not rest on this foundation. Standard setting on educational assessments will continue to be a fundamental concern because it inescapably involves the collection and synthesis of human judgment. This paper uses the due process analogy to develop suggestions for improving the synthesis of judgment, including: (1) refining clarity of purpose prior to setting standards; (2) pursuing new knowledge related to methods for selecting and training standard setting participants; (3) reevaluating participant consensus as a criterion for successful standard setting; (4) reevaluating the desirability of various "adjustments" used in standard setting; and (5) collecting and expanding professional guidelines for standard setting. An appendix identifies the standard setting references in the "Standards for Educational and Psychological Tests." (Contains 27 references.) (Author/SLD)

The report of the National Academy of Education (NAE), "Setting Performance Standards for Student Achievement," provides an interpretation of NAE investigations into the procedures surrounding establishment of achievement levels for the National Assessment of Educational Progress (NAEP). The NAE report synthesizes the investigations of other research efforts commissioned by the NAE Panel on the Evaluation of the NAEP Trial State Assessment. The following documents have been reviewed: (1) the NAE report itself, with 10 studies commissioned in preparing the report; (2) "Setting Achievement Levels on the 1992 National Assessment of Educational Progress in Mathematics, Reading, and Writing" from the American College Testing Program, 1991; (3) "NAGB Policy Framework and Technical Procedures for Setting Appropriate Achievement Levels for the National Assessment of Educational Progress" (National Assessment Governing Board, 1991); and (4) "The Reliability and Validity of the 1992 NAEP Achievement Levels" from American College Testing, 1993. It is concluded that the NAE evaluation is a seriously inaccurate representation of the technical and procedural propriety of the NAEP levels-setting process. Errors and inaccuracies are described in the general areas of authority and credibility, conclusions about the use of the Angoff method, procedural propriety and results, and the validity of NAEP levels. Accurate conclusions are also reviewed, and some constructive suggestions are made. (Contains 35 references.) (SLD)

A commonly accepted rule for developing equated examinations using the common-items non-equivalent groups (CINEG) design is that items common to the two examinations being equated should be identical. The CINEG design calls for two groups of examinees to respond to a set of common items that is included in two examinations. In practice, this rule has been extended to include the order in which options appear in the two examinations. The performance of a common set of items in which the order of options for one test form was experimentally manipulated was examined to determine if reordering multiple-choice item options resulted in any significant effect on item difficulty. Data from 759 subjects (graduates of medical specialty residency training programs) were gathered as part of the annual administration of a certification examination in a medical specialty area. Each subject responded to 20 multiple-choice items with a projected visual as the stimulus for each item. Examinees had to select from about 30 choices the option that correctly identified the projected visual. Two response booklet forms, differing only in that the position of the 20 options was scrambled, were used. A total of 380 examinees responded to Booklet A, and 379 examinees responded to Booklet B. One examinee was randomly excluded from the analyses for Booklet A. It was found that reordering items often has significant but unpredictable effects on item performance. A linkage is made to previous research on the "response set" construction, and cautions are suggested regarding the effect of reordering options. (SLD)

A discussion of the increased use of qualitative methods in social science research looks at four emerging issues: the nature of research, the growth in status of the narrative in research, political influences on the research process, and the nature of the relationship between the philosophy of science and applied research practice. Some approaches to the problems are discussed. (MSE)

Lack of knowledge and interest in grading translates into a serious information breakdown in education. Reforming classroom assessment and grading practices will require educators' commitment to professional development, classroom-relevant training programs, help from professional organizations and testing experts, a coordinated vision, consistently developed and applied policies, teacher collaboration, and revamped student attitudes toward grades. (13 references) (MLH)

This response to a review essay on the limited presence of African American teachers suggests that the original work leaves out some issues surrounding teacher recruitment and retention and leaves some empirical work and interpretations unexamined. The examination of such "crises" in education is a promising avenue for research. (SLD)