Search

The many-faceted Rasch (MFR) model has been used to evaluate the quality of ratings on constructed response assessments; however, it can also be used to evaluate the quality of judgments from panel-based standard setting procedures. The current study illustrates the use of the MFR model for examining the quality of ratings obtained from a standard setting process employing the Multiple Yes-No, modified Angoff standard setting procedure to set the cut scores on the Advanced Placement Environmental Science exam. The panelists’ severity, judged item difficulties, rounds, and cut scores using the MFR model were examined, in addition to panelists’ characteristics (gender and level of course taught). Findings from the Rasch analyses conducted in this study provide evidence of acceptable quality of the ratings obtained from the panelists. Thus, the current study provides an important source of both internal and procedural validity evidence for the Advanced Placement Environmental Science standard setting as well as an illustration of the utility of the MFR model for evaluating standard setting judgments.

This study examined the amount of time that different ability-level examinees spend on questions they answer correctly or incorrectly across different pretest item blocks presented on a fixed-length, time-restricted computerized adaptive testing (CAT). Results indicate that different ability-level examinees require different amounts of time to answer pretest items, that is, field-test items, embedded in different item locations. Also, guessing behaviors and test-taking effort (TTE), measured using detection indices developed from this study, varied across the pretest items embedded in different locations. With these indices, unique response time patterns (RTP) and guessing patterns (GP) were identified for six ability groups. These indices could also be used to monitor the amount of effort and the guessing behaviors of examinees taking a CAT. The degree of the detection consistency among these indices could be used as reliability evidence to evaluate how reliable the indices are in detecting unusual test-taking behaviors. Once reliable detection was confirmed, a warning sign prompted from the computer screen could either encourage the examinee to take appropriate TTE or stop guessing so that the examinees could receive more accurate estimates of their ability.

Even when the scoring of an examination is based on item response theory (IRT), standard-setting methods seldom use this information directly when determining the minimum passing score (MPS) for an examination from an Angoff-based standard-setting study. Often, when IRT scoring is used, the MPS value for a test is converted to an IRT-based theta ability estimate by using the test characteristic curve. The purpose of this study is to investigate some approaches to determining the MPS from an Angoff-based standard-setting study when the test questions have been calibrated using IRT methods. Although the methods studied appear to converge on the same cut score, the cut scores derived from the IRT-based approaches are all within one standard error of the Angoff-based MPS value.

In an Angoff standard setting procedure, judges estimate the probability that a hypothetical randomly selected minimally competent candidate will answer correctly each item in the test. In many cases, these item performance estimates are made twice, with information shared with the panelists between estimates. Especially for long tests, this estimation process can be time-consuming and fatiguing for the judges. This study extended an item selection strategy earlier proposed to form subsets of test items. The results of this study suggest that 40% to 50% of test items may be sufficient to estimate an equivalent passing score in an Angoff setting study when those items are selected to represent the full test in content, discrimination, and difficulty.

In an Angoff standard-setting procedure, judges estimate the probability that a hypothetical randomly selected minimally competent candidate will answer correctly each item constituting the test. In many cases, these item performance estimates are made twice, with information shared with the judges between estimates. Especially for long tests, this estimation process can be time consuming and fatiguing for the participants. This study addressed the possibilities of reducing the number of items individual judges consider in an Angoff standard-setting study. The results of this study suggest that 50% of test items may be sufficient to estimate an equivalent passing score in an Angoff setting study. This could result in substantial savings of time and resources not only for the agencies that have been carrying out this activity but also for the practitioners (judges) who participate in standard-setting studies.

The present study examined the relationship between the validity theory of the past 50 years with actual validity practices. This process involved the comparison of validity requirements and theory as they appeared in the published test standards (by the American Psychological Association, the American Educational Research Association, and the National Council on Measurement in Education) to the practices of measurement professionals. Each of the four sets of test standards were operationally defined and then compared to information and opinions about validity evidence expressed in the Mental Measurements Yearbook test reviews. Results from this process indicate that the theoretical modifications to the concept of validity have been carried over into validity practice. However, it was also found that operational modifications to the concept of validity have been influenced by validity practice. Therefore, there appears to be a symbiotic relationship between theory and practice on the influence of validity.

Traditional standard-setting methods are not well suited for applications with polytomously scored performance assessments. The present article presents a standard-setting method-the dominant profile judgment (DPJ) method-designed for use with profiles of polytomous scores on exercises in a performance-based assessment. The method is direct, in that it guides standard-setfing panelists in the articulation of their standard-setting policies. Further, it allows complex policy statements that could incorporate compensatory and/or conjunctive components. A detailed description of the method is provided, Results of an application of this standard-setting method are presented. Recommendations for improvements in the method are discussed.

Previous research has shown self-adapted testing to be a promising alternative to computerized adaptive testing. The present study looked at the importance of item feedback in self-adapted testing. Computerized adaptive tests and self-adapted tests were compared under both feedback and no-feedback conditions. The self-adapted tests yielded higher test performance and lower anxiety than the computerized adaptive tests regardless of feedback condition. These findings indicate that item feedback is not necessary for the score differences between self-adapted testing and computerized adaptive testing to be ralized.

In self-adapted testing (SAT), examinees select difficulty levels of items administered. This study investigated three variations of prior information provided when taking a SAT: (a) No Information: Examinees selected item difficulty levels without prior information; (b) View: Examinees inspected a typical item from each difficulty level prior to the SAT; (c) Route: Examinees answered a typical item from each difficulty level and were informed prior to the SAT of the level that best matched their performance. No significant difference in test performance or level of anxiety was found as a function of providing examinees more information.

This study investigated the psychometric properties of an instrument (i.e., the Goals Inventory) that measured learning and performance goal orientations. Test-retest reliability estimates for the learning and performance goal scales were r = .73 and r = .76, respectively. Internal consistency estimates were assessed using Cronbach's alpha. These values were .80 and .75, respectively. Convergent and divergent validity were evaluated by comparing the Goals Inventory to measures of test anxiety, hope, and attributions for success and failure. All the theoretically explicit predictions of Dweck and Leggett's model were supported. Suggestions are made for the use and interpretation of the Goals Inventory's two subscales and for future research.

A multidimensional instrument, the American-International Relations Scale, purports to measure the acculturation of international students, scholars, and academicians to the white-dominant society. Data from 606 respondents, making a return rate of 67%, were collected. Factor analysis of completed 34-item questionnaires (N - 481), using varimax rotation, yielded three interpretable factors, tentatively labeled (a) Perceived Prejudice, (b) Acculturation, and (c) Language Usage. Factor loadings of items ranged between .33 and .89 on one of the respective factors, with no item loading saliently on two or more factors. Full scale and subscale internal-consistency reliabilities (alphas) were .89, .88, .79, and .82, respectively.

In an attempt to make commercial tests valid for users of both sexes and all race groups, many test publishers employ review procedures to remove potentially invalid (or biased) items from their tests. Statistical procedures to identify biased test items are often sophisticated and costly to test developers. Alternative approaches have employers review test items and select those that appear biased. To verify this methodology, a statistical and a subjective procedure were used to identify biased items in an elementary achievement test. The results show little agreement on items selected as biased. An implication to test developers is to adopt statistical support for a reviewer's selection of biased items.

The Nedelsky standard setting procedure utilizes an option elimination strategy to estimate the probability that a minimally competent candidate (MCC) will answer a multiple-choice item correctly. The purpose of this study was to investigate the accuracy of predicted item performance from the Nedelsky ratings. The results indicate that test taking behavior of MCCs does not match the underlying Nedelsky assumption that MCCs randomly guess among the options judges believe should be attractive. Further, the accuracy of predicted item performance appears to vary as a function of item difficulty and content domain. However, an analysis of the relationship between judges' rating of distractor difficulty and proportion of examinees selecting item distractors indicated that useful information about examinee item performance is obtainable from Nedelsky-based judgments.

One source of test invalidity in the area of licensure and certification testing is the reading level of the test. The purposes of this paper are to: (a) propose a model for evaluating the readability level of certification/licensure examinations; (b) evaluate selected readability indices in terms of their appropriateness for use in estimating the readability level of specific certification/licensure examinations; and (c) present an example of the application of this model to the readability analysis of a state licensure examination in cosmetology. Readability level is an important dimension of the validity of licensure and certification examinations. One potential basis for candidate appeal is the appropriateness of the readability level of the examination with respect to the tasks and functions of the profession.

Since 1960 educational researchers have been using the Achievement Anxiety Test (AAT), which presumably measures independent aspects of achievement anxiety: facilitating and debilitating components. In several earlier investigations, developers have reported construct and predictive validity of the subscales from their use of anxiety measures and performance criteria. However, the hypothesized independence of the subscales and underlying factor structure of the test have been overlooked. Because the two subscales have been used to provide a difference score, the status of the interrelationship between the subscales is of concern. This study was concerned with the concurrent validity of the AAT and with its factor structure to provide empirical evidence about the quality of AAT.

The Bias in Attitudes Survey (BIAS) is a recently devised scale which assesses attitudes and beliefs about sex roles. This study was concerned with the factorial validity of the BIAS. The results suggested that the BIAS in relation to other sex-role scales is measuring more complete and complex attitudes toward sex roles. Two factors, which explained 68% of the variance, represented both the traditional view of male/female roles and a more liberal, non-sexist view. The analysis of the male/female subgroups demonstrated a lack of parallelism in the factor structure of the scale for the two sexes.

Differential test performance by males and females was reported on a quantitative examination as a function of item arrangement (Plake, Ansorge, Parker, and Lowry, 1982). Males performed at a higher level than females on tests the items of which were arranged from easy to hard. Plake and Ansorge (1982) speculated that this male superiority may be a function of differential reinforcement in backgrounds of males and females. Using a Latin Square design, this study provided item performance feedback for a nonquantitative examination. Significant sex-by-order effects did not occur, but the lack of this effect could possibly have been due to the use of a nonquantitative examination (Plake and Ansorge, 1983). It still remains to be seen whether, in using a quantitative examination, differential effects of item feedback may be a source of explanation for evidence of sex-by-order effects. Implications for validity of test score interpretation are considered briefly.

Past research on gender differences in performance on mathematics achievement test items has focused on characteristics of test items on which males and females perform differentially. In most instances, such studies have employed a post-hoc content analyses of test items identified via statistical analyses to function differentially for males and females. This study, in contrast, used an experimental design approach to investigate the effect of item context on differential item performance on mathematics test items between males and females. The analyses were based on a modified one parameter IRT methodology. Although some significant differences in performance were found between and within item arrangements for males and females, given the number of statistical tests employed, these significances could be due to chance alone.

Scores reflecting the attitudes of fourth, fifth, and sixth-grade Mexican-American and Anglo students about reading were compared to assess the relevance and validity of the Estes Reading Attitude Scale for each group. The validity of application of results from the Estes Reading Attitude Scale when administered to students of differing ethnic backgrounds was examined by considering the factorial complexity of the instrument overall for Anglo and Mexican American children, as well as at three grade levels. The comparability of the internal consistency indices and of factorial structure between ethnic groups within grade levels, and the consistency of item interpretation (or lack of item "bias") between ethnic groups within grade level suggested that scores on the scale afforded a valid means for interpretation of attitudes held by elementary school pupils towards reading irrespective of their ethnic status (Anglo or Mexican-American).

Judgmental methods for estimating passing scores generally are accomplished by having testing specialists evaluate items in an intact test form. Alternatively, judges are asked to rate individual items from an item pool. Cutscores are determined by these ratings when the items are chosen from the pool to form the operational test. This approach to establishing cutscores assumes (a) stable definitions of minimal competency from the time of review to time of assembly into the operational form and (b) invariance of item rating judgments to test form contextual variables. This study was concerned with the impact of overall test length and difficulty on the expert judgments of item performance by using the Nedelsky method. Results suggest that the judges are fairly consistent in their ratings of items regardless of overall test length or difficulty.

A 24-item shortened version of the 98-item Mathematics Anxiety Rating Scale (MARS) was developed to provide a more efficient index of statistics or mathematics course-related anxiety. The revised scale, which yielded a coefficient alpha reliability estimated at .98, was correlated .97 with a full scale MARS. It shows a pattern of relationships with state, trait, and test anxiety parallel to that of full scale MARS. Further, the relationship with mathematics achievement was consistent for both the revised and full scale MARS. A principal axes factor analysis of revised MARS identified two clear factors, labeled "Learning Mathematics Anxiety" and "Mathematics Evaluation Anxiety." Revised MARS appears to be an attractive substitute for full scale MARS for statistical or mathematical course-related uses.

The methodology suggested in this paper employs a selection rule for identifying group members that generates groups that have a range of achievement within groups but equal distribution of raw scores between. An ANOVA procedure on arcsin transformed item difficulties, followed by an investigation of strength of association lend practical significance to significant items by groups interaction. A follow-up procedure that identifies items contributing to the items-by-groups interaction is described.

This study investigated the effects of providing item feedback on student achievement test performance and anxiety, and how these effects may be moderated by the amount of success and students experience on the initial items of the test. Introductory statistics students were randomly assigned to six forms of a computer-based algebra test that differed in terms of (a) the difficulty of the first five items, and (b) the type of item feedback provided. Although test performance was not affected significantly by differences among the test forms, student anxiety levels were significantly increased when administered the test form using difficult initial items and providing item feedback along with a running score total. Implications for the use of item feedback in computer-based testing are discussed.