Search

Assessment use is a cornerstone of successful counseling. Information from assessments is used for making several important therapeutic decisions, requiring the counselor to have information about assessment devices and approaches. Tests are being published at a remarkable rate; it is a challenge for counselors to stay well informed. Products and programs from the Buros Institute of Mental Measurements serve test information needs of counselors and therapists. The "Mental Measurement Yearbook" and "Test in Print" series contain information about availability, quality, and utility of assessment devices. Counselors can identify tests potentially appropriate for their clinical practice and stay up-to-date on assessment of psychological constructs and educational outcomes through use of these volumes. In addition, the Institute sponsors topical symposia and volumes targeted at specific audiences; these can provide cutting edge assessment information to counselors and therapists. The counseling process is multifaceted and complex. Tests and other specific assessment approaches are useful assisting counselors in making appropriate clinical decisions; the Buros Institute's mission is to support well-informed assessment decisions. (JBJ)

An item selection strategy that is used to reduce the number of items considered by an individual judge in an Angoff standard setting study is examined. The results indicate that almost half of test items are enough for estimating an equivalent passing score in the Angoff standard setting procedure when those items are selected to represent the full test in content, discrimination and difficulty.

The present study examined the relationship between the validity theory of the past 50 years with actual validity practices. This process involved the comparison of validity requirements and theory as they appeared in the published test standards (by the American Psychological Association, the American Educational Research Association, and the National Council on Measurement in Education) to the practices of measurement professionals. Each of the four sets of test standards were operationally defined and then compared to information and opinions about validity evidence expressed in the Mental Measurements Yearbook test reviews. Results from this process indicate that the theoretical modifications to the concept of validity have been carried over into validity practice. However, it was also found that operational modifications to the concept of validity have been influenced by validity practice. Therefore, there appears to be a symbiotic relationship between theory and practice on the influence of validity., The changes in validity theory over the past 50 years have been given various labels. Shepard (1993) refers to these changes as an 'evolution,' whereas Geisinger (1992) refers to them [...]

Traditional standard-setting methods are not well suited for applications with polytomously scored performance assessments. The present article presents a standard-setting method--the dominant profile judgment (DPJ) method--designed for use with profiles of polytomous scores on exercises in a performance-based assessment. The method is direct. in that it guides standard-setting panelists in the articulation of their standard-setting policies. Further, it allows complex policy statements that could incorporate compensatory and/or conjunctive components. A detailed description of the method is provided. Results of an application of this standard-setting method are presented. Recommendations for improvements in the method are discussed., Educational assessments today look very different from the assessment practices of the past couple of decades. Many of the current assessments are performance based, requiring examinees to complete tasks that [...]

Previous research has shown self-adapted testing to be a promising alternative to computerized adaptive testing. The present study looked at the importance of item feedback in self-adapted testing. Computerized adaptive tests and self-adapted tests were compared under both feedback and no-feedback conditions. The self-adapted tests yielded higher test performance and lower anxiety than the computerized adaptive tests regardless of feedback condition. These findings indicate that item feedback is not necessary for the score differences between self-adapted testing and computerized adaptive testing to be realized.

This study investigated the psychometric properties of an instrument (i.e., the Goals Inventory) that measured learning and performance goal orientations. Test-retest reliability estimates for the learning and performance goal scales were r = .73 and r = .76, respectively. Internal consistency estimates were assessed using Cronbach's alpha. These values were .80 and .75, respectively. Convergent and divergent validity were evaluated by comparing the Goals Inventory to measures of test anxiety, hope, and attributions for success and failure. All the theoretically explicit predictions of Dweck and Leggett's model were supported. Suggestions are made for the use and interpretation of the Goals Inventory's two subscales and for future research.

A multidimensional instrument, the American-International Relations Scale, purports to measure the acculturation of international students, scholars, and academicians to the white-dominant society. Data from 606 respondents, making a return rate of 67%, were collected. Factor analysis of completed 34-item questionnaires (N = 481), using varimax rotation, yielded three interpretable factors, tentatively labeled (a) Perceived Prejudice, (b) Acculturation, and (c) Language Usage. Factor loading of items ranged between .33 and .89 on one of the respective factors, with no item loading saliently on two or more factors. Full scale and subscale internal-consistency reliabilities (alphas) were .89, .88, .79, and .82, respectively.

A study was conducted to examine the different prospective uses of item response theory (IRT) methodology in standard settings for determining the minimum passing score (MPS) values for tests. Results indicated that the methods used converged on the same cut score and these were all within one standard error of the Angoff-based MPS value.

Nebraska districts use different strategies for measuring student performance on the state's content standards. District assessments differ in type and technical quality. Six quality criteria were endorsed by the state. These criteria cover content and curricular validity, fairness, and appropriateness of score interpretations. District assessment portfolios document how well assessments meet these criteria. Districts receive ratings on how well their assessments meet each of the quality criteria and are given a rating from Unacceptable to Exemplary. This article presents these technical quality criteria and explains how they are (a) individually rated and (b) combined for the district's overall quality rating.

As part of Nebraska's assessment and accountability system, districts' local assessment systems are evaluated for their psychometric quality. This article provides an overview of a two‐stage evaluation strategy, discusses how it was applied in Nebraska, and presents results from the first three years of the evaluation process. Benefits of the method include an emphasis on formative evaluation and promotion of improved assessment quality at the local level. A limitation of the model is the inability to make refined comparisons of student performance across districts on the assessments. Results from the first three years suggest that greater specificity in the review criteria and additional reviewer calibration activities are needed.

Most states have adopted assessment and accountability systems that involve common measures of student performance. A state assessment system that allows school districts to choose the specific strategies they use to measure student performance on state‐adopted content standards presents a unique state accountability challenge. The authors propose an accountability model that addresses this challenge using a combination of student performance, technical quality, and noncognitive indicators of performance. They also describe a study that evaluated the proposed model using data from all school districts in a southern state.

Tests are used for making decisions about students, including decisions about promotion, in-grade retention, and high school graduation. This article provides criteria for evaluating the technical quality of tests used for high-stakes decisions in education. The intent is to provide educators with information for evaluating their testing program.

Past research on gender differences in performance on mathematics achievement test items has focused on characteristics of test items on which males and females perform differentially. In most instances, such studies have employed a post-hoc content analyses of test items identified via statistical analyses to function differentially for males and females. This study, in contrast, used an experimental design approach to investigate the effect of item context on differential item performance on mathematics test items between males and females. The analyses were based on a modified one parameter IRT methodology. Although some significant differences in performance were found between and within item arrangements for males and females, given the number of statistical tests employed, these significances could be due to chance alone.

One source of test invalidity in the area of licensure and certification testing is the reading level of the test. The purposes of this paper are to: (a) propose a model for evaluating the readability level of certification/licensure examinations; (b) evaluate selected readability indices in terms of their appropriateness for use in estimating the readability level of specific certification/licensure examinations; and (c) present an example of the application of this model to the readability analysis of a state licensure examination in cosmetology. Readability level is an important dimension of the validity of licensure and certification examinations. One potential basis for candidate appeal is the appropriateness of the readability level of the examination with respect to the tasks and functions of the profession.

Differential test performance by males and females was reported on a quantitative examination as a function of item arrangement (Plake, Ansorge, Parker, and Lowry, 1982). Males performed at a higher level than females on tests the items of which were arranged from easy to hard. Plake and Ansorge (1982) speculated that this male superiority may be a function of differential reinforcement in backgrounds of males and females. Using a Latin Square design, this study provided item performance feedback for a nonquantitative examination. Significant sex-by-order effects did not occur, but the lack of this effect could possibly have been due to the use of a nonquantitative examination (Plake and Ansorge, 1983). It still remains to be seen whether, in using a quantitative examination, differential effects of item feedback may be a source of explanation for evidence of sex-by-order effects. Implications for validity of test score interpretation are considered briefly.

The Bias in Attitudes Survey (BIAS) is a recently devised scale which assesses attitudes and beliefs about sex roles. This study was concerned with the factorial validity of the BIAS. The results suggested that the BIAS in relation to other sex-role scales is measuring more complete and complex attitudes toward sex roles. Two factors, which explained 68% of the variance, represented both the traditional view of male/female roles and a more liberal, non-sexist view. The analysis of the male/female subgroups demonstrated a lack of parallelism in the factor structure of the scale for the two sexes.

Scores reflecting the attitudes of fourth, fifth, and sixth-grade Mexican-American and Anglo students about reading were compared to assess the relevance and validity of the Estes Reading Attitude Scale for each group. The validity of application of results from the Estes Reading Attitude Scale when administered to students of differing ethnic backgrounds was examined by considering the factorial complexity of the instrument overall for Anglo and Mexican American children, as well as at three grade levels. The comparability of the internal consistency indices and of factorial structure between ethnic groups within grade levels, and the consistency of item interpretation (or lack of item "bias") between ethnic groups within grade level suggested that scores on the scale afforded a valid means for interpretation of attitudes held by elementary school pupils towards reading irrespective of their ethnic status (Anglo or Mexican-American).

A 24-item shortened version of the 98-item Mathematics Anxiety Rating Scale (MARS) was developed to provide a more efficient index of statistics or mathematics course-related anxiety. The revised scale, which yielded a coefficient alpha reliability estimated at .98, was correlated .97 with a full scale MARS. It shows a pattern of relationships with state, trait, and test anxiety parallel to that of full scale MARS. Further, the relationship with mathematics achievement was consistent for both the revised and full scale MARS. A principal axes factor analysis of revised MARS identified two clear factors, labeled "Learning Mathematics Anxiety" and "Mathematics Evaluation Anxiety." Revised MARS appears to be an attractive substitute for full scale MARS for statistical or mathematical course-related uses.

Since 1960 educational researchers have been using the Achievement Anxiety Test (AAT), which presumably measures independent aspects of achievement anxiety: facilitating and debilitating components. In several earlier investigations, developers have reported construct and predictive validity of the subscales from their use of anxiety measures and performance criteria. However, the hypothesized independence of the subscales and underlying factor structure of the test have been overlooked. Because the two subscales have been used to provide a difference score, the status of the interrelationship between the subscales is of concern. This study was concerned with the concurrent validity of the AAT and with its factor structure to provide empirical evidence about the quality of AAT.

The methodology suggested in this paper employs a selection rule for identifying group members that generates groups that have a range of achievement within groups but equal distribution of raw scores between. An ANOVA procedure on arcsin transformed item difficulties, followed by an investigation of strength of association lend practical significance to significant items by groups interaction. A follow-up procedure that identifies items contributing to the items-by-groups interaction is described.

How were the materials tested? How did the reactions of teachers compare to those of measurement students? How can these materials be used effectively?

Investigation of bias related to either gender or gender-related attributes has been hampered by the lack of a scale which would address the gender-relatedness of characteristics and which also would indicate how positively or negatively the stereotypical characteristics are viewed. The new scale, Positive Sex-role Inventory, was developed to have three dimensions: masculine, feminine and neutral. All of the characteristics are positive in terms of employability or market value. Empirical investigation of the instrument for the profession of counseling psychologist showed a three factor-structure and adequate estimates of internal consistency reliability.

This series of studies examined the effect of adhering to the universally endorsed item-writing practice of making the correct answer of a multiple-choice item grammatically consistent with the stem. Formats were designed to investigate whether the practice of providing relevant grammatical cues in itself created a sensitivity for item performance. ACT Assessment experimental social studies items were constructed to investigate grammatical compliance in two particular situations: plural-singular and vowel-consonant agreement. In no instance were grammatically inappropriate situations created. Results suggest some subject sensitivity to these cues. Rephrasing into neutral, but still precise and grammatically correct, language may eliminate this sensitivity.

Previous research using an examination with a quantitative emphasis revealed a significant sex effect and an item arrangement x sex interaction for a cognitive variable (number right) and an item arrangement effect for self-perception scores pertaining to difficulty, fairness, and anticipated performance on the test. Scores representing number of items right and self-perceptions were analyzed for nonquantitative examination that was assembled into three forms: easy-hard, spiral cyclical, random. Multivariate ANCOVA revealed no significant effects (α = .05) for the cognitive measure. However, significant (α = .05) sex and sex x order effects were found for perceptions scores which were not parallel to those reported previously for a quantitative examination. Implications for the valid interpretation of test scores also are briefly discussed.

In an attempt to make commercial tests valid for users of both sexes and all race groups, many test publishers employ review procedures to remove potentially invalid (or biased) items from their tests. Statistical procedures to identify biased test items are often sophisticated and costly to test developers. Alternative approaches have employers review test items and select those that appear biased. To verify this methodology, a statistical and a subjective procedure were used to identify biased items in an elementary achievement test. The results show little agreement on items selected as biased. An implication to test developers is to adopt statistical support for a reviewer's selection of biased items.

Combining the Livingston and Zieky (1982) correction to a cut score for an examination using correction-for-guessing scoring is reasonable if minimally competent candidates (MCCs) omit only those items for which they cannot eliminate any of the options. Otherwise, the corrected cut-score will be overly harsh, resulting in additional errors in classifications. In this study, omit behavior on a 48-item Mathematics Achievement Test of an empirically established group of MCCs was compared to the predictions of five expert judges. The results suggest that this group of examinees tended to respond to the items with an omit pattern similar to that predicted by the judges.

Procedures for validating the complexity level of test items include empirical examinations of item characteristics and subjective judgments of expert raters. These methods have yielded mixed and inconsistent results. This study employs a technique derived from cognitive psychology research in the examination of levels of item complexity. Using a microcomputer to administer test items, response time to a secondary task was used as an index of the level of item complexity. A series of three studies are reported that (1) employ the technique on items of marked discrepancy in complexity level and (2) contrast the sensitivity of the technique to more traditional analyses of level of item complexity. The results suggest that the technique has promise as a mechanism for validating or establishing level of item complexity.

A confirmatory factor analysis was used to determine the respective viability of factor structure models for the WAIS-R standardization data. The traditional Verbal-Performance split was compared to an alternative model proposed by Gutkin, Reynolds, and Galvin (1984). The confirmatory analysis indicated that the Gutkin model provides a superior fit to the data at all age levels and for the total sample. In addition, the results from Gutkin model analyses at each age level were iterated, yielding a refinement in the respective age-specific fits of the model. Implications for test interpretation are discussed.

To evaluate the potential bias (racial and sexual) of items in the Revised Children's Manifest Anxiety Scale, a scale used in the diagnosis and evaluation of emotional disturbance, a race-by-sex-by-items ANOVA was calculated from the responses of more than 5000 children with a Bonferroni-type adjusted follow-up test to identify specific biased items. Numerous indications of item bias were detected, though the direction of the bias tended to be balanced across groups. The race-by-sex-by-items interaction term, while highly statistically significant (p < .001), accounted for less than 1 percent of score variance.

A modified confirmatory factor analysis of the WAIS-R was used to refine the initial confirmatory factor analysis solutions identified by Plake, Gutkin, and Kroeten (1984) in order to have a more complete comparison of the factor structure models postulated by Wechsler (1981) and Gutkin, Reynolds, and Galvin (1984). Root mean square residuals of the final modified solutions were compared for the Wechsler and Gutkin Models at each of the nine age levels and for the total group. In most cases, the modified Gutkin model provided a better fit to the data than did the modified Wechsler model. Implications for score interpretation are discussed.

A national sample of administrators and teachers was administered a test of their knowledge of classroom assessment. The results indicated that administrators, on average, were more knowledgeable than teachers, and that both groups demonstrated considerable knowledge in certain areas. In other areas, however, both administrators and teachers were deficient in their knowledge of classroom assessment practices. Implications for training administrators and teachers are drawn.