Search

With the 2014 publication of the 5th revision of the "Standards for Educational and Psychological Testing," the cochairs of the Joint Committee for the revision process were asked to consider the role and importance of the "Standards" for the educational testing community, and in particular for members of the National Council on Measurement in Education. This article begins with an overview of the process used to create the 2014 revision of the "Standards" and a discussion of challenges faced by the Joint Committee in the revision process. The next section summarizes the major changes in the revised Standards. The final section considers uses of the revised "Standards" and is intended to stimulate discussion and debate on the role and importance of these revised "Standards" for the educational measurement community.

The many-faceted Rasch (MFR) model has been used to evaluate the quality of ratings on constructed response assessments; however, it can also be used to evaluate the quality of judgments from panel-based standard setting procedures. The current study illustrates the use of the MFR model for examining the quality of ratings obtained from a standard setting process employing the Multiple Yes-No, modified Angoff standard setting procedure to set the cut scores on the Advanced Placement Environmental Science exam. The panelists' severity, judged item difficulties, rounds, and cut scores using the MFR model were examined, in addition to panelists' characteristics (gender and level of course taught). Findings from the Rasch analyses conducted in this study provide evidence of acceptable quality of the ratings obtained from the panelists. Thus, the current study provides an important source of both internal and procedural validity evidence for the Advanced Placement Environmental Science standard setting as well as an illustration of the utility of the MFR model for evaluating standard setting judgments. (Contains 5 tables, 5 figures, and 4 notes.)

This study examined the amount of time that different ability-level examinees spend on questions they answer correctly or incorrectly across different pretest item blocks presented on a fixed-length, time-restricted computerized adaptive testing (CAT). Results indicate that different ability-level examinees require different amounts of time to answer pretest items, that is, field-test items, embedded in different item locations. Also, guessing behaviors and test-taking effort (TTE), measured using detection indices developed from this study, varied across the pretest items embedded in different locations. With these indices, unique response time patterns (RTP) and guessing patterns (GP) were identified for six ability groups. These indices could also be used to monitor the amount of effort and the guessing behaviors of examinees taking a CAT. The degree of the detection consistency among these indices could be used as reliability evidence to evaluate how reliable the indices are in detecting unusual test-taking behaviors. Once reliable detection was confirmed, a warning sign prompted from the computer screen could either encourage the examinee to take appropriate TTE or stop guessing so that the examinees could receive more accurate estimates of their ability. (Contains 10 tables and 1 figure.)

In many large-scale assessment programs, achievement level descriptors (ALDs) provide a critical role in communicating what scores on the assessment mean and in interpreting what examinees know and are able to do based on their test performance. Based on their test performance, examinees are often classified into performance categories. The descriptions of what these performance categories mean are communicated through the ALDs. In addition to providing supporting information for score interpretations, ALDs may also serve an important role in setting the cut scores on the score scale that determine the categorization of examinees' scores into the performance categories. The standard setting process is used to establish these cut scores. The purpose of this article is to provide a methodology to directly connect evidence-centered assessment design (ECD) to score interpretation and use through the development of ALDs and to explore how ECD-based ALDs can be leveraged in standard setting. (Contains 1 table, 2 figures and 2 footnotes.)

Standard setting is a systematic process that uses a combination of judgmental and empirical procedures to make recommendations about where on the score continuum "cut scores" should be placed. Cut scores divide the score scale into categories consistent with the descriptions of student performance associated with multiple levels of achievement. Critical decisions will be based on interpretations emerging from how students are categorized into each level. Ensuring that the methodology used to set the cut scores is sound and defensible is a prerequisite for valid interpretations of student performance. Innovations in standard setting methodology in the first decade of this century have been prolific; the number of articles, chapters and presentations are too numerous to cite or review here, but the fact that there have been at least three books just on standard setting this decade is testimony to the amount of research and development in this area (Cizek, 2001; Cizek & Bunch, 2007; Zieky, Perie, & Livingston, 2008). In this paper, the authors question whether all of this work, effort, and resources has been misdirected given the less-than-adequate work on one of the most critical inputs to the standard setting process: the achievement level descriptions. In other words, like the crude adage "garbage in, garbage out," innovations in standard setting methodology are attenuated until some necessary preconditions are met. The authors outline these necessary preconditions, their promise for improving standard setting, their promise for improving the accountability system and the reform effort for which standard setting is a means, and suggested research to make progress on those necessary preconditions. (Contains 2 tables, 1 figure and 7 footnotes.)

The National Assessment of Educational Progress (NAEP) program is a series of periodic assessments administered nationally to samples of students and designed to measure different content areas. This article describes a multi-year study that focused on the breadth of the development, administration, maintenance, and renewal of the assessments in the program. The methodology targeted data collection through documentation and onsite interviews with key personnel relative to evaluation criteria developed from the "Standards for Educational and Psychological Testing" (AERA, APA, & NCME, 1999). We present summary results of this study and discuss one of the overarching recommendations that can inform practice for all testing programs. (Contains 1 table and 1 figure.)

Even when the scoring of an examination is based on item response theory (IRT), standard-setting methods seldom use this information directly when determining the minimum passing score (MPS) for an examination from an Angoff-based standard-setting study. Often, when IRT scoring is used, the MPS value for a test is converted to an IRT-based theta ability estimate by using the test characteristic curve. The purpose of this study is to investigate some approaches to determining the MPS from an Angoff-based standard-setting study when the test questions have been calibrated using IRT methods. Although the methods studied appear to converge on the same cut score, the cut scores derived from the IRT-based approaches are all within one standard error of the Angoff-based MPS value. (Contains 5 tables.)

As an alternative to adaptation, tests may also be developed simultaneously in multiple languages. Although the items on such tests could vary substantially, scores from these tests may be used to make the same types of decisions about different groups of examinees. The ability to make such decisions is contingent upon setting performance standards for each exam that allow for comparable interpretations of test results. This article describes a standard setting process used for a multilingual high school literacy assessment constructed under these conditions. This methodology was designed to address the specific challenges presented by this testing program including maintaining equivalent expectations for performance across different student populations. The validity evidence collected to support the methodology and results is discussed along with recommendations for future practice.

In an Angoff standard setting procedure, judges estimate the probability that a hypothetical randomly selected minimally competent candidate will answer correctly each item in the test. In many cases, these item performance estimates are made twice, with information shared with the panelists between estimates. Especially for long tests, this estimation process can be time-consuming and fatiguing for the judges. This study extended an item selection strategy earlier proposed to form subsets of test items. The results of this study suggest that 40% to 50% of test items may be sufficient to estimate an equivalent passing score in an Angoff setting study when those items are selected to represent the full test in content, discrimination, and difficulty. (Contains 4 tables.)

The No Child Left Behind Act of 2001 allows states to assess students with "significant cognitive disabilities" on "alternative content standards" for determining adequate yearly progress. Alternative standards that align with regular content standards can allow for a continuum of performance expectations from very basic to approaching grade level, therefore tapping into the abilities of students with significant cognitive disabilities. Performance level descriptors were developed in an effort to identify students' level of attainment for each content standard. States must demonstrate that assessments align with articulated alternative standards and that alternative standards align with the state's content standards. The purpose of this study is to examine a methodology for aligning alternative content standards to regular content standards.

The purpose of this study was to examine the effect of pretest items on response time in an operational, fixed-length, time-limited computerized adaptive test (CAT). These pretest items are embedded within the CAT, but unlike the operational items, are not tailored to the examinee's ability level. If examinees with higher ability levels need less time to complete these items than do their counterparts with lower ability levels, they will have more time to devote to the operational test questions. Data were from a graduate admissions test that was administered worldwide. Data from both quantitative and verbal sections of the test were considered. For the verbal section, examinees in the lower ability groups spent systematically more time on their pretest items than did those in the higher ability groups, though for the quantitative section the differences were less clear.

Licensure testing programs in the United States (e.g., nursing) face an increasing challenge of measuring the competency of internationally trained candidates, both in relation to their clinical competence and their English language competence. To assist with the latter, professional licensing bodies often adopt well-established and widely available international English language proficiency measures. In this context, the National Council of State Boards of Nursing (NCSBN) sought to develop a nursing-specific passing standard on the International English Language Testing System that U.S. jurisdictions could consider in their licensure decisions for internationally trained candidates. Findings from a standard setting exercise were considered by NCSBN's Examination Committee in conjunction with other relevant information to produce a legally defensible passing standard on the test. This article reports in detail on the standard setting exercise conducted as part of this policy-making process; it describes the techniques adopted, the procedures followed, and the outcomes obtained. The study is contextualized within the current literature on standard setting. The latter part of the article describes the nature of the policy-making process to which the study contributed and discusses some of the implications of including a language literacy test as part of a licensure testing program. (Contains 5 footnotes and 9 tables.)

In 2001, the U.S. Citizenship and Immigration Services (USCIS) began the process of redesigning the U.S. naturalization tests due to concerns that the current testing procedure may not be sufficiently uniform, and that the test content may not be appropriate. A National Research Council committee issued recommendations, based largely on the Standards, to ensure that the redesigned tests would be scientifically and politically defensible; however, USCIS chose not to follow much of the committee's advice. This article summarizes some lessons for the measurement community and possible strategies for getting policymakers to pay more attention to the Standards.

In cut score setting processes, subject matter experts are asked to make judgments about the likely performance of examinees at a targeted skill level. When cut scores are used in K-12 settings to separate students who have and have not mastered certain skills, the target examinee may be characterized as the barely proficient or barely master student. This study examines how teachers who participate in cut score setting workshops conceptualize these target examinees. This article found that teachers' descriptions of target examinees were similar across subject matter and grade level, and also reflected the definitions provided by workshop facilitators. Findings have implications for interpreting cut scores.

This study addressed what standard-setting panelists think about when they make item performance estimates for a barely proficient student. This study extended previous studies by considering the factors that influenced panelists' decisions in an Angoff (1971)-based standard-setting study as a function of their item performance estimates. Panelists were divided into 3 groups based on their Round 1 item performance estimates (low, moderate, high). The study revealed that panelists in the high and moderate groups considered different factors than the panelists in the low group when they made decisions on the test items. Panelists in the low group appeared to think in a norm-referenced perspective, whereas the high and moderate groups thought in a criterion-referenced perspective.

In an Angoff standard-setting procedure, judges estimate the probability that a hypothetical randomly selected minimally competent candidate will answer correctly each item constituting the test. In many cases, these item performance estimates are made twice, with information shared with the judges between estimates. Especially for long tests, this estimation process can be time consuming and fatiguing for the participants. This study addressed the possibilities of reducing the number of items individual judges consider in an Angoff standard-setting study. The results of this study suggest that 50% of test items may be sufficient to estimate an equivalent passing score in an Angoff setting study. This could result in substantial savings of time and resources not only for the agencies that have been carrying out this activity but also for the practitioners (judges) who participate in standard-setting studies.

As part of Nebraska's assessment and accountability system, districts' local assessment systems are evaluated for their psychometric quality. This article provides an overview of a two-stage evaluation strategy, discusses how it was applied in Nebraska, and presents results from the first three years of the evaluation process. Benefits of the method include an emphasis on formative evaluation and promotion of improved assessment quality at the local level. A limitation of the model is the inability to make refined comparisons of student performance across districts on the assessments. Results from the first three years suggest that greater specificity in the review criteria and additional reviewer calibration activities are needed.

Nebraska districts use different strategies for measuring student performance on the state's content standards. District assessments differ in type and technical quality. Six quality criteria were endorsed by the state. These criteria cover content and curricular validity, fairness, and appropriateness of score interpretations. District assessment portfolios document how well assessments meet these criteria. Districts receive ratings on how well their assessments meet each of the quality criteria and are given a rating from Unacceptable to Exemplary. This article presents these technical quality criteria and explains how they are (a) individually rated and (b) combined for the district's overall quality rating.

Tests are used for making decisions about students, including decisions about promotion, in-grade retention, and high school graduation. This article provides criteria for evaluating the technical quality of tests used for high-stakes decisions in education. The intent is to provide educators with information for evaluating their testing program. (Contains 13 references.) (Author)

Proposed an accountably model that addresses the challenges of allowing school districts to choose the specific strategies they use to measure student performance and evaluated this model using data from multiple sources for all school districts in Florida. Findings identify three strategies that would be useful for this type of accountability system. (SLD)

Examined the reliability and accuracy of item performance estimates from an Angoff standard setting application with 29 panelists on 1 year and 30 in the next year. Results provide evidence that item performance estimates were both reasonable and reliable. Discusses factors that might have influenced the results. (SLD)

Applied the analytical judgment standard setting method to 90 papers from the 1996 Grade 8 National Assessment of Educational Progress science assessment. Compared sorting versus direct classification, long and short versions of the classification scale, and effects of discussion on cutscores. Results from 17 Georgia teachers and 8 Michigan teachers show direct classification to be more feasible than sorting. (SLD)

The relationship between the validity theory of the past 50 years and actual validity practices was studied by comparing published test standards with the practices of measurement professionals expressed in the "Mental Measurements Yearbook" test reviews. Results show a symbiotic relationship between theory and practice on the influence of validity. (SLD)

Credentialing programs were surveyed to determine the procedures they use to set performance standards on multiple-choice and open-ended assessments. Implications of the various standard-setting approaches for the National Assessment of Educational Progress are discussed, and it is asserted that generalizing from standard-setting in professional performance should be regarded with caution. (SLD)

Two studies of variations of the Angoff method (W. Angoff, 1971) involving nine elementary school teachers in each case compared a yes/no estimation with a proportion correct estimation for setting cut scores. Both methods yielded essentially equal cut scores, but judges found the yes/no method easier to implement. (SLD)

The development and validation of professional development resource materials for teachers by the National Council on Measurement in Education and associated professional groups is described. The developed module contains a context-orienting vignette, instruction, practice exercises, hands-on experiences, small group topics, and homework problems. (SLD)

The dominant profile judgment method, designed for use with profiles of polytomous scores on exercises in a performance-based assessment, is presented as a standard-setting method. The approach guides standard-setting panelists in articulating their standard-setting policies and allows for complex policy statements. (SLD)

Three national educational organizations articulated seven areas of teacher competency in student assessment. Among 555 teachers surveyed nationwide, the area of best performance was "administering, scoring, and interpreting test results," and the area of worst performance was "communicating test results." Development of inservice training materials is discussed. Includes standards. (SV)

Studied acculturation of international people (n=606). Found that Africans, Asians, and South Americans were significantly less acculturated than were Europeans. Significant differences emerged for permanent versus nonpermanent U.S. residence status, length of residence in United States, and religion. Participants related their identity to values, religion, physical appearance, and language of national origins. (Author/NB)

Summarizes the past contributions of the Buros Institute to the counseling and psychology communities, and previews the emergence of a more timely and responsive array of products and services. Contends that, as the psychological community continues through the 1990s, the Buros Institute will continue to provide clinicians, practitioners, and researchers with evaluative information for assessment instruments. (LLL)

Several methods for determining a passing score on an examination from individual raters' estimates of minimal pass levels were compared through simulation. The methods used differed in the weighting estimates for each item received in the aggregation process. Reasons why the simplest procedure is most preferred are discussed. (SLD)

Discusses the unique advantages provided by computer-based (CB) testing. Describes the various forms of CB tests used in higher education and the variety of testing applications of computers in colleges and universities. Presents psychometric issues and concerns related to CB testing along with relevant research findings. (Author/PVV)

Examined Gottfredson's postulation that individuals are more likely to sacrifice prestige than sex type preference when career compromise is needed in college students (n=246). Results indicated prestige more often used as preferred factor than sex type, but career compromise behavior was affected by gender and degree of contrast in prestige and sex type levels. (Author/ABL)

Literature on acculturation is reviewed, and the effects of acculturation on the mental health of Hispanics and Asian Americans are discussed. In this study we attempted to understand acculturation processes at the group level (Hispanics versus Asian Americans) and at the individual level (within-group heterogeneity). The Majority-Minority Relations Survey measured acculturation attitudes of two American ethnic minority groups, Hispanics and Asian Americans, in a Midwestern university. Data were obtained from 282 participants for a 54% return rate. Instrument analysis included assessment of generalizability (using Pearson correlation coeeficient procedures), LISREL confirmatory factor analysis, and tests of internal consistency reliabilities (using coefficient alpha). Significant effects for ethnicity, Asian culture subgroups, and for the sociocultural variables of generational status, voluntary immigration versus political asylum, and religion were indicated by parametric and nonparametric tests. Therefore, although differences between the Hispanics and the Asian Americans were indicated, within-group differences for both minority groups were also influenced by select sociocultural variables.

Two studies examined the effect of making the correct answer of a multiple choice test item grammatically consistent with the item. American College Testing Assessment experimental items were constructed to investigate grammatical compliance to investigate grammatical compliance for plural-singular and vowel-consonant agreement. Results suggest some subject sensitivity to these cues. (Author/BS)

Investigated effects of psychological sex role orientation and level of cognitive development on performance in mathematics in 18 male and 51 female undergraduates. Revealed significant effects for level of cognitive development and for masculine by feminine sex role orientation interaction. Results indicated lower mathematics performance for androgynous and undifferentiated subjects. (Author/NB)

Examined sex bias in hiring by personnel directors. Personnel directors (N=86) rated bogus job applications on their qualifications and likelihood of being considered for the position, expected performance, and expected success in the job. Found higher access ratings were given to the sex-stereotypical applicant when the applicant's credentials matched job demands. (Author/ABL)

This research addresses the question of the comparability of equal scores on the same test from children in different grades. The results indicate that the scores are comparable, and support the assignment of Iowa Tests of Basic Skills levels that match the child's developmental level. (Author)

The effect of item context on differential item performance based on gender on mathematics test items was studied, using 404 male and 375 female adults. The analyses were based on a modified one-parameter item response theory methodology. Gender differences emerged; however, they may be due to chance. (TJH)