Search

Over the past four decades, there has been incremental growth in computer-based testing (CBT) as a viable alternative to paper-and-pencil testing. However, the transition to CBT is neither easy nor inexpensive. As Drasgow, Luecht, and Bennett (2006) noted, many design engineering, test development, operations/logistics, and psychometric changes are required to develop a successful operational program. Early research on CBT almost exclusively focused on theoretical issues such as improving measurement efficiency by achieving adequate levels of test score reliability using as few items as possible. However, it was soon evident that practical issues--such as ensuring content representation, making sure all examinees have sufficient time to complete the test, implementation of new item types, and controlling the degree to which items were exposed to examinees--needed to be addressed, too. In the past few years, research on CBT has focused on developing models that achieve desired levels of measurement efficiency while simultaneously satisfying other important goals, such as minimizing item exposure and maintaining content validity. In addition, there has been a growing awareness among practitioners that basic CBT research using small samples or simulation studies needs to be vetted using cost-benefit analysis, as well as engineering design and implementation criteria to ensure that feasibility, scalability, and efficiency are evaluated in more concrete ways than by merely reporting a reduction of error variances for theoretical examinee scores (Luecht, 2005a, 2005b).

The purpose of this report is to provide an update on the state of the research on testing accommodations, as well as to identify promising areas of research to further clarify and enhance understanding of current and emerging issues. The research described encompasses empirical studies of score comparability and validity studies, as well as investigations into accommodations use and perceptions of their effectiveness. Taken together, the current research explores many of the issues surrounding test accommodations practices in both breadth and depth. Insofar as reporting on the findings of current research studies is a primary goal of this analysis, a second goal is to also identify areas requiring continued investigation in the future. (Contains 37 tables and 1 figure.)

The National Assessment of Educational Progress (NAEP) seeks to include all students in the United States in the sampling frame from which students are selected to participate in the assessment. However, some students with disabilities (SWD) are either unable to take NAEP tests under standard testing conditions or are unable to perform at their best under standard testing conditions. In many testing situations, accommodations to standard testing conditions are given to SWD to improve measurement of their knowledge, skills, and abilities. This practice is in the pursuit of more valid test score interpretation; however, it produces the ultimate psychometric oxymoron--an accommodated standardized test. In this paper, I review validity issues related to test accommodations and summarize some empirical studies in this area. The focus of the paper is on accommodations for reading tests because some types of accommodations on these tests are particularly controversial. The specific accommodations emphasized in this review are extended time and oral (read-aloud) accommodations. A review of professional standards, validity theory, and recent empirical research in this area suggests that extended time accommodations may be appropriate for reading tests, but read-aloud accommodations are likely to alter the construct measured. Suggestions for determining when to provide accommodations and how to report scores from accommodated test administrations are provided. (Contains 3 tables, 2 figures, and 2 footnotes.) [This paper is one of a set of research-oriented papers commissioned by National Assessment Governing Board (NAGB) to serve as background information for attendees of the NAGB Conference on Increasing the Participation of Students with Disabilities (SD) and limited English proficient (LEP) Students in NAEP. This paper was also published as: Center for Educational Assessment Research Report No. 515. Amherst, MA: School of Education, University of Massachusetts Amherst.]

The purposes of this research study were to develop and field test anchor-based judgmental methods for enabling test specialists to estimate item difficulty statistics. The study consisted of three related field tests. In each, researchers worked with six Law School Admission Test (LSAT) test specialists and one or more of the LSAT subtests. The three field tests produced a number of conclusions. A considerable amount was learned about the process of extracting test specialists' estimates of item difficulty. The ratings took considerably longer to obtain than had been expected. Training, initial ratings, and discussion took a considerable amount of time. Test specialists felt they could be trained to estimate item difficulty accurately and, to some extent, they demonstrated this. Average error in the estimates of item difficulty varied from about 11% to 13 %. Also the discussions were popular with the panelists, and almost always resulted in improved item difficulty estimates. By the end of the study, the two expected frameworks that developers thought they might provide test specialists, had merged to one. Test specialists seemed to benefit from the descriptions of items located at three levels of difficulty and from information about the item statistics of many items. Four appendixes describe tasks and contain the field test materials. (Contains 8 tables and 18 references.) (SLD)

The primary objective of this study was to investigate how incorporating prior information improves estimation of item parameters in two small samples. The factors that were investigated were sample size and the type of prior information. To investigate the accuracy with which item parameters in the Law School Admission Test (LSAT) are estimated, the item parameter estimates were compared with known item parameter values. By randomly drawing small samples of varying sizes from the population of test takers, the relationship between sample size and the accuracy with which item parameters are estimated was studied. Data used were from the Reading Comprehension subtest of the LAST. Results indicate that the incorporation of ratings of item difficulty provided by subject matter specialists/test developers produced estimates of item difficulty statistics that were more accurate than that obtained without using such information. The improvement was observed for all item response models, including the model used in the LSAT. (SLD)

International comparative educational studies make use of test instruments originally developed in English by international panels of experts, but that are ultimately administered in the language of instruction of the students. The comparability of the different language versions of these assessments is a critical issue in validating the comparative inferences drawn from the test results. This study analyzed data from the 1999 Third International Mathematics and Science Study (TIMSS) science assessment to evaluation the consistency of the structure of the item response data across different test versions. Individual differences multidimensional scaling analyses were used to evaluate data structure. The findings suggest that slight structural differences exist across countries, and that these differences are related to differences in item difficulty. The implications of these findings for better understanding of international comparisons of educational achievement, and for future research in this area, are discussed. (Contains 1 figure, 7 tables, and 26 references.) (Author/SLD)

Many students in the United States who are required to take educational tests are not fully proficient in English. To address this problem, a state-mandated testing program created dual language English-Spanish versions of some of their tests. In this study, the psychometric properties of the English and dual language versions of a fourth-grade mathematics test were explored. Analyses of the consistency of test structure across the two forms were conducted using structural equation modeling and multidimensional scaling. Analyses of differential item functioning (DIF) were conducted using Poly-SIBTEST. The results suggest slight structural differences across the two versions of the test. Part of this difference was attributed to overall proficiency differences across the two studied groups, and part was attributed to DIF. The implications of the findings for future research in this area are discussed. (Contains 4 figures, 11 tables, and 28 references.) (Author/SLD)

Measurement specialists routinely assume examinee responses to test items are independent of one another. However, previous research has shown that many contemporary tests contain item dependencies and not accounting for these dependencies leads to misleading estimates of item, test, and ability parameters. In this study, methods for detecting local item dependence (LID) are reviewed, and the use of testlets to account for LID in context-dependent item sets is discussed. LID detection methods and testlet-based item calibrations are applied to data from a large-scale, high stakes admissions test, and the results are evaluated with respect to test score reliability and examinee proficiency estimation. Data were from two forms of the Medical College Admission Test (MCAT) for 8,494 and 8,026 examinees. Results suggest the presence of LID impacts estimation of examinee proficiency. The practical effects of the presence of LID on passage-based tests are discussed, as are issues regarding the calibration of context-dependent item sets using item response theory. (Contains 3 figures, 11 tables, and 30 references.) (Author/SLD)

To provide a better understanding of the structure of the Medical College Admission Test (MCAT) and to determine if there are structural differences across selected groups of MCAT examinees, several dimensionality analyses were conducted on data from recent administrations of the MCAT. The first set of analyses focused on the global structure of the MCAT, and the second set appraised the consistency of the structure of data across groups of testtakers that differed with respect to sex, repeater/nonrepeater status, orientation to the English language, and race/ethnicity. Data from two forms of the MCAT were used. Forms 15A and 15B were administered in 1994 to 16,520 examinees, and Forms 23A and 23B were administered in 1996 to 12,625 examinees. Results suggest that appraisals of the MCAT structure should be conducted at the parcel level rather than at the item level. Parcel-level results suggest that a dominant factor underlies the MCAT. This is probably a "general intelligence" factor. The results also suggest additional factors that represent the principal disciplines measured on the MCAT. After the general factor, the next structural layer of the MCAT separates test material measuring science from test material measuring verbal reasoning and writing skills. The next structural level depicts three factors: science, verbal reasoning, and writing skills. These three factors were supported by all analyses. Results also support the distinction between the science disciplines, and, in general, analyses support the current content structure of the MCAT reported in the test blueprint. From a statistical perspective, results suggest it might be possible to scale the biological and physical sciences along a single continuum. With respect to the consistency of the MCAT structure across selected groups of testtakers, results supported the hypothesis of structural invariance across groups. In general, multidimensional scaling analyses indicated that all dimensions were relevant for accounting for the variation in the data for each group. Some exceptions are discussed. An appendix contains depictions of the confirmatory factor analysis models and parceling schemes. (Contains 13 figures, 30 tables, and 20 references.) (SLD)

Setting standards on educational tests is extremely challenging. The psychometric literature is replete with methods and guidelines for setting standards on educational tests; however, little attention has been paid to the process of setting standards on computerized adaptive tests (CATs). This lack of attention is unfortunate because CATs are becoming more widely used, and setting standards on these tests is typically more difficult than setting standards on nonadaptive (linear) tests. This paper discusses some of the issues to be addressed when setting standards on CATs, presents the results of a standard setting study conducted on a computerized adaptive placement test, and discusses the implications of the findings for future research and practice in this area. Thirteen mathematics experts participated in the standard-setting study using ACCUPLACER (College Board) scores. The results of the study suggest that standards can be set on CATs using subsets of items from a CAT item pool, and that methods designed to gather test-centered standard setting data more quickly than traditional methods show promise for setting standards on CATs. (Contains 2 figures, 5 tables, and 14 references.) (Author/SLD)

Assessing people who operate in different languages necessitates the use of multiple language versions of an assessment. However, different language versions of an assessment are not necessarily equivalent. In this paper, the psychometric properties of different language versions on an international employee attitude survey are evaluated. This survey was administered to more than 50,000 employees of a large telecommunications company using both paper-and-pencil and Web administration formats. The structural equivalence of the survey was evaluated across language versions, cultural groups, and administration formats using multidimensional scaling. The statistical equivalence of English, French, and Spanish versions of the survey items was evaluated using analysis of covariance. The results indicate the structure of the survey is consistent across the groups studied, and that the different language versions of the items functioned similarly. The implications of the results for future research in this area are discussed. (Contains 3 figures, 4 tables, and 38 references.) (Author/SLD)

This study investigated the attitudes and opinions of 970 students with and without learning disabilities regarding the use of computers for school-related work. Using a quasi-experimental design with three non-equivalent groups, within and between subjects effects were studied using a survey instrument. The students in grades 5 through 12 at three school sites completed pre- and post-test surveys at the beginning and end of the school year. One site served as the experimental group, while the other two were control groups. The experimental condition consisted of the installation of a campus-wide computer network for use by all students at the experimental site. A 24-item scale measured participants' attitudes about the general use of computers in schools and the use of computers by students with special needs. The most significant variables related to students' attitudes and opinions were their past experiences using computers and their school affiliation. These data also showed there was no relationship between the installation of a campus-wide computer network and changes in students' attitudes and opinions about computer use in special education. There were no significant differences in attitudes toward computers between students with and without learning disabilities. (Contains 25 references.) (Author/CR)

Seven specific guidelines included in the taxonomy proposed by T. Haladyna and S. Downing (1998) for writing multiple-choice test items were evaluated. These specific guidelines are: (1) avoid the complex multiple-choice, K-type format; (2) state the stem in question format; (3) word the stem positively; (4) avoid the phrase "all of the above"; (5) avoid the phrase "none of the above"; (6) avoid specific determiners such as "always" or "never"; and (7) keep the length of options fairly consistent. These guidelines were evaluated by comparing statistical indices of item quality across items that do and do not violate one or more of these guidelines. The items and their statistics were taken from a recently administered, high-stakes, large-scale licensure examination, the Uniform Certified Public Accountant Examination. Only 1 of the 285 items evaluated violated the guidelines to avoid the phrases, "none of the above" and "all of the above"; and the determiners, "always" and "never." The only guideline supported by the data was avoiding the K-type item, since K-type items on this test tended to be more difficult and to have lower discrimination statistics. Results do not support the "state the stem in the question" guideline. Relatively few items violated more than one guideline. (Contains 8 tables and 10 references.) (SLD)

In many cross-cultural research studies, assessment instruments are translated or adapted for use in multiple languages. However, it cannot be assumed that different language versions of an assessment are equivalent across languages. A fundamental issue to be addressed is the comparability or equivalence of the construct measured by each language version of the assessment. This paper presents and critiques several methods for evaluating structural equivalence across different language versions of a test or questionnaire. Applications of these techniques to large-scale, cross-lingual tests are presented and discussed. Simulated data are also used to evaluate the methods. It is concluded that weighted multidimensional scaling and confirmatory factor analysis are effective for helping evaluate construct equivalence across groups. Qualifications for using these procedures to evaluate construct equivalence are provided. (Contains 2 figures, 6 tables, and 42 references.) (Author/SLD)

Multidimensional scaling (MDS) is a versatile technique for understanding the structure of multivariate data. Recent studies have applied MDS to the problem of evaluating content validity. This paper describes the importance of evaluating test content and the logic of using MDS to analyze data gathered from subject matter experts employed in content validation studies. Some recent applications of the procedure are reviewed, and illustrations of the results are presented. Suggestions for gathering content validity data and using MDS to analyze them are presented. (Contains 3 exhibits, 7 figures, and 24 references.) (Author/SLD)

The primary purpose of this study was to assess the effect of item bundling on multidimensional data. A second purpose was to compare three methods for assessing dimensionality. Eight multidimensional data sets consisting of 100 items and 1,000 examinees were simulated varying in terms of dimensionality, inter-dimensional correlation, and number of items loading on each dimension. Analyses were also performed on two samples of examinees from the November 1996 administration of the Uniform Certified Public Accountant examination. The items from both data sets were grouped into bundles that varied in size and content. Principal components factor analysis, maximum likelihood factor analysis, and multidimensional scaling were used to analyze the item bundles as well as the items themselves. Results suggest that item bundling tends to obscure multidimensionality, but analyses on the items themselves overestimate dimensionality. Multidimensional scaling also appeared better able to recover the underlying dimensionality of the data than the other two techniques. (Contains 13 tables and 17 references.) (Author/SLD)

The most popular methods for setting passing scores and other standards on educational tests rely heavily on subjective judgment. This paper presents and evaluates a new procedure for setting and evaluating standards on tests based on cluster analysis of test data. The clustering procedure was applied to a statewide mathematics proficiency test administered to 818 seventh-grade students in a small urban/suburban school district. Content area subscores were derived from the test specifications to serve as clustering variables. Subsequent course grades in mathematics were used to validate the cluster solutions and the stability of the solutions were evaluated using two random samples. The three-cluster (K-means) solution provided relatively homogeneous groupings of students that were consistent across the two samples and were congruent with school mathematics grades. Standards for "intervention,""proficient," and "excellent" levels of student performance were derived from these results. These standards were similar to those established by the local school district. The results suggest that cluster analytic techniques may be useful for helping set standards on educational tests, as well as for evaluating standards set by other methods. Suggestions for future research are provided. (Contains 2 figures, 7 tables, and 23 references.) (Author/SLD)

Evaluating the comparability of a test administered in different languages is a difficult, if not impossible, task. Comparisons are problematic because observed differences in test performance between groups who take different language versions of a test could be due to a difference in difficulty between the tests, to cultural differences in test taking behavior, or to a difference in proficiency between the language groups. The international certification testing programs conducted by Novell, Inc. are exceptional examples of the complex psychometric demands inherent in multiple language assessment programs. Novell's international certification program includes tests administered in 12 languages. Many of these tests are computerized adaptive (CAT), complicating comparisons across tests and languages. This paper reports the results of a study comparing English and German language versions of a high-stakes Novell CAT certification exam. The two versions of the test were compared by analyses including separate and concurrent item response theory calibrations. Results with 1,668 English-language candidates and 922 German-language candidates indicate that the English and German CATs are highly similar, and that the tests appear to be unidimensional in both the English and German versions. It is also concluded that the German candidate sample was more proficient than the English sample, and that 2 of 15 items functioned differentially across the languages. The source of the differential item functioning was identified post hoc using bilingual subject matter experts. The comparability of the passing scores, and other critical validity issues are discussed. (Contains 4 tables, 11 figures, and 20 references.) (Author/SLD)

Procedures for evaluating differential item functioning (DIF) are commonly used to investigate the statistical equivalence of items that are translated from one language to another. However, the methodology developed for detecting DIF is designed to evaluate the functioning of the same items administered to two groups. In evaluating the differential functioning of dual language versions of a test item (translation DIF), the items being compared (i.e., an original item and its translated version) are not the same. Thus, studies of translation DIF may not fulfill the requirements of currently available DIF detection procedures. This paper discusses the complex issues involved in evaluating translation DIF. An important, but often overlooked, issue is the dimensionality of the construct measured across the two languages. It is concluded that the dimensionality issues must be addressed first in studies of translation DIF. The development of an adequate research design is another important issue in studies of translation DIF. The design must be able to control for extraneous language proficiency effects. Some suggestions and examples of such research designs are proposed. (Contains 29 references.) (Author/SLD)

Test developers continue to struggle with the technical and logistical problems inherent in assessing achievement across different languages. Many testing programs offer separate language versions of a test to evaluate the achievement of examinees in different language groups. However, comparison of individuals who took different language versions of a test are not valid unless the score scales for the different versions are linked or equated. This paper discusses the psychometric problems involved in cross-lingual assessment, reviews linking models that have been proposed to enhance score comparability, and provides suggestions for developing and evaluating a model for linking different language versions of a test. Attempts to link different language versions of a test onto a common scale are classified into three general research design categories: (1) separate monolingual group designs, usually linked through item response theory; (2) bilingual group designs; and (3) matched monolingual group designs. (Contains 4 figures and 47 references.) (Author/SLD)