Search

This paper explores the application of a technique for hierarchical item response theory (IRT) calibration of complex constructed response tasks that has promise both as a calibration tool and as a means of evaluating the isomorphic equivalence of complex constructed response tasks. Isomorphic tasks are explicitly and rigorously designed to be highly similar in domain-relevant characteristics and evaluation standards. A related task model was used in which each item was modeled with a separate item response function, but the isomorphic tasks were related through a hierarchical model. The model was implemented in software that conducted Bayesian Markov Chain Monte Carlo (MCMC) estimation to estimate the joint posterior of all model parameters by integrating over the posterior distribution of model parameters given the data. The study analyzed operational data from a high-stakes assessment consisting of a number of complex constructed response tasks. The MCMC estimation procedure was conducted through 100,000 iterations. The item characteristic curves (ICCs) for the six isomorphic families were determined. In general, the families of isomorphic tasks showed considerable similarity in the item response functions for their respective members as well as for the family response function for the isomorphic set. Results suggest that efforts to construct complex constructed response tasks that are isomorphic equivalent tasks can range somewhat in their degree of success, with some being consistently equivalent, some being more variable, and others being largely consistent but with notable deviations. (SLD)

This study explored the application of hierarchical model calibration as a means of reducing, if not eliminating, the need for pretesting of automatically generated items from a common item model prior to operational use. Ultimately the successful development of automatic item generation (AIG) systems capable of producing items with highly similar statistical properties may permit the development and implementation of adaptive on-the-fly testing. The study applied the related siblings model to mathematics item data from an experimental administration associated with a national testing program to explore the application of the model for calibrating operational data incorporating multiple items generated both from AIG and manual item generation. The sample consisted of 3,793 examinees in grade 8, distributed among 4 test forms. Results suggest that the inclusion of AIG generated items in item families will have a tendency to have item characteristic curves that are somewhat more variable than if the family consisted of the same item under repeated administration. However, this increased variability is neither assured, nor in most cases, even particularly pronounced. (Contains 2 figures, 1 table, and 13 references.) (SLD)

As the automated scoring of constructed responses reaches operational status, monitoring the scoring process becomes a primary concern, particularly if automated scoring is intended to operate completely unassisted by humans. Using actual candidate selections from the Architectural Registration Examination (n=326), this study uses Kohonen Self-Organizing Maps (SOM) to build on previous research (D. Williamson, A. Hone, S. Miller, and I. Bejar, 1998) suggesting that classification trees are a useful means of validity maintenance. Classification trees can assist in identifying sources of disagreement between human and automated scoring, identify tendencies for human graders to overlook elementary or complex solutions, and provide significant efficiency in future case selection for human intervention. Since classification trees require a criterion value of score discrepancy between human and automated scores, Kohonen SOM provide an advantage in the ability to classify solutions in similar groups through neural networks without requiring prior human grading. Results suggest that Kohonen SOM could be used to classify solutions prior to human grading and classification tree analyses, thus providing a 43% reduction in the human grading required. However, further analyses are needed to establish whether classification trees would produce similar results with a reduced sample on the basis of Kohonen SOM classifications. (Contains 3 figures, 3 tables, and 14 references.) (Author/SLD)

As the automated scoring of constructed responses reaches operational status, the issue of monitoring the scoring process becomes a primary concern, particularly when the goal is to have automated scoring operate completely unassisted by humans. Using a vignette from the Architectural Registration Examination and data for 326 cases with both human and computer scores available, this study reports on the usefulness of an approach based on classification trees (L. Breiman, J. Friedman, R. Olshen, and C. Stone, 1984) as a means of quality control. Five studies were carried out analyzing different aspects of the "training set" and making efforts to cross-validate the results of the analysis by applying the resulting classification trees to data that had not been used in the development of the tree. The application of classification trees led to valuable insights with implications for operational quality control processes. Furthermore, classification tree methods were shown to be able to select cases for future quality control processes accurately and efficiently, thereby suggesting that future quality control selection procedures may be completely automated. However, further analyses are needed to establish whether classification trees can be relied on to identify cases that are the most likely to require some adjustment without incurring the potentially costly error of ignoring solutions that are likely to require adjustment. (Contains 10 tables, 7 figures, and 13 references.) (Author/SLD)

Generative response modeling is an approach to test development and response modeling that calls for the creation of items in such a way that the parameters of the items on some response model can be anticipated through knowledge of the psychological processes and knowledge required to respond to the item. That is, the computer would not merely retrieve an item from a database, as is the case in adaptive testing, but would compose it, or assist in doing so, according to the desired specifications. This approach to assessment has implications for both the economics and validity of computer administered tests. To illustrate the concept, a system for measuring writing skills is outlined in which the examinee is expected to rewrite sentences, rather than just recognize errors in a sentence, using a multiple-choice format. The possibility of estimating the psychometric parameters of items based on a psychological analysis of the response process is examined, and shown to be feasible. A Monte Carlo study with 100 simulated examinees at each of 6 ability levels is presented, which investigated the possibility of compensating for that imprecision when estimating ability or proficiency. It is concluded that a generative approach is feasible, and can be a mechanism for taking advantage of the considerable investment required for computer-based testing. (Contains 5 figures, 7 tables, and 59 references.) (Author/SLD)

Response generative modeling (RGM) is an approach to psychological measurement that involves a "grammar" capable of assigning a psychometric description to every item in a universe of items and is capable of generating all the items in that universe. The article discusses the rationale behind RMG and its roots, explores how it relates to validity, and assesses its feasibility in a wide variety of domains. A brief review of possible theoretical approaches to a psychologically sound approach to test construction and modeling concludes the discussion. RGM links item construction and response modeling in a single package, so that linkage (the predictions about response behavior) is challenged every time a test is administered. The administration of a test then becomes a psychological experiment, a fact that may, in turn, lead to the improvement of both theories and tests. One table and seven figures illustrate the discussion. (Contains 133 references.) (SLD)

The history of the Educational Testing Service (ETS) Factor Kits is summarized. The original ETS Factor Kit was developed in 1954 and contained 51 items, three each for each of 15 factors and six for a 16th factor. The next edition was developed in 1963 and included adaptations (clones) of the defining tests instead of the exact copies. These tests marked 24 factors. The current edition of the ETS Factor Kit was developed in 1976 and consists of 72 tests marking 23 cognitive factors. Some limitations of paper-and-pencil versions of the kits are identified, and computer-administered versions being developed are described. Information is given about a study comparing computer and paper-and-pencil tests. The Factor Kit tests were intended to be used as markers in factor-analytic studies of cognition and have been widely used in psychological research. Tests that could be used to determine a number of major factors were assembled in "kits" for factorial research. Limitations of the format restricted the kinds of cognitive processes that could be assessed and the ways in which tests could be scored. Questions of test misuse arose. Creating computer-administered versions posed a number of problems in the areas of timing, confirmation and correction of responses, and pacing. For the computer administered versions, system features are described. A small pilot study compared the two formats using data for 30 secondary school students aged 13 to 19 years, who took part of each kit of 10 tests in each format. Results suggest that the factors measured by these 10 tests were not affected by the use of the computer version. Versions of the computer-administered kit for field testing are anticipated in 1992. Three tables provide details about the 1954, 1963, and 1976 editions of the Factor Kits. A 17-item list of references is included. (SLD)

As automated scoring of complex constructed-response examinations reaches operational status, the process of evaluating the quality of resultant scores, particularly in contrast to scores of expert human graders, becomes as complex as the data itself. Using a vignette from the Architectural Registration Examination (ARE), this article explores the potential utility of Classification and Regression Trees (CART) and Kohonen Self-Organizing Maps (SOM) as tools to facilitate subject matter expert (SME) examination of the fine-grained (feature level) quality of automated scores for complex data, with implications for the validity of resultant scores. This article explores both supervised and unsupervised learning techniques, with the former being represented by CART (Breiman, Friedman, Olshen, & Stone, 1984) and the latter by SOM (Kohonen, 1989). Three applications comprise this investigation, the first of which suggests that CART can facilitate efficient and economical identification of specific elements of complex responses that contribute to automated and human score discrepancies. The second application builds on the first by exploring CART for efficiently and accurately automating case selection for human intervention to ensure score validity. The final application explores the potential for SOM to reduce the need for SMEs in evaluating automated scoring. Although both supervised and unsupervised methodologies examined were found to be promising tools for facilitating SME roles in maintaining and improving the quality of automated scoring, such applications remain unproven, and further studies are necessary to establish the reliability of these techniques.

Information provided by typical and improved conventional classroom achievement tests was compared with information provided by an adaptive test covering the same subject matter. Both tests were administered to over 700 college students in a general biology course. Using the same scoring method, adaptive testing was found to yield substantially more precise estimates of achievement level than the classroom test throughout the entire range of achievment, even in the range where the improved conventional test was designed to be optimal. Adaptive testing also made it possible to reduce the length of the test. An analysis of the effects of expanding an adaptive test item pool indicated that improved precision of measurement could result from the addition of only slightly more discriminating items. A comparison of response pattern information values (observed information) with test information values (theoretical information) showed that the observed information consistently underestimated theoretical information, although the pattern of results from the two procedures was quite similar. It was concluded that the adaptive measurement of classroom achievement results in scores which are less likely to be confounded by errors of measurement and, therefore, are more likely to reflect the true level of achievement. (Author/MV)

The applicability of item characteristic curve (ICC) theory to a multiple choice test item pool used to measure achievement is described. The rationale for attempting to use ICC theory in an achievement framework is summarized, and the adequacy for adaptive testing of a classroom achievement test item pool in a college biology class is studied. Using criteria usually applied to ability measurement item pools, the item difficulties and discriminations in this achievement test pool were found to be similar to those used in adaptive testing pools for ability testing. Studies of the dimensionality of the pool indicate that it is primarily unidimensional. Analysis of the item parameters of items administered to two different samples reveals the possibility of a deviation from invariance in the discrimination parameter but a high degree of invariance for the difficulty parameter. The pool as a whole, as well as two subpools, are judged to be adequate for use in adaptive testing. It is also concluded that the ICC model is not inappropriate for application to typical college classroom achievement tests similar to the one studied. (Author/MV)

This study was conducted to establish the existence of grade inflation and show that the ability of college students has remained constant. The second purpose was to examine the effect of grade inflation on the validity of the Scholastic Aptitude Test (SAT). Two types of analyses were performed to gather data. A longitudinal analysis of selected characterisitcs of SAT scores and grade point average (GPA) over a fifteen year period was conducted. The second analysis focuses on selected schools to evaluate the effect of grade inflation on the validity of the SAT in those colleges. The conclusions reached were increases in GPA were due to grade inflation since there was no concomitant increase in SAT scores. The rate of grade inflation seems to have diminished since 1974 as indicated by levelling-out of GPA means. If grade inflation is seen as the addition of a constant to grades, it will not affect validity since this would not affect the correlation of GPA and SAT. In the presence of declining validity of High School Record, the SAT has become a more valuable tool for predicting academic success in college. (Author/DWH)

The study reports a feasibility study for using Item Response Theory (IRT) as a means of equating the Test of Standard Written English (TSWE). The study focused on the possibility of pre-equating, that is, deriving the equating transformation prior to the final administration of the test. The three-parameter logistic model was postulated as the response model and its fit assessed at the item, subscore, and total score level. Minor problems were found at each of these levels but, on the whole, the three-parameter model was found to portray the data well. The adequacy of the equating provided by IRT procedures was investigated in two TSWE forms. It was concluded that pre-equating does not appear to present problems beyond those inherent to IRT-equating. (Author)

The Test of English as a Foreign Language (TOEFL) was used in this study, which attempted to develop a new methodology for assessing the speededness of right-scored tests. Traditional procedures of assessing speededness have assumed that the test is scored under formula-scoring instructions; this approach is not always appropriate. In this study, the approach used capitalized on the use of item response theory in equating TOEFL. Two indices were investigated and validated against artificial data and real data. Both indices were able to identity data that contained the speed component. The application of these indices to several test forms suggested that speed did not play a significant role in TOEFL as a whole. However, the second section appeared to be affected by speed in administrations which included pretest items, resulting in a slightly shorter time allowed per item. It was recommended that the item-based index be used on an operational basis for a trial period. It was also concluded that the time limit for Section 2 be increased when new items are being pretested, or that the number of items be reduced accordingly. (GDC)

The feasibility of reducing scoring costs for the Test of Spoken English (TSE) by using one rater was investigated. Currently, two raters are used. It was found that, because of the possibility of different standards used by potential raters, it does not appear feasible to use a single rater as the sole determiner of speaking proficiency under the current system. Other possible alternatives were also examined. One approach was the development of a quality control index which would predict the extent of the disagreement between two raters, but the index that was developed could not be validated. The best predictors of rater disagreement were the identities of the raters. Their disagreements, however, resulted from the differing standards they used. Raters agreed substantially about the ordering of examinees, but varied slightly in the severity of their ratings. (Author/GDC)

This paper considers the feasibility of incorporating research results from cognitive science into the modeling of performance on psychometric tests and the construction of test items. The paper focuses on the feasibility of modeling performance on a three-dimensional rotation task within the context of Item Response Theory (IRT). To test the feasibility of psychometrically modeling performance on this item type, an 80-item, three-dimensional rotation test was constructed using eight basic Shepard-Metzler figures. An inexpensive computer system was also developed to administer the test and record performance, including response-time data. Data were collected on high school juniors and seniors. As expected, angular disparity was a potent determinant of item difficulty. The applicability of IRT to these data was investigated by dichotomizing response time at several points and applying standard item parameter estimation procedures. It was concluded that an approach to psychometric modeling that explicitly incorporates information on the mental models that examiners use in solving an item is not only workable, but also essential for future developments in psychometrics. (Author/LPG)

The purpose of this report is to distill curricular and testing implications of brain research. The report will focus on three topics. One topic is the possibility that brain lateralization--that is, the degree to which the two brain hemispheres specialize in different types of information processing--is an individual differences variable that should be measured, just as we measure individual differences in verbal and mathematical aptitudes. Second, since one of the better-established findings is that verbal and spatial abilities tend to reside in different hemispheres, the possibility of incorporating spatial ability into admissions testing is explored. Third, the calls for curricular reform based on brain research are examined. (Author)

A review of the implications for the validity of the Scholastic Aptitude Test (SAT) of scientific developments accompanying the revival of cognitive psychology provides insights into the importance of such changes. A distinction can be made between a process-oriented or diagnostic test and an outcomes-oriented test such as the SAT. Since the SAT does not aim to be a diagnostic test, the implications of cognitive theories for test construction that would otherwise be applicable are not emphasized. Nevertheless, an analysis based on quantitative items illustrates how tests with a diagnostic orientation could be based on cognitive principles. An explication was developed for the linkage of cognitive principles with psychometric considerations for an outcomes-oriented test. The essence of that linkage is an accounting of item difficulty. The understanding of differential difficulty among items can be viewed as an additional requirement for construct validity, which until recently, focused almost exclusively on an accounting of the co-variation of test scores in terms of "abilities." Although work from this enlarged validational perspective on SAT items is limited, there is work based on similar items that suggests that the SAT will fare well when relevant studies are conducted. Future studies should be based on an enlarged validational framework. A nine-page list of references is included. (TJH)

This report summarizes the results of research designed to study the psychometric and technological feasibility of adaptive testing to assess spatial ability. Data was collected from high school students on two types of spatial items: three-dimensional cubes and hidden figure items. The analysis of the three-dimensional cubes focused on the fit of the simplest possible item response model capable of modeling response time; the analysis of the hidden figure item focused on the feasibility of generating items from an algorithm in such a way that the psychometric characteristics of the generated items were predictable. The results for the three-dimensional cube items suggested that angular disparity can be used effectively to control the difficulty of true items, but this was not the case for false items. That is, true and false items appear to measure different aspects of performance, and as a result, a multidimensional item response model may be necessary to fully account for performance. The analysis of the hidden figure item 3 showed that an item generation algorithm can be formulated to produce items of similar psychometric characteristics. The practical and theoretical implication of the results are discussed. (Author/JAZ)

In this study, the ability of test development staff to predict the difficulty of analogy items was explored. The nature of the item attributes that contributed to test writers' predictions of difficulty as well as actual item difficulty was also investigated. The two expert test writers studied were quite good at predicting item difficulty. Item attributes such as vocabulary difficulty and rationale difficulty contributed to item difficulty. However, a statistical model of item difficulty did not capture all the information that test writers used to judge item difficulty. This research contributes to the construct validation of tests in two ways. First, identification of some item attributes that are associated with item difficulty clarifies what skills and processes are likely to be involved in solving analogies. Second, the expertise of test writers, a crucial ingredient in ensuring the validity of the test, is demonstrated. (Contains 2 figures, 6 tables, and 13 references.) (Author/SLD)

Argues that synergy between computer-based instruction and automated assessment is possible because of the common needs in assessment and instruction; outlines a framework for characterizing performance; and examines procedures developed as part of an ongoing project to develop fully automated scoring of architectural design for a licensing exam. (Contains 38 references.) (JLB)