Search

A study was conducted to examine patterns of test preparation for the Law School Admission Test (LSAT) during the 1990-1991 testing year. This represented an extension of an earlier study that examined self-report test preparation methods for the June and October 1989 test administrations. In this study, five types of analysis were performed: (1) determining the response rate for each test date to see if there were differences across administrations; (2) comparing respondents and nonrespondents in terms of mean LSAT score and mean age; (3) evaluating the utilization rates for different ways of test preparation; (4) evaluating the extent to which test takers used multiple test preparation methods; and (5) comparing users and nonusers for each method in terms of mean LSAT score and mean age. Overall, patterns in the results of the comparison of respondents and nonrespondents were consistent across administrations. The mean LSAT was higher for respondents than nonrespondents, and the mean age was higher for nonrespondents. Results indicate that caution should be exercised in generalizing findings of this study to nonrespondents. Patterns of use among the various methods of test preparation appeared to remain relatively stable across administrations. The most popular method was use of the sample questions in the Law Services Information Book, followed closely by use of another book not published by the Law Services, and the use of the sample test in the Law Services Information Book. These three methods were selected by 45% to 50% of respondents. Commercial test preparation courses were selected by as few as 28% of the respondents in February and as many as 40% in October. LSAT scores were generally higher for respondents who used these methods, and lower for those who used undergraduate courses, self-study, other methods, or no methods. (Contains 17 tables and 1 reference.) (SLD)

An analysis of the skills necessary for performance on the Test of English as a Foreign Language (TOEFL) tends to support the view that there are important, although subtle, secondary dimensions present in the test. This research explored the feasibility of an item response theory (IRT) based method of modeling examinee performance on these secondary ability dimensions. Both exploratory multidimensional IRT (MIRT) and confirmatory multidimensional IRT (CMIRT) models were investigated in the study. The work performed included the application of unidimensional IRT, MIRT, and CMIRT models in two TOEFL forms to evaluate the extent to which model fit is enhanced by using a multidimensional model and to determine to what extent the additional fitted ability dimensions correspond to meaningful cognitive processes or content areas. Results indicate that the MIRT and CMIRT procedures were successful in modeling secondary ability dimensions on TOEFL and that they provide corroborative evidence in interpreting the structure of the test that is consistent with previous structure interpretations. The data also illustrate how the consistent Akaike information criterion can identify the best competing models of test structure. Four figures (plots) and seven tables illustrate the discussion. (Contains 34 references.) (SLD)

Two procedures were developed to determine whether examinees in a given test center were affected by a testing irregularity on the Listening Comprehension section of the Test of English as a Foreign Language (TOEFL). One approach employed analysis of covariance (ANCOVA) on Listening Comprehension (Section 1) means using scores on Structure and Written Expression (Section 2) and scores on Reading and Vocabulary (Section 3) as covariates. The second procedure entailed a Bayesian approach that used prior information about performance at the center in question. Analyses using these two procedures were carried out with simulated data and data from actual testing irregularities at 5 centers involving 639 affected examinees and numerous comparisons. Results indicated that both ANCOVA and Bayesian approaches provided useful information. They usually agreed, but differences occurred in specific situations as discussed. Although both procedures should be incorporated into the operational procedures for resolving testing irregularities, results should be interpreted with caution, particularly if they produce discrepant results. An appendix contains five tables of summary statistics. (Contains 1 figure, 10 tables, and 3 references.) (Author/SLD)

A study was conducted to evaluate six goodness-of-fit procedures using data simulation techniques. The procedures evaluated included the weighted average absolute deviation (WAAD), the mean absolute deviation (MAD), Bock's chi-square (BCHI), Yen's chi-square (YCHI), Wright and Mead's chi-square (WCHI), and the likelihood ratio chi-square (LCHI) statistics. These procedures were evaluated using data generated according to three different item response theory models and a factor analytic model. Three different distributions of ability and sample sizes were used. The results of this study yielded the following conclusions: (1) sample sizes of 500 to 1000 seemed to yield the best results; (2) the largest sample size (N=2000) seemed to make the fit procedures too sensitive; (3) shifts in the mean of the ability distribution caused minor fluctuations, but did not appear to be a major concern; (4) the chi-square statistics performed better than did the two non-chi-square statistics; and (5) the likelihood ratio chi-square procedure appeared to yield the best results. (Author/DWH)

A new procedure is proposed for developing, evaluating, and implementing routing procedures for use with individualized instruction programs. An item response theory model for describing the relationship between performance on sequentially arranged units of instruction, and procedures for using the model to evaluate sequential relationships and for making routing decisions are discussed. A procedure for estimating the parameters of the model is described. Empirical data are presented to support the validity of the model. The model and procedures described appear to be useful and to merit continued research efforts directed toward their development. (DWH)

A latent trait model is described that is appropriate for use with tests that measure more than one dimension, and its application to both real and simulated test data is demonstrated. Procedures for estimating the parameters of the model are presented. The research objectives are to determine whether the two-parameter logistic model more adequately explains multidimensional test data than does the unidimensional version of the model, and to determine whether the results yielded by the application of the model are consistent with the results of another, more established multivariate data reduction procedure, factor analysis. The results of the analyses of the simulation data indicated that the parameters of the multidimensional model could be accurately estimated. The results of the goodness of fit analyses indicated that the multidimensional model could more adequately model simulated multidimensional response data than could the unidimensional model. The results of the analyses of the real data indicated that the multidimensional model also more adequately modeled multidimensional real data than did the unidimensional model. It was concluded that the use of a multidimensional model does seem to be feasible, and that more research is warranted. (Author/PN)

Item response theory (IRT) has proven to be a very powerful and useful measurement tool. However, most of the IRT models that have been proposed, and all of the models commonly used, require the assumption of unidimensionality, which prevents their application to a wide range of tests. The few models that have been proposed for use with multidimensional data have not been developed to the point that they can be applied in actual testing situations. The purpose of this report is to present a model for use with multidimensional data and to discuss some of its characteristics. This discussion will include information on the interpretation of the model parameters, the sufficient statistics for the model parameters, and the information function for the model. In addition, the estimation of the parameters of the model using the maximum likelihood estimation technique is also discussed. (Author/PN)

A two-stage study was conducted to compare the ability estimates yielded by tailored testing procedures based on the one-parameter logistic (1PL) and three-parameter logistic (3PL) models. The first stage of the study employed real data, while the second stage employed simulated data. In the first stage, response data for 3,000 examinees were obtained for the 40 item ACT Assessment Mathematics Usage subtest. The first 2,000 cases were used to obtain item parameter estimates for both models. Using these estimates, 1PL and 3PL tailored tests were simulated using the response data for the remaining 1,000 cases. Both tailored testing procedures employed maximum likelihood ability estimation and maximum information item selection procedures. The two sets of ability estimates were then compared. In the second stage, response data for 3,000 cases were simulated using the 3PL item parameter estimates from the first stage as true parameters. True abilities were selected from the standard normal distribution. The first 2,000 cases were used for 1PL and 3PL calibration of the items, and the remaining 1,000 cases were used to simulate 1PL and 3PL tailored tests. The two sets of ability estimates were compared to each other and to the true ability parameters. Results of both stages of the study indicated that the 1PL and 3PL tailored tests yielded highly correlated ability estimates, and there was no apparent advantage in terms of ability estimation to using one of the models over the other. Because the 1PL procedure was less expensive to use, it was the recommended model for this application. (Author)

A live tailored achievement testing study was conducted to compare procedures based on the one- and three-parameter logistic models. Previous studies yielded inconclusive results because of the procedures by which item calibrations were linked and because of the item selection procedures. Using improved procedures, 83 college students were tested in a test-retest design. Comparisons were based upon (1) test-retest reliability; (2) ability estimates yielded by the procedures; (3) the information yielded by the procedures; (4) the number of items the methods administered; (5) goodness of fit of the models based on mean square deviations; and (6) the correlations of estimated true scores, based on ability estimates. In addition, an attitude survey was administered after each test session to determine student attitudes toward the tailored tests. Results indicated that both tailored tests had higher reliabilities than a conventional paper-and-pencil test over the same material. The three-parameter procedure had higher test information than the one-parameter procedure and the conventional test. The attitude survey results indicated generally favorable student attitudes toward tailored testing. (Author/CP)

A study was conducted to compare the quality of the item parameter estimates obtained from the ANCILLES and LOGIST estimation procedures using goodness of fit as a criterion. Statistics used to compare the fit included a chi-square statistic and a mean square deviation statistic. Other analyses performed included comparisons of the distributions of the parameter estimates obtained from the procedures, and a set of meta-analyses performed on the chi-square statistics obtained for the items. The data for the study were composed of 50 items and 2,000 cases obtained using a stratified random sample of 357 items and 4,000 cases of the Iowa Tests of Educational Development. Results indicated that there are qualitative differences in the estimates obtained from these two procedures. While the parameter estimate distributions obtained from these two procedures were similar, lack of fit occurred for significantly more items for ANCILLES than for LOGIST. Lack of fit for ANCILLES appeared to be stongly related to item difficulty, while for LOGIST it was related to item discrimination. Although LOGIST is more expensive to use than ANCILLES, ANCILLES yielded lack of fit significantly more often than LOGIST, and did not yield item parameter estimates for two items. (Author/RL)

A study was conducted to compare tailored testing procedures based on a Bayesian ability estimation technique and on a maximum likelihood ability estimation technique. The Bayesian tailored testing procedure selected items so as to minimize the posterior variance of the ability estimate distribution, while the maximum likelihood tailored testing procedure selected items so as to maximize the item information for the current ability estimate. Results of the analyses for the two procedures indicated that the optimal test length of Bayesian procedure was 14 items, while the optimal test length of the maximum likelihood procedure was 12 items. No difference was found between the procedures in terms of the reliability of the ability estimates. The Bayesian procedure yielded greater mean total test information than the maximum likelihood. The goodness of fit comparison indicated that the Bayesian procedure yielded poorer fit of the 3PL model to the data than did the maximum likelihood procedure. It was concluded that for large scale tailored testing, a maximum likelihood tailored testing procedure with item selection based on information is the procedure of choice. (Author/GK)

To assess the effects of correlated abilities on test characteristics, and to explore the effects of correlated abilities on the use of a multidimensional item response theory model which does not explicitly account for such a correlation, two tests were constructed. One had two relatively unidimensional subsets of items, the other had all two-dimensional items. For each test, response data were generated according to a multidimensional two-parameter logistic model using four groups of 2000 simulated examinees, differing in the degree of inter-dimension ability correlation. To evaluate the effects on observed test characteristics, the simulated response data were analyzed using item analysis and factor analysis techniques. To assess the effects on the use of the multidimensional model, the model parameters were estimated, and compared to the true parameters. Results of the study indicated that the presence of correlated abilities has important implications. It is necessary to consider latent item structure as well as latent ability structure in test construction and analysis. Use of multidimensional item response theory models that do not explicitly account for correlated abilities may result in misinterpretation of the underlying dimensions. Research is needed to determine the nature of the misinterpretation and to perhaps develop an item response theory analogue to factor rotation. (Author/BS)

Several special cases of the general Rasch model, varying in complexity, were investigated to determine whether they could successfully model realistic multidimensional item response data. Whether the parameters of the model could be readily interpreted was also investigated. The models investigated included: (1) the vector model; (2) the product term model; (3) the vector and product term model; (4) the reduced vector and product term model; and (5) the item cluster model. Of the models investigated, all but the reduced vector and product term model and the item cluster model were rejected as being incapable of modelling realistic multidimensional data. The item cluster model appears to be a useful model, but its applications may be limited in scope. Of the models studied, the reduced vector and product term model was found to be the most capable of modelling realistic multidimensional data. Although the estimation of the parameters of the reduced vectors and product term model may be more difficult than it would be for other models, this model appears to be the model that is most worth pursuing. (Author/PN)

A review of the literature was made of the various procedures for sequencing instructional units in individualized instruction programs. Procedures for validating sequences of instructional units, or learning hierarchies, were found to fall into two general categories. One category included procedures based on coefficients of dependence, while the other category contained procedures based on more complete descriptions of the relationships between units of instruction, usually a mathematical model. Procedures based on coefficients were found to provide insufficient information for optimal sequencing of units. Procedures based on mathematical models were also found to be inadequate, but the model-based approach was found to have potential for dealing with the problem of sequencing. Recommended future research emphasizes improving model-based procedures so that they provide more information useful for determining the optimal order of instructional units. (Author/PN)

This paper reviews the existing multidimensional item response theory (IRT) models and demonstrates how one of the models can be applied to estimation of abilities from a test measuring more than one dimension. The purposes of this paper were threefold. First, the fundamental concepts required when considering multidimensional models for the interaction of a person and a test item were defined. These concepts included the multidimensional latent space, the item difficulty function, and the item discrimination function. These definitions were conceived as multidimensional generalizations of similar concepts in unidimensional IRT models. Second, six existing multidimensional models were reviewed and, on the basis of their similarities, were classified into three general categories. The characteristics of these categories were described, and the general Rasch model was selected for further study on the basis of ease of parameter estimation. Third, estimation procedures for the parameters of the general Rasch model were described and applied to a set of simulation data that had been generated according to a two-dimensional special case of the model. The results indicated that a very close correspondence had been obtained between the estimated item parameters and those used to generate the simulation data. (PN)

A confirmatory approach to assessing test structure using multidimensional item response theory (MIRT) was developed and evaluated. The approach involved adding to the exponent of the MIRT model an item structure matrix that allows the user to specify the ability dimensions measured by an item. Various combinations of item structures were fit to two sets of simulation data with known true structures, and the results were evaluated using a likelihood ratio chi-square statistic and two information-based model selection criteria. The results of these analyses support the use of the confirmatory MIRT approach, since it was found that the procedures could recover true item structures. It was also found that adding an additional ability dimension that forces together items that ought not to be together noticeably deteriorates the quality of the solution. On the other hand, imposing structures different from, but not inconsistent with, the true structures does not necessarily yield worse fit. Finally, in terms of model fit statistics, the consistent Akaike information criterion performed better than the simple Akaike information criterion, while the likelihood ratio chi-square was clearly inadequate. (Contains 12 tables and 30 references.) (Author/SLD)

A study was conducted to evaluate the feasibility of using item response theory (IRT) equating to reduce test form overlap of the Graduate Record Examinations (GRE) Subject Test in Mathematics. Monte Carlo methods were employed to compare double-part equating with 20-item common item blocks to triple-part equating with 10-item common item blocks. The two methods were evaluated using a circular design that allowed a form to be equated to itself through a series of other forms. The design was replicated five times. Comparisons between scores on equated forms and scores on the base form indicated that triple-part equating did at least as well as double-part equating. This suggests that it may be reasonable to use IRT equating with the GRE Mathematics Test with smaller common item blocks than are used with the linear equating procedures currently employed, as long as there are more of them. This would result in a substantial reduction in the advantage given to repeat examinees and would significantly decrease the number of items on any one test form affected by compromised security. Items that need to be resolved before the procedure can be implemented are discussed. (Contains 4 figures, 3 tables, and 10 references.) (Author/SLD)