Search

The contention is made that group performance data are useful in the construction and interpretation of criterion-referenced tests. The Mastery Learning Test Model, which was developed for analyzing criterion-referenced test data, is described. An estimate of the proportion of students in an instructional group having achieved the referent objectives is usable as a prior probability in interpreting individual responses. Considering instructional group performance enhances estimates of individual performance. Correlational data from a set of test items and a representative population of students are used to estimate the required item parameters. (Author)

This paper examines the effect of using unidimensional item response theory (IRT) item parameter estimates of multidimensional items to create weakly parallel test forms using target information curves. To date, all computer-based algorithms that have been devised to create parallel test forms assume that the items are unidimensional. This paper focuses on one such algorithm, which was developed by R. M. Luecht and T. M. Hirsch. Unidimensional item parameter estimates were obtained by calibrating response data generated from two-dimensional item parameters. Using these unidimensional estimates, three sets of test items from a pool of 200 multidimensional items were selected for each of two different test lengths for three differently shaped target information functions. The item parameter estimates were obtained by calibrating five forms of the EAAP Mathematics usage test using the multidimensional IRT calibration program NOHARM. Response data were generated for 2,000 abilities. Observed score differences for each triad, based on the multidimensional item parameters, were then compared. Despite the multidimensionality of the selected items, the created forms appear to be quite parallel both unidimensionally and multidimensionally. Two tables and seven figures are included. (SLD)

This discussion of new methods for calibrating item response theory (IRT) models looks into new optimization procedures, such as the Genetic Algorithm (GA) to improve on the use of the Newton-Raphson procedure. The advantages of using a global optimization procedure like GA is that this kind of procedure is not easily affected by local optima and saddle points. Because these procedures do not use gradient information, they can be implemented easily to higher dimensional data, even though they converge more slowly than the Newton-Raphson approach. However, the two approaches can be combined to exploit the advantages of both. That is, GA can be used to find a suitable starting point close to the global optima, and then Newton-Raphson can be used to speed up the convergence. The focus in this paper is on calibrating the unidimensional three-parameter logistic model (3PL) because that is the model most widely used in large-scale standardized tests. Using recent 3PL model estimates from recent Test of English as a Foreign Language administrations to generate examinee responses, the effectiveness of the new method is demonstrated using simulated data. How to implement the new methods with multidimensional data is discussed. (Contains 3 tables, 2 figures, and 10 references.) (SLD)

This paper presents a detailed description of maximum parameter estimation for item response models using the general EM algorithm. In this paper the models are specified using a univariate discrete latent ability variable. When the latent ability variable is discrete the distribution of the observed item responses is a finite mixture, and the EM algorithm for finite mixtures can be used. Maximum likelihood estimates of the item parameters and of the discrete probabilities of the latent ability distribution are given using the EM algorithm for finite mixtures. Results are presented in general for both dichotomous and polytomous item response models. The relation between the EM estimates and the Bock Aitken marginal maximum likelihood estimates is discussed. Estimates for the item parameters will depend on the specific form of the item response functions, and will usually require iterative numerical procedures. The EM algorithm is the same as the Bock-Aitken algorithm (R. D. Bock and M. Aitken, 1981) for marginal maximum likelihood estimation of the item parameters. (Contains 28 references.) (Author/SLD)

A statistical model for investigating predictive validity at highly selective institutions is described. When the selection ratio is small, one must typically deal with a data set containing relatively large amounts of missing data on both criterion and predictor variables. Standard statistical approaches are based on the strong assumption that the missing data are missing at random (MAR) (i.e., the missing data can be accounted for in terms of the observed measures), and there are no unmeasured variables that underlie the missing data process. The proposed model represents an attempt to account for any unmeasured selection variables by assuming that applicants are first placed into admission categories by the institution and then selected within each category in terms of the observed predictor variables. Thus, although the MAR assumption may not hold for the set of all applicants, it may very well hold within each admission category. The model uses the EM algorithm to obtain estimates of validity separately within each category. The model is quite general and can be used when there are missing data on the predictor and criterion variables, and even if the admission category is not known for each applicant. The proposed model is illustrated in terms of a real life data set for a selective secondary school with over 2,000 applicants. Four tables present analysis data. (Author/SLD)

In this paper, algorithms are described for obtaining the maximum likelihood estimates of the parameters in log-linear models. Modified versions of the iterative proportional fitting and Newton-Raphson algorithms are described that work on the minimal sufficient statistics rather than on the usual counts in the full contingency table. This is desirable if the contingency table becomes too large to store. Special attention is given to log-linear Item Response Theory (IRT) models that are used for the analysis of educational and psychological test data. To calculate the necessary expected sufficient statistics and other marginal sums of the table, a method is described that avoids summing large numbers of elementary cell frequencies by writing them out in terms of multiplicative model parameters and applying the distributive law of multiplication over summation. These algorithms are used in the computer program LOGIMO, and are illustrated with simulated data for 10,000 cases. Two tables, 3 graphs, and a 34-item list of references are included. (Author/SLD)

Diagnosing cognitive errors possessed by examinees can be considered as a pattern classification problem that is designed to classify a sequential input of stimuli into one of several predetermined groups. The sequential inputs in this paper's context are item responses, and the predetermined groups are various states of knowledge resulting from misconceptions or different degrees of incomplete knowledge in a domain. In this study, the foundations of a combinatorial algorithm that will provide the universal set of states of knowledge will be introduced. Each state of knowledge is represented by a list of "can/cannot" cognitive tasks and processes (cognitively relevant attributes or latent variables) that are usually unobservable. A Boolean descriptive function is introduced as a mapping between the attribute space spanned by latent attribute variables and the item response space spanned by the item score variables. This function uncovers the unobservable content of a "black box." Once all possible classes are retrieved explicitly and expressed by a set of ideal item response patterns described by a "can/cannot" list of latent attributes, the notion of bug distributions and statistical pattern classification techniques will enable the accurate diagnosis of students' states of knowledge. Moreover, investigations on algebraic properties of these logically-derived-ideal-response patterns will provide insight into the structures of the test and dataset. There are 11 references and three illustrative tables. (Author/SLD)

The probability integral of the multivariate normal distribution (ND) has received considerable attention since W. F. Sheppard's (1900) and K. Pearson's (1901) seminal work on the bivariate ND. This paper evaluates the formula that represents the "n x n" correlation matrix of the "chi(sub i)" and the standardized multivariate normal density function. C. W. Dunnett and M. Sobel's formula for the univariate ND function, and R. E. Bohrer and M. J. Schervish's error-bounded algorithm for evaluating "F(sub n)" for general "rho(sub ij)" are discussed. Computationally, the latter algorithm is restricted to "n = 7"; even at "n = 7", it can take up to 24 hours for it to compute a single probability with 10(sup -3) accuracy on a computer than is capable of about 1-2 million scalar floating point operations/second. This report presents a fast and general approximation (APX) for rectangular regions of the multivariate ND function based on C. E. Clark's (1961) APX to the moments of the maximum of "n" jointly normal random variables. The performance of this APX compared to special cases in which the exact results are known and error-bounded reduction formulae show that the APX's accuracy is adequate for many practical applications where multivariate normal probabilities are required. The computational speed of the Clark APX is unparalleled. The error bound for the APX is about 10(sup -3) regardless of dimensionality, and accuracy increases with increases in "rho". The Clark algorithm provides a generalization of Dunnett's (1955) results to the case of general "rho(sub ij)", a natural application of which would be a generalization of Dunnett's test to the case of unequal sample sizes among the "k + 1" groups (i.e., multiple treatment groups compared to a single control group). One data table is included. (RLC)

Four discussions of test construction based on item response theory (IRT) are presented. The first discussion, "Test Design as Model Building in Mathematical Programming" (T. J. J. M. Theunissen), presents test design as a decision process under certainty. A natural way of modeling this process leads to mathematical programming. General models of test construction are discussed, with information about algorithms and heuristics; ideas about the analysis and refinement of test constraints are also considered. The second paper, "Methods for Simultaneous Test Construction" (Ellen Boekkooi-Timminga), gives an overview of simultaneous test construction using zero-one programming. The item selection process is based on IRT. Some objective functions and practical constraints are presented, the construction of parallel tests is considered, and two tables are provided. The third paper, "Automated Test Construction Using Minimax Programming" (Wim J. van der Linden), proposes the use of the minimax principle in IRT test construction and indicates how this results in test information functions deviating less systematically from the target function than for the usual criterion of minimal test length. An alternative approach and some practical constraints are considered. The final paper, "A Procedure To Assess Target Information Functions" (Henk Kelderman), discusses the concept of an information function and its properties. An interpretable function of information is chosen: the probability of a wrong order of the ability estimates of two subjects. (SLD)

This paper is a theoretical discussion of several functions of knowledge systems based on the idea of partial matching, that is, comparison of two or more descriptions by identification of their similarities. Several knowledge system functions are described in terms of partial or best matchings including analogical reasoning, inductive inference, predicate discovery, pattern-directed inference, semantic interpretation, and speech and image understanding. It is difficult to determine admissible algorithms for these functions; economical solutions seem to be possible only for globally organized knowledge bases. Examples are provided and directions for research are discussed. (Author/SD)

A method of automatically selecting items for inclusion in a test with constraints on item content and statistical properties was applied to real data. Tests constructed manually from the same data and constraints were compared to tests constructed automatically. Results show areas in which automated assembly can improve test construction. (SLD)