Search

The contention is made that group performance data are useful in the construction and interpretation of criterion-referenced tests. The Mastery Learning Test Model, which was developed for analyzing criterion-referenced test data, is described. An estimate of the proportion of students in an instructional group having achieved the referent objectives is usable as a prior probability in interpreting individual responses. Considering instructional group performance enhances estimates of individual performance. Correlational data from a set of test items and a representative population of students are used to estimate the required item parameters. (Author)

This paper examines the effect of using unidimensional item response theory (IRT) item parameter estimates of multidimensional items to create weakly parallel test forms using target information curves. To date, all computer-based algorithms that have been devised to create parallel test forms assume that the items are unidimensional. This paper focuses on one such algorithm, which was developed by R. M. Luecht and T. M. Hirsch. Unidimensional item parameter estimates were obtained by calibrating response data generated from two-dimensional item parameters. Using these unidimensional estimates, three sets of test items from a pool of 200 multidimensional items were selected for each of two different test lengths for three differently shaped target information functions. The item parameter estimates were obtained by calibrating five forms of the EAAP Mathematics usage test using the multidimensional IRT calibration program NOHARM. Response data were generated for 2,000 abilities. Observed score differences for each triad, based on the multidimensional item parameters, were then compared. Despite the multidimensionality of the selected items, the created forms appear to be quite parallel both unidimensionally and multidimensionally. Two tables and seven figures are included. (SLD)

The reliability of four branching algorithms commonly used in computer adaptive testing (CAT) was examined. These algorithms were: (1) maximum likelihood (MLE); (2) Bayesian; (3) modal Bayesian; and (4) crossover. Sixty-eight undergraduate college students were randomly assigned to one of the four conditions using the HyperCard-based CAT program, HyperCAT. As a way to control for order effects, half of the students were randomly assigned to take the paper-and-pencil test first, followed 3 weeks later by the CAT, while the other half took the CAT first. Investigative analyses showed no initial group differences by algorithm for the paper-and-pencil test and for CAT-estimated ability. In addition, there was no order effect. The internal consistency coefficient for the paper-and-pencil test was 0.73. The marginal reliability for the CAT was 0.97. Correlations between the paper-and-pencil scores and theta estimates of ability ranged from 0.48 to 0.79. Reliability was highest for the MLE algorithm, followed by the Bayesian, modal Bayesian, and crossover algorithms, respectively. Given the constraints of MLE branching algorithms (e.g., the examinee must get at least one item correct and one item incorrect), and the alleged biasedness associated with Bayesian branching strategies, the results suggest that the modal Bayesian testing may provide an acceptable alternative. Six tables present study data. Three figures and 14 references are included. (Author/SLD)

This discussion of new methods for calibrating item response theory (IRT) models looks into new optimization procedures, such as the Genetic Algorithm (GA) to improve on the use of the Newton-Raphson procedure. The advantages of using a global optimization procedure like GA is that this kind of procedure is not easily affected by local optima and saddle points. Because these procedures do not use gradient information, they can be implemented easily to higher dimensional data, even though they converge more slowly than the Newton-Raphson approach. However, the two approaches can be combined to exploit the advantages of both. That is, GA can be used to find a suitable starting point close to the global optima, and then Newton-Raphson can be used to speed up the convergence. The focus in this paper is on calibrating the unidimensional three-parameter logistic model (3PL) because that is the model most widely used in large-scale standardized tests. Using recent 3PL model estimates from recent Test of English as a Foreign Language administrations to generate examinee responses, the effectiveness of the new method is demonstrated using simulated data. How to implement the new methods with multidimensional data is discussed. (Contains 3 tables, 2 figures, and 10 references.) (SLD)

This paper presents a detailed description of maximum parameter estimation for item response models using the general EM algorithm. In this paper the models are specified using a univariate discrete latent ability variable. When the latent ability variable is discrete the distribution of the observed item responses is a finite mixture, and the EM algorithm for finite mixtures can be used. Maximum likelihood estimates of the item parameters and of the discrete probabilities of the latent ability distribution are given using the EM algorithm for finite mixtures. Results are presented in general for both dichotomous and polytomous item response models. The relation between the EM estimates and the Bock Aitken marginal maximum likelihood estimates is discussed. Estimates for the item parameters will depend on the specific form of the item response functions, and will usually require iterative numerical procedures. The EM algorithm is the same as the Bock-Aitken algorithm (R. D. Bock and M. Aitken, 1981) for marginal maximum likelihood estimation of the item parameters. (Contains 28 references.) (Author/SLD)

A statistical model for investigating predictive validity at highly selective institutions is described. When the selection ratio is small, one must typically deal with a data set containing relatively large amounts of missing data on both criterion and predictor variables. Standard statistical approaches are based on the strong assumption that the missing data are missing at random (MAR) (i.e., the missing data can be accounted for in terms of the observed measures), and there are no unmeasured variables that underlie the missing data process. The proposed model represents an attempt to account for any unmeasured selection variables by assuming that applicants are first placed into admission categories by the institution and then selected within each category in terms of the observed predictor variables. Thus, although the MAR assumption may not hold for the set of all applicants, it may very well hold within each admission category. The model uses the EM algorithm to obtain estimates of validity separately within each category. The model is quite general and can be used when there are missing data on the predictor and criterion variables, and even if the admission category is not known for each applicant. The proposed model is illustrated in terms of a real life data set for a selective secondary school with over 2,000 applicants. Four tables present analysis data. (Author/SLD)

In this paper, algorithms are described for obtaining the maximum likelihood estimates of the parameters in log-linear models. Modified versions of the iterative proportional fitting and Newton-Raphson algorithms are described that work on the minimal sufficient statistics rather than on the usual counts in the full contingency table. This is desirable if the contingency table becomes too large to store. Special attention is given to log-linear Item Response Theory (IRT) models that are used for the analysis of educational and psychological test data. To calculate the necessary expected sufficient statistics and other marginal sums of the table, a method is described that avoids summing large numbers of elementary cell frequencies by writing them out in terms of multiplicative model parameters and applying the distributive law of multiplication over summation. These algorithms are used in the computer program LOGIMO, and are illustrated with simulated data for 10,000 cases. Two tables, 3 graphs, and a 34-item list of references are included. (Author/SLD)

Diagnosing cognitive errors possessed by examinees can be considered as a pattern classification problem that is designed to classify a sequential input of stimuli into one of several predetermined groups. The sequential inputs in this paper's context are item responses, and the predetermined groups are various states of knowledge resulting from misconceptions or different degrees of incomplete knowledge in a domain. In this study, the foundations of a combinatorial algorithm that will provide the universal set of states of knowledge will be introduced. Each state of knowledge is represented by a list of "can/cannot" cognitive tasks and processes (cognitively relevant attributes or latent variables) that are usually unobservable. A Boolean descriptive function is introduced as a mapping between the attribute space spanned by latent attribute variables and the item response space spanned by the item score variables. This function uncovers the unobservable content of a "black box." Once all possible classes are retrieved explicitly and expressed by a set of ideal item response patterns described by a "can/cannot" list of latent attributes, the notion of bug distributions and statistical pattern classification techniques will enable the accurate diagnosis of students' states of knowledge. Moreover, investigations on algebraic properties of these logically-derived-ideal-response patterns will provide insight into the structures of the test and dataset. There are 11 references and three illustrative tables. (Author/SLD)

The probability integral of the multivariate normal distribution (ND) has received considerable attention since W. F. Sheppard's (1900) and K. Pearson's (1901) seminal work on the bivariate ND. This paper evaluates the formula that represents the "n x n" correlation matrix of the "chi(sub i)" and the standardized multivariate normal density function. C. W. Dunnett and M. Sobel's formula for the univariate ND function, and R. E. Bohrer and M. J. Schervish's error-bounded algorithm for evaluating "F(sub n)" for general "rho(sub ij)" are discussed. Computationally, the latter algorithm is restricted to "n = 7"; even at "n = 7", it can take up to 24 hours for it to compute a single probability with 10(sup -3) accuracy on a computer than is capable of about 1-2 million scalar floating point operations/second. This report presents a fast and general approximation (APX) for rectangular regions of the multivariate ND function based on C. E. Clark's (1961) APX to the moments of the maximum of "n" jointly normal random variables. The performance of this APX compared to special cases in which the exact results are known and error-bounded reduction formulae show that the APX's accuracy is adequate for many practical applications where multivariate normal probabilities are required. The computational speed of the Clark APX is unparalleled. The error bound for the APX is about 10(sup -3) regardless of dimensionality, and accuracy increases with increases in "rho". The Clark algorithm provides a generalization of Dunnett's (1955) results to the case of general "rho(sub ij)", a natural application of which would be a generalization of Dunnett's test to the case of unequal sample sizes among the "k + 1" groups (i.e., multiple treatment groups compared to a single control group). One data table is included. (RLC)

Four discussions of test construction based on item response theory (IRT) are presented. The first discussion, "Test Design as Model Building in Mathematical Programming" (T. J. J. M. Theunissen), presents test design as a decision process under certainty. A natural way of modeling this process leads to mathematical programming. General models of test construction are discussed, with information about algorithms and heuristics; ideas about the analysis and refinement of test constraints are also considered. The second paper, "Methods for Simultaneous Test Construction" (Ellen Boekkooi-Timminga), gives an overview of simultaneous test construction using zero-one programming. The item selection process is based on IRT. Some objective functions and practical constraints are presented, the construction of parallel tests is considered, and two tables are provided. The third paper, "Automated Test Construction Using Minimax Programming" (Wim J. van der Linden), proposes the use of the minimax principle in IRT test construction and indicates how this results in test information functions deviating less systematically from the target function than for the usual criterion of minimal test length. An alternative approach and some practical constraints are considered. The final paper, "A Procedure To Assess Target Information Functions" (Henk Kelderman), discusses the concept of an information function and its properties. An interpretable function of information is chosen: the probability of a wrong order of the ability estimates of two subjects. (SLD)

Analyzes the properties of genetic sequences that cause the failure of classical algorithms used for data compression. A lossless algorithm, which compresses the information contained in DNA and RNA sequences by detecting regularities such as palindromes, is presented. This algorithm combines substitutional and statistical methods and appears to give the highest compression of DNA. (19 references) (KRN)

Presents an inference algorithm that produces a weighted finite automata (WFA), in particular, the grayness functions of graytone images. Image-data compression results based on the new inference algorithm produces a WFA with a relatively small number of edges. Image-data compression results alone and in combination with wavelets are discussed. (nine references) (KRN)

Presents two new algorithms for performing arithmetic coding without employing multiplication and discusses their implementation requirements. The first algorithm, suitable for an alphabet of arbitrary size, reduces the worst case excess length to under 0.8%. The second algorithm, suitable only for alphabets of less than 12 symbols, allows even greater reduction. (nine references) (KRN)

Presents a new image compression algorithm that employs some of the most successful approaches to adaptive lossless compression to perform adaptive online (single pass) vector quantization with variable size codebook entries. Results of tests of the algorithm's effectiveness on standard test images are given. (12 references) (KRN)

Describes a detailed algorithm for fast text compression. Related to the PPM (prediction by partial matching) method, it simplifies the modeling phase by eliminating the escape mechanism and speeds up coding by using a combination of quasi-arithmetic coding and Rice coding. Details of the use of quasi-arithmetic code tables are given, and their compression performance is analyzed. (11 references) (Author/KRN)

A method of automatically selecting items for inclusion in a test with constraints on item content and statistical properties was applied to real data. Tests constructed manually from the same data and constraints were compared to tests constructed automatically. Results show areas in which automated assembly can improve test construction. (SLD)

This simulation study involved the design of two two-stage tests in which the routing tests and the second-stage measurement testlets took the form of a multidimensional knapsack problem with prespecified target informations and constraints to be enumerated using the algorithm of E. Balas. Two conventional tests of similar length to the two-stage tests were developed by random selection of items from the same item pool. Item responses of 1,000 simulees on 240 items were generated using the computer program DATAGEN based on the three-parameter item response model. Analysis of the degree in which the true abilities were recovered showed that the inaccuracies and root square mean errors for the two-stage designs were lower than those for their conventional counterparts. Fidelity coefficients were also higher and compared well with the results of previous adaptive test procedures. Three tables and three graphs accompany the discussion. (Author/SLD)

The graded response model for two-stage testing was applied to an attitudes toward science scale using real-data simulation. The 48-item scale was administered to 920 students at a grade-8 equivalent in Singapore. A two-stage 16-item computerized adaptive test was developed. In two-stage testing an initial, or routing, test is followed by a second-stage testlet of greater or lesser difficulty based on performance. A conventional test of the same length as the adaptive two-stage test was selected from the 48-item pool. Responses to the conventional test, the routing test, and a testlet were simulated. The algorithm of E. Balas (1965) and the multidimensional knapsack problem of optimization theory were used in test development. The simulation showed the efficiency and accuracy of the two-stage test with the graded response model in estimating attitude trait levels, as evidenced by better results from the two-stage test than its conventional counterpart and the reduction to one-third of the length of the original measure. Six tables and three graphs are included. (SLD)

This paper is a theoretical discussion of several functions of knowledge systems based on the idea of partial matching, that is, comparison of two or more descriptions by identification of their similarities. Several knowledge system functions are described in terms of partial or best matchings including analogical reasoning, inductive inference, predicate discovery, pattern-directed inference, semantic interpretation, and speech and image understanding. It is difficult to determine admissible algorithms for these functions; economical solutions seem to be possible only for globally organized knowledge bases. Examples are provided and directions for research are discussed. (Author/SD)

This paper discusses the use of latent class structure as a modelling framework for tests in which much of the data conforms to a relatively small number of systematic patterns. Application of this framework to the analysis of tests has been limited because available parameter estimation algorithms can only handle a relatively small number of items. This paper proposes a modification of the constrained maximum likelihood estimation algorithm of tests with many more items. A number of special cases of the latent class structure model and a series of problems which arise in the development of any latent class model are reviewed and discussed. A few problems with no satisfactory general solution are noted. The proposed procedures are used to develop and test models for signed-number addition test data gathered by Tatsuoka and Birenbaum. The models are shown to represent most features of the data well, including some significantly negative interitem correlations and differences between groups in covariance structure. The ability of these simple models to represent non-unidimensional data, and to represent differences between groups in covariance structure and means simultaneously, suggests that the latent class structure framework may be very useful in dealing with a number of troublesome psychometric issues. (Author/JAZ)

A new heuristic factorization scheme uses learning to improve the efficiency of determining the symbolic factorization of multivariable polynomials with interger coefficients and an arbitrary number of variables and terms. The factorization scheme makes extensive use of artificial intelligence techniques (e.g., model-building, learning, and automatic classification) in an attempt to reduce the amount of searching for the irreducible factors of the polynomial. The approach taken to polynomial factorization is quite different from previous attempts because: (1) it is distinct from numerical techniques; (2) possibilities for terms in a factor are generated from the terms in the polynomial; and (3) a reclassification technique is used to allow the application of different sets of heuristics to a polynomial during factorization attempts on it. Data presented show the importance of learning to the efficiency of operation of the scheme. Factorization times of polynomials factored by both the scheme described in this paper and Wang's implementation of Berlekamp's algorithm are given and compared, and an analysis of avariance experiment provides an indication of the significant sources of variation influencing the factorization time. (Author/DGC)

Human behavior shows a variety of serially ordered action sequences. This paper presents a theory of serial order which describes how sequences of actions might be learned and performed. In this theory, parallel interactions across time (coarticulation) and parallel interactions across space (dual-task interference) are viewed as two aspects of a common underlying process. It describes a dynamical system which is embodied as a "parallel distributed processing" or "connectionist" network. The trajectories of this dynamical system come to follow desired paths corresponding to particular action sequences as a result of a learning process during which constraints are imposed on the system. These constraints enforce sequentiality where necessary, and as they are relaxed, performance becomes more parallel. The theory is applied to the problem of coarticulation in speech production and simulation experiments are presented. (Author/JAZ)

This paper reports on a project which has developed the general latent class model as a framework for representation of item responses. This framework can be used to represent data in applications such as mastery tests and other kinds of achievement tests, where there is reason to believe that current foundations are deficient. Methods of estimation for the latent class model have been improved and hypothesis tests addressing issues important in developing specific models for test data have been devised. These hypothesis tests include a test for monotone homogeneity of items, tests of invariance of item parameters between groups and over time, a test for the significance of inclusion of a new state in a model, and other tests. A nonparametric approach to maximum likelihood estimation of item response functions for monotonely homogeneous sets of items has been devised. The use of this general approach has been illustrated by developing models which successfully represent signed-number addition test data gathered by Tatsuoka and Birenbaum (1979). A number of technical issues relating to these models are discussed. (Author/JAZ)