Search

Responds to a negative evaluation of the National Assessment of Educational Progress (NAEP) by the National Academy of Sciences (NAS) and asserts that a review of the evidence for the NAEP performance standards indicates that there is support for the current approach to NAEP standard setting. Considers the scholarship of the NAS evaluation inadequate. (SLD)

One of the highlights in the observed-score equating literature is a theorem by Lord in his 1980 monograph, Applications of Item Response Theory to Practical Testing Problems. The theorem states that observed scores on two different tests cannot be equated unless the scores are perfectly reliable or the forms are strictly parallel (Lord, 1980, Chapter 13, Theorem 13.3.1). Because the first condition is impossible and equating under the second condition is unnecessary, the theorem is rather sobering. [ABSTRACT FROM AUTHOR]

Switching a testing program from a linear to an adaptive format increases its efficiency considerably. The gain in efficiency can be used to shorten the length of the test or increase the accuracy of the scores. The gain is especially relevant to testing programs in which a battery of tests has to be administered in a single session but the testing time has to remain feasible. Examples of such programs are diagnostic testing for instructional purposes (e.g., Boughton, Yao, & Lewis 2006; Yao & Boughton, 2007) and large-scale assessments of education. These programs generally involve the reporting of profiles of scores of students, schools, or districts. In order to use such profiles for decision making, each of their individual scores should have satisfactory accuracy. The more advantageous combination of testing time and score accuracy made possible by the use of a battery of adaptive instead of linear tests has been highlighted earlier, for instance, in Brown and Weiss (1977) and Giallucca andWeiss (1979). [ABSTRACT FROM AUTHOR]

The last century saw a tremendous progression in the refinement and use of standardized linear tests. The first administered College Board exam occurred in 1901 and the first Scholastic Assessment Test (SAT) was given in 1926. Since then, progressively more sophisticated standardized linear tests have been developed for a multitude of assessment purposes, such as college placement, professional licensure, higher-education admissions, and tracking educational standing or progress. Standardized linear tests are now administered around the world. For example, the Test of English as a Foreign Language (TOEFL) has been delivered in approximately 88 countries. [ABSTRACT FROM AUTHOR]

The intuitive principle underlying adaptive testing is that a test has better measurement properties if the difficulties of its items match the ability of the examinee. Items that are too easy or difficult have predictable responses and cannot provide much information about the ability of the examinee. The first to formalize this principle was Birnbaum (1968). The information measure he used was Fisher΄s well-known information in the sample. For dichotomous response models, the measure is defined as 2.1 ]> where Pi(θ) is the probability of a correct response to item i = 1, …, n for an examinee with ability θ, Ii (θ) is the information in the examinee΄s response to item i, and I(θ) is the information in his or her joint responses to the test. [ABSTRACT FROM AUTHOR]

Although multidimensional item response theory (IRT) (e.g., McDonald, 1962, 1997; Reckase, 1985, 1997; Samejima, 1974) has been available for some time, it has been applied much less frequently in adaptive testing than unidimensional IRT. The main reason for this was a lack of computational power. However, recently this condition has changed dramatically. Even for the more time-intensive Bayesian treatments of multidimensional models, regular PCs now have plenty of computational power to deal with them in a large variety of applications. [ABSTRACT FROM AUTHOR]

In existing adaptive testing programs, each successive item in the test is chosen to optimize an objective. Examples of well-known objectives aremaximizing the information in the test at the ability estimate for the test taker or minimizing the deviation of its information froma target value at the estimate. In addition, item selection is required to realize a set of content specifications for the test. For example, item content may be required to follow a certain taxonomy or the answer-key distribution for the test must not deviate too much from uniformity. Content specifications are generally defined in terms of combinations of attributes the items in the test should have. They are typically realized by imposing a set of constraints on the item-selection process. The presence of both an objective and a set of constraints in adaptive testing leads to the notion of adaptive testing as constrained (sequential) optimization problem; for a more formal introduction to this notion, see van der Linden (this volume, chap. 2). [ABSTRACT FROM AUTHOR]

Item response theory (IRT) models with random person parameters have become a common choice among practitioners in the field of educational and psychological measurement. Though initially the choice for such models was motivated by an attempt to get rid of the statistical problems inherent in the incidental nature of the person parameters (Bock & Lieberman, 1970), the insight soon emerged that such models more adequately represent cases where the focus is not on the measurement of individual persons but on the estimation of characteristics of populations. Early examples of models with random person parameters in the literature are those proposed by Andersen and Madsen (1977) and Sanathanan and Blumenthal (1978), who were interested in estimates of the mean and variance in a population of persons, and by Mislevy (1991), who provided tools for inference from a response model with a regression structure on the person parameters introduced to account for sampling persons differing background variables. [ABSTRACT FROM AUTHOR]

To implement the methods of variance estimation described in this book, one needs to have computer software of known quality and capability. One can write original software for this purpose or purchase a commercially available software package. [ABSTRACT FROM AUTHOR]

Transformations find wide areas of application in the statistical sciences. It often seems advantageous to conduct an analysis on a transformed data set rather than on the original data set. Transformations are most often motivated by the need or desire to [ABSTRACT FROM AUTHOR]

Missing data due to nonresponse, edit failure, and other factors appear in all surveys of human populations. Standard methods of handling missing data can result in an inflation in the estimator variance relative to the variance that would have occurred had all data been observed. This final appendix defines the extra variability and summarizes several methods that can be used to ensure that it properly reflected in the variance estimates. [ABSTRACT FROM AUTHOR]

We shall now introduce measurement (or response) errors and look briefly at the properties of variance estimators when the data are contaminated by such errors. [ABSTRACT FROM AUTHOR]

Inferences from large, complex sample surveys usually derive from the pivotal quantity [ABSTRACT FROM AUTHOR]

In this book, we have studied several practical methods for variance estimation for complex sample surveys. And at this point we address briefly the ultimate question: which of the various methods can be recommended and under what circumstances can they be recommended? [ABSTRACT FROM AUTHOR]

The method of systematic sampling, first studied by the Madows (1944), is used widely in surveys of finite populations. When properly applied, the method picks up any obvious or hidden stratification in the population and thus can be more precise than random sampling. In addition, systematic sampling is implemented easily, thus reducing costs. [ABSTRACT FROM AUTHOR]

In this chapter, we discuss the possibility of a simple mathematical relationship connecting the variance or relative variance of a survey estimator to the expectation of the estimator. If the parameters of the model can be estimated from past data or from a small subset of the survey items, then variance estimates can be produced for all survey items simply by evaluating the model at the survey estimates rather than by direct computation. We shall call this method of variance estimation the method of generalized variance functions (GVF). [ABSTRACT FROM AUTHOR]

In sample surveys of both simple and complex designs, it is often desirable or necessary to employ estimators that are nonlinear in the observations. Ratios, differences of ratios, correlation coefficients, regression coefficients, and poststratified means are common examples of such estimators. Exact expressions for the sampling variances of nonlinear estimators are not usually available and, moreover, neither are simple, unbiased estimators of the variance. [ABSTRACT FROM AUTHOR]

In the foregoing chapters,wediscussed three replication-based methods of variance estimation. Here we close our coverage of replication methods with a presentation of Efron's (1979) bootstrap method, which has sparked a massive amount and variety of research in the past quarter century. For example, see Bickel and Freedman (1984), Booth, Butler, and Hall (1994), Chao and Lo (1985, 1994), Chernick (1999), Davison and Hinkley (1997), Davison, Hinkley, and Young (2003), Efron (1979, 1994), Efron andTibshirani (1986, 1993, 1997), Gross (1980), Hinkley (1988), Kaufman (1998), Langlet, Faucher, and Lesage (2003), Li, Lynch, Shimizu, and Kaufman (2004), McCarthy and Snowden (1984), Rao,Wu, and Yue (1992), Roy and Safiquzzaman (2003), Saigo, Shao, and Sitter (2001), Shao and Sitter (1996), Shao and Tu (1995), Sitter (1992a, 1992b), and the references cited by these authors. [ABSTRACT FROM AUTHOR]

In Chapters 2 and 3, we discussed variance estimating techniques based on random groups and balanced half-samples. Both of these methods are members of the class of variance estimators that employ the ideas of subsample replication. Another subsample replication technique, called the jackknife, has also been suggested as a broadly useful method of variance estimation. As in the case of the two previous methods, the jackknife derives estimates of the parameter of interest from each of several subsamples of the parent sample and then estimates the variance of the parent sample estimator from the variability between the subsample estimates. [ABSTRACT FROM AUTHOR]

Efficiency considerations often lead the survey designer to stratify to the point where only two primary units are selected from each stratum. In such cases, only two independent random groups (or replicates or half-samples) will be available for the estimation of variance, and confidence intervals for the population parameters of interest will necessarily be wider than desired. To overcome this problem, several techniques have been suggested. One obvious possibility is to apply the first version of rule (iv), Section 2.4.1, letting the random group method operate within the strata instead of across them. Variations on the collapsed stratum method offer the possibility of ignoring some of the stratification, thus increasing the number of available random groups. A bias is incurred, however, when the variance is estimated in this manner. Other proposed techniques, including jackknife and half-sample replication, aim to increase the precision of the variance estimator through some form of "pseudoreplication." [ABSTRACT FROM AUTHOR]

The random group method of variance estimation amounts to selecting two or more samples from the population, usually using the same sampling design for each sample; constructing a separate estimate of the population parameter of interest from each sample and an estimate from the combination of all samples; and computing the sample variance among the several estimates. Historically, this was one of the first techniques developed to simplify variance estimation for complex sample surveys. It was introduced in jute acreage surveys in Bengal by Mahalanobis (1939, 1946), who called the various samples interpenetrating samples. Deming (1956), the United Nations Subcommission on Statistical Sampling (1949), and others proposed the alternative term replicated samples. Hansen, Hurwitz, and Madow (1953) referred to the ultimate cluster technique in multistage surveys and to the random group method in general survey applications. Beginning in the 1990s, various writers have referred to the resampling technique. All of these terms have been used in the literature by various authors, and all refer to the same basic method. We will employ the term random group when referring to this general method of variance estimation. [ABSTRACT FROM AUTHOR]

The theory and applications of survey sampling have grown dramatically in the last 60 years. Hundreds of surveys are now carried out each year in the private sector, the academic community, and various governmental agencies, both in the United States and abroad. Examples include market research and public opinion surveys; surveys associated with academic research studies; and large nationwide surveys about labor force participation, health care, energy usage, and economic activity. Survey samples now impinge upon almost every field of scientific study, including agriculture, demography, education, energy, transportation, health care, economics, politics, sociology, geology, forestry, and so on. Indeed, it is not an overstatement to say that much of the data undergoing any form of statistical analysis are collected in surveys. [ABSTRACT FROM AUTHOR]

In this paper we explore the idea that by combining different sources of information in a hospital environment, valuable e-services can be developed for reducing cost and improving quality of service. Companies, like Philips Medical Systems, may have a competitive advantage, because they have a large installed base which may provide valuable information already, and because they can change their products to provide additional information. To optimally benefit from this advantage, we created a platform that enables quick development of e-services. The platform enables uniform access to data, combines static with live data, and supports transparent composition of existing, into new services. We discuss the requirements, design, and implementation of the platform, and we show its use in a case study that addresses asset management and utilization services for mobile medical equipment. [ABSTRACT FROM AUTHOR]

The orthogonal matrices used in defining half-sample replicates in Chapter 3 are known in mathematics as Hadamard matrices. A Hadamard matrix H is a k × k matrix all of whose elements are +1 or −1 that satisfies H′H = kI, where I is the identity matrix of order k. The order k is necessarily 1, 2, or 4t, with t a positive integer. [ABSTRACT FROM AUTHOR]