Search

Five years of longitudinal data for general chemistry student assessments at the University of Georgia have been analyzed using item response theory (IRT). Our analysis indicates that minor changes in question wording on exams can make significant differences in student performance on assessment questions. This analysis encompasses data from over 6100 students, giving an extremely small statistical uncertainty. IRT provided us with a new insight into student performance on our assessments that is also important to the chemical education community. In this paper, IRT, in conjunction with computerized testing, indicates how nuances in question wording impact student performance on assessments. (Contains 1 table and 3 figures.)

This article highlights issues associated with the use of the differential functioning of items and tests (DFIT) methodology for assessing measurement invariance (or differential functioning) with Likert-type data. Monte Carlo analyses indicate relatively low sensitivity of the DFIT methodology for identifying differential item functioning (DIF) under some conditions of differential functioning with previously recommended significance values. The differential test functioning index was extremely insensitive to differential functioning under all study conditions. The authors recommend alternative noncompensatory DIF cutoff values used to evaluate the significance of DIF for different DIF effect sizes. Additionally, contrasts between polytomous and dichotomous data are drawn, and problems with determining measurement invariance at the scale, rather than item, level for Likert scale data are highlighted. (Contains 6 tables, 6 figures and 9 notes.)

If measurement invariance does not hold over 2 or more measurement occasions, differences in observed scores are not directly interpretable. Golembiewski, Billingsley, and Yeager (1976) identified 2 types of psychometric differences over time as beta change and gamma change. Gamma change is a fundamental change in thinking about the nature of a construct over time. Beta change can be described as respondents' change in calibration of the response scale over time. Recently, researchers have had considerable success establishing measurement invariance using confirmatory factor analytic (CFA) techniques. However, the use of item response theory (IRT) techniques for assessing item parameter drift can provide additional useful information regarding the psychometric equivalence of a measure over time that is not attainable with traditional CFA techniques. This article marries the terminology commonly used in CFA and IRT techniques and illustrates real advantages for identifying beta change over time with IRT methods rather than typical CFA methods, utilizing a longitudinal assessment of job satisfaction as an example.

In recent years, confirmatory factor analytic (CFA) techniques have become the most common method of testing for measurement equivalence/invariance (ME/I). However, no study has simulated data with known differences to determine how well these CFA techniques perform. This study utilizes data with a variety of known simulated differences in factor loadings to determine how well traditional tests of ME/I can detect these specific simulated differences. Results show that traditional CFA tests of ME/I perform well under ideal situations but that large sample sizes, a sufficient number of manifest indicators, and at least moderate communalities are crucial for assurance that ME/I conditions exist.

Conducted a regression analysis to assess the contributions of passage and no-passage factors to item variance on the Scholastic Aptitude Test reading comprehension task. Results show that no-passage factors play a larger role than do passage factors, accounting for as much as three-fourths of systematic variance in item difficulty and more than half of total variance. (SLD)

Describes a Statistical Analysis System (SAS) MACRO for computing various indices of interrater agreement, including a new generalizability coefficient, for categorical data in a single-facet, crossed design. (Author/SLD)

An equation derived by W. A. Nicewander and J. M. Price relating statistical power to reliability of dependent variable measures when true score regression is homogeneous across treatment conditions is enhanced through overcoming the problem of directly estimating the squared linear correlation between true scores for X and Y. (SLD)

The effectiveness of two iterative methods of item response theory (IRT) item bias detection was examined in a simulation study. A modified form of the iterative item parameter linking method of F. Drasgow and an adaptation of the test purification procedure of F. M. Lord were compared. (SLD)

Investigated the effect of computer vs. paper-and-pencil administration on two components of socially desirable responding (SDR), impression management (IM), and self-deception (SD). Ss' degree of anonymity was also manipulated. Independent variables were expected to affect only IM scores, with the computer anonymous condition resulting in the least amount of IM. Results indicated that IM and SD scores were influenced by main effects of both administration and anonymity manipulations. In contrast to previous research, computer administration produced the greatest amount of IM. The findings are discussed relative to results reported Martin and Nagao (1989) on the impact of computer administration of interviews and recent research on the dimension of SDR. (Reprinted by permission of the publisher.)

The consequences of using item response theory (IRT) item bias detecting procedures with multidimensional IRT item data are examined. Limitations in procedures for detecting item bias are discussed. (SLD)

Procedures for implementing parallel analysis (PA) criteria in practice were compared, examining regression equation methods that can be used to estimate random data eigenvalues from known values of the sample size and number of variables. More internally accurate methods for determining PA criteria are presented. (SLD)

A method for estimating the first eigenvalue of random data correlation matrices is reported, and its precision is demonstrated via comparison to the method of S. J. Allen and R. Hubbard (1986). Data generated in Monte Carlo simulations with 10 sample sizes reaching up to 1,000 were used. (SLD)

The component stability of Owens's Biographical Questionnaire (BQ) is reexamined in light of a reanalysis of the original data used to develop the 118-item BQ. Results indicate that components for men are stable over time and geographical location. In addition, the stability of a set of components for women is now demonstrated. The present results suggest that the differences found by Eberhardt and Muchinsky (1982) in the components obtained from women are due to methodological differences between studies and not to changing life experiences of women over the time period between the studies. The components for women that were obtained from a reanalysis of the original data used by Owens and Schoenfeldt (1979) do replicate. (Reprinted by permission of the publisher.)

A study of students in a military academy found that facial attractiveness is a contributing factor to long-term success in an organization. The study found no correlation between attractiveness and academy performance, or between attractiveness and rank attained after 12 years. However, for those who remained on active duty beyond 12 years, attractiveness became a significant factor. The study indicates that organizations foster stereotypes. In this case, attractive people appear more likely to remain in an organization and are more likely to prosper there.

The more useful methods for assessing the fakeability of self-report measures given in Gordon and Gross (1978) are presented along with descriptions of two alternative measures of faking. These measures all assume that faking is being examined in a repeated measures design where subjects typically are asked to respond honestly on one administration of a scale, and are asked to fake on another administration. Since faking that is due to constant discrepancy over an entire group is of minor importance in most practical situations, the discussion focuses primarily on individual differences in faking. The most useful measure of individual differences in faking was operationalized by the variance of difference scores across subjects. This measure is shown to be less sensitive to detecting forms of individual differences in faking that are measured by the proposed within-subject measures. The proposed methods are described, and empirical support for their sensitivity to individual differences in faking is presented. [ABSTRACT FROM PUBLISHER]

The problem of determining test bias in prediction using regression models is reexamined. Past ap proaches have made use of separate regression anal yses in each subgroup, moderated multiple regression analysis using subgroup coding, and hierarchical mul tiple regression strategies. Although it is agreed that hierarchical multiple regression analysis is preferable to either of the former methods, the approach pre sented here differs with respect to the hypothesis test ing procedure to be employed in such an analysis. This paper describes the difficulties in testing hy potheses about the existence of bias in prediction us ing step-up methods of analysis. Some shortcomings of previously recommended approaches for testing these hypotheses are discussed. Finally, a step-down hierarchical multiple regression procedure is recom mended. Analysis of real data illustrates the potential usefulness of the step-down procedure. [ABSTRACT FROM PUBLISHER]

The article focuses on the new AlphaTest program that provides a user-friendly Windows interface for testing hypotheses to allow tests for equivalence among a user-defined set of coefficient alpha values. Further, a superior algorithm based on the incomplete gamma integral is used to evaluate the upper tail probability of the obtained chi-square value. Meanwhile, the program and documentation are available as a free self-installing executable file that works on any version of Windows with Microsoft .Net Framework 2.0 installed.