Search

The authors undertook a comprehensive examination of the construct validity of an assessment center in this study by (a) gathering many different types of evidence to evaluate the strength of the inference between predictor measures and constructs (e.g., reliability, accuracy, convergent and discriminant relationships), (b) introducing a theoretically relevant intervention (frame-of-reference [FOR] training) aimed at improving construct validity, and (c) examining the effect of this intervention on criterion-related validity (something heretofore unexamined in the assessment center literature). Results from 58 assessees and 122 assessors suggested that FOR training was effective at improving the reliability, accuracy, convergent and discriminant validity, and criterion-related validity of assessment center ratings. Findings are discussed in terms of implications and future directions for both FOR training and assessment center practice.

This study investigated within-source interrater reliability of supervisor, peer, and subordinate feedback ratings made for managerial development. Raters provided 360-degree feedback ratings on a sample of 153 managers. Using generalizability theory, results indicated that little within-source agreement exists; a large portion of the error variance is attributable to the combined rater main effect and Rater x Ratee effect; more raters are needed than currently used to reach acceptable levels of reliability; supervisors are the most reliable with trivial differences between peers and subordinates when the numbers of raters and items are held constant; and peers are the most reliable, followed by subordinates, followed by supervisors, under conditions commonly encountered in practice. Implications for the validity, design, and maintenance of 360-degree feedback systems are discussed along with directions for future research in this area.

The authors investigated the relative validity of personality testing and the assessment center (AC) in order to evaluate the incremental validity of both assessment methods in the prediction of managerial performance. Managers (N = 68) from a forestry products organization were evaluated with an AC and selected traits on the Personality Research Form (D. N. Jackson, 1984). Performance and promotability served as criteria. Overall, personality was found to be at least equivalent to the AC in predicting performance. Personality testing resulted in significant incremental validity over that of the AC in the prediction of performance, but the converse was also true. This suggested that personality and the AC assess different domains, with each uniquely and significantly predicting performance. Promotability was not significantly predicted. Practical implications and future directions were discussed.

Videotaping of assessment center exercises has become an increasingly common practice, yet little is known about the impact of video technology on rating accuracy. This study compared ratings of a group discussion made after live observation (direct), after viewing a video (indirect), or after viewing a video with opportunities to pause and rewind (controlled). Results indicated some differences in observational accuracy but not in rating accuracy. Implications for the use of video technology in assessment centers are discussed. (Reprinted by permission of the publisher.)

The author investigates whether the extent to which participants recognize rating dimensions in assessment centers has an effect on performance. Results showed that people who more accurately identify dimensions perform better. Convergent validity of dimension ratings is greater when participants accurately perceive that the same dimension is being evaluated in 2 exercises. In the discussion, the author considers how the ability to adequately perceive rating dimensions (i.e. transparency) may influence criterion validity. (Reprinted by permission of the publisher.)

The authors examined the effect of 2 different assessment center scoring methods on the cross-situational consistency of assessor ratings. A scoring method that focused on dimensions yielded similar heterotrait-monomethod and monotrait-heteromethod correlations compared with the more typical within-exercise procedure. As in previous research, dimension ratings correlated higher with different dimensions in the same exercise than with ratings of the same dimension in other exercises. In light of this as well as other studies, the authors concluded that different changes appear necessary to increase convergent validity. (Reprinted by permission of the publisher.)

A robust finding in assessment center research is that ratings cluster according to exercises and not dimensions. Seeking further understanding of exercise effects, we proposed two exercise-based factors, exercise form and exercise content, as sources of variance in assessment center ratings. Exercise designs were manipulated so that two levels of form (leaderless group discussions and role play exercises) were crossed with two levels of content (cooperative and competitive task designs). Analysis of variance and confirmatory factor analysis procedures applied to multitrait-multimethod ratings of 89 high school student assessees revealed that most of the variance in the ratings was explained by exercises and not dimensions. Exercise form accounted for 16% of method variance, whereas exercise content had near zero effect. The form effect was primarily due to higher correlations of dimension ratings across role play exercises. Practical implications and future directions for exercise research design research are discussed. (Reprinted by permission of the publisher.)

The construct validity of final self- and peer evaluations in an assessment center was examined within a nomological network of conceptually related and unrelated variables. Data included self-, peer, and assessor evaluations, cognitive ability and personality measures, and job advancement. The evidence for construct validity was stronger for peer than for self-evaluations, and for more easily observable dimensions than for dimensions requiring greater inferential judgment. Self- and peer evaluations were associated with assessor ratings of management potential, whereas only peer evaluations predicted job advancement. Implications for the use of self- and peer evaluations in assessment centers and the need for further research are discussed. (Reprinted by permission of the publisher.)

The authors investigated temporal trends in the validity of an assessment center consisting of a group discussion and an analysis--presentation exercise for predicting career advancement as measured by average salary growth over a 7-year period in a sample of 679 academic graduates. The validity of the overall assessment rating for persons with tenure of 7 years, corrected for initial differences in starting salaries and restriction in range, was .39. There was a considerable time variation in the validity of both the overall assessment rating and the assessment center dimensions. In accordance with findings from research in managerial effectiveness and development, the interpersonal effectiveness dimension became valid only after a number of years, whereas the firmness dimension was predictive in the whole period and increased with time. For comparison, validity trends for 2 types of interviews and a mental test were also studied.

This article reports a study of the construct validity of in-baskets designed to measure managerial abilities. Data were collected using a randomized pretest-posttest experimental design with two alternate in-basket forms. Results indicated that (a) there was little convergent validity and evidence of method bias both within and between alternate in-basket forms and (b) brief training improved in-basket performance on the perceptiveness and delegation dimensions and on overall performance. Both lines of evidence call into question the validity of inferring individual differences in managerial ability from in-basket scores. (Reprinted by permission of the publisher.)

Undergraduates (N = 131) were trained as assessors who, evaluated the performance of confederates in an assessment center simulation on 3, 6, or 9 dimensions. Number of dimensions significantly affected some assessment center judgments but not others. Ss who rated a small number of dimensions classified behaviors more accurately and made more accurate ratings than did Ss who rated a large number of dimensions. Number of dimensions did not affect the accuracy of assessors' observations nor the discriminant validity of their dimension ratings. Given these results and the findings of others (Hinrichs & Haanpera, 1976; Russell, 1985; Sackett & Hakel, 1979; Schmitt, 1977), developers of assessment centers should limit the cognitive demands placed on assessors by, for example, minimizing the number of dimensions assessors are required to process. (Reprinted by permission of the publisher.)

Meta-analysis (Hunter, Schmidt, & Jackson, 1982) of 50 assessment center studies containing 107 validity coefficients revealed a corrected mean and variance of .37 and .017, respectively. Validities were sorted into five categories of criteria and four categories of assessment purpose. Higher validities were found in studies in which potential ratings were the criterion, and lower validities were found in promotion studies. Sufficient variance remained after correcting for artifacts to justify searching for moderators. Validities were higher when the percentage of female assessees was high, when several evaluation devices were used, when assessors were psychologists rather than managers, when peer evaluation was used, and when the study was methodologically sound. Age of assessees, whether feedback was given, days of assessor training, days of observation, percentages of minority assessees, and criterion contamination did not moderate assessment center validities. The findings suggest that assessment centers show both validity generalization and situational specificity. (Reprinted by permission of the publisher.)

Assessment center ratings of eight abilities from each of five situational exercises were examined for their cross-situational consistency and discriminant validity. A series of confirmatory factor analyses revealed that the ratings were largely (if not totally) situation specific, and that assessors failed to distinguish among the eight target abilities. These results combined with previous research suggest that the assessment center method measures mainly situation-specific performance, not cross-situational managerial abilities. We suggest that the intended constructs might be better measured if more ability-related behaviors were elicited within each exercise and if the cognitive demands placed on assessors were reduced. (Reprinted by permission of the publisher.)

Assessment center ratings on 275 police recruits correlated .14 (p