Search

The authors undertook a comprehensive examination of the construct validity of an assessment center in this study by (a) gathering many different types of evidence to evaluate the strength of the inference between predictor measures and constructs (e.g., reliability, accuracy, convergent and discriminant relationships), (b) introducing a theoretically relevant intervention (frame-of-reference [FOR] training) aimed at improving construct validity, and (c) examining the effect of this intervention on criterion-related validity (something heretofore unexamined in the assessment center literature). Results from 58 assessees and 122 assessors suggested that FOR training was effective at improving the reliability, accuracy, convergent and discriminant validity, and criterion-related validity of assessment center ratings. Findings are discussed in terms of implications and future directions for both FOR training and assessment center practice.

Ratings from an operational assessment centre (AC) were examined as a function of the order that candidates participated in the assessments. Assessment scheduling had a significant multivariate effect on the preconsensus exercise-based ratings. The effect was small, however, accounting for at most 1% of the rating variance. Differences in exercise order were seen as being unlikely to result in serious unfairness to applicants, although programme designers should be sensitive to the possibility of early vs. late-day performance differences., The assessment centre (AC) method has been used for many years to help make a variety of employment decisions (Thornton, 1992). Use of the method typically requires that candidates participate [...]

This study investigated within-source interrater reliability of supervisor, peer, and subordinate feedback ratings made for managerial development. Raters provided 360-degree feedback ratings on a sample of 153 managers. Using generalizability theory, results indicated that little within-source agreement exists; a large portion of the error variance is attributable to the combined rater main effect and Rater x Ratee effect; more raters are needed than currently used to reach acceptable levels of reliability; supervisors are the most reliable with trivial differences between peers and subordinates when the numbers of raters and items are held constant; and peers are the most reliable, followed by subordinates, followed by supervisors, under conditions commonly encountered in practice. Implications for the validity, design, and maintenance of 360-degree feedback systems are discussed along with directions for future research in this area.

Results of a study on management development revealed that such programs may not be realizing their strategic potential. A total of 123 managers who attended a management development program at Cranfield School of Management completed questionnaires. Findings showed that, even though perceived value of management development is on the rise, its practical application has not been satisfactory. Organizations have not been implementing management development strategies or aligning management development with strategic requirements. This suggests that the problem does not actually lie with the management development programs themselves but with the organizations that send people to attend these courses. Management development professionals should be more proactive in marketing the value of these programs.

A study was conducted to examine participation in management development programs of managers from Central and Eastern Europe. Specifically, it sought to determine the link between reasons for selection and motivations for their decision to attend. Respondents completed a questionnaire at the start and the middle of a six-week development program held in Canada. They reported greater value from the program when the participants were chosen due to usefulness of the material and contacts to their employer. Findings also revealed relationships between four motivations for attendance and self-reported program value. The two most common motivations were to gain a deeper understanding of market economies and to achieve greater managerial effectiveness. The other two were to secure a certificate and to learn skills in starting own business.

This study investigates the degree to which subgroup (Black-White) mean differences on various assessment center exercises (e.g., in-basket, role play) may be a function of the type of exercise employed; and furthermore, begins to explore why these different types of exercises result in subgroup differences. The sample consisted of 633 participants who completed a managerial assessment center that evaluated them on 14 ability dimensions across 7 different types of assessment exercises. In addition, each participant completed a cognitive ability measure. The results suggest that subgroup differences varied by type of assessment exercise; and furthermore that the subgroup difference appeared to be a function of the cognitive component of the exercise. Lastly, preliminary support is found that the validity of some of the assessment center exercises in predicting supervisor ratings of job performance is based, in part, on their cognitive component; however, evidence of incremental validity does exist. (Reprinted by permission of the publisher.)

The effectiveness of training programs on the managerial skills of some 60 middle managers included in a management development program was assessed through the use of a 360 feedback method. Conventional methods to evaluate effectiveness of training, which rely on reactions of participants, are deemed inefficient, necessitating the use of the 360 method. Results show that training improved the success of 360 feedback in aiding managerial skill development at the general profile of skill level but not at the individual level.

Process design principles from organization science offer insights for constructing learning systems that produce consistent, high-quality management education. We incorporated key process design concepts into a structure for graduate and undergraduate management and organizational behavior courses and then tested predictions using path analysis and comparative measures of dispersion. Results indicate that variety designed into a learning system can overcome extensive individual differences in student inputs and yield uniformly high levels of perceived personal effectiveness, organizational effectiveness, ability to apply course materials, and satisfaction with both course results and the educational process. (Reprinted by permission of the publisher.)

Introduction The paper is concerned with the contribution of management development to the culture change in the health service which the Griffiths Report (DHSS, 1983) and subsequent reforms (DHSS, 1986; [...]

This paper discusses the roles of validity, cut score choice, and adverse impact on selection system utility using data from two concurrent validation studies. We contrast an assessment center and published aptitude test on several metrics, including validity, testing costs, adverse impact, and utility. The assessment center produced slightly lower validity than the aptitude test while costing roughly 10 times as much per candidate. In spite of these advantages for the aptitude test, the assessment center produced so much less adverse impact its operational utility would be higher given cut scores likely to be chosen in this organization. Potential concerns with applying net utility models to this type of situation are discussed in comparison to gross utility models. (Reprinted by permission of the publisher.)

Two hundred fifteen organizations in the United States provided information about multiple aspects of their assessment centers, including design, usage, and their adherence to professional guidelines and research-based suggestions for the use of this method. Results reveal that centers are usually conducted for selection, promotion, and development purposes. Supervisor recommendation plays a sizable role in choosing center participants. Most often, line managers act as assessors; they typically arrive at participant ratings through a consensus process. In general, respondents indicate close adherence to recommendations for center design and assessor training. Recommendations involving other practices (e.g., informing participants, evaluating assessors, validating center results) are frequently not followed. Furthermore, methods thought to improve predictive validity of center ratings are underutilized. Variability in center practices according to industry and center purpose was revealed. We encourage practitioners to follow recommendations for center usage, and researchers to work to better understand moderators of center validity. (Reprinted by permission of the publisher.)

Overall previous research has found assessment centres to have low construct validity. Criticism of the analytical procedures used in such research have stimulated a re-evaluation of this issue. In the present study we have examined two variables whose influence on assessment centre construct validity has been neglected. These variables are assessor type (or profession, i.e. psychologist or manager) and the number of distinguishable dimension categories required to adequately represent the ratings. Using Marsh's (1989) approach to confirmatory factor analysis for the assessment centre ratings, we have found that two different models fit the psychologist and manager data. While all the predetermined dimensions were reflected in the psychologists' ratings (i.e. high construct validity), only two broad dimension categories: performance skills and interpersonal skills, accounted for the managers' ratings. Theoretical and practical implications of the findings are discussed., The increased popularity of the assessment centre (AC) in recent years stems partly from its high criterion related validity (Gaugler, Rosenthal, Thornton & Bentson, 1987; McEvoy & Beatty, 1989; Tziner, [...]

The authors investigated the relative validity of personality testing and the assessment center (AC) in order to evaluate the incremental validity of both assessment methods in the prediction of managerial performance. Managers (N = 68) from a forestry products organization were evaluated with an AC and selected traits on the Personality Research Form (D. N. Jackson, 1984). Performance and promotability served as criteria. Overall, personality was found to be at least equivalent to the AC in predicting performance. Personality testing resulted in significant incremental validity over that of the AC in the prediction of performance, but the converse was also true. This suggested that personality and the AC assess different domains, with each uniquely and significantly predicting performance. Promotability was not significantly predicted. Practical implications and future directions were discussed.

Recent research has investigated the use of the Management Charter Initiative (MCI) Level 1 competence-based management development programme within the context of National Health Service reforms. The research raised considerable doubts about the use of MCI standards within management development programmes. There was particular concern about the segmented nature of the standards, as well as the collection of data for the assessment process against strictly defined criteria.

Videotaping of assessment center exercises has become an increasingly common practice, yet little is known about the impact of video technology on rating accuracy. This study compared ratings of a group discussion made after live observation (direct), after viewing a video (indirect), or after viewing a video with opportunities to pause and rewind (controlled). Results indicated some differences in observational accuracy but not in rating accuracy. Implications for the use of video technology in assessment centers are discussed. (Reprinted by permission of the publisher.)

This study addresses the question of whether developmental assessment centers are effective interventions. Career outcomes of assessment center participants (n = 151) were evaluated with respect to center ratings, developmental activity, and acceptance of feedback. Although comparison between a subgroup of assessees (n = 113) and a naturally occurring control group of non-assessees (n = 167) showed no differences in terms of career advancement, findings within the assessee sample did show that developmental activity related to advancement. Findings also showed that assessors' ratings of career motivation, rather than ratings of knowledge or skill (KS) dimensions, predicted both developmental activity and career advancement. (Reprinted by permission of the publisher.)

Manufacturing applicants' perceptions of two selection devices were examined. In Study 1, applicants (n=3,984) completed cognitive ability tests and a survey of reactions. In Study 2, a subset of applicants from Study 1 (n=194) participated in an assessment center and completed the survey. Applicants reacted favorably to the procedures but viewed the assessment center as more face valid than the cognitive tests. Applicants who perceived the selection techniques more favorably were also more satisfied with the selection process, the job, and the organization. Although applicants' perceptions of the procedures were related to job acceptance intentions, applicants' liking of the job and organization explained the largest unique variance. In future studies, applicants' job acceptance intentions and attitudes toward the job and organization should be assessed before and after administration of selection devices; not controlling for prior impressions resulted in overestimation of the contribution of applicants' perceptions of selection procedures. (Reprinted by permission of the publisher.)

The author investigates whether the extent to which participants recognize rating dimensions in assessment centers has an effect on performance. Results showed that people who more accurately identify dimensions perform better. Convergent validity of dimension ratings is greater when participants accurately perceive that the same dimension is being evaluated in 2 exercises. In the discussion, the author considers how the ability to adequately perceive rating dimensions (i.e. transparency) may influence criterion validity. (Reprinted by permission of the publisher.)

The authors examined the effect of 2 different assessment center scoring methods on the cross-situational consistency of assessor ratings. A scoring method that focused on dimensions yielded similar heterotrait-monomethod and monotrait-heteromethod correlations compared with the more typical within-exercise procedure. As in previous research, dimension ratings correlated higher with different dimensions in the same exercise than with ratings of the same dimension in other exercises. In light of this as well as other studies, the authors concluded that different changes appear necessary to increase convergent validity. (Reprinted by permission of the publisher.)

The following article describes an organization development intervention as part of a Unite Nations funded evaluation mission to Gambia in West Africa. The mission was led by the author of this report and was directed to evaluate the Gambian Management Development Institute in Banjul, the capital city. The evaluation methodology used relied heavily on an O.D. action research approach which incorporated the active involvement of the Institute's leadership, as well as those responsible for the United Nation's Development Program funding to help develop institutional capacity. The mission accomplished its evaluation purpose easily and also achieved considerable success in promoting team work and developing action research skills in the officials responsible. The process described is defined as an effort to build in self corrective responses to the project's problems rather than using an evaluation mission to find fault. The end result was the development of a strategy for Gambianization and institution building that helped reconnect the key professionals whose cooperation would aid in project success., The Management Development Institute in Gambia, West Africa was reviewed by a team sent to form an action research intervention program. The evaluation team discovered five legitimate mutually exclusive frameworks within the institution. Upon this discovery, an action team was formed to troubleshoot problem areas. The action team was empowered and a team was formed. A future role and structure in which to work was formed for the action team. Overall, the action team research is a viable means for troubleshooting.

The importance of internal, cognitive or covert support mechanisms to ensure successful application of managerial skills is highlighted. This durability factor has often been left out of the managerial success equation. But consistent application of managerial qualities demands a supportive network to ensure durability. Thus, the introduction of cognitive behavior modification-based management consultations is seen to alter maladaptive thoughts, behaviors and affect which hinder job performance. A skillbuilding managerial framework is laid out for the effective implementation of such cognitive behavioral techniques, even in hard business environments.

Successful managers develop practical theories or frameworks to guide them in problem solving and decision making. A practical theory can be a key element in focusing and handling change and in managerial development. A managerial framework is most useful when it conceptualizes and outlines a vision for people based on the assumptions and values of managers and employees. The development of a managerial framework in a long-term care hospital is described, with focus on performance appraisal, training and career development.

Assessors evaluated assessees' performances in one of three performance conditions in an assessment center simulation: a consistent performance condition, a within-assessee performance variation condition, or a between-assessee performance variation condition. Contrast effects influenced ratings of assessees when there was variation in performance within assessees. For each assessee, the low target performance was rated lower when the assessee's prior performance had been dissimilar (high) than when the assessee's prior performance had been similar (low). Contrast effects also influenced assessors' ratings of the target assessee when there was variation in performance among assessees. The low performing target assessee was rated significantly lower when he was evaluated with two high performers than when he was evaluated with two low performers. In addition, ratings obtained when there was performance variation among assessees were more accurate than those obtained when there was no performance variation, suggesting that under some circumstances contrast effects may not lead to inferior judgments. (Reprinted by permission of the publisher.)

The effects of participation in an assessment center program was studied in relation to the self-assessments made by candidates just before attendance in the assessment center program, immediately after participation in the assessment center program, and a half year after attendance in the assessment center program. The self-assessments of the candidates at these three different time periods were compared to the ratings they received from assessors during the assessment program in terms of accuracy. Unsuccessful candidates were observed to have greatly misjudged their performance effectiveness both before and after participation in the assessment center program, indicating the lack of adequate feedback systems after assessment.

A robust finding in assessment center research is that ratings cluster according to exercises and not dimensions. Seeking further understanding of exercise effects, we proposed two exercise-based factors, exercise form and exercise content, as sources of variance in assessment center ratings. Exercise designs were manipulated so that two levels of form (leaderless group discussions and role play exercises) were crossed with two levels of content (cooperative and competitive task designs). Analysis of variance and confirmatory factor analysis procedures applied to multitrait-multimethod ratings of 89 high school student assessees revealed that most of the variance in the ratings was explained by exercises and not dimensions. Exercise form accounted for 16% of method variance, whereas exercise content had near zero effect. The form effect was primarily due to higher correlations of dimension ratings across role play exercises. Practical implications and future directions for exercise research design research are discussed. (Reprinted by permission of the publisher.)

The construct validity of final self- and peer evaluations in an assessment center was examined within a nomological network of conceptually related and unrelated variables. Data included self-, peer, and assessor evaluations, cognitive ability and personality measures, and job advancement. The evidence for construct validity was stronger for peer than for self-evaluations, and for more easily observable dimensions than for dimensions requiring greater inferential judgment. Self- and peer evaluations were associated with assessor ratings of management potential, whereas only peer evaluations predicted job advancement. Implications for the use of self- and peer evaluations in assessment centers and the need for further research are discussed. (Reprinted by permission of the publisher.)

The construct validity of assessment center final dimension ratings was examined within a nomological network of cognitive and personality measures. Four hundred forty-one employees of a large mid-western petroleum company were assessed on 11 dimensions in two broad categories and completed four tests. Results showed that several cognitive ability measures related more strongly to performance-style dimension ratings than to interpersonal-style dimension ratings, providing evidence for convergent and discriminant validity. Correlation analysis and factor analysis support the two a priori interpersonal- and performance-style categories. The results suggest that final dimension ratings posses construct validity and that assessors can differentiate between two broad categories of assessment dimensions. (Reprinted by permission of the publisher.)

Although research has established the criterion-related validity of assessment centers for selection purposes, the construct validity of dimension ratings has not been demonstrated. A quasi-experimental design was used to investigate the influence of retranslated behavior checklists on the construct validity of dimension ratings for two assessment center exercises. Assessor use of behavior checklists increased the average convergent (i.e., same dimension across exercise) validity from .24 to .43 while decreasing the average discriminant (i.e., different dimension within exercise) validity (.47 to .41). Behavior checklist sums were moderately correlated with corresponding dimension ratings and demonstrated a comparable level of construct validity. It is suggested that using behavior checklists may improve dimension construct validity by reducing the cognitive demands placed on raters. (Reprinted by permission of the publisher.)

Validity coefficients for a single assessment center implemented in multiple locations are presented. Correction for range restriction, criterion unreliability, and sampling error did not account for a large portion of the variability in these validity coefficients. The type of assessor used, the center's administrative arrangement, and prior assessor-assessee contact moderated these validities. It is suggested that the quality of the implementation of a selection procedure when there is local latitude in its implementation is an important factor in determining the procedure's effectiveness. (Reprinted by permission of the publisher.)