Search

Supervisors' opportunity to observe incumbents' performance (i.e., how often a supervisor typically sees an employee's performance) has been suggested to be important for accurate performance rating and to be a moderator of criterion‐related validity. Here we test these suggestions and present empirical evidence for the effects of opportunity to observe. In Study 1, supervisors in a multi‐occupation/organization criterion‐related validation study for a biodata measure indicated the opportunity they had to observe incumbents. The data were split according to different levels of opportunity to observe. Higher validities were found when opportunity to observe was maximal. In Study 2, this finding was replicated using a cognitive ability test. These results suggest that psychologists should consider measuring opportunity to observe in criterion‐related validity studies.

We investigated the effects of empirical keying on scoring personality measures. To our knowledge, this is the first published study to investigate the use of empirical keying for personality in a selection context. We hypothesized that empirical keying maximizes use of the information provided in responses to personality items. We also hypothesized that it reduces faking since the relationship between response options and performance is not obvious to respondents. Four studies were used to test the hypotheses. In Study 1, the criterion-related validity of empirically keyed personality measures was investigated using applicant data from a law enforcement officer predictive validation study. A combination of training and job performance measures was used as criteria. In Study 2, two empirical keys were created for long and short measures of the five factors. The criterion-related validities of the empirical keys were investigated using Freshman GPA (FGPA) as a criterion. In Study 3, one set of the empirical keys from Study 2 was applied to experimental data to examine the effects of empirical keying on applicant faking and on the relationship of personality scores and cognitive ability. In Study 4, we examined the generalizability of empirical keying across different organizations. Across the studies, option- and item-level empirical keying increased criterion-related validities for academic, training, and job performance. Empirical keying also reduced the effects of faking. Thus, both hypotheses were supported. We recommend that psychologists using personality measures to predict performance should consider the use of empirical keying as it enhanced validity and reduced faking.

Abstract. Varimax rotated principal component scores (VRPCS) have previously been offered as a possible solution to the non-orthogonality of scores for the Big Five factors. However, few researchers have examined the reliability and validity of VRPCS. To address this gap, we use a lab study and a field study to investigate whether using VRPCS increase orthogonality, reliability, and criterion-related validity. Compared to the traditional unit-weighting scoring method, the use of VRPCS enhanced the reliability and discriminant validity of the Big Five factors, although there was little improvement in criterion-related validity. Results are discussed in terms of the benefit of using VRPCS instead of traditional unit-weighted sum scores.

Summary In recent years, there has been an increasing emphasis on the role of theory in organizational behavior (OB) research. Authors are strongly encouraged to develop “theory” in their manuscripts and to make “theoretical contributions.” This trend is in stark contrast to the process used in other fields of science. Our counterparts in those fields follow the scientific method and define theory as a concise and elegant hypothesis that has survived extensive empirical testing. Rather than being based on extensive empirical research, many of OB's theories are based on a limited number of primary studies (at best) or speculation and conjecture (at worst). OB researchers are discouraged from testing other researchers' theories or replicating previously published work. Consequently, many OB theories do not meet the criteria for a true scientific theory. We propose that OB researchers should re-embrace the scientific method and focus on creating a body of empirical research that could be used in the future to establish true scientific theories (through extensive hypothesis testing, empirical research, and conceptual replications) rather than concocting pseudotheories. Research in the personnel selection subfield of OB provides a reasonable exemplar of this model, yet it has been derided of late for being too atheoretical. Copyright © 2016 John Wiley & Sons, Ltd.

The positive relationship between complexity of work and the validity of general mental ability (GMA) measures across a variety of occupations is well supported by research and provides important practical and theoretical support for cognitive ability measures. However, there is currently no research demonstrating a systematic relationship between the size of the validities of any personality measure and the personality requirements of jobs, thus leaving open to question the predictive and construct validity of personality measures for applicant selection. We compared the validities of two biodata measures – one scored to measure social competence and one to measure GMA – across six job families that varied in social requirements. The validity of the GMA measure generalized at approximately the same level across the job families while the validity of the social competence measure decreased as social requirements diminished.

A conceptual rationale is presented for the use of a high-fidelity simulation: a video-based test (VBT), in which applicants view job-related scenarios, respond orally, and are later scored on five dimensions by trained raters. Using applicant data, we empirically investigate the criterion-related validity, reliability, and utility of the VBT. Overall, we found that the VBT predicted performance in training and on the job. In addition, different raters strongly agreed with one another on each applicant's scores. Despite somewhat higher development costs, the VBT demonstrated high utility. We suggest that organizations consider the use of VBTs as a selection instrument for jobs that require the use of applied social and interpersonal skills.

Five quasi-rational biodata scales were developed by empirically keying biodata items to predict scores on a measure of the Big Five dimensions of personality. The criterion-related validities of the quasi-rational scales were compared to empirical and rational biodata keying methods using supervisory ratings of job performance as the criterion. Empirical keying outperformed the quasi-rational and rational methods (which had similar validities).

This study examined the effects of item format (single-stimulus vs. forced-choice) and response motivation (honest vs. applicant) on scores for personality scales measuring Conscientiousness and Openness to Experience. Consistent with the hypotheses, cognitive ability was related to forced-choice personality scores in the applicant condition but not in the honest condition. Cognitive ability was unrelated to single-stimulus personality scores in both the applicant and honest conditions. The results suggest that controlling for cognitive ability can reduce the incremental predictive validity of forced-choice personality scales in applicant settings. Findings are discussed in terms of the importance of considering how item format influences the construct and criterion-related validity of personality tests used to make selection decisions.

This study tested the hypothesis that personality-based job analysis (PBJA) ratings are correlated with subject matter expert (SME) personality scale scores (through a self-serving bias). Some support was found for the hypothesis. However, PBJA ratings were somewhat predictive of the criterion-related validity of scale scores for the big five personality dimensions.

This study investigated whether scores of item response accuracy (AC) and item response latencies (RLs) from a computer based artificial language test (ALT) uniquely predicted grades in a Spanish language course at a U.S. law enforcement training academy. As expected, ALT–RLs added to the prediction of course grades provided by ALT–AC. This finding held even after controlling for scores on a logical reasoning test. Results are discussed in terms of the potential benefits of using RLs to predict performance in training.

The purpose of this study was to investigate whether worker-oriented job analysis importance ratings were influenced by subject matter experts' (SME) standing (as measured by self-rated performance) on a competency. This type of relationship (whereby SMEs indicate that the traits they have are important for successful job performance) is an example of the self-serving bias (which is widely described in the social cognition literature and rarely described in the industrial/organizational psychology literature). An archival dataset covering 57 clerical and technical occupations with 26,682 participants was used. Support was found for the relationship between self-rated performance and importance ratings. Significant relationships (typically in the .30s) were observed for all 31 competencies that were studied. Controls were taken to account for common method bias and differences in the competencies required for each of the 57 occupations. Past research has demonstrated the effects of the self-serving bias on personality-based job analysis ratings. This study was the first to extend these findings to traditional job analysis, which covers other competencies in addition to personality. In addition, this study is the first to use operational field data instead of laboratory data.