Search

Abstract. Using two meta-analytic datasets, we investigated the effect that two scale-item characteristics – number of item response categories and item response-category label format – have on the reliability of multi-item rating scales. The first dataset contained 289 reliability coefficients harvested from 100 samples that measured Big Five traits. The second dataset contained 2,524 reliability coefficients harvested from 381 samples that measured a wide variety of constructs in psychology, marketing, management, and education. We performed moderator analyses on the two datasets with the two item characteristics and their interaction. As expected, as the number of item response categories increased, so did reliability, but more importantly, there was a significant interaction between the number of item response categories and item response-category label format. Increasing the number of response categories increased reliabilities for scale-items with all response categories labeled more so than for other item response-category label formats. We explain that the interaction may be due to both statistical and psychological factors. The present results help to explain why findings on the relationships between the two scale-item characteristics and reliability have been mixed.

Using meta-analytic data, we investigated three attributes of personality scales – their readability, the average number of words per item, and whether adjectives or statements were used – to determine the effects that these qualities have on scale reliability (Cronbach’s α). From a large pool of studies examined in three prior meta-analyses, we extracted the relevant data for seven personality traits measured in 167 independent samples (N = 55,593), and obtained measures of the traits’ readability, the average number of words per item, and whether adjectives or statements were used as the item format. We found strong and consistent evidence that using fewer words per item produced higher scale reliabilities, and there was also evidence in favor of using adjectives, as opposed to statements, as the item format when measuring broad traits. In contrast, the results for readability were inconsistent and mostly nonsignificant. These results have practical implications for scale developers as well as researchers, and they raise the possibility that shorter and more “de-contextualized” items do a better job of assessing personality traits.