Bias in job analysis survey ratings attributed to order effects.

When respondents complete long surveys, evidence suggests that they may change their response behavior as they become fatigued. The present study examines question order effects in a block‐randomized job analysis survey that collected task ratings (importance, frequency, and needed at certification) for a certification program. Analyses showed that average task ratings were lower for tasks presented later in the survey for both complete and partial responses, though the effect of question ordering was stronger for non‐completers in two rating categories (importance and needed at certification). Additionally, rater variance was lower for questions that appeared later in the survey. The results highlight the value of using randomization to mitigate order effects on survey response quality. Practitioner points: Survey respondents alter their response behavior in lengthy surveys, but little work has examined this phenomenon in job analysis surveys.Ratings decreased for questions that appeared later in the survey, especially for participants who did not ultimately complete the survey.Rater variance also decreased as questions appeared later in the survey.Survey administrators should randomize question order and keep surveys as short as possible. [ABSTRACT FROM AUTHOR]