A Propensity-Adjusted Interviewer Performance Indicator.

Fair evaluation of interviewers based on their cooperation rates is complicated in most surveys. Depending on the mode(s) of data collection and the methods used to assign sampled cases to interviewers, different interviewers often work cases that are more or less difficult to interview. Further, in centralized telephone facilities, interviewers are given cases affected by the results of earlier attempts by other interviewers. This paper proposes and evaluates an interviewer performance indicator that attempts to repair these weaknesses and can be used in all data collection modes involving interviewers. For each contact made by an interviewer, the deviation between the outcome of that contact (1 = successful interview, 0 = other) and the predicted probability of a successful interview for that contact (according to auxiliary information and survey paradata) is recorded. Each interviewer then receives a score that is the mean of these deviations across all contacts. This performance indicator gives larger credit to the interviewer who obtains success on difficult cases versus relatively easy cases. The indicator also gives smaller penalties to failures on very difficult cases. We analyze face-to-face and telephone contacts from three different surveys to illustrate the computation of this indicator and its properties. We find that calling-history paradata are the strongest predictors of obtaining interviews in both modes (especially for face-to-face contacts), and produce propensity-adjusted performance indicators that more effectively distinguish among interviewers. [ABSTRACT FROM PUBLISHER]