Search

Behavior coding is used to detect issues in questionnaires by flagging interactions between survey respondents and interviewers when their interactions deviate from standardized interviewing. It has become a popular method for evaluating survey questionnaire quality and has been used mostly in Western countries, with little research being done in Asia. However, because of its dependence on overt behaviors observed during the coding, it was hypothesized that it might not be as effective in a country where different communicative norms are present. We conducted an experiment in Korea using two sets of survey questionnaires with embedded problematic questions in the experimental group to investigate the validity of behavior coding as reference to the control group. Using 200 audio recordings from a telephone survey, we investigated whether behavior-coded data were helpful in identifying embedded flaws. The results indicated behavior coding in Korean surveys may not be as effective as in Western countries.

Cognitive interviewing is a method to develop culturally appropriate survey questions and scale items. We conducted two rounds of cognitive interviews with 24 pregnant women in Ethiopia and Kenya to assess the appropriateness, acceptability, and comprehension of general and micronutrient supplement adherence-specific social support scales. We stopped the first round of cognitive interviews after receiving negative feedback from interviewers and participants about their distressing and uncomfortable experiences with cognitive probes and challenges related to cultural perspectives on social support. Through an iterative process, we made substantial changes to the cognitive interview guides and items from both social support scales. In the second round, the revised cognitive interviewing process substantially improved interviewer and participant experiences and increased comprehension and appropriateness of both social support scales. This study confirms the importance of cultural adaptation of the cognitive interviewing process as well as social support scales.

Expert reviews are frequently used as a questionnaire evaluation method but have received little empirical attention. Questions from two surveys are evaluated by six expert reviewers using a standardized evaluation form. Each of the questions has validation data available from records. Large inconsistencies in ratings across the six experts are found. Despite the lack of reliability, the average expert ratings successfully identify questions that had higher item nonresponse rates and higher levels of inaccurate reporting. This article provides empirical evidence that experts are able to discern questions that manifest data quality problems, even if individual experts vary in what they rate as being problematic. Compared to a publicly available computerized question evaluation tool, ratings by the human experts positively predict questions with data quality problems, whereas the computerized tool varies in success in identifying these questions. These results indicate that expert reviews have value in identifying question problems that result in lower survey data quality. [ABSTRACT FROM PUBLISHER]

This article explores the use of expert respondent debriefings to evaluate the quality of survey data. In this case study, subject matter experts observed 169 interviews and conducted qualitative respondent debriefings on select cases in a field test of a census coverage survey. By comparing "true" residence status for each person as determined by the debriefing against the residence status obtained by the questionnaire alone, the authors determined whether the questionnaire was collecting accurate information. For the 473 people for whom survey data was available, the questionnaire failed only five times to assign the correct residence status code in a way that would have detrimentally affected coverage estimates. The respondent debriefing technique helped pinpoint specific problems in the questionnaire, as well as confirm that the questionnaire performed adequately in most cases. This article describes the expert respondent debriefing methodology that was used in a face-to-face interview setting and discusses how it could be adapted to telephone interviewing. [ABSTRACT FROM AUTHOR]

Behavior coding is a widely used method for pretesting survey questionnaires. It is based on the idea that certain types of interviewer-respondent interactions indicate question flaws if the number of such interactions exceeds some specified percentage of tile pretest interviews. In this article, we examine statistical approaches for assessing tile reliability of different observed percentages of problematic interactions. [ABSTRACT FROM AUTHOR]