Search

ispartof: Communications In Statistics-Simulation And Computation pages:1-19 status: Published online

When monitoring industrial processes, a Statistical Process Control tool, such as a multivariate Hotelling T 2 chart is frequently used to evaluate multiple quality characteristics. However, research into the use of T 2 charts for survey fieldwork–essentially a production process in which data sets collected by means of interviews are produced–has been scant to date. In this study, using data from the eighth round of the European Social Survey in Belgium, we present a procedure for simultaneously monitoring six response quality indicators and identifying outliers: interviews with anomalous results. The procedure integrates Kernel Density Estimation (KDE) with a T 2 chart, so that historical “in-control” data or reference to the assumption of a parametric distribution of the indicators is not required. In total, 75 outliers (4.25%) are iteratively removed, resulting in an in-control data set containing 1,691 interviews. The outliers are mainly characterized by having longer sequences of identical answers, a greater number of extreme answers, and against expectation, a lower item nonresponse rate. The procedure is validated by means of ten-fold cross-validation and comparison with the minimum covariance determinant algorithm as the criterion. By providing a method of obtaining in-control data, the present findings go some way toward a way to monitor response quality, identify problems, and provide rapid feedbacks during survey fieldwork.

Nonresponse is a repeatedly reported concern in survey research. In this article, we investigate the technique of reissuing nonrespondents to another interviewer and attempting to convert them into respondents, using data of Rounds 7 and 8 of the European Social Survey (ESS) in Belgium. The results show no marked differences between respondents interviewed by the more and the less successful interviewers, indicating that the latter are not more successful in persuading more reluctant respondents to participate. Sample units that were unsuccessfully approached in the initial phase by an interviewer with a high response rate are more difficult to convert during the reissue phase. Sample units that were unsuccessfully approached in the initial phase by an interviewer with a low response rate are easier to convert during the reissue phase.

Interviewer characteristics are usually assumed fixed over the fieldwork period. The number of sample units that require the interviewers’ attention, however, can vary strongly over the fieldwork period. Different workload levels produce different constraints on the time interviewers have available to contact, recruit and interview each target respondent, and may also induce different motivational effects on interviewers’ behavior as they perform their different tasks. In this article we show that fine-grained, time-varying operationalizations of project-specific workload can be useful to explain differences in interviewers’ field efforts and achieved response outcomes over the fieldwork period. We derive project-specific workload for each interviewer on each day of fieldwork in two rounds of the European Social Survey in Belgium from contact history and assignment paradata. Project-specific workload is measured as (1) the number of sample units which have been and remain assigned on any day t (assigned case workload), and (2) the number of sample units for which interviewer activity has started and not yet ceased on any day t (active case workload). Capturing temporal variation in interviewers’ workloads in a direct way, the time-varying operationalizations, are better predictors than are the interviewer-level operationalizations of typical (active or potential) workload that are derived from them, as well as the traditional total-count workload operationalization.

A growing body of literature points to the possibilities offered by postsurvey interviewer observations as a source of paradata to obtain insights into data quality. However, their utility in predicting actual behavior of respondents has attracted limited scholarly attention so far. Using data from Round 7 of the European Social Survey, we aim to clarify the relationship between interviewer observations and response styles (straightlining, item nonresponse, use of middle and extreme categories). We contribute to previous research by empirically verifying whether interviewer observations in face-to-face interviews can be effectively used as accurate predictors of data quality. Findings suggest that interviewers’ assessments of response behavior to some extent reflect response styles and can potentially be a valuable tool for the evaluation and improvement of survey data quality, although this seems mainly restricted to the most overt (i.e., easily observable) response styles.

When assessing interview response quality to identify potentially low-quality interviews, both numerical and categorical response quality indicators (mixed indicators) are usually available. However, research on how to use them simultaneously is very rare. In the current article, we extend the application of conventional multivariate control charts to include response quality indicators that are of a mixed type. We analyze data from the eighth round of the European Social Survey in Belgium, characterized by six numerical and two categorical response quality indicators. First, we employ a principal component analysis mix procedure (PCA Mix) to transform the mixed quality indicators into principal components. The principal component scores are subsequently used to construct a Hotelling T2 statistic. To deal with the non-multivariate normal nature of the principal component scores obtained from the PCA Mix, a nonparametric bootstrap method is then applied to calculate the control limit for the T2 statistic. Second, we suggest tools to interpret an identified outlier in terms of finding the responsible original indicator(s). Third, we present a cyclic procedure for determining the “in-control” data, by iteratively removing the outliers until the process is considered as being in control. Lastly, we identify the most important indicators that discriminate the outliers from the in-control data. Our results imply that multivariate control charts based on relevant projection tools such as PCA Mix in combination with the bootstrap technique have great potential for use in evaluating interview response quality and identifying outliers.

The proportion of elderly people in general population samples is increasing. Therefore, it is becoming more important to pay special attention to older respondents when assessing the quality of data. The main hypothesis of the current article is that interviewer effects are higher in the older age-group. We use data collected in 13 countries during Round 7 of the European Social Survey. The results support the supposition that older respondents tend to need more clarification, are more prone to misunderstand the questions, and are likely to have longer interviews. In line with the expectations, we also observe that among older respondents, particularly those aged 71 and above, interviewer effects are more common than among younger respondents.

Purpose In survey methodology, it is well-known that interviewers can have an impact on the registered answers. This paper aims to focus on one type of interviewer effect that arises from the differences between interviewers in the systematic effects of each interviewer on the answers. In the first case, the authors evaluate interviewer effects on the measurement of alcohol consumption in European countries. The second case is about the interviewer effects on the respondents’ tendency to (non)differentiate their responses and the consequences of this response style for the correlation between variables. Design/methodology/approach The interviewer effects are evaluated by means of interviewer variance analysis. Because respondents are nested within interviewers, we can specify a two- or three-level random intercept model to calculate the proportion of variance explained by the interviewers. Data from the seventh round of the European Social Survey are used. Findings The results in the first case show that the substantive conclusions about the effect of gender and education on the alcohol measures continue to hold when interviewer effects are taken into account. The results of the second case make clear that interviewer effects on attitudinal questions are considerable. There is also a significant effect of the interviewers on the degree that respondents differentiate their responses. The results also illustrate that correlations between attitudinal variables are influenced. This also implies that the results of statistical procedures using a correlation or covariance matrix can be strongly influenced by the tendency to (non)differentiate and the interviewers’ impact on this tendency. Originality/value The results clearly demonstrate that there are considerable differences between countries concerning the impact of the interviewers on substantive variables. Cross-national differences are striking and the importance and necessity to evaluate interviewer effects in a cross-national survey becomes clear.

Adaptive and responsive survey designs rely on monitoring indicators based on paradata. This process can better inform fieldwork management if the indicators are paired with a benchmark, which relies on empirical information collected in the first phase of the fieldwork or, for repeated or longitudinal surveys, in previous rounds or waves. We propose the “fieldwork power” (fieldwork production per time unit) as an indicator for monitoring, and we simulate this for the European Social Survey (ESS) Round 7 in Belgium and in the Czech Republic. We operationalize the fieldwork power as the weekly number of completed interviews and of contacts, the ratio of the number of completed interviews to the number of contact attempts and to the number of refusals. We use a repeated measurement multilevel model, with surveys in the previous rounds of the European Social Survey as the macro level and the weekly fieldwork power as repeated measurements to create benchmarks. We also monitor effort and data quality metrics. The results show how problems in the fieldwork evolution can be detected by monitoring the fieldwork power and by comparing it with the benchmarks. The analysis also proves helpful regarding post-survey fieldwork evaluation, and links effort, productivity, and data quality. ispartof: Journal of Official Statistics vol:33 issue:3 pages:659-686 status: published

Interviewer characteristics affect nonresponse and measurement errors in face-to-face surveys. Some studies have shown that mismatched sociodemographic characteristics – for example gender – affect people’s behaviour when interacting with an interviewer at the door and during the survey interview, resulting in more nonresponse. We investigate the effect of sociodemographic (mis)matching on nonresponse in two successive rounds of the European Social Survey in Belgium. As such, we replicate the analyses of the effect of (mis)matching gender and age on unit nonresponse on the one hand, and of gender, age and education level(mis)matching on item nonresponse on the other hand. Recurring effects of sociodemographic (mis)match are found for both unit and item nonresponse. ispartof: Social Science Research vol:67 pages:229-238 ispartof: location:United States status: published

Suitable auxiliary data is important in order to assess the potentially detrimental effects of nonresponse on survey estimates. Whereas sufficient individual-level auxiliary data is rarely available, aggregated data is often quite readily accessible. We investigate whether municipality-level data from official administrative registries is useful to explain and predict individuals’ survey response outcomes in two rounds of the European Social Survey (ESS) in Belgium. This study was prompted by the recent publication of the 2011 Belgian Census. This is the first Belgian census produced by linking various national administrative databases instead of a nationwide survey. It offers free and easy access to a broad range of data at the municipality level. We find no consistent results in the two rounds of the ESS, and the overall usefulness of aggregated data appears limited. Individual-level data available from other auxiliary sources is more strongly and more consistently related to survey response outcomes. Nonresponse analyses in the ESS and other surveys would benefit from access to individual-level administrative registry data, but as this is usually restricted, alternative auxiliary data sources such as interviewer observations will need to suffice. ispartof: Statistical Journal of the IAOS vol:33 issue:3 pages:645-659 status: published

© Statistics Sweden. In this article we examine the interviewer effects on different aspects of response styles, namely non-differentiation and straightlining, which in general refers to the tendency to provide the same answers to questions in a block of questions. According to research about response styles, the impact of the interviewer on this kind of response behavior is rare. Five blocks of items in the questionnaire in the sixth round of the European Social Survey (2012) are used in the analysis. These data also allow for an evaluation of the differences between countries in terms of non-differentiation and straightlining. Five different measurements of these aspects of response style are used in the analysis. To disentangle the impact of respondents and interviewers on these aspects of response style, a three-level random intercept model is specified. The results clearly show interviewer effects on the respondent’s tendency to select a response category that is the same as the response category for the previous item. In some countries the proportion of explained variance due to differences between interviewers is larger than the proportion of variance explained by the differences between respondents. ispartof: Journal of Official Statistics vol:33 issue:2 pages:409-426 status: published

© The Author(s) 2019. Despite general agreement regarding the usefulness of statistical process control (SPC) tools for monitoring paradata, using SPC from an early phase of the survey fieldwork is rather rare. This study focuses on one type of paradata—interview duration—to fill this void. First, we establish a procedure based on the idea of enabling fieldwork monitoring for the seventh round of the European Social Survey in Belgium from its start. The impact of respondent characteristics on interview duration is controlled for by multiple regression. Moreover, we simulate the real conditions of an ongoing survey data collection process by cumulating data and repeating the identification of problematic interviews each week, on the basis that “new” data are available. Second, for each interview we record and track the results with regard to whether or not it is problematic over the fieldwork period, to examine the consistency of our findings. We find that as more data becomes available, the results concerning whether an interview is problematic changes in only 0.3% of the cases. Out of the 27 interviews identified as problematic when all information was available, 25 were immediately identified once relevant information was available. Overall, these findings suggest that SPC tools are reliable and efficient in a survey context, and accordingly have great potential for allowing survey practitioners to focus on the interviews for which further examination is needed immediately, rather than when the data collection has been completed. ispartof: Sage Open vol:9 issue:2 status: Published online

Previous research shows that interviewers to some extent fail to expend the effort that is needed to collect high-quality survey data. We extend the idea of interviewer satisficing to a related task, in which the interviewers themselves answer survey questions. We hypothesize that interviewers who self-administer the questionnaire in a careless manner, also will not apply themselves fully to the task of administering survey interviews. Based on interviewer and respondent data from the sixth round of the European Social Survey in Belgium, we find support for some of the hypothesized associations between (suboptimal) response characteristics of interviewers in the “task as respondent” and the same (suboptimal) response characteristics recorded for their respondents, specifically with regard to interview speed, multiple response, and item nonresponse to the household income question. ispartof: Field Methods vol:29 issue:2 pages:140-153 status: published

The confounding of selection and measurement effects between different modes is a disadvantage of mixed-mode surveys. Solutions to this problem have been suggested in several studies. Most use adjusting covariates to control selection effects. Unfortunately, these covariates must meet strong assumptions, which are generally ignored. This article discusses these assumptions in greater detail and also provides an alternative model for solving the problem. This alternative uses adjusting covariates, explaining measurement effects instead of selection effects. The application of both models is illustrated by using data from a survey on opinions about surveys, which yields mode effects in line with expectations for the latter model, and mode effects contrary to expectations for the former model. However, the validity of these results depends entirely on the (ad hoc) covariates chosen. Research into better covariates might thus be a topic for future studies.

In surveys carried out by interviewers trained according to the key principle of standardized interviewing it is assumed that the interviewer has only limited impact on the time a respondent needs to answer questions. In the paper the effects of interviewers and respondent characteristics on interview speed are analyzed simultaneously by means of a three-level random coefficient model. Data from the fifth round of the European Social Survey (ESS) are used. In twelve participating countries (CAPI) timers were implemented at several places in the questionnaire. Based on this time information the interview speed (number of questions asked per minute) was measured for each respondent during five modules of the questionnaire. The results support most of the expectations concerning the effects of the respondent characteristics. However, the results also indicate that, for all countries, interviewers strongly determine the interview speed and that interview length is not a simple linear function of the number of questions in a questionnaire.

In order to investigate the advantage of mixed-mode (MM) surveys, selection effects between the modes should be evaluated. Selection effects refer to differences in respondent compositions on the target variables between the modes. However, estimation of selection effects is not an easy task because they may be completely confounded with measurement effects between the modes (differences in measurement error). Publications concerning the estimation of these mode effects are scarce. This article presents and compares three methods that allow measurement effects and selection effects to be evaluated separately. The first method starts from existing publications that avoid the confounding problem by introducing a set of mode-insensitive variables into the analysis model. However, this article will show that this method involves unrealistic assumptions in most practical research. The second and the third methods make use of an MM sample extended by comparable single-mode data. The assumptions, advantages, and...

Summary Non-response is a common source of error in many surveys. Because surveys often are costly instruments, quality-cost trade-offs play a continuing role in the design and analysis of surveys. The advances of telephone, computers, and Internet all had and still have considerable impact on the design of surveys. Recently, a strong focus on methods for survey data collection monitoring and tailoring has emerged as a new paradigm to efficiently reduce non-response error. Paradata and adaptive survey designs are key words in these new developments. Prerequisites to evaluating, comparing, monitoring, and improving quality of survey response are a conceptual framework for representative survey response, indicators to measure deviations thereof, and indicators to identify subpopulationsthatneedincreasedeffort.Inthispaper,wepresentanoverviewofrepresentativeness indicators or R-indicators that are fit for these purposes. We give several examples and provide guidelines for their use in practice.

Resume Les enquetes a mode de collecte mixte, dans lesquelles le choix du mode de reponse est laisse aux personnes interrogees, sont de plus en plus repandues. Cette possibilite du choix du mode de reponse peut induire une confusion entre deux sortes d'effets : les effets de selection et les effets de mesure. L'estimation separee de ces deux effets est impossible a partir des seules donnees d'une enquete a mode de collecte mixte. Dans cet article, nous montrons comment la meme estimation devient possible a partir d'une comparaison entre les resultats d'une enquete a mode de collecte mixte et ceux d'un enquete a mode de collecte unique basees sur le meme questionnaire. La methode que nous proposons permet d'estimer l'effet du mode de reponse sur la moyenne et la variance d'une variable de reponse continue. Elle est illustree par l'estimation des effets du mode de reponse sur six questions relatives aux opinions vis-a-vis des enquetes. Summary Mixed-mode surveys, in which different respondents complete the survey by different modes, become increasingly popular. That said, such surveys may lead to a confounding of two forms of mode effects, i.e., selection effects and measurement effects. Exclusive focus on mixed-mode data almost precludes disentangling both effects. In this paper, we show how this problem can be circumvented merely by comparing mixed-mode data with a comparable single-mode sample. The proposed method allows estimating the mode effects on the mean and variance of a continuous variable. As an illustration, the authors estimate mode effects on six items on opinions about surveys.

Whether, and to what degree, the publication of poll results might influence the opinions of the public is a much-debated subject. While many effect-studies of polls have looked at the voting context, this study is an empirical investigation into the effects of poll information on personal opinions and perceptions of collective opinion on five nonelectoral, though politically relevant issues (independence of the Flemish region in Belgium, retirement income, immigration policy, purchasing power, and health-care solidarity). An experimental study was conducted in which poll information was integrated into a survey of a randomly selected group of respondents in order to test the overall impact of poll results on opinions, as well as the differential effects depending on individual characteristics, such as opinion strength, political interest, and perceived poll influence. Furthermore, a two-wave panel design was used to investigate the duration and repeatability of the poll effects. It was successively found in both waves and consistently for all issues that exposure to poll information primarily affected the perceptions of collective opinion, while the personal opinions on political issues remained unaffected. This poll effect on the perceptions of collective opinion was observed not only shortly after being exposed to the poll information, but also three months later.

Survey designs in which data from different groups of respondents are collected by different survey modes have become increasingly popular. However, such mixed-mode (MM) designs lead to a confounding of selection effects and measurement effects (measure- ment error) caused by mode differences. Consequently, MM data have poor quality. Nevertheless, comparing MM data with data from a compa- rable single-mode survey allows researchers to measure selection effects and measurement effects separately. The authors develop a method to evaluate mode effects and illustrate this method with data from a Dutch MM experiment within the European Social Survey program. In this experiment, respondents could choose between three modes: a Web sur- vey, a telephone interview, or a face-to-face interview. Mode effects on three political variables are evaluated: interest in politics, perceived com- plexity of politics, and voter turnout in the last national election.

The hold period between the initial refusal and the follow-up conversion attempt may be used as a strategic tool to improve conversion rates. We argue that longer hold periods result in better conversion rates, particularly among hard-to-convert refusals. In this article we will first investigate to what extend and in which manner survey authorities exert this elapsed time as an active survey instrument. Contact sheet data show that different national survey coordinators deploy different strategies with regard to this hold period. In the Netherlands, intentional survey tactics can be disentangled, whereas Swiss contact data show that the length of the hold period is a mere result other fieldwork conditions. Actual conversion success seems to be consistently dependent upon the hold period between the initial refusal and the follow-up contact.

The current study experimentally investigates the differ- ences in data quality between a face-to-face and a web survey. Based on satisficing theory, it was hypothesized that web survey respondents would be more likely to satisfice for a multitude of reasons, thereby pro- ducing data of lower quality. The data show support for the hypothesis. Web survey respondents were shown to produce a higher "don't know" response rate, to differentiate less on rating scales, and to produce more item nonresponse than face-to-face survey respondents.

Respondent's opinions about survey research are regarded as an expression at the individual level of the general survey-taking climate. Based on a review of the research about ‘surveys on surveys’ five relevant dimensions of the respondent's opinion about surveys are distinguished: survey enjoyment, survey value, survey cost, survey reliability, and survey privacy. Items related to these aspects are used to develop an ‘Opinions about Survey Scale’. The results of an analysis of the predictive validity of the scale support the expectations based on the theory of planned behavior. Opinions about surveys are associated with the intention to participate in future surveys and with hesitation to fill out the questionnaire of a mail survey. Opinions are not related with participation in a follow-up mail survey. The results of the wave analysis of a mail survey are a clear indication that nonrespondents have more negative opinions about different aspects of surveys. This also means that the measurement of opinions is biased. However, because of this bias it is worthwhile and useful to measure these opinions.

Personalizing correspondence has often shown to significantly increase survey response rates in mail surveys. This study experimentally tests whether personalization of email invitations acts correspondingly to web survey response rates. Also, it is investigated whether personalization influences data quality. The results of the study, using a large student sample, show that personalization significantly increases the web survey response rate by 8.6 percentage points. The data quality does not appear to be affected in any major way by personalizing the email invitations. However, the analyses do show that respondents of the personalization condition tend to respond with more social desirability bias to sensitive questions. Therefore, it is concluded that personalization has positive effects on the survey response rate, but one should carefully consider whether or not to personalize when a survey on sensitive topics is conducted.

In one year’s time, the number of respondents in the Flemish Survey of Cultural Shifts expressing a wish to be paid for participating in surveys doubled (28% in 2002, compared to 14% in 2001). Analyses show this is mainly the result of survey-specific characteristics such as length of the survey, invasion of privacy issues, and cognitive efforts needed to fill in the questionnaire. Person-specific characteristics such as the level of individualistic attitudes and the age of the respondent also have an effect but less so than the survey-related issues.

It is common practice to limit web survey access to individuals in the sample through the use of access codes such as personal identification numbers, user names, and passwords. Different technically equivalent methods exist to control survey access. However, these methods are not necessarily methodologically equivalent. In this study, an experimental design was set up to evaluate the effect of three login procedures: a manual, a semiautomatic, and an automatic login procedure. It was found that the automatic login produced the lowest data quality, whereas the manual and semiautomatic login produced data of a higher and comparable level of quality. Nevertheless, the manual and the semiautomatic login procedures differ from each other in certain respects. The study concludes that the semiautomatic login is a useful alternative to control web survey access.

Web surveys generally use some sort of access control to prevent uninvited respondents from taking part in the survey and to prevent multiple completions by the same (invited) respondent. Restriction of survey access can be accomplished in several technically equivalent ways. However, these methods may not be equivalent from a methodological viewpoint. In this article, an exploration of possible effects of access control mechanisms is undertaken. Two different modes of access control were experimentally manipulated. Five hundred respondents were assigned to the automatic login condition. Another 500 respondents were assigned to the manual login condition. It was expected that the automatic login procedure would generate higher response rates but a lower degree of data quality. The results show that using a manual login procedure does not decrease response rates, whereas it does increase the overall degree of data quality.

This note demonstrates an application of a multilevel multinomial model. We use that model to analyse interviewer effects on various components of unit nonresponse to a face-to-face survey: refusals and noncontacts. The model allows for an analysis of these two interviewer effects and a possible connection between both at the same time. Our results show that both the chances for refusals as for noncontacts are subject to interviewer effects. We also find some evidence for a relation between both interviewer effects: interviewers who obtain more refusals are also more likely to report noncontacts. That result is however at least partly dependent on an outlier, an interviewer with a high number of refusals and noncontacts.

Unit nonresponse in the second wave of a panel survey is related to several respondent characteristics and to the interviewer of that wave. More striking is the effect of the interviewer of the first wave, who is not involved in the second interview. To analyze both interviewer effects simultaneously, the authors use a multilevel cross-classified model. In that analysis, the effect of the interviewer of the second wave almost disappears. That effect turns out to be at least partly spurious due to a correlation of both interviewer effects. The authors conclude that the interviewer of the first interview is very important regarding participation in the subsequent waves of a panel survey.

Une évaluation d'une typologie de répondants avec un modèle logit multiniveau et multinomial. Dans cet article, nous examinons une typologie de répondants à une enquête, basée sur les rapports des interviewers. Dans de tels rapports, l'interviewer évalue la motivation et les capacités des interviewés à répondre aux questions. Dans un article précédent, cette information a été utilisée pour créer une typologie de répondants de trois types. Le premier a largement les capacités cognitives requises et il est bien motivé. Le deuxième type ressemble au premier, mais avec des capacités et motivations moindres. Le troisième type a des problèmes à répondre au questionnaire. Il est possible de valider cette typologie empiriquement en confrontant les caractéristiques sociologiques des répondants avec les indicateurs de qualité des données recueillies. Nous faisons une validation de cette typologie avec une analyse multiniveau, contrôlant ainsi pour le biais des interviewers. L'analyse multiniveau prend en compte la structure hiérarchique des données d'enquête et offre la meilleure possibilité d'analyser les effets d'interviewer. Nous utilisons un modèle logit multiniveau et multinomial avec les caractéristiques des répondants et leur comportement de réponse comme variables indépendantes. la typologie comme variable dépendante et l'interviewer comme identificateur du niveau 2. Nos résultats montrent une variabilité significative des interviewers, ce qui remet en question l'utilité des rapports d'interviewers dans la construction d'une typologie de répondants.