Search

Implementing innovations in surveys often results in uncertainty concerning how different design decisions will affect key performance indicators such as response rates, nonresponse bias, or survey costs. Thus, responsive survey designs have been developed to better cope with such situations. In the present study, we propose a responsive survey design that relies on experimentation in the earlier phases of the survey to decide between different design choices of which—prior to data collection—their impact on performance indicators is uncertain. We applied this design to the European Values Study 2017/2018 in Germany that advanced its general social survey-type design away from the traditional face-to-face mode to self-administered modes. These design changes resulted in uncertainty as to how different incentive strategies and mode choice sequences would affect response rates, nonresponse bias, and survey costs. We illustrate the application and operation of the proposed responsive survey design, as well as an efficiency issue that accompanies it. We also compare the performance of the responsive survey design to a traditional survey design that would have kept all design characteristics static during the field period.

Congruence on policies between political parties and voters is a frequently assumed re­quirement for democracy. To be able to study this, we should be able to calculate accurate and precise measures of policy congruence in political systems. This could then tell us more about the political system we study, and the “distances” that exist between parties and voters on either issues or broader ideological dimensions. Here, I draw on experimental data from a Voting Advice Application to show that the wording of the issues can influence the degree of congruence one measures. Yet, this comes with the complication that this influence depends on the type of issue, the characteristics of the voters themselves, and the party the congruence is calculated with. These findings should serve as a warning for those who aim to measure congruence that even minor changes in question-wording can (but do not have to) cause relatively large changes in congruence, especially when many parties are involved and the differences between the congruences are small., methods, data, analyses, Vol 17, No 1 (2023)

In this research note, we address the potentials of using interviewer-observed paradata, typically collected during face-to-face-only interviews, in mixed-mode and innovative data collection methods that involve an interviewer at some stage (e.g., during the initial contact or during the interview). To this end, we first provide a systematic overview of the types and purposes of the interviewer-observed paradata most commonly collected in face-to-face interviews—contact form data, interviewer observations, and interviewer evaluations—using the methodology of evidence mapping. Based on selected studies, we illustrate the main purposes of interviewer-observed paradata we identified—including fieldwork management, propensity modeling, nonresponse bias analysis, substantive analysis, and survey data quality assessment. Based on this, we discuss the possible use of interviewer-observed paradata in mixed-mode and innovative data collection methods. We conclude with thoughts on new types of interviewer-observed paradata and the potential of combining paradata from different survey modes.

Reluctance of respondents to participate in surveys has long drawn the attention of survey researchers. Yet, little is known about what drives a respondent’s decision to answer the survey invitation early or late during the field period. Moreover, we still lack evidence on response timing in longitudinal surveys. That is, the questions on whether response timing is a rather stable respondent characteristic and what—if anything—affects change in response timing across different interviews remain open. We relied on data from a mixed-mode general population panel survey collected between 2014 and 2016 to study the stability of response timing across 18 panel waves and factors that influence the decision to participate early or late in the field period. Our results suggest that the factors which had effects on response timing are different in the mail and web modes. Moreover, we found that experience with prior panel waves affected the respondent’s decision to participate early or late. Overall, the present study advocates understanding response timing as a metric variable and, consequently, the need to reflect this in modeling strategies.

Objective: With response rates of large-scale surveys having decreased significantly over the years and rebounds seeming unlikely, many studies now examine how response rates vary with methodological design and incentives. This investigation delves into how individual-level factors shape survey participation. Specifically, we examine the influence of individuals’ democratic engagement and their trust in institutions on intent to participate in surveys, both directly and indirectly through their perceptions of surveys. Methods: We collected survey data from a probability sample of adults (N = 1343) in Mannheim, Germany, from November 2019 to March 2020. Structural equation models were estimated to test the hypothesized relationships. Results: The analyses support most, but not all, hypothesized relationships. Democratic engagement bolstered intent to participate, directly as well as indirectly through perceptions of surveys. Institutional trust, on the other hand, only influenced the outcome measure indirectly. Perceptions of surveys had a strong bearing overall effect on intent to participate. Conclusion: The study's results suggest that the response rates and larger issues related to the perceived legitimacy of public opinion and survey research might be intertwined with orientations related to people's civic and political life. The article discusses potential ways survey researchers can counteract distrust in surveys.

Conducting quantitative research (e.g., surveys, a large number of interviews, experiments) with the participation of political elites is typically challenging. Given that a population of political elites is typically small by definition, a particular challenge is obtaining a sufficiently high number of observations and, thus, a certain response rate. This paper focuses on two questions related to this challenge: (1) What are best practices for designing the study? And (2) what are best practices for soliciting the participation of political elites? To arrive at these best practices, we (a) examine which factors explain the variation in response rates across surveys within and between large-scale, multi-wave survey projects by statistically analyzing a newly compiled dataset of 342 political elite surveys from eight projects, spanning 30 years and 58 countries, (b) integrate the typically scattered findings from the existing literature and (c) discuss results from an original expert survey among researchers with experience with such research (n = 23). By compiling a comprehensive list of best practices, systematically testing some widely held believes about response rates and by providing benchmarks for response rates depending on country, survey mode and elite type, we aim to facilitate future studies where participation of political elites is required. This will contribute to our knowledge and understanding of political elites’ opinions, information processing and decision making and thereby of the functioning of representative democracies.

Recent years have seen a growing number of studies investigating the accuracy of nonprobability online panels; however, response quality in nonprobability online panels has not yet received much attention. To fill this gap, we investigate response quality in a comprehensive study of seven nonprobability online panels and three probability-based online panels with identical fieldwork periods and questionnaires in Germany. Three response quality indicators typically associated with survey satisficing are assessed: straight-lining in grid questions, item nonresponse, and midpoint selection in visual design experiments. Our results show that there is significantly more straight-lining in the nonprobability online panels than in the probability-based online panels. However, contrary to our expectations, there is no generalizable difference between nonprobability online panels and probability-based online panels with respect to item nonresponse. Finally, neither respondents in nonprobability online panels nor respondents in probability-based online panels are significantly affected by the visual design of the midpoint of the answer scale.

Despite the fact that third parties are present during a substantial amount of face-to-face interviews, bystander influence on respondents’ response behavior is not yet fully understood. We use nine waves of the German Family Panel pairfam and apply fixed effects panel regression models to analyze effects of third-party presence on items regarding the sharing of household tasks between partners. We find that both male and female respondents report doing a smaller share of household tasks when their partner is present during the interview as compared to when their partner is not present. Similarly, if the respondent’s partner is present, both partners’ reports correspond more, so that they are less prone to resulting in unrealistically high sums. These results indicate that for items concerning household labor, partner presence does not compromise data quality but may in fact improve it.

Open survey questions are often used to evaluate closed questions. However, they can fulfil this function only if there is a strong link between answers to open questions and answers to related closed questions. Using reasons for non-voting reported in the German Longitudinal Election Study 2013, we investigated this link by examining whether the reported reasons for non-voting may be substantive reasons or ex-post legitimations. We tested five theoretically derived hypotheses about respondents who gave, or did not give, a specific reason. Results showed that (a) answers to open questions were indeed related to answers to closed questions and could be used in explanatory turnout models to predict voting behavior, and (b) the relationship between answers to open and closed questions and the predictive power of reasons given in response to the open questions were stronger in the post-election survey (reported behavior) than in the pre-election survey (intended behavior).

IR-tree models assume that categorical item responses can best be explained by multiple response processes. In the present article, guidelines are provided for the development and interpretation of IR-tree models. In more detail, the relationship between a tree diagram, the model equations, and the analysis on the basis of pseudo-items is described. Moreover, it is shown that IR-tree models do not allow conclusions about the sequential order of the processes, and that mistakes in the model specification can have serious consequences. Furthermore, multiple-group IR-tree models are presented as a novel extension of IR-tree models to data from heterogeneous units. This allows, for example, to investigate differences across countries or organizations with respect to core parameters of the IR-tree model. Finally, an empirical example on organizational commitment and response styles is presented.

Previous research has shown that question characteristics, such as the shape of rating scales, can affect how respondents interpret and respond to questions. For example, earlier studies reported different response distributions for questions employing rating scales in the form of a ladder and in the form of a pyramid. The current experiment, implemented in a probability-based online panel (N = 4,377), re-visits and extends this research by examining how the two graphical layouts (ladder vs. pyramid) affect response behavior and data quality of a question on subjective social stratification. In line with the earlier results, we found that respondents rated their social status lower in the pyramid than in the ladder condition. No differences between the two layouts were found regarding response effort, however, the ladder layout was associated with higher criterion validity. Therefore, we recommend employing the ladder layout when measuring subjective social stratification.

As response rates continue to decline, the need to learn more about the survey participation process remains an important task for survey researchers. Search engine data may be one possible source for learning about what information some potential respondents are looking up about a survey when they are making a participation decision. In the present study, we explored the potential of search engine data for learning about survey participation and how it can inform survey design decisions. We drew on freely available Google Trends (GT) data to learn about the use of Google Search with respect to our case study: participation in the Family Research and Demographic Analysis (FReDA) panel survey. Our results showed that some potential respondents were using Google Search to gather information on the FReDA survey. We also showed that the additional data obtained via GT can help survey researchers to discover topics of interest to respondents and geographically stratified search patterns. Moreover, we introduced different approaches for obtaining data via GT, discussed the challenges that come with these data, and closed with practical recommendations on how survey researchers might utilize GT data to learn about survey participation. Da Response Rates in Umfragen immer weiter sinken, bleibt es eine wichtige Aufgabe für methodische Forschung, mehr über den Teilnahmeprozess zu lernen. Search Engine Data können eine mögliche Quelle sein, um herauszufinden, welche Informationen potenzielle Befragte über eine Umfrage suchen, wenn sie eine Teilnahmeentscheidung treffen. In der vorliegenden Studie untersuchten die Autor*innen das Potenzial von Suchmaschinendaten, um etwas über die Teilnahme an Umfragen zu erfahren und wie diese Daten in Entscheidungen über die Gestaltung von Umfragen einfließen können. Sie stützten sich auf frei verfügbare Daten von Google Trends (GT), um mehr über die Nutzung der Google-Suche in Bezug auf eine Fallstudie zu erfahren: die Teilnahme an der Panel-Umfrage Family Research and Demographic Analysis (FReDA). Die Ergebnisse zeigten, dass einige potenzielle Befragte die Google-Suche nutzten, um Informationen über die FReDA-Umfrage einzuholen. Die Autoren zeigen ebenfalls, dass die über GT gewonnenen zusätzlichen Daten den Umfrageforschern helfen können, Themen, die für die Befragten von Interesse sind, sowie geografisch geschichtete Suchmuster zu entdecken. Darüber hinaus stellen die Autoren verschiedene Ansätze für die Beschaffung von Daten über GT vor, erörtern die mit diesen Daten verbundenen Herausforderungen und geben abschließend praktische Empfehlungen, wie Umfrageforscher GT-Daten nutzen können, um mehr über die Teilnahme an ihren Umfragen zu erfahren.

Preventing panel members from attriting is a fundamental challenge for panel surveys. Research has shown that response behavior in earlier waves (response or nonresponse) is a good predictor of panelists' response behavior in upcoming waves. However, response behavior can be described in greater detail by considering the time until the response is returned. In the present study, we investigated whether respondents who habitually return their survey late and respondents who switch between early and late response in multiple waves are more likely to attrit from a panel. Using data from the GESIS Panel, we found that later response is related to a higher likelihood of attrition (AME = 0.087) and that response-time stability is related to a lower likelihood of attrition (AME = −0.013). Our models predicted most cases of attrition; thus, survey practitioners could potentially predict future attriters by applying these models to their own data.

Since 1996, reports on cohabiting same-sex partnerships have been collected in the German Microcensus. However, it is unclear how reliable these reports are. Compared with other data sources, the Microcensus shows only a small number of cohabiting same-sex couples (less than 0.5% of all cohabiting couples in 2016), so under-reporting is assumed. But, because the "true" values are unknown, it is difficult to determine whether under-reporting is actually occurring. In this paper a procedure is proposed where the response behaviour of respondents is analysed depending on the composition of their household. It was found that non-response to the question about a partner in the household is highest among respondents in whose household there is a possible same-sex partner. This indicates under-reporting of cohabiting same-sex couples, and this under-reporting decreased only slightly, at most, over the period considered here (1996 to 2016). This is not the case for registered same-sex partnerships: a comparison with the Census shows that these are reliably recorded in the German Microcensus., GESIS Papers

Attention checks detect inattentiveness by instructing respondents to perform a specific task. However, while respondents may correctly process the task, they may choose to not comply with the instructions. We investigated the issue of noncompliance in attention checks in two web surveys. In Study 1, we measured respondents’ attitudes toward attention checks and their self-reported compliance. In Study 2, we experimentally varied the reasons given to respondents for conducting the attention check. Our results showed that while most respondents understand why attention checks are conducted, a nonnegligible proportion of respondents evaluated them as controlling or annoying. Most respondents passed the attention check; however, among those who failed the test, 61% seem to have failed the task deliberately. These findings reinforce that noncompliance is a serious concern with attention check instruments. The results of our experiment showed that more respondents passed the attention check if a comprehensible reason was given.

A question asking for respondents' sex is one of the standard sociodemographic characteristics collected in a survey. Until now, it typically consisted of a simple question (e.g., "Are you…?") with two answer categories ("male" and "female"). In 2019, Germany implemented the additional sex designation divers for intersex people. In survey methodology, this has led to an ongoing discussion how to include a third category in questionnaires. We investigate respondents' understanding of the third category, and whether introducing it affects data quality. Moreover, we investigate the understanding of the German term Geschlecht for sex and gender. To answer our research questions, we implemented different question wordings asking for sex/gender in a non-probability-based online panel in Germany and combined them with open-ended questions. Findings and implications for surveys are discussed. Der Geschlechtseintrag divers wurde zu Beginn 2019 in Deutschland für intersexuelle Menschen eingeführt, also für Menschen, die auf Basis ihrer primären oder sekundären Geschlechtsmerkmale weder eindeutig männlich noch weiblich sind. Im März 2019 haben die Autorinnen in einer Web Probing-Studie Daten zum Verständnis der Kategorie erhoben und ausgewertet. Die Ergebnisse zeigen, dass etwa die Hälfte der Befragten ein grundsätzliches Verständnis des Begriffs divers haben, ihn also beispielsweise als "nicht männlich oder weiblich" definieren können. Allerdings zeigte sich auch, dass einige Befragte divers als Kategorie für Transgender hielten, also beispielsweise für Menschen, die biologisch gesehen Männer sind, sich aber als Frau begreifen. Diese Assoziation war sogar wesentlich häufiger als das korrekte Verständnis als Geschlechtseintrag für Intersexualität. Dieser Befund deckt sich mit einem weiteren Schwerpunkt des Artikels, der zeigt, dass die meisten Befragten keinen Unterschied zwischen ihrem biologischen und gefühlten Geschlecht machen. Dies haben wir in einer Auswertung der Begriffe "Geschlecht", "offiziell eingetragenes Geschlecht" und "Geschlechtsidentität" untersucht. Des Weiteren zeigt der Artikel, dass der Anteil an negativen Reaktionen auf den Begriff, in Form von Survey Abbruch oder feindseligen Kommentaren über nicht-binäre Geschlechtszugehörigkeit, gering ist.

The ever-growing number of respondents completing web surveys via smartphones is paving the way for leveraging technological advances to improve respondents’ survey experience and, in turn, the quality of their answers. Smartphone surveys enable researchers to incorporate audio and voice features into web surveys, that is, having questions read aloud to respondents using pre-recorded audio files and collecting voice answers via the smartphone’s microphone. Moving from written to audio and voice communication channels might be associated with several benefits, such as humanizing the communication process between researchers and respondents. However, little is known about respondents’ willingness to undergo this change in communication channels. Replicating and extending earlier research, we examine the extent to which respondents are willing to use audio and voice channels in web surveys, the reasons for their (non)willingness, and respondent characteristics associated with (non)willingness. The results of a web survey conducted in a nonprobability online panel in Germany ( N = 2146) reveal that more than 50% of respondents would be willing to have the questions read aloud (audio channel) and about 40% would also be willing to give answers via voice input (voice channel). While respondents mostly name a general openness to new technologies for their willingness, they mostly name preference for written communication for their nonwillingness. Finally, audio and voice channels in smartphone surveys appeal primarily to frequent and competent smartphone users as well as younger and tech-savvy respondents.

In cross-sectional surveys, as well as in longitudinal panel studies, systematic gender dif­ferences in survey participation are routinely observed. Since there has been little research on this issue, this study seeks to reveal this association for web-based online surveys and computer-assisted telephone interviews in the context of a sequential mixed-mode design with a push-to-web method. Based on diverse versions of benefit–cost theories relating to deliberative and heuristic decision-making, several hypotheses are deduced and then tested by longitudinal data in the context of a multi-wave panel study on the educational and occu­pational trajectories of juveniles. Employing event history data on the survey participation of young panelists living in German-speaking cantons in Switzerland and matching them with geographical data at the macro level and panel characteristics at the meso level, none of the hypotheses is confirmed empirically. It is concluded that indirect measures of an individual’s perceptions of a situation, and of the benefits and costs as well as the process and mechanisms of the decision relating to survey participation, are insufficient to explain this gender difference. Direct tests of these theoretical approaches are needed in future., methods, data, analyses, Online First

Measuring change over time is one of the main purposes of longitudinal surveys. With an increase in the use of web as a mode of data collection it is important to assess whether the web mode differs from other modes with respect to the number of changes and events that are captured. We examine whether telephone and web data collection modes are compara­ble with respect to measuring changes over time or experiencing events. Using experimen­tal data from a two-wave pilot of the Swiss Household Panel, we investigate this question for several variables in the domain of work and family. We find differences for the work-related variables, with web respondents more likely to re­port changes. These differences do not disappear once the socio-demographic composition of the sample is taken into consideration. This suggests that these differences are not driven by observed different characteristics of the respondents who may have self-selected into one or the other mode. Contrary to work-related variables, a termination of a relationship was more common in the telephone group. This shows that one mode does not necessarily measure more change or events than another; it may depend on the variable in question. In addition, the difference in the protocol mattered: a web respondent in a household that participated fully by web sometimes differed from a web respondent in a household that had a household interview by phone. Nonetheless, the telephone group differed more from the various web protocols that the web protocols among themselves. With more household panel surveys introducing web questionnaires in combination with more traditional face-to-face and telephone interviews, this study adds to our understand­ing of the potential consequences of mixing modes with respect to longitudinal data analy­sis., methods, data, analyses, Online First

Although list-style open-ended questions generally help us gain deeper insights into respondents’ thoughts, opinions, and behaviors, the quality of responses is often compromised. We tested a dynamic and a follow-up design to motivate respondents to give higher quality responses than with a static design, but without overburdening them. Our results showed that a follow-up design achieved longer responses with more themes and theme areas than a static design. In contrast, the dynamic design produced the shortest answers with the fewest themes and theme areas. No differences in item nonresponse and only minor differences in additional response burden were found among the three list-style designs. Our study shows that design features and timing are crucial to clarify the desired response format and motivate respondents to give high-quality answers to list-style open-ended questions.

Between April 2019 and September 2020, 23,001 people aged 15 or over responded to questions about their health and living conditions for the German Health Update (GEDA 2019/2020-EHIS). The results are representative of the German resident population aged 15 or above. The response rate was 21.6%. The study used a questionnaire based on the third wave of the European Health Interview Survey (EHIS), which was carried out in all EU member states. EHIS consists of four modules on health status, health care provision, health determinants, and socioeconomic variables. The data are collected in a harmonised manner and therefore have a high degree of international comparability. They constitute an important source of information for European health policy and health reporting and are made available by the Statistical Office of the European Union (Eurostat). They also form the basis of the Federal Health Reporting undertaken in Germany. Data collection began in April 2019, just under a year before the beginning of the SARS-CoV-2 pandemic, and continued into its initial phase, as of March 2020. As such, data from the current GEDA wave can also be used to conduct research into the health impact of the SARS-CoV-2 pandemic.

Knowledge questions frequently are used in survey research to measure respondents’ topic-related cognitive ability and memory. However, in self-administered surveys, respondents can search external sources for additional information to answer a knowledge question correctly. In this case, the knowledge question measures accessible and procedural memory. Depending on what the knowledge question aims at, the validity of this measure is limited. Thus, in this study, we conducted three experiments using a web survey to investigate the effects of task difficulty, respondents’ ability, and respondents’ motivation on the likelihood of searching external sources for additional information as a form of over-optimizing response behavior when answering knowledge questions. We found that the respondents who are highly educated and more interested in a survey are more likely to invest additional efforts to answer knowledge questions correctly. Most importantly, our data showed that for these respondents, a more difficult question design further increases the likelihood of over-optimizing response behavior.

Cognitive online pretests have, in recent years, become recognized as a promising tool for evaluating questions prior to their use in actual surveys. While existing research has shown that cognitive online pretests produce similar results to face-to-face cognitive interviews with regard to the problems detected and the item revisions suggested, little is known about the ideal design of a cognitive online pretest. This study examines whether the number of open-ended probing questions asked during a cognitive online pretest has an effect on the quality and depth of respondents’ answers as well as on respondents’ satisfaction with the survey. We conducted an experiment in which we varied the number of open-ended probing questions that respondents received during a cognitive online pretest. The questionnaire consisted of 26 survey questions, and respondents received either 13 probing questions ( n = 120, short version) or 21 probing questions ( n = 120, long version). The findings suggest that asking a greater number of open-ended probes in a cognitive online pretest does not undermine the quality of respondents’ answers represented by the response quality indicators: (1) amount of probe nonresponse, (2) number of uninterpretable answers, (3) number of dropouts, (4) number of words, (5) response times, and (6) number and type of themes covered by the probes. Furthermore, the respondents’ satisfaction with the survey is not affected by the number of probes being asked.

Background Participation in epidemiologic studies is steadily declining, which may result in selection bias. It is therefore an ongoing challenge to clarify the determinants of participation to judge possible selection effects and to derive measures to minimise that bias. We evaluated the potential for selection bias in a recent population-based cohort study with low baseline participation and investigated reasons for nonparticipation. Methods LIFE-Adult is a cohort study in the general population of the city of Leipzig (Germany) designed to gain insights into the distribution and development of civilisation diseases. Nine thousand one hundred forty-five participants aged 40–79 years were randomly sampled in 2011–2014. We compared LIFE-Adult participants with both the Leipzig population and nonparticipants using official statistics and short questionnaire data. We applied descriptive statistics and logistic regression analysis to evaluate the determinants of study participation. Results Thirty-one percent of the invited persons participated in the LIFE-Adult baseline examination. Study participants were less often elderly women and more often married, highly educated, employed, and current nonsmokers compared to both the Leipzig population and nonparticipants. They further reported better health than nonparticipants. The observed differences were considerable in education and health variables. They were generally stronger in men than in women. For example, in male study participants aged 50–69, the frequency of high education was 1.5 times that of the general population, and the frequency of myocardial infarction was half that of nonparticipants. Lack of time and interest, as well as health problems were the main reasons for nonparticipation. Conclusions Our investigation suggests that the low baseline participation in LIFE-Adult is associated with the typical selection of study participants with higher social status and healthier lifestyle, and additionally less disease. Notably, education and health status seem to be crucial selection factors. Consequently, frequencies of major health conditions in the general population will likely be underestimated. A differential selection related to sex might also distort effect estimates. The extent of the assessment, the interest in the research topic, and health problems of potential participants should in future be considered in LIFE-Adult and in similar studies to raise participation and to minimise selection bias. Electronic supplementary material The online version of this article (10.1186/s12874-019-0779-8) contains supplementary material, which is available to authorized users.

Many surveys aim to achieve high response rates to keep bias due to nonresponse low. However, research has shown that the relationship between the nonresponse rate and nonresponse bias is small. In fact, high response rates may lead to measurement error, if respondents with low response propensities provide survey responses of low quality. In this paper, we explore the relationship between response propensity and measurement error, specifically, motivated misreporting, the tendency to give inaccurate answers to speed through an interview. Using data from four surveys conducted in several countries and modes, we analyze whether motivated misreporting is worse among those respondents who were the least likely to respond to the survey. Contrary to the prediction of our theoretical model, we find only limited evidence that reluctant respondents are more likely to misreport.

Do web surveys still yield lower response rates compared with other survey modes? To answer this question, we replicated and extended a meta-analysis done in 2008 which found that, based on 45 experimental comparisons, web surveys had an 11 percentage points lower response rate compared with other survey modes. Fundamental changes in internet accessibility and use since the publication of the original meta-analysis would suggest that people’s propensity to participate in web surveys has changed considerably in the meantime. However, in our replication and extension study, which comprised 114 experimental comparisons between web and other survey modes, we found almost no change: web surveys still yielded lower response rates than other modes (a difference of 12 percentage points in response rates). Furthermore, we found that prenotifications, the sample recruitment strategy, the survey’s solicitation mode, the type of target population, the number of contact attempts, and the country in which the survey was conducted moderated the magnitude of the response rate differences. These findings have substantial implications for web survey methodology and operations.

Summary The paper analyses the sources of income measurement error in surveys with a unique data set. We use the Austrian 2008–2011 waves of the European Union ‘Statistics on income and living conditions’ survey which provide individual information on wages, pensions and unemployment benefits from survey interviews and officially linked administrative records. Thus, we do not have to fall back on complex two-sample matching procedures like related studies. We empirically investigate four sources of measurement error, namely social desirability, sociodemographic characteristics of the respondent, the survey design and the presence of learning effects. We find strong evidence for a social desirability bias in income reporting, whereas the presence of learning effects is mixed and depends on the type of income under consideration. An Owen value decomposition reveals that social desirability is a major explanation of misreporting in wages and pensions, whereas sociodemographic characteristics are most relevant for mismatches in unemployment benefits.

In light of issues such as increasing unit nonresponse in surveys, several studies argue that social media sources such as Twitter can be used as a viable alternative. However, there are also a number of shortcomings with Twitter data such as questions about its representativeness of the wider population and the inability to validate whose data you are collecting. A useful way forward could be to combine survey and Twitter data to supplement and improve both. To do so, consent within a survey is first needed. This study explores the consent decisions in three large representative surveys of the adult British population to link Twitter data to survey responses and the impact that demographics and survey mode have on these outcomes. Findings suggest that consent rates for data linkage are relatively low, and this is in part mediated by mode, where face-to-face surveys have higher consent rates than web versions. These findings are important to understand the potential for linking Twitter and survey data but also to the consent literature generally.

The collection and use of paradata is gaining in importance, especially in web surveys. From a research ethics’ perspective, respondents should be asked for their consent to the collection and use of web paradata. In this context, a positive attitude toward paradata use has been deemed to be a prerequisite for respondents’ willingness to share their paradata. The present study aimed to identify factors affecting respondents’ attitudes toward paradata use. Our findings revealed that adequately informing survey respondents about what paradata are and why they are used was an important determinant of their attitudes toward paradata use. Moreover, we found that respondents with a positive attitude toward the survey were more likely to have a favorable opinion of paradata use. Our findings suggest that a thorough understanding of the factors that contribute to a positive attitude toward paradata use provides the basis for improved paradata consent procedures, which in turn will increase rates of consent to paradata use and help attenuate the risk of consent bias in web surveys.

The Social Desirability–Gamma Short Scale—the English-language adaptation of the Kurzskala Soziale Erwünschtheit–Gamma (KSE-G)—measures two aspects of the Gamma factor of socially desirable responding (SDR): exaggerating positive qualities (PQ+) and minimizing negative qualities (NQ−). The items of the German-language source version were translated into English using the TRAPD approach. Our empirical validation shows that the reliability and validity coefficients of the English adaptation are comparable to those of the German source instrument. Moreover, the results of measurement invariance testing suggest metric measurement invariance of the scale for the United Kingdom and Germany, thus implying comparability of correlations based on the latent factors across the two countries.

Longitudinal social surveys are widely used to understand which factors enable or constrain access to higher education. One such data resource is the Next Steps survey comprising an initial sample of 16,122 pupils aged 13-14 attending English state and private schools in 2004, with follow up annually to age 19-20 and a further survey at age 25. The Next Steps data is a potentially rich resource for studying inequalities of access to higher education. It contains a wealth of information about pupils' social background characteristics - including household income, parental education, parental social class, housing tenure and family composition - as well as longitudinal data on aspirations, choices and outcomes in relation to education. However, as with many longitudinal social surveys, Next Steps suffers from a substantial amount of missing data due to item non-response and sample attrition which may seriously compromise the reliability of research findings. Helpfully, Next Steps data has been linked with more robust administrative data from the National Pupil Database (NPD), which contains a more limited range of social background variables, but has comparatively little in the way of missing data due to item non-response or attrition. We analyse these linked datasets to assess the implications of missing data for the reliability of Next Steps. We show that item non-response in Next Steps biases the apparent socioeconomic composition of the Next Steps sample upwards, and that this bias is exacerbated by sample attrition since Next Steps participants from less advantaged social backgrounds are more likely to drop out of the study. Moreover, by the time it is possible to measure access to higher education, the socioeconomic background variables in Next Steps are shown to have very little explanatory power after controlling for the social background and educational attainment variables contained in the NPD. Given these findings, we argue that longitudinal social surveys with much missing data are only reliable sources of data on access to higher education if they can be linked effectively with more robust administrative data sources. This then raises the question - why not just use the more robust datasets?

Methodological studies usually gauge response quality in narrative open-ended questions with the proportion of nonresponse, response length, response time, and number of themes mentioned by respondents. However, not all of these indicators may be comparable and appropriate for evaluating open-ended questions in a cross-national context. This study assesses the cross-national appropriateness of these indicators and their potential bias. For the analysis, we use data from two web surveys conducted in May 2014 with 2,685 respondents and in June 2014 with 2,689 respondents and compare responses from Germany, Great Britain, the United States, Mexico, and Spain. We assess open-ended responses for a variety of topics (e.g., national identity, gender attitudes, and citizenship) with these indicators and evaluate whether they arrive at similar or contradictory conclusions about response quality. We find that all indicators are potentially biased in a cross-national context due to linguistic and cultural reasons and that the bias differs in prevalence across topics. Therefore, we recommend using multiple indicators as well as items covering a range of topics when evaluating response quality in open-ended questions across countries.

Although waiting room surveys are frequently conducted, methodological studies on this topic are scarce. Behaviour of surveyors in waiting rooms can easily be controlled, and these surveys also allow for collection of paradata; relevant information on the circumstances of a request to participate in survey research. In this paper, we present the results of an experiment systematically manipulating surveyor’s handshakes and verbal introduction of their names. Patients scheduled for radiological examinations were approached to take part in a survey. An observer noted circumstances in the waiting room (CT or MRI) such as the number of people present.In the CT waiting room, willingness to participate was higher when no other people were filling out the survey than when there were other people filling out the survey. Thus, scarcity effects seemed to play a major role in the decision to participate. In addition, a patient waiting alone was more likely to fully complete the questionnaire, than patients accompanied by one or more caregivers. There was no effect of the surveyor’s handshake or verbal name introduction on survey participation, which is a fortunate outcome in light of social distances measures fighting COVID-19.

Response probabilities are used in adaptive and responsive survey designs to guide data collection efforts, often with the goal of diversifying the sample composition. However, if response probabilities are also correlated with measurement error, this approach could introduce bias into survey data. This study analyzes the relationship between response probabilities and data quality in grid questions. Drawing on data from the probability-based GESIS panel, we found low propensity cases to more frequently produce item nonresponse and nondifferentiated answers than high propensity cases. However, this effect was observed only among long-time respondents, not among those who joined more recently. We caution that using adaptive or responsive techniques may increase measurement error while reducing the risk of nonresponse bias., Survey Research Methods, Vol 15 No 1 (2021)

Invitation letters to web surveys often contain information on how long it will take to complete a web survey. When the stated length in an invitation of a survey is short, it could help to convince respondents to participate in the survey. When it is long respondents may choose not to participate, and when the actual length is longer than the stated length there may be a risk of dropout. This paper reports on an Randomised Control Trial (RCT) conducted in a cross-sectional survey conducted in the Netherlands. The RCT included different version of the stated length of a survey and inclusion of a Quick Response (QR) code as ways to communicate to potential respondents that the survey was short or not. Results from the RCT show that there are no effects of the stated length on actual participation in the survey, nor do we find an effect on dropout. We do however find that inclusion of a QR code leads respondents to be more likely to use a smartphone, and find some evidence for a different composition of our respondent sample in terms of age.

Survey researchers are often confronted with the question of how long to set the length of the field period. Longer fielding time might lead to greater participation yet requires survey managers to devote more of their time to data collection efforts. With the aim of facilitating the decision about the length of the field period, we investigated whether a longer fielding time reduces the risk of nonresponse bias to judge whether field periods can be ended earlier without endangering the performance of the survey. By using data from six waves of a probability-based mixed-mode (online and mail) panel of the German population, we analyzed whether the risk of nonresponse bias decreases over the field period by investigating how day-by-day coefficients of variation develop during the field period. We then determined the optimal cut-off points for each mode after which data collection can be terminated without increasing the risk of nonresponse bias and found that the optimal cut-off points differ by mode. Our study complements prior research by shifting the perspective in the investigation of the risk of nonresponse bias to panel data as well as to mixed-mode surveys, in particular. Our proposed method of using coefficients of variation to assess whether the risk of nonresponse bias decreases significantly with each additional day of fieldwork can aid survey practitioners in finding the optimal field period for their mixed-mode surveys.

Probing questions, essentially open-ended comment boxes that are attached to a traditional closed-ended question, are increasingly used in online surveys. They give respondents an opportunity to share information that goes beyond what can be captured through standardized response categories. However, even when probes are non-mandatory, they can add to perceived response burden and incur a cost in the form of lower respondent cooperation. This paper seeks to measure this cost and reports on a survey experiment that was integrated into a short questionnaire on a German salary comparison site (N = 22,306). Respondents were randomly assigned to one of three conditions: a control without a probing question; a probe that was embedded directly into the closed-ended question; and a probe displayed on a subsequent page. For every meaningful comment gathered, the embedded design resulted in 0.1 break-offs and roughly 3.7 item missings for the closed-ended question. The paging design led to 0.2 additional break-offs for every open-ended answer it collected. Against expectations, smartphone users were more likely to provide meaningful (albeit shorter) open-ended answers than those using a PC or laptop. However, smartphone use also amplified the adverse effects of the probe on break-offs and item non-response to the closed-ended question. Despite documenting their hidden cost, this paper argues that the value of the additional information gathered by probes can make them worthwhile. In conclusion, it endorses the selective use of probes as a tool to better understand survey respondents, methods, data, analyses, Online First

Identifying inattentive respondents in self-administered surveys is a challenging goal for survey researchers. Instructed response items (IRIs) provide a measure for inattentiveness in grid questions that is easy to implement. The present article adds to the sparse research on the use and implementation of attention checks by addressing three research objectives. In a first study, we provide evidence that IRIs identify respondents who show an elevated use of straightlining, speeding, item nonresponse, inconsistent answers, and implausible statements throughout a survey. Excluding inattentive respondents, however, did not alter the results of substantive analyses. Our second study suggests that respondents’ inattentiveness partially changes as the context in which they complete the survey changes. In a third study, we present experimental evidence that a mere exposure to an IRI does not negatively or positively affect response behavior within a survey. A critical discussion on using IRI attention checks concludes this article.

With the increasing use of smartphones in web surveys, considerable efforts have been devoted to reduce the amount of screen space taken up by questions. An emerging stream of research in this area is aimed at optimizing the design elements of rating scales. One suggestion that has been made is to completely abandon verbal labels and use only numeric labels instead. This approach deliberately shifts the task of scale interpretation to the respondents and reduces the information given to them with an intention to reduce their response burden while still preserving the scale meaning. Following prior research, and by drawing on the established model of the cognitive response process, we critically tested these assumptions. Based on a web survey experiment, we found that omitting verbal labels and using only numeric labels instead pushed respondents to focus their responses on the endpoints of a rating scale. Moreover, drawing on response time paradata, we showed that their response burden was not reduced when presented with only numeric labels; quite the opposite was the case, especially when respondents answered the scale with only numeric labels for the first time, which seemed to entail additional cognitive effort. Based on our findings, we advise against using only numeric labels for rating scales in web surveys.

In this article, we examine two types of item nonresponse in a face-to-face population survey: ‘don’t know’ (DK) and ‘item refusal’ (REF). Based on the cognitive model of survey response, the theory of survey satisficing and previous research, we derive explanatory variables on three levels: question, respondent and interviewer characteristics. The results of our cross-classified model show that while the two levels question and respondents’ characteristics affected both types of item nonresponse, interviewer characteristics affected only DK answers. Our results also confirm that DK and REF are substantially different item nonresponse types resulting from distinguishable disruptions of the cognitive response process. Since most results are in line with prior theoretical predictions, they suggest that survey practitioners are well-advised by continuing to follow the large body of practical guidance derived from the theories tested here.

Complex interactive test items are becoming more widely used in assessments. Being computer-administered, assessments using interactive items allow logging time-stamped action sequences. These sequences pose a rich source of information that may facilitate investigating how examinees approach an item and arrive at their given response. There is a rich body of research leveraging action sequence data for investigating examinees' behavior. However, the associated timing data have been considered mainly on the item-level, if at all. Considering timing data on the action-level in addition to action sequences, however, has vast potential to support a more fine-grained assessment of examinees' behavior. We provide an approach that jointly considers action sequences and action-level times for identifying common response processes. In doing so, we integrate tools from clickstream analyses and graph-modeled data clustering with psychometrics. In our approach, we (a) provide similarity measures that are based on both actions and the associated action-level timing data and (b) subsequently employ cluster edge deletion for identifying homogeneous, interpretable, well-separated groups of action patterns, each describing a common response process. Guidelines on how to apply the approach are provided. The approach and its utility are illustrated on a complex problem-solving item from PIAAC 2012.

The European Values Study (EVS) was first conducted in 1981 and then repeated in 1990, 1999, 2008, and 2017, with the aim of providing researchers with data to investigate whether European individual and social values are changing and to what degree. The EVS is traditionally carried out as a probability-based face-to-face survey that takes around 1 hour to complete. In recent years, large-scale population surveys such as the EVS have been challenged by decreasing response rates and increasing survey costs. In the light of these challenges, six countries that participated in the last wave of the EVS tested the application of self-administered mixed-modes (Denmark, Finland, Germany, Iceland, the Netherlands, and Switzerland). With the present data brief, we will introduce researchers to the latest wave of the EVS, the implemented mode experiments, and the EVS data releases. In our view, it is pivotal for data use in substantive research to make the reasoning behind design changes and country-specific implementations transparent as well as to highlight new research opportunities.

Open-ended questions are an important methodological tool for social science researchers, but they suffer from large variations in response quality. In this contribution, we discuss the state of research and develop a systematic approach to the mechanisms of quality generation in open-ended questions, examining the effects from respondents and interviewers as well as those arising from their interactions. Using data from an open-ended question on associations with foreigners living in Germany from the ALLBUS 2016, we first apply a two-level negative binomial regression to model influences on response quality on the interviewer and respondent level and their interaction. In a second regression analysis, we assess how qualitative variation (information entropy) in responses on the interviewer level is related to interviewer characteristics and data quality. We find that respondents’ education, age, gender, motivation and topic interest influence response quality. The interviewer-related variance in response length is 36%. Whereas interviewer characteristics (age, gender, education, experience) do not have a direct effect, they impact on response quality due to interactions between interviewer and respondent characteristics. Notably, an interviewer’s experience has a positive effect on response quality only in interaction with highly educated respondents., methods, data, analyses, Online First

Survey scientists increasingly face the problem of high-dimensionality in their research as digitization makes it much easier to construct high-dimensional (or "big") data sets through tools such as online surveys and mobile applications. Machine learning methods are able to handle such data, and they have been successfully applied to solve predictive problems. However, in many situations, survey statisticians want to learn about causal relationships to draw conclusions and be able to transfer the findings of one survey to another. Standard machine learning methods provide biased estimates of such relationships. We introduce into survey statistics the double machine learning approach, which gives approximately unbiased estimators of parameters of interest, and show how it can be used to analyze survey nonresponse in a high-dimensional panel setting. The double machine learning approach here assumes unconfoundedness of variables as its identification strategy. In high-dimensional settings, where the number of potential confounders to include in the model is too large, the double machine learning approach secures valid inference by selecting the relevant confounding variables. Wissenschaftlerinnen und Wissenschaftler im Feld "Umfrageforschung" sehen sich in ihrer Forschung zunehmend mit dem Problem der hohen Dimensionalität konfrontiert, da es durch die Digitalisierung viel einfacher geworden ist, hochdimensionale (oder "große") Datensätze mit Hilfe von Tools wie Online-Umfragen und mobilen Anwendungen zu erstellen. Methoden des maschinellen Lernens sind in der Lage, mit solchen Daten umzugehen, und sie wurden bereits erfolgreich zur Lösung von Vorhersageproblemen eingesetzt. In vielen Situationen möchten Umfragestatistiker*innen jedoch kausale Zusammenhänge erkennen, um Schlussfolgerungen ziehen und die Ergebnisse einer Umfrage auf eine andere übertragen zu können. Standardmethoden des maschinellen Lernens liefern verzerrte Schätzungen solcher Zusammenhänge. Die Autor*innen führen in die Umfragestatistik den Ansatz des doppelten maschinellen Lernens ein, der annähernd unverzerrte Schätzer der interessierenden Parameter liefert, und zeigen, wie er zur Analyse von Umfrage-Nonresponse in einem hochdimensionalen Panel-Umfeld eingesetzt werden kann.

Increasing use of web surveys and different devices for survey completion calls for examination of device effects on survey response quality. Whereas most previous studies are based on web panels, subgroups (e.g. students) or short questionnaires designed for device experiments, compelling participants to respond through specific devices, this study is based on two large nationally representative cross sectional samples (ISSP 2018 and ISSP 2019) where completion device was optional. Seven indicators of response quality are applied, allowing a comparison among survey participants answering the questionnaire by smartphone, tablet and by PC. Results are in line with previous findings, and do not give support to any assumptions of device effects in the quality of responses. Respondents’ self-evaluated engagement in survey completion do not differ across devices, and only small and non-systematic differences between devices on satisficing indicators such as the tendency to agree regardless of question content (acquiescence), non-substantive answers, selection of mid-point response options and primacy effects, and straightlining are identified. Controlling the associations between response device and the response quality indicators for self-selection biases did not change the overall result.

Response Styles stellen eine Herausforderung für die empirische Surveyforschung dar. Antwortverhalten, welches nicht mit dem Inhalt der Frage assoziiert ist, kann nicht nur Anteils- und Durchschnittswerte beeinflussen, sondern auch modellbasierte Parameterschätzung verzerren. In dieser Pilotstudie soll der Frage nachgegangen werden, ob sich über Machine Learning Verfahren ein empirisch basierter Ansatz zu einheitlichen Identifikation von den gängigsten Response Style Mustern konstruieren lässt. Dies soll aufwändige Kontrollverfahren, welche jeweils nur einzelne Muster finden können oder mit erheblichem Mehraufwand verbunden sind, ersetzen und somit eine universellere und praxistauglichere Option zu bisherigen Ansätzen darstellen. Der hier dargelegte Ansatz basiert auf der Kalibrierung des ML-Verfahrens anhand von synthetisierten Daten, welche der formalen Definition von Response Styles (RS) entsprechen und einem Anteil von empirischen Daten (European Social Survey), welche nicht von RS betroffen sind. Das hierauf trainierte Modell kann auf empirisch erhobene Daten angewendet werden, um RS-Muster in Survey-Daten zuverlässig entdecken und bearbeiten zu können. Die Ergebnisse der Studie legen mit Fehlerklassifikationsraten von 0.3 bis 3.5 % den ML-Ansatz als eine vielversprechende Alternative zu bisherigen Verfahren nahe. Response styles represent a challenge for empirical survey research. Response behavior which is not associated with the content of a survey question can not only influence proportional and average values but also distort model-based parameter estimation. In this pilot study we investigate whether machine learning (ML) methods can be used to construct an empirically based approach to a uniform identification of the most common response style patterns. This method could replace more complex control procedures, which can only find some patterns or are associated with considerable additional effort. Our approach could, therefore, be a more universal and practical option. The method presented here is based on the calibration of the ML model using synthesized data that correspond to a formal definition of response styles (RS) and a proportion of empirical data (European Social Survey) which is not affected by RS. The trained model can then be applied to empirically collected data in order to reliably detect and process RS patterns in survey data. With error classification rates of 0.3 to 3.5 %, the results of this study suggest that our ML-based response style detection is a promising alternative to previous methods.

Previous research shows that a high proportion of respondents engage in other activities while answering surveys. In this study, we examine the effect of multitasking in reporting sensitive information and socially undesirable behavior (e.g., substance use, mental health, gambling) along with reporting of knowledge/awareness of publicly funded programs. The dataset comes from a dual-frame random digit dial telephone survey of adults in a Midwestern state (N = 1,761) who were asked about their attitudes and behaviors toward gambling and health-related behaviors. The results of the study reveal that nearly half of the respondents engaged in multitasking activities (46.9%). In addition, it was found that multitaskers disclosed more socially undesirable information and reported lower levels of knowledge than non-multitaskers. The implications of these findings and how they fit in with previous work are discussed.

The Federal Employee Viewpoint Survey (FEVS) is a Web-based organizational climate survey administered annually by the U.S. Office of Personnel Management. The survey targets full- or part-time, permanently employed civilian personnel from more than 80 U.S. government agencies. Despite increases in the scope, publicity, and uses of FEVS data, its response rate has gradually declined over time. In an effort to gauge the causes for nonresponse, this paper discusses results from an experiment fielded during the 2017 FEVS in which a portion of sampled individuals was given the opportunity to opt out from the survey. Before effectively being added to the “unsubscribe” list, however, the individual was asked to cite the primary reason for choosing not to take the FEVS. Using auxiliary information from the sampling frame, this article argues that inferences based on this item are valuable because opters out are generally representative of the larger pool of nonrespondents.