Search

Grid questions are frequently employed in web surveys due to their assumed response efficiency. In line with this, many previous studies have found shorter response times for grid questions compared to item-by-item formats. Our contribution to this literature is to investigate how altering the question format affects response behavior and the depth of cognitive processing when answering both grid question and item-by-item formats. To answer these questions, we implemented an experiment with three questions in an eye-tracking study. Each question consisted of a set of ten items which respondents answered either on a single page (large grid), on two pages with five items each (small grid), or on ten separate pages (item-by-item). We did not find substantial differences in cognitive processing overall, while the processing of the question stem and the response scale labels was significantly higher for the item-by-item design than for the large grid in all three questions. We, however, found that when answering an item in a grid question, respondents often refer to surrounding items when making a judgement. We discuss the findings and limitations of our study and provide suggestions for practical design decisions.

Combining surveys and digital trace data can enhance the analytic potential of both data types. We present two studies that examine factors influencing data sharing behaviour of survey respondents for different types of digital trace data: Facebook, Twitter, Spotify and health app data. Across those data types, we compared the relative impact of four factors on data sharing: data sharing method, respondent characteristics, sample composition and incentives. The results show that data sharing rates differ substantially across data types. Two particularly important factors predicting data sharing behaviour are the incentive size and data sharing method, which are both directly related to task difficulty and respondent burden. In sum, the paper reveals systematic variation in the willingness to share additional data which need to be considered in research designs linking surveys and digital traces.

In two studies, we examined whether open science practices, such as making materials, data, and code of a study openly accessible, positively affect public trust in science. Furthermore, we investigated whether the potential trust-damaging effects of research being funded privately (e.g. by a commercial enterprise) may be buffered by such practices. After preregistering six hypotheses, we conducted a survey study (Study 1; N = 504) and an experimental study (Study 2; N = 588) in two German general population samples. In both studies, we found evidence for the positive effects of open science practices on trust, though it should be noted that in Study 2, results were more inconsistent. We did not however find evidence for the aforementioned buffering effect. We conclude that while open science practices may contribute to increasing trust in science, the importance of making use of open science practices visible should not be underestimated.

Learning effects due to repeated interviewing, also known as panel conditioning, are a major threat to response quality in later waves of a panel study. To date, research has not provided a clear picture regarding the circumstances, mechanisms, and dimensions of potential panel conditioning effects. In particular, the effects of conditioning frequency, that is, different levels of experience within a panel, on response quality are underexplored. Against this background, we investigated the effects of panel conditioning by using data from the GESIS Panel, a German mixed-mode probability-based panel study. Using two refreshment samples, we compared three panel cohorts with differing levels of experience on several response quality indicators related to the mechanisms of reflection, satisficing, and social desirability. Overall, we find evidence for both negative (i.e., disadvantageous for response quality) and positive (i.e., advantageous for response quality) panel conditioning. Highly experienced respondents were more likely to satisfice by speeding through the questionnaire. They also had a higher probability of refusing to answer sensitive questions than less experienced panel members. However, more experienced respondents were also more likely to optimize the response process by needing less time compared to panelists with lower experience levels (when controlling for speeding). In contrast, we did not find significant differences with respect to the number of “don’t know” responses, nondifferentiation, the selection of first response categories and mid-responses, and the number of nontriggered filter questions. Of the observed differences, speeding showed the highest magnitude with an average increase of 6.0 percentage points for highly experienced panel members compared to low experienced panelists.

In times of declining response rates and over-surveying, improving our understanding of why people participate in surveys is more important than ever. Previous research showed that online panel participants have intrinsic (e.g., topic interest, altruism) and extrinsic (e.g., incentives) participation reasons. Our study expands this research by implementing an experiment using two common forms of survey measurement: ranking and rating. The experiment was fielded in a professional respondents’ sample from a German online panel ( n = 407) and in an address-based sample (mail and online) of German non-professional respondents ( n = 1,137). Besides extrinsic and intrinsic motivations, the experiment included various study design features (i.e., mode, length, data security) and the mood during the time of contact as possible reasons for participation. The results confirm previous findings regarding the motivations of online panelists but also show important differences between professional and non-professional respondents. Specifically, the main participation reasons of professionals are topic interest (intrinsic) and incentives (extrinsic), while non-professionals are primarily motivated by intrinsic reasons (topic interest and purpose of study). This notion is also supported by the Latent Class Analyses, which showed that three of the four classes for professional respondents had a high probability of naming incentives as one of their main reasons for survey participation, whereas none of the four non-professional clusters rated incentives highly. The differences between the two samples highlight that professional panel members have different motivation structures than participants in general population surveys. This may undermine generalizability, but it also provides opportunities for targeted recruitment.

Previous studies have shown many instances where non-probability surveys were not as accurate as probability surveys. However, because of their cost advantages, non-probability surveys are widely used, and debate is ongoing as to the settings for which they are appropriate. To contribute to this debate, we evaluate the accuracy of non-probability surveys by investigating the common claim that estimates of bivariate and multivariate relationships are more robust to sample bias. We compare demographic, attitudinal, and behavioral variables across eight German probability and non-probability surveys with demographic and political benchmarks from the Microcensus and a high-quality face-to-face survey. In the analyses, we compare three types of statistical inference: univariate point estimates, bivariate Pearson’s r coefficients, and 24 different multiple regression models. The results show that in univariate comparisons, non-probability surveys were clearly less accurate than probability surveys when compared with the population benchmarks. These differences in accuracy were smaller in the bivariate and even more so in the multivariate comparisons across surveys. In addition, the outcome of bivariate and multivariate comparisons largely depended on the variables included in the estimation, with some variables being more prone to error. Adjusting the non-probability surveys by raking showed mixed results.

A major challenge in web-based cross-cultural data collection is varying response rates, which can result in low data quality and non-response bias. Country-specific factors such as the political and demographic, economic, and technological factors as well as the socio-cultural environment may have an effect on the response rates to web surveys. This study evaluates web survey response rates using meta-analytical methods based on 110 experimental studies from seven countries. Three dependent variables, so-called effect sizes, are used: the web response rate, the response rate to the comparison survey mode, and the difference between the two response rates. The meta-analysis indicates that four country-specific factors (political and demographic, economic, technological, and socio-cultural) impact the magnitude of web survey response rates. Specifically, web surveys achieve high response rates in countries with high population growth, high internet coverage, and a high survey participation propensity. On the other hand, web surveys are at a disadvantage in countries with a high population age and high cell phone coverage. This study concludes that web surveys can be a reliable alternative to other survey modes due to their consistent response rates and are expected to be used more frequently in national and international settings.

In the context of the current “replication crisis” across the sciences, failures to reproduce a finding are often viewed as discrediting it. This paper shows how such a conclusion can be incorrect. In 1981, Schuman and Presser showed that including the word “freedom” in a survey question significantly increased approval of allowing a speech against religion in the USA. New experiments in probability sample surveys (n = 23,370) in the USA and 10 other countries showed that the wording effect replicated in the USA and appeared in four other countries (Canada, Germany, Taiwan, and the Netherlands) but not in the remaining countries. The effect appeared only in countries in which the value of freedom is especially salient and endorsed. Thus, public support for a proposition was enhanced by portraying it as embodying a salient principle of a nation’s culture. Instead of questioning initial findings, inconsistent results across countries signal limits on generalizability and identify an important moderator.

Objective: With response rates of large-scale surveys having decreased significantly over the years and rebounds seeming unlikely, many studies now examine how response rates vary with methodological design and incentives. This investigation delves into how individual-level factors shape survey participation. Specifically, we examine the influence of individuals’ democratic engagement and their trust in institutions on intent to participate in surveys, both directly and indirectly through their perceptions of surveys. Methods: We collected survey data from a probability sample of adults (N = 1343) in Mannheim, Germany, from November 2019 to March 2020. Structural equation models were estimated to test the hypothesized relationships. Results: The analyses support most, but not all, hypothesized relationships. Democratic engagement bolstered intent to participate, directly as well as indirectly through perceptions of surveys. Institutional trust, on the other hand, only influenced the outcome measure indirectly. Perceptions of surveys had a strong bearing overall effect on intent to participate. Conclusion: The study's results suggest that the response rates and larger issues related to the perceived legitimacy of public opinion and survey research might be intertwined with orientations related to people's civic and political life. The article discusses potential ways survey researchers can counteract distrust in surveys.

Satisficing response behavior can be a threat to the quality of survey responses. Past research has provided broad empirical evidence on the existence of satisficing and its consequences on data quality, however, relatively little is known about the extent of satisficing over the course of a panel study and its impact on response quality in later waves. Drawing on panel conditioning research, we use question design experiments to investigate whether learning effects across waves of a panel study cause changes in the extent of satisficing and if so, whether general survey experience (process learning) or familiarity with specific question content (content learning) accounts for those changes. We use data from a longitudinal survey experiment comprising six panel waves administered within a German non-probability online access panel. To investigate the underlying mechanism of possible learning effects, the experimental study randomly assigned respondents to different frequencies of receiving identical question content over the six panel waves. Our results show the existence of satisficing in every panel wave, which is in its magnitude similar to the extent of satisficing in the probability-based GESIS Panel that we use as a benchmark study. However, we did not find changes in the extent of satisficing across panel waves, nor did we find moderation effects of the interval between the waves, respondents’ cognitive ability, or motivation. Additional validity analyses showed that satisficing does not only affect the distribution of individual estimates by 15 percent or more but also can have an effect on associations between variables.

Today, there are more survey results available than ever before. This increase in survey data is, however, accompanied by a decline in survey quality. Thus, it is more likely than in the past that citizens and politicians get a biased picture of public opinion when relying on survey results. Those misperceptions can have worrying consequences for political discourse and decision-making. With the present study, we aim to investigate to what extent the public draws on survey quality information when evaluating the trustworthiness of a survey result. To explore this research question, we implemented a vignette experiment in an online panel survey (n = 3,313) in which each respondent was confronted with four different, randomly assigned descriptions of a survey and then asked to evaluate the trustworthiness of the respective survey result. The survey descriptions varied regarding the methodological information provided (i.e., sample size, sampling method, and sample balance). The results showed that survey quality information only had a minor effect on the perceptions of trust compared to respondents’ characteristics, such as pre-existing opinions on the topic or general trust in science. Yet, trust in the survey result was significantly influenced by the sample size and sample balance, but not by the sampling method. Finally, in line with information processing theory, the relevance of survey quality information increases with the cognitive abilities of the respondent.

Filter questions are used to administer follow-up questions to eligible respondents while allowing respondents who are not eligible to skip those questions. Filter questions can be asked in either the interleafed or the grouped formats. In the interleafed format, the follow-ups are asked immediately after the filter question; in the grouped format, follow-ups are asked after the filter question block. Underreporting can occur in the interleafed format due to respondents’ desire to reduce the burden of the survey. This phenomenon is called motivated misreporting. Because smartphone surveys are more burdensome than web surveys completed on a computer or laptop, due to the smaller screen size, longer page loading times, and more distraction, we expect that motivated misreporting is more pronounced on smartphones. Furthermore, we expect that misreporting occurs not only in the filter questions themselves but also extends to data quality in the follow-up questions. We randomly assigned 3,517 respondents of a German online access panel to either the PC or the smartphone. Our results show that while both PC and smartphone respondents trigger fewer filter questions in the interleafed format than the grouped format, we did not find differences between PC and smartphone respondents regarding the number of triggered filter questions. However, smartphone respondents provide lower data quality in the follow-up questions, especially in the grouped format. We conclude with recommendations for web survey designers who intend to incorporate smartphone respondents in their surveys.

Learning effects due to repeated interviewing, which are referred to as panel conditioning, are a major threat to response quality in later waves of a panel study. Up to date, research has not provided a clear picture regarding the circumstances, mechanisms, and dimensions of potential panel conditioning effects. Especially the effects of conditioning frequency, that is, different levels of experience within a panel, on response quality are underexplored. Against this background, we investigated the effects of panel conditioning by using data from the GESIS Panel, a German mixed-mode probability-based panel study. Using two refreshment samples, we compared three panel cohorts with differing levels of experience with respect to several response quality indicators related to the mechanisms of reflection, satisficing, and social desirability. Overall, we find evidence for both negative (i.e., disadvantageous for response quality) as well as positive (i.e., advantageous for response quality) panel conditioning. Highly experienced respondents were more likely to satisfice by selecting mid-point responses or by speeding through the questionnaire. They also had a higher probability of refusing to answer sensitive questions than less experienced panel members. However, more experienced respondents were also more likely to optimize the response processes by needing less time compared to panelists with lower experience levels (when controlling for speeding). In contrast, we did not find significant differences with respect to the number of “don’t know” responses, non-differentiation, the selection of first response categories, and the number of non-triggered filter questions. Of the observed differences, speeding showed the highest magnitude with an average increase of 5.9 percentage points for highly experienced panel members compared to low experienced panelists.

As our modern world has become increasingly digitalized, various types of data from different data domains are available that can enrich survey data. To link survey data to other sources, consent from the survey respondents is required. This article compares consent to data linkage requests for seven data domains: administrative data, smartphone usage data, bank data, biomarkers, Facebook data, health insurance data, and sensor data. We experimentally explore three factors of interest to survey designers seeking to maximize consent rates: consent question order, consent question wording, and incentives. The results of the study using a German online sample (n = 3,374) show that survey respondents have a relatively high probability of consent to share smartphone usage data, Facebook data, and biomarkers, while they are least likely to share their bank data in a survey. Of the three experimental factors, only the consent question order affected consent rates significantly. Additionally, the study investigated the interactions between the three experimental manipulations and the seven data domains, of which only the interaction between the data domains and the consent question order showed a consistent significant effect.

Attention checks detect inattentiveness by instructing respondents to perform a specific task. However, while respondents may correctly process the task, they may choose to not comply with the instructions. We investigated the issue of noncompliance in attention checks in two web surveys. In Study 1, we measured respondents’ attitudes toward attention checks and their self-reported compliance. In Study 2, we experimentally varied the reasons given to respondents for conducting the attention check. Our results showed that while most respondents understand why attention checks are conducted, a nonnegligible proportion of respondents evaluated them as controlling or annoying. Most respondents passed the attention check; however, among those who failed the test, 61% seem to have failed the task deliberately. These findings reinforce that noncompliance is a serious concern with attention check instruments. The results of our experiment showed that more respondents passed the attention check if a comprehensible reason was given.

The COVID-19 pandemic has spotlighted the importance of high-quality data for empirical health research and evidence-based political decision-making. To leverage the full potential of these data, a better understanding of the determinants and conditions under which people are willing to share their health data is critical. Building on the privacy theory of contextual integrity, the privacy calculus, and previous findings regarding different data types and recipients, we argue that established social norms shape the acceptance of novel practices of data collection and use. To investigate the willingness to share health data, we conducted a preregistered vignette experiment. The scenarios experimentally varied the vignette dimensions by data type, recipient, and research purpose. While some findings contradict our hypotheses, the results indicate that all three dimensions affected respondents’ data sharing decisions. Additional analyses suggest that institutional and social trust, privacy concerns, technical affinity, altruism, age, and device ownership influence the willingness to share health data.

This paper reports findings from a crowdsourced replication. Eighty-five independent teams attempted a computational replication of results reported in an original study of policy preferences and immigration by fitting the same statistical models to the same data. The replication involved an experimental condition. Random assignment put participating teams into either the transparent group that received the original study and code, or the opaque group receiving only a methods section, rough results description and no code. The transparent group mostly verified the numerical results of the original study with the same sign and p-value threshold (95.7%), while the opaque group had less success (89.3%). Exact numerical reproductions to the second decimal place were far less common (76.9% and 48.1%), and the number of teams who verified at least 95% of all effects in all models they ran was 79.5% and 65.2% respectively. Therefore, the reliability we quantify depends on how reliability is defined, but most definitions suggest it would take a minimum of three independent replications to achieve reliability. Qualitative investigation of the teams’ workflows reveals many causes of error including mistakes and procedural variations. Although minor error across researchers is not surprising, we show this occurs where it is least expected in the case of computational reproduction. Even when we curate the results to boost ecological validity, the error remains large enough to undermine reliability between researchers to some extent. The presence of inter-researcher variability may explain some of the current “reliability crisis” in the social sciences because it may be undetected in all forms of research involving data analysis. The obvious implication of our study is more transparency. Broader implications are that researcher variability adds an additional meta-source of error that may not derive from conscious measurement or modeling decisions, and that replications cannot alone resolve this type of uncertainty.

Identifying inattentive respondents in self-administered surveys is a challenging goal for survey researchers. Instructed response items (IRIs) provide a measure for inattentiveness in grid questions that is easy to implement. The present article adds to the sparse research on the use and implementation of attention checks by addressing three research objectives. In a first study, we provide evidence that IRIs identify respondents who show an elevated use of straightlining, speeding, item nonresponse, inconsistent answers, and implausible statements throughout a survey. Excluding inattentive respondents, however, did not alter the results of substantive analyses. Our second study suggests that respondents’ inattentiveness partially changes as the context in which they complete the survey changes. In a third study, we present experimental evidence that a mere exposure to an IRI does not negatively or positively affect response behavior within a survey. A critical discussion on using IRI attention checks concludes this article.

In this article, we examine two types of item nonresponse in a face-to-face population survey: ‘don’t know’ (DK) and ‘item refusal’ (REF). Based on the cognitive model of survey response, the theory of survey satisficing and previous research, we derive explanatory variables on three levels: question, respondent and interviewer characteristics. The results of our cross-classified model show that while the two levels question and respondents’ characteristics affected both types of item nonresponse, interviewer characteristics affected only DK answers. Our results also confirm that DK and REF are substantially different item nonresponse types resulting from distinguishable disruptions of the cognitive response process. Since most results are in line with prior theoretical predictions, they suggest that survey practitioners are well-advised by continuing to follow the large body of practical guidance derived from the theories tested here.

In der Regel konnen Befragte bei der Teilnahme an Webbefragungen selbst wahlen, mit welchem Endgerat sie an der Befragung teilnehmen mochten. Aufgrund des technischen Fortschritts und der zunehmenden Netzabdeckung entscheiden sich immer mehr Personen an Webbefragungen mittels Smartphones teilzunehmen, wahrend in der Regel weiterhin die grosere Zahl der Befragten mit ihrem PC antworten. Unsere Studie untersucht, auf der Basis von zwei Studierendenbefragungen, Unterschiede im Hinblick auf die Bearbeitungszeit sowie das Antwortverhalten zwischen PC und Smartphonenutzenden. Des Weiteren untersuchen wir Unterschiede zwischen optimiertem und nicht optimiertem visuellem Fragedesign. Unsere Untersuchung zeigt, dass Befragte auf Smartphones langere Bearbeitungszeiten als Befragte auf PCs aufweisen, dies aber kaum Auswirkungen auf das Antwortverhalten hat. Bei der Gegenuberstellung von optimiertem und nicht optimiertem Fragebogendesign konnte eine signifikant hohere Abbruchrate unter Verwendung eines Smartphones mit nicht optimiertem Design festgestellt werden. Die Ergebnisse unserer Studie deuten darauf hin, dass eine Datenerhebung mit Smartphones eine echte Alternative zu traditionellen Webbefragungen mittels PCs darstellt – gerade, wenn man in Betracht zieht, dass die Erhebung mit Smartphones auch die Moglichkeit einer Auswertung von zusatzlichen Daten (z. B. Sensor- oder GPS-Daten) bietet.

Awareness that culture may influence survey response behavior has a long history. Early recognition that social surveys may be influenced by cultural processes dates back to the late 1940s when cross-national survey research studies were first fielded. During the ensuing decades, the rapid growth and development of social survey methodologies, coupled with an increasingly globalized world, and expanding opportunities for cross-national and multi-ethnic investigations, created awareness of the importance of understanding differences between communities and societies and how these differences can influence data collection and survey errors. The purpose of this chapter is to provide an overview of three of the more well-known frameworks for conceptualizing cultural values and orientations (i.e., Hofstede’s cultural orientations; Inglehart’s national values, and Schwartz’s universal values), and then present the available evidence that links these various cultural processes with the total survey error framework through selected survey response and nonresponse behaviors.

This study aims to assess whether respondent inattentiveness causes systematic and unsystematic measurement error that influences survey data quality. To determine the impact of (in)attentiveness o...

In this article, we present the results of three question design experiments on inter-item correlations, which tested a grid design against a single-item design. The first and second experiments examined the inter-item correlations of a set with five and seven items, respectively, and the third experiment examined the impact of the question design on inter-item correlations of follow-up questions. We found that the inter-item correlations in grid designs were higher for both the set of five and seven items than for questions in a single-item design. Additionally, the inter-item correlations of the set of follow-up questions were higher if the preceding set of questions was presented in a grid. According to the findings of the present research, respondents seem to apply the interpretative near means related heuristic when answering grid questions, and also when answering follow-up questions preceeded by grid questions.

In an era of mass migration, social scientists, populist parties and social movements raise concerns over the future of immigration-destination societies. What impacts does this have on policy and social solidarity? Comparative cross-national research, relying mostly on secondary data, has findings in different directions. There is a threat of selective model reporting and lack of replicability. The heterogeneity of countries obscures attempts to clearly define data-generating models. P-hacking and HARKing lurk among standard research practices in this area.This project employs crowdsourcing to address these issues. It draws on replication, deliberation, meta-analysis and harnessing the power of many minds at once. The Crowdsourced Replication Initiative carries two main goals, (a) to better investigate the linkage between immigration and social policy preferences across countries, and (b) to develop crowdsourcing as a social science method. The Executive Report provides short reviews of the area of social policy preferences and immigration, and the methods and impetus behind crowdsourcing plus a description of the entire project. Three main areas of findings will appear in three papers, that are registered as PAPs or in process.

Zentraler Vorteil von Onlinebefragungen auf mobilen Endgeräten (Tablet, Smartphone) ist ihre Allgegenwart und ihr technologisches Potenzial. Umfragemethodisch sind solche Befragungen jedoch eine Herausforderung und die Konsequenzen für die Datenqualität nicht ignorierbar.