Search

The measurement of political solidarities and related concepts is an important endeavor in numerous scientific disciplines, such as political and social science research. European surveys, such as the Eurobarometer, frequently measure these concepts for people’s home country and Europe raising questions with respect to the order of precedence. Research has shown that the order of asking questions can have a profound impact on answer behavior compromising data quality. In this study, we therefore investigated the occurrence of question order effects in a German-European context using two questions on political solidarities. For this purpose, we conducted an experiment in a German online panel (N = 874) and analyzed response behavior and effort in terms of response times. In contrast to previous research, we found no empirical evidence for question order effects impacting people’s responses. Even though there were no response time differences between the question order conditions, the first question always took longer to respond to than the remaining one. Overall, our findings indicate the robustness of questions on political solidarities against question order effects. One potential explanation is that people have comparatively strong (or crystallized) attitudes when it comes to political solidarities.

Congruence on policies between political parties and voters is a frequently assumed re­quirement for democracy. To be able to study this, we should be able to calculate accurate and precise measures of policy congruence in political systems. This could then tell us more about the political system we study, and the "distances" that exist between parties and voters on either issues or broader ideological dimensions. Here, I draw on experimental data from a Voting Advice Application to show that the wording of the issues can influence the degree of congruence one measures. Yet, this comes with the complication that this influence depends on the type of issue, the characteristics of the voters themselves, and the party the congruence is calculated with. These findings should serve as a warning for those who aim to measure congruence that even minor changes in question-wording can (but do not have to) cause relatively large changes in congruence, especially when many parties are involved and the differences between the congruences are small.

Knowledge questions frequently are used in survey research to measure respondents’ topic-related cognitive ability and memory. However, in self-administered surveys, respondents can search external sources for additional information to answer a knowledge question correctly. In this case, the knowledge question measures accessible and procedural memory. Depending on what the knowledge question aims at, the validity of this measure is limited. Thus, in this study, we conducted three experiments using a web survey to investigate the effects of task difficulty, respondents’ ability, and respondents’ motivation on the likelihood of searching external sources for additional information as a form of over-optimizing response behavior when answering knowledge questions. We found that the respondents who are highly educated and more interested in a survey are more likely to invest additional efforts to answer knowledge questions correctly. Most importantly, our data showed that for these respondents, a more difficult question design further increases the likelihood of over-optimizing response behavior.

With the increasing use of smartphones in web surveys, considerable efforts have been devoted to reduce the amount of screen space taken up by questions. An emerging stream of research in this area is aimed at optimizing the design elements of rating scales. One suggestion that has been made is to completely abandon verbal labels and use only numeric labels instead. This approach deliberately shifts the task of scale interpretation to the respondents and reduces the information given to them with an intention to reduce their response burden while still preserving the scale meaning. Following prior research, and by drawing on the established model of the cognitive response process, we critically tested these assumptions. Based on a web survey experiment, we found that omitting verbal labels and using only numeric labels instead pushed respondents to focus their responses on the endpoints of a rating scale. Moreover, drawing on response time paradata, we showed that their response burden was not reduced when presented with only numeric labels; quite the opposite was the case, especially when respondents answered the scale with only numeric labels for the first time, which seemed to entail additional cognitive effort. Based on our findings, we advise against using only numeric labels for rating scales in web surveys.

The collection and use of paradata is gaining in importance, especially in web surveys. From a research ethics’ perspective, respondents should be asked for their consent to the collection and use of web paradata. In this context, a positive attitude toward paradata use has been deemed to be a prerequisite for respondents’ willingness to share their paradata. The present study aimed to identify factors affecting respondents’ attitudes toward paradata use. Our findings revealed that adequately informing survey respondents about what paradata are and why they are used was an important determinant of their attitudes toward paradata use. Moreover, we found that respondents with a positive attitude toward the survey were more likely to have a favorable opinion of paradata use. Our findings suggest that a thorough understanding of the factors that contribute to a positive attitude toward paradata use provides the basis for improved paradata consent procedures, which in turn will increase rates of consent to paradata use and help attenuate the risk of consent bias in web surveys.

To collect high-quality data, survey designers aim to develop questions that each respondent can understand as intended. A critical step to this end is designing questions that minimize the respondents' burden by reducing the cognitive effort required to comprehend and answer them. One promising technique for identifying problematic survey questions is eye tracking. This article investigates the potential of eye movements and pupil dilations as indicators for evaluating survey questions. Respondents were randomly assigned to either a problematic or an improved version of six experimental questions. By analyzing fixation times, fixation counts, and pupil diameters, it was examined whether these parameters could be used to distinguish between the two versions. Identifying the improved version worked best by comparing fixation times, whereas in most cases, it was not possible to differentiate between versions on the basis of pupil data. Limitations and practical implications of the findings are discussed.

Methodological studies usually gauge response quality in narrative open-ended questions with the proportion of nonresponse, response length, response time, and number of themes mentioned by respondents. However, not all of these indicators may be comparable and appropriate for evaluating open-ended questions in a cross-national context. This study assesses the cross-national appropriateness of these indicators and their potential bias. For the analysis, we use data from two web surveys conducted in May 2014 with 2,685 respondents and in June 2014 with 2,689 respondents and compare responses from Germany, Great Britain, the United States, Mexico, and Spain. We assess open-ended responses for a variety of topics (e.g., national identity, gender attitudes, and citizenship) with these indicators and evaluate whether they arrive at similar or contradictory conclusions about response quality. We find that all indicators are potentially biased in a cross-national context due to linguistic and cultural reasons and that the bias differs in prevalence across topics. Therefore, we recommend using multiple indicators as well as items covering a range of topics when evaluating response quality in open-ended questions across countries.

Cognitive online pretests have, in recent years, become recognized as a promising tool for evaluating questions prior to their use in actual surveys. While existing research has shown that cognitive online pretests produce similar results to face-to-face cognitive interviews with regard to the problems detected and the item revisions suggested, little is known about the ideal design of a cognitive online pretest. This study examines whether the number of open-ended probing questions asked during a cognitive online pretest has an effect on the quality and depth of respondents' answers as well as on respondents' satisfaction with the survey. We conducted an experiment in which we varied the number of open-ended probing questions that respondents received during a cognitive online pretest. The questionnaire consisted of 26 survey questions, and respondents received either 13 probing questions (n = 120, short version) or 21 probing questions (n = 120, long version). The findings suggest that asking a greater number of open-ended probes in a cognitive online pretest does not undermine the quality of respondents' answers represented by the response quality indicators: (1) amount of probe nonresponse, (2) number of uninterpretable answers, (3) number of dropouts, (4) number of words, (5) response times, and (6) number and type of themes covered by the probes. Furthermore, the respondents' satisfaction with the survey is not affected by the number of probes being asked.

In light of issues such as increasing unit nonresponse in surveys, several studies argue that social media sources such as Twitter can be used as a viable alternative. However, there are also a number of shortcomings with Twitter data such as questions about its representativeness of the wider population and the inability to validate whose data you are collecting. A useful way forward could be to combine survey and Twitter data to supplement and improve both. To do so, consent within a survey is first needed. This study explores the consent decisions in three large representative surveys of the adult British population to link Twitter data to survey responses and the impact that demographics and survey mode have on these outcomes. Findings suggest that consent rates for data linkage are relatively low, and this is in part mediated by mode, where face-to-face surveys have higher consent rates than web versions. These findings are important to understand the potential for linking Twitter and survey data but also to the consent literature generally.

Identifying inattentive respondents in self-administered surveys is a challenging goal for survey researchers. Instructed response items (IRIs) provide a measure for inattentiveness in grid questions that is easy to implement. The present article adds to the sparse research on the use and implementation of attention checks by addressing three research objectives. In a first study, we provide evidence that IRIs identify respondents who show an elevated use of straightlining, speeding, item nonresponse, inconsistent answers, and implausible statements throughout a survey. Excluding inattentive respondents, however, did not alter the results of substantive analyses. Our second study suggests that respondents' inattentiveness partially changes as the context in which they complete the survey changes. In a third study, we present experimental evidence that a mere exposure to an IRI does not negatively or positively affect response behavior within a survey. A critical discussion on using IRI attention checks concludes this article.

Attention checks detect inattentiveness by instructing respondents to perform a specific task. However, while respondents may correctly process the task, they may choose to not comply with the instructions. We investigated the issue of noncompliance in attention checks in two web surveys. In Study 1, we measured respondents’ attitudes toward attention checks and their self-reported compliance. In Study 2, we experimentally varied the reasons given to respondents for conducting the attention check. Our results showed that while most respondents understand why attention checks are conducted, a nonnegligible proportion of respondents evaluated them as controlling or annoying. Most respondents passed the attention check; however, among those who failed the test, 61% seem to have failed the task deliberately. These findings reinforce that noncompliance is a serious concern with attention check instruments. The results of our experiment showed that more respondents passed the attention check if a comprehensible reason was given.

Complex interactive test items are becoming more widely used in assessments. Being computer-administered, assessments using interactive items allow logging time-stamped action sequences. These sequences pose a rich source of information that may facilitate investigating how examinees approach an item and arrive at their given response. There is a rich body of research leveraging action sequence data for investigating examinees’ behavior. However, the associated timing data have been considered mainly on the item-level, if at all. Considering timing data on the action-level in addition to action sequences, however, has vast potential to support a more fine-grained assessment of examinees’ behavior. We provide an approach that jointly considers action sequences and action-level times for identifying common response processes. In doing so, we integrate tools from clickstream analyses and graph-modeled data clustering with psychometrics. In our approach, we (a) provide similarity measures that are based on both actions and the associated action-level timing data and (b) subsequently employ cluster edge deletion for identifying homogeneous, interpretable, well-separated groups of action patterns, each describing a common response process. Guidelines on how to apply the approach are provided. The approach and its utility are illustrated on a complex problem-solving item from PIAAC 2012.

IR-tree models assume that categorical item responses can best be explained by multiple response processes. In the present article, guidelines are provided for the development and interpretation of IR-tree models. In more detail, the relationship between a tree diagram, the model equations, and the analysis on the basis of pseudo-items is described. Moreover, it is shown that IR-tree models do not allow conclusions about the sequential order of the processes, and that mistakes in the model specification can have serious consequences. Furthermore, multiple-group IR-tree models are presented as a novel extension of IR-tree models to data from heterogeneous units. This allows, for example, to investigate differences across countries or organizations with respect to core parameters of the IR-tree model. Finally, an empirical example on organizational commitment and response styles is presented.

Response probabilities are used in adaptive and responsive survey designs to guide data collection efforts, often with the goal of diversifying the sample composition. However, if response probabilities are also correlated with measurement error, this approach could introduce bias into survey data. This study analyzes the relationship between response probabilities and data quality in grid questions. Drawing on data from the probability-based GESIS panel, we found low propensity cases to more frequently produce item nonresponse and nondifferentiated answers than high propensity cases. However, this effect was observed only among long-time respondents, not among those who joined more recently. We caution that using adaptive or responsive techniques may increase measurement error while reducing the risk of nonresponse bias.

In mobility panels, respondents may use a strategy of soft-refusal to lower their response burden, e.g. by claiming they did not leave their house even though they actually did. Soft-refusal leads to poor data quality and may complicate research, e.g. focused on people with actual low mobility. In this study we develop three methods to detect the presence of soft-refusal in mobility panels, based on respectively (observed and predicted) out-of-home activity, straightlining and speeding. For each indicator, we explore the relation with reported immobility and panel attrition. The results show that speeding and straightlining in a questionnaire is strongly related to reported immobility in a (self-reported) travel diary. Using a binary logit model, respondents who are predicted to leave their home but report no trips are identified as possible soft refusers. To reveal different patterns of soft-refusal and assess how these patterns influence the probability to drop out of the panel, a latent transition model is estimated. The results show four behavioral patterns with respect to soft-refusal ranging from a large class of reliable respondents who score positive on all three soft-refusal indicators, to a small ‘high-risk’ class of respondents who score poorly on all indicators. This ‘high-risk’ group also reports the highest immobility and has the highest attrition rate. The model also shows that respondents who do not drop out of the panel, tend to stay in the same behavioral pattern over time. The amount of soft-refusal expressed by a respondent therefore seems to be a stable behavioral trait., Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public., Transport and Planning, Transport and Logistics, Industrial Design Engineering, Corporate Innovations, Transport and Planning

Public Significance Statement Increasingly, researchers use frequently repeated self-report measures to assess individuals' experiences in the context of their daily lives. We find signs of changes in response behavior, affect, and emotional awareness that are triggered by these frequent assessments. The possibility of such reactive changes over time is important to consider when collecting repeated self-report measures in daily life.Since the introduction of the experience sampling method (ESM), there have been concerns that the repeated assessments typically related to this method may alter the behavior, thoughts, or feelings of participants. Previous studies have offered mixed results with some studies reporting reactive changes, while others failed to find such effects. Our aim was to investigate under which circumstances ESM induces reactive effects. Students (N = 151) were randomly assigned to receive a questionnaire containing 30 or 60 items three, six, or nine times per day for 14 days. A random sample of 50 participants took part in qualitative interviews after the end of the data collection. We investigated changes over time in the data, while taking into account the sampling protocol and characteristics of participants, and analyzed qualitative reports of measurement reactivity. Decreases in completion time, within-person variance of ratings and subjective reports of habituation point toward the existence of a habituation period. While participants reported increases in emotional awareness in interviews, ESM measures indicated a decrease in emotional awareness over time. Changes in behavior were rare in quantitative and qualitative reports. Positive affect was decreasing over time in the ESM data, and various changes in affect, emotion regulation, and thoughts were reported in interviews. Individual characteristics and sampling protocol had inconsistent effects on changes over time. The results suggest that ESM induces changes in within-person variability

The trial production of natural gas hydrate reservoirs remains poor. Reasonable reservoir reconstruction, which can improve formation permeability, is an important approach to increasing the efficiency and enhancing production. In this work, water jet slotting is proposed to reconstruct an natural gas hydrate reservoir near a wellbore. The spatial slots formed by water jet slotting not only directly constitute high-permeability channels, but also generate disturbances to the surrounding in-situ sediment. Water jet slotting disturbances to nearby sediment was investigated using a three dimensional flow-structure coupling model to evaluate the proposed reconstruction method. The reservoir at the SH2 site in the Shenhu area of the South China Sea was used as the reference. A horizontal slotting arrangement along the vertical well was adopted. The results demonstrate that water jet slotting can change the primary stress state of the sediment around the wellbore, and generate a dominant stress relaxation zone and small stress concentration zone. Within the stress relaxation zone, the in-situ compressive stress was remarkably reduced or even transformed into tensile stress, accompanied by sediment displacement and volumetric expansion strain. This is conducive to loosening the sediment around the wellbore and improving the permeability characteristics. In addition, the influence of the water jet slotting parameters including slot radius, spacing, and number on disturbances to the nearby sediment was studied. Reservoir responses to water jet slotting under balanced and unbalanced bottom-hole pressures were compared and analyzed. This study provides a reference for natural gas hydrate reservoir reconstruction using water jet slotting., Engineering Thermodynamics

Since the introduction of the experience sampling method (ESM), there have been concerns that the repeated assessments typically related to this method may alter the behavior, thoughts, or feelings of participants. Previous studies have offered mixed results with some studies reporting reactive changes, while others failed to find such effects. Our aim was to investigate under which circumstances ESM induces reactive effects. Students ( N = 151) were randomly assigned to receive a questionnaire containing 30 or 60 items three, six, or nine times per day for 14 days. A random sample of 50 participants took part in qualitative interviews after the end of the data collection. We investigated changes over time in the data, while taking into account the sampling protocol and characteristics of participants, and analyzed qualitative reports of measurement reactivity. Decreases in completion time, within-person variance of ratings and subjective reports of habituation point toward the existence of a habituation period. While participants reported increases in emotional awareness in interviews, ESM measures indicated a decrease in emotional awareness over time. Changes in behavior were rare in quantitative and qualitative reports. Positive affect was decreasing over time in the ESM data, and various changes in affect, emotion regulation, and thoughts were reported in interviews. Individual characteristics and sampling protocol had inconsistent effects on changes over time. The results suggest that ESM induces changes in within-person variability, completion times, affect, or emotional awareness over time. Further research is needed to explore whether observed changes affect the validity of ESM data.

As response rates continue to decline, the need to learn more about the survey participation process remains an important task for survey researchers. Search engine data may be one possible source for learning about what information some potential respondents are looking up about a survey when they are making a participation decision. In the present study, we explored the potential of search engine data for learning about survey participation and how it can inform survey design decisions. We drew on freely available Google Trends (GT) data to learn about the use of Google Search with respect to our case study: participation in the Family Research and Demographic Analysis (FReDA) panel survey. Our results showed that some potential respondents were using Google Search to gather information on the FReDA survey. We also showed that the additional data obtained via GT can help survey researchers to discover topics of interest to respondents and geographically stratified search patterns. Moreover, we introduced different approaches for obtaining data via GT, discussed the challenges that come with these data, and closed with practical recommendations on how survey researchers might utilize GT data to learn about survey participation., Da Response Rates in Umfragen immer weiter sinken, bleibt es eine wichtige Aufgabe für methodische Forschung, mehr über den Teilnahmeprozess zu lernen. Search Engine Data können eine mögliche Quelle sein, um herauszufinden, welche Informationen potenzielle Befragte über eine Umfrage suchen, wenn sie eine Teilnahmeentscheidung treffen. In der vorliegenden Studie untersuchten die Autor*innen das Potenzial von Suchmaschinendaten, um etwas über die Teilnahme an Umfragen zu erfahren und wie diese Daten in Entscheidungen über die Gestaltung von Umfragen einfließen können. Sie stützten sich auf frei verfügbare Daten von Google Trends (GT), um mehr über die Nutzung der Google-Suche in Bezug auf eine Fallstudie zu erfahren: die Teilnahme an der Panel-Umfrage Family Research and Demographic Analysis (FReDA). Die Ergebnisse zeigten, dass einige potenzielle Befragte die Google-Suche nutzten, um Informationen über die FReDA-Umfrage einzuholen. Die Autoren zeigen ebenfalls, dass die über GT gewonnenen zusätzlichen Daten den Umfrageforschern helfen können, Themen, die für die Befragten von Interesse sind, sowie geografisch geschichtete Suchmuster zu entdecken. Darüber hinaus stellen die Autoren verschiedene Ansätze für die Beschaffung von Daten über GT vor, erörtern die mit diesen Daten verbundenen Herausforderungen und geben abschließend praktische Empfehlungen, wie Umfrageforscher GT-Daten nutzen können, um mehr über die Teilnahme an ihren Umfragen zu erfahren.

Although list-style open-ended questions generally help us gain deeper insights into respondents’ thoughts, opinions, and behaviors, the quality of responses is often compromised. We tested a dynamic and a follow-up design to motivate respondents to give higher quality responses than with a static design, but without overburdening them. Our results showed that a follow-up design achieved longer responses with more themes and theme areas than a static design. In contrast, the dynamic design produced the shortest answers with the fewest themes and theme areas. No differences in item nonresponse and only minor differences in additional response burden were found among the three list-style designs. Our study shows that design features and timing are crucial to clarify the desired response format and motivate respondents to give high-quality answers to list-style open-ended questions.

Preventing panel members from attriting is a fundamental challenge for panel surveys. Research has shown that response behavior in earlier waves (response or nonresponse) is a good predictor of panelists' response behavior in upcoming waves. However, response behavior can be described in greater detail by considering the time until the response is returned. In the present study, we investigated whether respondents who habitually return their survey late and respondents who switch between early and late response in multiple waves are more likely to attrit from a panel. Using data from the GESIS Panel, we found that later response is related to a higher likelihood of attrition (AME = 0.087) and that response-time stability is related to a lower likelihood of attrition (AME = −0.013). Our models predicted most cases of attrition; thus, survey practitioners could potentially predict future attriters by applying these models to their own data.

In this article, we examine two types of item nonresponse in a face-to-face population survey: 'don’t know' (DK) and 'item refusal' (REF). Based on the cognitive model of survey response, the theory of survey satisficing and previous research, we derive explanatory variables on three levels: question, respondent and interviewer characteristics. The results of our cross-classified model show that while the two levels question and respondents' characteristics affected both types of item nonresponse, interviewer characteristics affected only DK answers. Our results also confirm that DK and REF are substantially different item nonresponse types resulting from distinguishable disruptions of the cognitive response process. Since most results are in line with prior theoretical predictions, they suggest that survey practitioners are well-advised by continuing to follow the large body of practical guidance derived from the theories tested here.

In recent years, conjoint experiments have been in vogue across the social sciences. A rea­son for the conjoint experiments’ popularity is that they allow researchers to estimate the causal effects of many components of stimuli simultaneously. However, for conjoint experi­ments to produce valid results, respondents need to be able to process and understand the wide range of dimensions presented to them in the experiment. If the information process­ing is too demanding or too complicated, respondents are likely to turn to satisficing strate­gies, leading to poorer data quality and subsequently decreasing the researcher’s ability to make accurate causal inferences. One factor that may lead to the adoption of satisficing strategies is the presentation format used for the conjoint experiment (i.e., presenting the information within a text paragraph or a table). In the present paper, a direct replication of the single conjoint presentation format experiment described in Shamon, Dülmer, and Giza's (2019) paper in Sociological Methods & Research is presented, and extending their work to paired conjoint experiment. The results of the direct replication showed that re­spondents evaluated the questionnaire more favorably when reading the table format but were, on the other hand, less likely to participate in subsequent panel waves. Albeit the number of break-offs, refusals, and non-responses did not differ between the two formats, respondents who saw the table format evaluated the scenarios with more consistency and less dimension reduction, thus favoring the table presentation format. For paired conjoint experiments, the presentation format did not affect survey evaluations or panel participa­tion but the table format heavily outperformed the text format on every data quality mea­sure except for dimension reduction. Conceptually, albeit not directly replicating the find­ings in Shamon, Dülmer, and Giza (2019), the present manuscript concludes that the table format appears pr

A question asking for respondents' sex is one of the standard sociodemographic characteristics collected in a survey. Until now, it typically consisted of a simple question (e.g., "Are you…?") with two answer categories ("male" and "female"). In 2019, Germany implemented the additional sex designation divers for intersex people. In survey methodology, this has led to an ongoing discussion how to include a third category in questionnaires. We investigate respondents' understanding of the third category, and whether introducing it affects data quality. Moreover, we investigate the understanding of the German term Geschlecht for sex and gender. To answer our research questions, we implemented different question wordings asking for sex/gender in a non-probability-based online panel in Germany and combined them with open-ended questions. Findings and implications for surveys are discussed., Der Geschlechtseintrag divers wurde zu Beginn 2019 in Deutschland für intersexuelle Menschen eingeführt, also für Menschen, die auf Basis ihrer primären oder sekundären Geschlechtsmerkmale weder eindeutig männlich noch weiblich sind. Im März 2019 haben die Autorinnen in einer Web Probing-Studie Daten zum Verständnis der Kategorie erhoben und ausgewertet. Die Ergebnisse zeigen, dass etwa die Hälfte der Befragten ein grundsätzliches Verständnis des Begriffs divers haben, ihn also beispielsweise als "nicht männlich oder weiblich" definieren können. Allerdings zeigte sich auch, dass einige Befragte divers als Kategorie für Transgender hielten, also beispielsweise für Menschen, die biologisch gesehen Männer sind, sich aber als Frau begreifen. Diese Assoziation war sogar wesentlich häufiger als das korrekte Verständnis als Geschlechtseintrag für Intersexualität. Dieser Befund deckt sich mit einem weiteren Schwerpunkt des Artikels, der zeigt, dass die meisten Befragten keinen Unterschied zwischen ihrem biologischen und gefühlten Geschlecht machen. Dies haben wir in einer Auswertung der Begriffe "Geschlecht", "offiziell eingetragenes Geschlecht" und "Geschlechtsidentität" untersucht. Des Weiteren zeigt der Artikel, dass der Anteil an negativen Reaktionen auf den Begriff, in Form von Survey Abbruch oder feindseligen Kommentaren über nicht-binäre Geschlechtszugehörigkeit, gering ist.

Survey scientists increasingly face the problem of high-dimensionality in their research as digitization makes it much easier to construct high-dimensional (or "big") data sets through tools such as online surveys and mobile applications. Machine learning methods are able to handle such data, and they have been successfully applied to solve predictive problems. However, in many situations, survey statisticians want to learn about causal relationships to draw conclusions and be able to transfer the findings of one survey to another. Standard machine learning methods provide biased estimates of such relationships. We introduce into survey statistics the double machine learning approach, which gives approximately unbiased estimators of parameters of interest, and show how it can be used to analyze survey nonresponse in a high-dimensional panel setting. The double machine learning approach here assumes unconfoundedness of variables as its identification strategy. In high-dimensional settings, where the number of potential confounders to include in the model is too large, the double machine learning approach secures valid inference by selecting the relevant confounding variables., Wissenschaftlerinnen und Wissenschaftler im Feld "Umfrageforschung" sehen sich in ihrer Forschung zunehmend mit dem Problem der hohen Dimensionalität konfrontiert, da es durch die Digitalisierung viel einfacher geworden ist, hochdimensionale (oder "große") Datensätze mit Hilfe von Tools wie Online-Umfragen und mobilen Anwendungen zu erstellen. Methoden des maschinellen Lernens sind in der Lage, mit solchen Daten umzugehen, und sie wurden bereits erfolgreich zur Lösung von Vorhersageproblemen eingesetzt. In vielen Situationen möchten Umfragestatistiker*innen jedoch kausale Zusammenhänge erkennen, um Schlussfolgerungen ziehen und die Ergebnisse einer Umfrage auf eine andere übertragen zu können. Standardmethoden des maschinellen Lernens liefern verzerrte Schätzungen solcher Zusammenhänge. Die Autor*innen führen in die Umfragestatistik den Ansatz des doppelten maschinellen Lernens ein, der annähernd unverzerrte Schätzer der interessierenden Parameter liefert, und zeigen, wie er zur Analyse von Umfrage-Nonresponse in einem hochdimensionalen Panel-Umfeld eingesetzt werden kann.

Invitation letters to web surveys often contain information on how long it will take to complete a web survey. When the stated length in an invitation of a survey is short, it could help to convince respondents to participate in the survey. When it is long respondents may choose not to participate, and when the actual length is longer than the stated length there may be a risk of dropout. This paper reports on an Randomised Control Trial (RCT) conducted in a cross-sectional survey conducted in the Netherlands. The RCT included different version of the stated length of a survey and inclusion of a Quick Response (QR) code as ways to communicate to potential respondents that the survey was short or not. Results from the RCT show that there are no effects of the stated length on actual participation in the survey, nor do we find an effect on dropout. We do however find that inclusion of a QR code leads respondents to be more likely to use a smartphone, and find some evidence for a different composition of our respondent sample in terms of age.

Although waiting room surveys are frequently conducted, methodological studies on this topic are scarce. Behaviour of surveyors in waiting rooms can easily be controlled, and these surveys also allow for collection of paradata; relevant information on the circumstances of a request to participate in survey research. In this paper, we present the results of an experiment systematically manipulating surveyor’s handshakes and verbal introduction of their names. Patients scheduled for radiological examinations were approached to take part in a survey. An observer noted circumstances in the waiting room (CT or MRI) such as the number of people present. In the CT waiting room, willingness to participate was higher when no other people were filling out the survey than when there were other people filling out the survey. Thus, scarcity effects seemed to play a major role in the decision to participate. In addition, a patient waiting alone was more likely to fully complete the questionnaire, than patients accompanied by one or more caregivers. There was no effect of the surveyor’s handshake or verbal name introduction on survey participation, which is a fortunate outcome in light of social distances measures fighting COVID-19.

Survey researchers are often confronted with the question of how long to set the length of the field period. Longer fielding time might lead to greater participation yet requires survey managers to devote more of their time to data collection efforts. With the aim of facilitating the decision about the length of the field period, we investigated whether a longer fielding time reduces the risk of nonresponse bias to judge whether field periods can be ended earlier without endangering the performance of the survey. By using data from six waves of a probability-based mixed-mode (online and mail) panel of the German population, we analyzed whether the risk of nonresponse bias decreases over the field period by investigating how day-by-day coefficients of variation develop during the field period. We then determined the optimal cut-off points for each mode after which data collection can be terminated without increasing the risk of nonresponse bias and found that the optimal cut-off points differ by mode. Our study complements prior research by shifting the perspective in the investigation of the risk of nonresponse bias to panel data as well as to mixed-mode surveys, in particular. Our proposed method of using coefficients of variation to assess whether the risk of nonresponse bias decreases significantly with each additional day of fieldwork can aid survey practitioners in finding the optimal field period for their mixed-mode surveys.

Between April 2019 and September 2020, 23,001 people aged 15 or over responded to questions about their health and living conditions for the German Health Update (GEDA 2019/2020-EHIS). The results are representative of the German resident population aged 15 or above. The response rate was 21.6%. The study used a questionnaire based on the third wave of the European Health Interview Survey (EHIS), which was carried out in all EU member states. EHIS consists of four modules on health status, health care provision, health determinants, and socioeconomic variables. The data are collected in a harmonised manner and therefore have a high degree of international comparability. They constitute an important source of information for European health policy and health reporting and are made available by the Statistical Office of the European Union (Eurostat). They also form the basis of the Federal Health Reporting undertaken in Germany. Data collection began in April 2019, just under a year before the beginning of the SARS-CoV-2 pandemic, and continued into its initial phase, as of March 2020. As such, data from the current GEDA wave can also be used to conduct research into the health impact of the SARS-CoV-2 pandemic.

In der Studie Gesundheit in Deutschland aktuell (GEDA 2019/2020-EHIS) beantworteten 23.001 Menschen im Alter ab 15 Jahren zwischen April 2019 und September 2020 Fragen zur Gesundheit und zur Lebenssituation. Die Ergebnisse sind repräsentativ für die in Deutschland lebende Wohnbevölkerung ab 15 Jahren. Die Responserate lag bei 21,6 %. Die Fragebogeninhalte basieren auf der dritten Welle der Europäischen Gesundheitsbefragung (European Health Interview Survey, EHIS), die in allen EU-Mitgliedstaaten durchgeführt wurde. Sie umfasst die vier Module Gesundheitszustand, Gesundheitsversorgung, Gesundheitsdeterminanten und sozioökonomische Variablen. Die harmonisiert erhobenen EHIS-Daten besitzen ein hohes Maß an internationaler Vergleichbarkeit. Sie stellen eine wichtige Informationsgrundlage für die europäische Gesundheitspolitik und -berichterstattung dar und werden vom Statistischen Amt der Europäischen Union (Eurostat) zur Verfügung gestellt. Die Daten sind Grundlage für die Gesundheitsberichterstattung des Bundes. Der Zeitraum der Datenerhebung ab April 2019 berücksichtigte knapp ein Jahr vor der SARS-CoV-2-Pandemie und fiel dann ab März 2020 in die Anfangsphase der Pandemie. Somit stehen mit der aktuellen GEDA-Welle Daten für die Erforschung von gesundheitlichen Auswirkungen im zeitlichen Zusammenhang mit der SARS-CoV-2-Pandemie zur Verfügung.

Since 1996, reports on cohabiting same-sex partnerships have been collected in the German Microcensus. However, it is unclear how reliable these reports are. Compared with other data sources, the Microcensus shows only a small number of cohabiting same-sex couples (less than 0.5% of all cohabiting couples in 2016), so under-reporting is assumed. But, because the "true" values are unknown, it is difficult to determine whether under-reporting is actually occurring. In this paper a procedure is proposed where the response behaviour of respondents is analysed depending on the composition of their household. It was found that non-response to the question about a partner in the household is highest among respondents in whose household there is a possible same-sex partner. This indicates under-reporting of cohabiting same-sex couples, and this under-reporting decreased only slightly, at most, over the period considered here (1996 to 2016). This is not the case for registered same-sex partnerships: a comparison with the Census shows that these are reliably recorded in the German Microcensus.

In cross-sectional surveys, as well as in longitudinal panel studies, systematic gender dif­ferences in survey participation are routinely observed. Since there has been little research on this issue, this study seeks to reveal this association for web-based online surveys and computer-assisted telephone interviews in the context of a sequential mixed-mode design with a push-to-web method. Based on diverse versions of benefit-cost theories relating to deliberative and heuristic decision-making, several hypotheses are deduced and then tested by longitudinal data in the context of a multi-wave panel study on the educational and occu­pational trajectories of juveniles. Employing event history data on the survey participation of young panelists living in German-speaking cantons in Switzerland and matching them with geographical data at the macro level and panel characteristics at the meso level, none of the hypotheses is confirmed empirically. It is concluded that indirect measures of an individual’s perceptions of a situation, and of the benefits and costs as well as the process and mechanisms of the decision relating to survey participation, are insufficient to explain this gender difference. Direct tests of these theoretical approaches are needed in future.

Measuring change over time is one of the main purposes of longitudinal surveys. With an increase in the use of web as a mode of data collection it is important to assess whether the web mode differs from other modes with respect to the number of changes and events that are captured. We examine whether telephone and web data collection modes are compara­ble with respect to measuring changes over time or experiencing events. Using experimen­tal data from a two-wave pilot of the Swiss Household Panel, we investigate this question for several variables in the domain of work and family. We find differences for the work-related variables, with web respondents more likely to re­port changes. These differences do not disappear once the socio-demographic composition of the sample is taken into consideration. This suggests that these differences are not driven by observed different characteristics of the respondents who may have self-selected into one or the other mode. Contrary to work-related variables, a termination of a relationship was more common in the telephone group. This shows that one mode does not necessarily measure more change or events than another; it may depend on the variable in question. In addition, the difference in the protocol mattered: a web respondent in a household that participated fully by web sometimes differed from a web respondent in a household that had a household interview by phone. Nonetheless, the telephone group differed more from the various web protocols that the web protocols among themselves. With more household panel surveys introducing web questionnaires in combination with more traditional face-to-face and telephone interviews, this study adds to our understand­ing of the potential consequences of mixing modes with respect to longitudinal data analy­sis.

It is sometimes observed and frequently assumed that top incomes in household surveys worldwide are poorly measured and that this problem biases the measurement of income inequality. This paper tests this assumption and compares the performance of reweighting and replacing methods designed to correct inequality measures for top-income biases generated by data issues such as unit or item non-response. Results for the European Union’s Statistics on Income and Living Conditions survey indicate that survey response probabilities are negatively associated with income and bias the measurement of inequality downward. Correcting for this bias with reweighting, the Gini coefficient for Europe is revised upwards by 3.7 percentage points. Similar results are reached with replacing of top incomes using values from the Pareto distribution when the cut point for the analysis is below the 95th percentile. For higher cut points, results with replacing are inconsistent suggesting that popular parametric distributions do not mimic real data well at the very top of the income distribution.

Previous research has shown that question characteristics, such as the shape of rating scales, can affect how respondents interpret and respond to questions. For example, earlier studies reported different response distributions for questions employing rating scales in the form of a ladder and in the form of a pyramid. The current experiment, implemented in a probability-based online panel (N = 4,377), re-visits and extends this research by examining how the two graphical layouts (ladder vs. pyramid) affect response behavior and data quality of a question on subjective social stratification. In line with the earlier results, we found that respondents rated their social status lower in the pyramid than in the ladder condition. No differences between the two layouts were found regarding response effort, however, the ladder layout was associated with higher criterion validity. Therefore, we recommend employing the ladder layout when measuring subjective social stratification.

The ever-growing number of respondents completing web surveys via smartphones is paving the way for leveraging technological advances to improve respondents’ survey experience and, in turn, the quality of their answers. Smartphone surveys enable researchers to incorporate audio and voice features into web surveys, that is, having questions read aloud to respondents using pre-recorded audio files and collecting voice answers via the smartphone’s microphone. Moving from written to audio and voice communication channels might be associated with several benefits, such as humanizing the communication process between researchers and respondents. However, little is known about respondents’ willingness to undergo this change in communication channels. Replicating and extending earlier research, we examine the extent to which respondents are willing to use audio and voice channels in web surveys, the reasons for their (non)willingness, and respondent characteristics associated with (non)willingness. The results of a web survey conducted in a nonprobability online panel in Germany (N = 2146) reveal that more than 50% of respondents would be willing to have the questions read aloud (audio channel) and about 40% would also be willing to give answers via voice input (voice channel). While respondents mostly name a general openness to new technologies for their willingness, they mostly name preference for written communication for their nonwillingness. Finally, audio and voice channels in smartphone surveys appeal primarily to frequent and competent smartphone users as well as younger and tech-savvy respondents.

Survey research is still confronted by a trend of increasing nonresponse rates. In this context, several methodological advances have been made to stimulate participation and avoid bias. Yet, despite the growing number of tools and methods to deal with nonresponse, little is known about whether nonresponse biases show similar trends as nonresponse rates and what mechanisms (if any) drive changes in bias. Our article focuses on biases in cohort distributions in the U.S. and German general social surveys from 1980 to 2012 as one of the key variables in the social sciences. To supplement our cross-national comparison of these trends, we decompose changes into within-cohort change (WCC) and between-cohort change. We find that biases in cohort distributions have remained relatively stable and at a relatively low level in both countries. Furthermore, WCC (i.e., survey climate) accounts for the major part of the change in nonresponse bias.

The risk of multitasking is high in online surveys. However, knowledge on the effects of multitasking on answer quality is sparse and based on suboptimal approaches. Research reports inconclusive results concerning the consequences of multitasking on task performance. However, studies suggest that especially sequential-multitasking activities are expected to be critical. Therefore, this study focusses on sequential-on-device-multitasking activities (SODM) and its consequences for data quality. Based on probability-based data, this study aims to reveal the prevalence of SODM based on the javascript function OnBlur, to reflect its determinants and to examine the consequences for data quality. Results show that SODM was detected for 25% of all respondents and that respondent attributes and the device used to answer the survey are related to SODM. Moreover, it becomes apparent that SODM is significantly correlated to data quality measures. Therefore, I propose SODM behavior as a new instrument for researching suboptimal response behavior.

The risk of multitasking is high in online surveys. However, knowledge on the effects of multitasking on answer quality is sparse and based on suboptimal approaches. Research reports inconclusive results concerning the consequences of multitasking on task performance. However, studies suggest that especially sequential-multitasking activities are expected to be critical. Therefore, this study focusses on sequential-on-device-multitasking activities (SODM) and its consequences for data quality. Based on probability-based data, this study aims to reveal the prevalence of SODM based on the javascript function OnBlur, to reflect its determinants and to examine the consequences for data quality. Results show that SODM was detected for 25% of all respondents and that respondent attributes and the device used to answer the survey are related to SODM. Moreover, it becomes apparent that SODM is significantly correlated to data quality measures. Therefore, I propose SODM behavior as a new instrument for researching suboptimal response behavior.

Das Verbundprojekt "Studienerfolg und Studienabbruch bei Bildungsausländern in Deutschland im Bachelor- und Masterstudium" (SeSaBa) wurde vom Bundesministerium für Bildung und Forschung (BMBF) im Rahmen der Förderlinie "Studienerfolg und Studienabbruch I" gefördert und widmete sich der Frage nach den soziologischen und psychologischen Determinanten des Studienerfolgs bei internationalen Studierenden. Zur Beantwortung der Forschungsfragen wurde der International Student Survey entwickelt. Dieser Methodenbericht beschreibt die quantitative Entwicklung und Konzeption (z.B. die Entwicklung der Fragebögen, das Incentivierungskonzept) des International Student Surveys. Weiterhin dokumentiert er dessen Durchführung (z.B. den Ablauf der Feldphase) sowie dessen Rücklauf und Repräsentativität. Der International Student Survey umfasst eine Kohorte von internationalen Studierenden im Bachelor- und Masterstudium, die im Wintersemester 2017/2018 ihr Studium in Deutschland im ersten Fachsemester an einer staatlichen oder staatlich anerkannten Hochschule in Deutschland (mit Ausnahme der Musik- und Kunsthochschulen) begonnen haben. Internationale Studierende werden verstanden als Personen, die keine deutsche Staatsbürgerschaft besitzen und an einer Schule im Ausland oder einem deutschen Studienkolleg ihre Hochschulzugangsberechtigung (HZB) erhalten haben.

The paper analyses the sources of income measurement error in surveys with a unique data set. We use the Austrian 2008-2011 waves of the European Union 'Statistics on income and living conditions' survey which provide individual information on wages, pensions and unemployment benefits from survey interviews and officially linked administrative records. Thus, we do not have to fall back on complex two-sample matching procedures like related studies. We empirically investigate four sources of measurement error, namely social desirability, sociodemographic characteristics of the respondent, the survey design and the presence of learning effects. We find strong evidence for a social desirability bias in income reporting, whereas the presence of learning effects is mixed and depends on the type of income under consideration. An Owen value decomposition reveals that social desirability is a major explanation of misreporting in wages and pensions, whereas sociodemographic characteristics are most relevant for mismatches in unemployment benefits.

Open survey questions are often used to evaluate closed questions. However, they can fulfil this function only if there is a strong link between answers to open questions and answers to related closed questions. Using reasons for non-voting reported in the German Longitudinal Election Study 2013, we investigated this link by examining whether the reported reasons for non-voting may be substantive reasons or ex-post legitimations. We tested five theoretically derived hypotheses about respondents who gave, or did not give, a specific reason. Results showed that (a) answers to open questions were indeed related to answers to closed questions and could be used in explanatory turnout models to predict voting behavior, and (b) the relationship between answers to open and closed questions and the predictive power of reasons given in response to the open questions were stronger in the post-election survey (reported behavior) than in the pre-election survey (intended behavior).

Conducting quantitative research (e.g., surveys, a large number of interviews, experiments) with the participation of political elites is typically challenging. Given that a population of political elites is typically small by definition, a particular challenge is obtaining a sufficiently high number of observations and, thus, a certain response rate. This paper focuses on two questions related to this challenge: (1) What are best practices for designing the study? And (2) what are best practices for soliciting the participation of political elites? To arrive at these best practices, we (a) examine which factors explain the variation in response rates across surveys within and between large-scale, multi-wave survey projects by statistically analyzing a newly compiled dataset of 342 political elite surveys from eight projects, spanning 30 years and 58 countries, (b) integrate the typically scattered findings from the existing literature and (c) discuss results from an original expert survey among researchers with experience with such research (n = 23). By compiling a comprehensive list of best practices, systematically testing some widely held believes about response rates and by providing benchmarks for response rates depending on country, survey mode and elite type, we aim to facilitate future studies where participation of political elites is required. This will contribute to our knowledge and understanding of political elites’ opinions, information processing and decision making and thereby of the functioning of representative democracies.

Survey researchers are actively seeking powerful auxiliary data sources capable of correcting for possible nonresponse bias in survey estimates of the general population. While several auxiliary data options exist, concerns about their usefulness for addressing nonresponse bias remain. One underutilized—but potentially rich—source of auxiliary data for nonresponse bias adjustment is federal administrative records. While federal records are routinely used to study nonresponse in countries where it is possible to directly link them (via a unique identifier) to population-based samples, such records are not widely used for this purpose in countries which lack a unique identifier to facilitate direct linkage. In this article, we examine the utility of indirectly linked administrative data from a federal employment database for nonresponse bias adjustment in a general population survey in Germany. In short, we find that the linked administrative variables have stronger correlations with the substantive survey variables than do standard paradata variables and that incorporating linked administrative data in nonresponse weighting adjustments reduces relative nonresponse bias to a greater extent than paradata-only weighting adjustments. However, for the majority of weighted survey estimates, including the administrative variables in the weighting adjustment procedure has minimal impact on the point estimates and their variances. We conclude with a general discussion of these findings and comment on the logistical issues associated with this type of linkage relevant to survey practice.

Statistics on adult education and training (AET) are often considered as insufficient because they fail to deliver a comprehensive and consistent picture of this field of education. This study addresses a specific problem in AET statistics that is varying participation rates of adults in AET depending on underlying data sources. We elaborate potential causes for deviations in survey design and the measurement of participation in sample based AET statistics with reference to the Total Survey Error (TSE) approach. Our analysis compares AET participation rates and patterns from four representative German surveys and reveals substantial differences in participation rates and mixed results for patterns of participation in AET. We find similar relationships for the influence of employment and educational attainment. The relationship with region, gender, and age shows to some extent deviations that conclude in contradictory statements on probabilities of participation. The discussion addresses consequences for the interpretation of survey results on AET participations and draws conclusions for the further development of AET statistics.

Recent years have seen a growing number of studies investigating the accuracy of nonprobability online panels; however, response quality in nonprobability online panels has not yet received much attention. To fill this gap, we investigate response quality in a comprehensive study of seven nonprobability online panels and three probability-based online panels with identical fieldwork periods and questionnaires in Germany. Three response quality indicators typically associated with survey satisficing are assessed: straight-lining in grid questions, item nonresponse, and midpoint selection in visual design experiments. Our results show that there is significantly more straight-lining in the nonprobability online panels than in the probability-based online panels. However, contrary to our expectations, there is no generalizable difference between nonprobability online panels and probability-based online panels with respect to item nonresponse. Finally, neither respondents in nonprobability online panels nor respondents in probability-based online panels are significantly affected by the visual design of the midpoint of the answer scale.

The European Values Study (EVS) was first conducted in 1981 and then repeated in 1990, 1999, 2008, and 2017, with the aim of providing researchers with data to investigate whether European individual and social values are changing and to what degree. The EVS is traditionally carried out as a probability-based face-to-face survey that takes around 1 hour to complete. In recent years, large-scale population surveys such as the EVS have been challenged by decreasing response rates and increasing survey costs. In the light of these challenges, six countries that participated in the last wave of the EVS tested the application of self-administered mixed-modes (Denmark, Finland, Germany, Iceland, the Netherlands, and Switzerland). With the present data brief, we will introduce researchers to the latest wave of the EVS, the implemented mode experiments, and the EVS data releases. In our view, it is pivotal for data use in substantive research to make the reasoning behind design changes and country-specific implementations transparent as well as to highlight new research opportunities.

In this paper, we examine rates of careless responding and reactions towards detection methods (i.e., attention check item and instruction) in an experimental setting based on two different samples. First, we use a quota sample (with monetary incentive), a central data source for internet-based surveys in sociological and political research. Second, we include a voluntary opt-in panel (without monetary incentive) well suited to conduct survey experiments (e.g., factorial surveys). Respondents’ reactions towards the detection items are analyzed by objective, non-reactive indicators (i.e., break-off, item nonresponse, and measurement quality), and two self-report scales. Our reaction analyses reveal that the detection methods that we applied are not only well suited to identify careless respondents but also exert a motivational rather than demotivating influence on respondents’ answer behavior and, hence, contribute to data quality.