Search

This article leverages a five-treatment response mode experiment (paper-only, web-only, sequential web-mail, choice, and choice-plus [choice with a promised incentive for responding online]) that was conducted within a nationally representative survey. Because this survey's sample was drawn from respondents to another nationally representative survey, we have rich frame data that includes multiple indicators of comfort using the internet for our sample members and we can compare their response behavior across two surveys. We find that the paper-only treatment yielded a lower response rate than most of the other treatments, but there were not significant differences between the response rates for the other treatments. Among our mixed-mode treatments, the sequential web-mail treatment had the highest percentage of response by web and the lowest cost per response. When focusing on the subgroups that we expected to be the least—and the most—comfortable with the internet, we found that the paper-only treatment generally performed worse than the others, even among subgroups expected not to be comfortable with the internet. We generally did not find significant differences in the effect of response mode treatment on the response rate or percentage of response by web between the subgroups who were the most and least comfortable with the internet. In terms of the consistency of response mode choice over time, our results suggest that some people respond consistently—but also that response mode preferences are weak enough that they can be influenced by the way in which the modes are offered. We ultimately recommend using a sequential web-mail design to minimize costs while still providing people who cannot or will not respond by web with another response mode option. We also find evidence that there may be a growing lack of interest in responding by paper; more research is needed in this area. [ABSTRACT FROM AUTHOR]

When deciding which modes to offer, researchers consider cost, known respondent contact information, and potential mode effects. For a short survey on employment, we evaluate the effect of adding one of two new electronic data collection modes to a mailed questionnaire. We sent a survey to principals who previously responded to the National Center for Education Statistics' (NCES) National Teacher and Principal Survey (NTPS) asking about their current job status. This questionnaire, known as the Principal Follow-up Survey (PFS), has typically been administered as a short paper form that is mailed to NTPS respondents. In 2022, the PFS introduced two new modes of completion, and principals were randomly assigned to receive: (i) a paper form only; (ii) a paper form, as well as emails with a direct link to complete a web survey; or (iii) a paper form, as well as invitations by text message to complete an automated two-way short message service text survey by responding to texted "yes/no" questions. This article compares overall response rates and time-to-response by mode to determine respondent preferences for completing short surveys. Adding either electronic mode significantly increased response rates and decreased the number of days in which completed surveys were received, compared to offering only a paper questionnaire. Although email and text messages are both forms of electronic communication that may be accessible on a smartphone, the added text message survey resulted in higher response rates than the added web survey. This suggests that respondents interact differently with emails and text messages they receive and that offering an option to complete a survey by text message can increase the speed and efficiency of data collection for short surveys. [ABSTRACT FROM AUTHOR]

In theory, offering each sample member the mode sequence that maximizes their response propensity should increase the response rate and/or reduce the amount of nonresponse follow-up relative to a design that offers all sample members the same mode sequence. However, for this sort of tailoring to be feasible in a cross-sectional survey, it must be possible to use data available prior to data collection (e.g. on the sampling frame) to predict sample members' "mode-sensitivity"—the effect of the offered mode sequence on response propensity. Using data from randomized experiments incorporated into the 2016 and 2019 cycles of the National Household Education Survey, we evaluate whether data appended to an address-based sampling (ABS) frame can accurately predict the sensitivity of household-level response behavior to the initial offer of a paper questionnaire instead of a web instrument and whether a modeled-mode design that tailors the offered mode sequence (web-push vs. paper-only) based on the resulting predictions improves household-level data collection outcomes relative to a uniform web-push design. We find that several characteristics available on the ABS frame show statistically significant interactions with the offered mode sequence in determining the probability of response to initial survey mailings. Consequently, relative to a uniform web-push design, the modeled-mode design increased the response rate to early mailings and reduced the number of mailings required per response. However, the modeled-mode design did not meaningfully increase the final response rate, nor did it lead to a substantial reduction in indicators of nonresponse bias. [ABSTRACT FROM AUTHOR]

This paper builds on prior research on alternative protocols for conducting mixed-mode surveys by mail where respondents are asked to complete either a web questionnaire or a paper questionnaire. We present the results from a 4 × 2 factorial experiment in which the first factor, the survey protocol, varied the modes initially offered (either web alone or both web and paper) as well as the order in which they are offered in subsequent follow-up contacts (either concurrently or sequentially). This factor was crossed with a second factor offering various promised incentive amounts and response options (a $5 prepaid incentive with a promise of either $10 or $20 for response). Our research sought to evaluate the differences between protocols/incentive treatments relative to their costs, response rates, respondent choice of web versus paper response, and consistency with benchmark estimates in other large-scale surveys. Specifically, we introduce a new protocol referred to as choice+ (i.e. choice plus) that presents both web and paper response options concurrently at the initial request, but promises a bonus ($10) incentive for choosing the web option. Results presented are from the 2015 Residential Energy Consumption Survey National Pilot study—an experimental component of the main Residential Energy Consumption Survey sponsored by the U.S. Energy Information Administration. [ABSTRACT FROM AUTHOR]

Relatively low response rates in mixed mode studies remain a concern. Whether targeting protocols to match respondents' likely mode is an effective strategy remains unclear. For those without a clear likely mode, how the details about sequencing influence response rates, mode, field work effort, and potential response bias remain important questions. This article describes a targeted sequential design implemented in a 2016 mixed mode supplement with individuals aged 30 years and older in the Panel Study of Income Dynamics, the longest running national panel study in the United States. Respondents predicted to be likely to respond by web were invited to a web study and sent a paper copy after six weeks (web-first); those likely to respond by paper were also invited to participate by web but told that a paper copy would be sent shortly (signal-andsend paper). An embedded experiment measured the impact of the two protocols among a group of respondents with no clear likely mode. More than 40 percent of individuals with no likely mode are under the age of 40 years, and the group falls between the likely web and paper groups in terms of education and internet use and includes more women and single respondents. Compared to the likely web and paper groups, those with no likely mode had lower response rates and required more fieldwork effort. Among those randomly assigned, the signal-and-send protocol increased response over the web-first protocol from weeks 4 through 7. By week 16, both protocols yielded similar response rates, field effort and distributions of respondent characteristics. Among those responding, cases randomized to web-first were more likely than those randomized to signal-and-send paper to respond by web. We discuss implications for targeted protocols in mixed mode panel surveys. [ABSTRACT FROM AUTHOR]

Multimode data collection has emerged as a common approach for conducting household surveys in the United States. A number of different data collection schemes have been investigated, with an emphasis on collecting as many respondents by the Web prior to going to paper data collection to reduce costs. Despite this, little research has been conducted on the approaches to weighting data from multimode surveys. The typical approach assumes that all respondents should be treated the same regardless of mode even though it is well known that the response patterns by mode vary substantially. We examine an adaptive mode adjustment to address these differences and propose an imbalance measure to help determine the adjustment factor using ideas from responsive design. We then compare the effects of the alternative weighting method in two recent sequential mixed-mode surveys and show it appears to reduce bias while only slightly increasing variances of the estimates. [ABSTRACT FROM AUTHOR]

Given the promise of the web push plus e-mail survey design for providing cost-effective and high-quality data (Patrick et al. 2018 , 2019) as an alternative to a paper-and-pencil mailed survey design for the longitudinal Monitoring the Future (MTF) study, the current study sought to further enhance the web push condition. The MTF sample is based on US nationally representative samples of 12th grade students surveyed annually. The MTF control group for the current study included participants who completed the in-school baseline survey in the 12th grade and were selected to participate in their first follow-up survey in 2017 via mailed surveys (N  = 1,222). A supplementary sample (N = ∼2,450) was assigned to one of the two sequential mixed-mode conditions. Those in condition 1 (N = 1,198), or mail push, were invited to complete mailed surveys and later given a web survey option. Those in condition 2 (N  = 1,173), or enhanced web push, were invited to complete a web survey (the same as in the 2014 study, but with the addition of text messages and quick response (QR) codes and the web survey was optimized for mobile devices) and then later given a mailed survey option. Research aims were to examine response rates across conditions, as well as how responses were distributed across mode (paper, web), devices (computer, smartphone, table), and method of accessing the web survey (hand-entered URL, QR code, e-mail link, SMS link). Response rates differed significantly: the MTF control group was 34.2 percent, mail push was 35.4 percent, and enhanced web push was 42.05 percent. The higher response rate in the enhanced web push condition suggests that the additional strategies were effective at bringing in more respondents. Key estimates produced by the enhanced web push condition did not differ from those of the MTF control group. [ABSTRACT FROM AUTHOR]

During the COVID-19 pandemic, many survey programs switched to self-administered modes of data collection, often offering respondents both web-based and paper-based questionnaires. However, there is little guidance as to when to offer which mode, especially when the aim is to recruit participants for a panel survey. This study examines the effectiveness of different mode-sequence designs by analyzing an experiment implemented in the recruitment wave of the German panel study "Family Research and Demographic Analysis." We randomly assigned 108,256 individuals aged 18–49 years to one of three mode-sequence-design conditions: concurrent, web-first including a paper-based questionnaire with the second reminder (web-first-2), and web-first including a paper-based questionnaire with the third reminder (web-first-3). A fourth, simulated group did not receive a paper-based questionnaire (web-only). We analyzed how different mode-sequence designs affected outcome rates, sample composition, response distributions, data quality, share of paper-based questionnaires, postage costs, number of postal mailings in the recruitment wave, and participation in the first regular wave. Our results show no differences in response distributions and small differences in sample composition across the four mode-sequence designs. As the web-first-2 and simulated web-only designs yielded comparatively good response and recruitment rates at reasonable costs, we recommend their use when surveying adults between 18 and 49 years old. [ABSTRACT FROM AUTHOR]

Frames of residential mailing addresses based on U.S. Postal Service (USPS) sources are often used for selecting samples of housing units for surveys to be conducted in-person, by mail, by web, or using mixed modes. Address lists designed for mail delivery require some modifications for sampling purposes, which therefore require familiarity with aspects of the address files themselves. This paper takes a detailed look at the address components of these files along with the vendors from which such frames and samples are available. More specifically, this paper describes the types of address records and the components of an address record that are available. It discusses how vendors differ in both the services they provide with respect to sampling frames as well as in how those sampling frames are updated and maintained. The paper also details ways in which address-based frames can be enhanced with auxiliary data such as geocodes, area-level demographic variables, and commercial indicators at the individual address level. The match rate, append rate, accuracy, and utility of auxiliary variables are described, along with potential uses in survey design and estimation. Information contained within the paper will be useful for survey designers who have an interest in tailoring address lists for a given survey's target population and budget. [ABSTRACT FROM AUTHOR]

The national Monitoring the Future (MTF) study examines substance use among adolescents and adults in the United States and has used paper questionnaires since it began in 1975. The current experiment tested three conditions as compared to the standard MTF follow-up protocol (i.e., MTF Control) for the first MTF follow-up survey at ages 19/20 years (i.e., one or two years after high school graduation). The MTF Control group included participants who completed in-school baseline surveys in the 12th grade in 2012-2013 and who were selected to participate in the first follow-up survey in 2014 (n=2,451). A supplementary sample of participants who completed the 12th grade baseline survey in 2012 or 2013 but were not selected to participate in the main MTF follow-up (n=4,950) were recruited and randomly assigned to one of three experimental conditions: (1) Mail Push, (2) Web Push, (3) Web Push+E-mail. Results indicated that the overall response rate was lower in Condition 2 compared to MTF Control and to Condition 1; there were no differences between Condition 3 and other conditions. Web response was highest in Condition 3; among web responders, smartphone response was also highest in Condition 3. Subgroup differences also emerged such that, for example, compared to white participants, Hispanics had greater odds of web (versus paper) response and blacks had greater odds of smartphone (versus computer or tablet) response. Item nonresponse was lowest in the Web Push conditions (compared to MTF Control) and on the web survey (compared to paper). Compared to MTF Control, Condition 3 respondents reported higher rates of alcohol use in the past 30 days. The total cost was lowest for Condition 3. Overall, the Condition 3 Web Push+E-mail design is promising. Future research is needed to continue to examine the implications of web and mobile response in large, national surveys. [ABSTRACT FROM AUTHOR]

We encourage authors to continue to submit high-quality manuscripts on cutting-edge topics in survey statistics and methodology as well as innovative and relevant applications (combining both) to I JSSAM i . Of course, we continue to accept longer original research papers and review papers on topics related to survey methodology, survey statistics, and applications. In June 2021, I JSSAM i received its opening impact factor (1.957 for 2020), ranking I JSSAM i as 46/125 in Statistics and Probability and 29/52 in Social Sciences Mathematical Methods. [Extracted from the article]

This paper examines existing methods of evaluating sample quality, showing that their practical utility and applicability to large-scale cross-project comparisons depends on whether they require auxiliary individual-level data. Among those methods that do not demand any such additional data, we differentiate between two approaches that rely on (i) external criteria, that is, comparisons of sample estimates to benchmarks derived from external population statistics, and (ii) internal criteria, that is, comparisons of subsample estimates to a theoretically derived aprioristic value. Our analyses demonstrate the advantages and limitations of both approaches based on an evaluation of 1,125 national surveys carried out in Europe between 2002 and 2016 within four survey projects: the Eurobarometer, European Quality of Life Survey, European Social Survey, and International Social Survey Programme. We show that the prevailing absence of design weights in cross-national survey datasets severely limits the applicability of external criteria evaluations. In contrast, using internal criteria without any weights proves acceptable because incorporating design weights in calculations of internal sample quality has only minor consequences for estimates of sample bias. Furthermore, applying internal criteria, we find that around 75 percent of samples in the four analyzed projects are not significantly biased. We also identify surveys with extremely high sample bias and investigate its potential sources. The paper concludes with recommendations regarding future research, which are directed at secondary data users, as well as producers of cross-national surveys. [ABSTRACT FROM AUTHOR]

In recent years, web-push strategies have been developed in cross-sectional mixed-mode surveys to improve response rates and reduce the costs of data collection. However, pushing respondents into the more cost-efficient web mode has rarely been examined in the context of panel surveys. This study evaluates how a web-push intervention affects the willingness of panel members to switch survey modes from mail to web. We tested three web-push strategies in a German probability-based mixed-mode panel by randomly assigning 1,895 panelists of the mail mode to one of three conditions: (1) the web option was offered to panelists concurrently with the paper questionnaire including a promised €10 incentive for completing the survey on the web, (2) the web option was presented sequentially two weeks before sending the paper questionnaire and respondents were also promised an incentive of €10, or (3) same sequential web-first approach as for condition 2, but with a prepaid €10 incentive instead of a promised incentive. The study found that a sequential presentation of the web option significantly increases the web response in a single survey but may not motivate more panelists to switch to the web mode permanently. Contrary to our expectation, offering prepaid incentives neither improves the web response nor the proportion of mode switchers. Overall, all three web-push strategies show the potential to effectively reduce survey costs without causing differences in panel attrition after five consecutive waves. Condition 2, the sequential web-first design combined with a promised incentive was most effective in pushing respondents to switch to the web mode and in reducing costs. [ABSTRACT FROM AUTHOR]

Government statistical agencies compose a population statistic for a given domain using a sample of units nested in that domain. Subsequent modeling of these domain survey estimates is often used to "borrow strength" across a dependence structure among the domains to improve estimation accuracy and efficiency. This paper focuses on models jointly defined for sample-based point estimates along with their sample-based estimates of variances. Bias may be present in the sample-based (observed) variances due to small sample sizes or the estimation procedure. We propose a new formulation that extends existing joint model formulations to allow for a multiplicative bias in observed variances. Our approach capitalizes on the unbiasedness property of point estimates. We utilize a nonparametric mixture construction that allows the data to discover distinct bias regimes. As a consequence of the better variance estimation, domain point estimates are more robustly estimated under a joint model for the domain point estimates and their associated variances. We compare the performances of alternative models in application to estimates of total employment from the Current Employment Statistics survey conducted by the US Bureau of Labor Statistics, and in simulations. [ABSTRACT FROM AUTHOR]

The paper introduces a new item count technique (ICT) for dealing with sensitive features. One of the aims of the proposed method is to eliminate both the "ceiling" and "floor" effects that can occur in the original ICT. In the new model, in contrast to the classical one and recently proposed models, the two groups into which the sample is split are treated similarly. Therefore, there is no traditional distinction between the control and treatment groups. The control variable can be either a qualitative count variable or a continuous variable. To our knowledge, ICT methods with continuous control have not been proposed before. Although our new model differs substantially in the approach, it requires only a slight modification of the questionnaire design, which in no way leaks respondents' privacy. Substantial gain in statistical efficiency is achieved as compared to the earlier proposed Poisson and negative binomial ICTs. The new method also reduces the bias of maximum likelihood estimators and allows to obtain bootstrap confidence intervals with a substantially smaller width. The statistical properties of the method are investigated theoretically and by simulations. [ABSTRACT FROM AUTHOR]

This research note compares sequential and concurrent web-mail mixed-mode approaches and incentives for a survey on US consumers' use of financial products, especially when their finances are tight. The sample (n = 2,000) was drawn from credit bureau data. We examine the effects on response rates, survey costs, and possible nonresponse bias in an experiment varying two factors in a 2x2 design: (1) using concurrent or sequential web and then mail survey modes, and (2) different incentive amounts given to initial survey nonrespondents ($5 versus $10). The sequential (web-first) design had a significantly lower response rate (3.9 percent) at week five—before the paper questionnaire was mailed—than the concurrent group (11.0 percent). This difference was nearly fully eliminated by the end of the field period. The higher incentive brought in slightly more respondents in the concurrent arm and slightly fewer respondents in the sequential arm, but neither difference is statistically significant. Compared with the sample frame, respondents in both groups were generally older and had many of the characteristics that come from being older: higher credit scores, more open credit cards, lower credit card utilization, greater likelihood of having a mortgage, and lower likelihood of being delinquent on credit card payments. Given the lower initial response rate and the need for more follow-up mailing, the sequential mixed-mode approach resulted in a higher cost per complete survey. [ABSTRACT FROM AUTHOR]

This article presents the results of an adaptive design experiment in the recruitment of households and individuals for a two-stage national probability web or mail mixed-mode survey, the American Family Health Study (AFHS). In the screening stage, we based the adaptive design's subgroup differentiation on Esri Tapestry segmentation. We used tailored invitation materials for a subsample where a high proportion of the population was Hispanic and added a paper questionnaire to the initial mailing for a subsample with rural and older families. In the main-survey stage, the adaptive design targeted the households where a member other than the screening respondent was selected for the survey. The adaptations included emailing and/or texting, an additional prepaid incentive, and seeking screening respondents' help to remind the selected individuals. The main research questions are (i) whether the adaptive design improved survey production outcomes and (ii) whether combining adaptive design and postsurvey weighting adjustments improved survey estimates compared to performing postsurvey adjustments alone. Unfortunately, the adaptive designs did not improve the survey production outcomes. We found that the weighted AFHS estimates closely resemble those of a benchmark national face-to-face survey, the National Survey of Family Growth, although the adaptive design did not additionally change survey estimates beyond the weighting adjustments. Nonetheless, our experiment yields useful insights about the implementation of adaptive design in a self-administered mail-recruit web or mail survey. We were able to identify subgroups with potentially lower response rates and distinctive characteristics, but it was challenging to develop effective protocol adaptations for these subgroups under the constraints of the two primary survey modes and the operational budget of the AFHS. In addition, for self-administered within-household selection, it was difficult to obtain contact information from, reach, and recruit selected household members that did not respond to the screening interview. [ABSTRACT FROM AUTHOR]

Statistical modeling has been, and continues to be, at the core of statistical inferential activity. No good can come off of sophisticated modeling of data from a badly designed study, and hence, the design principles have been at the core of our field. However, use of models in sample surveys has been controversial and has been limited to certain applications, such as small-area estimation. It is clear, however, from Professor Valliant's paper that almost every practical problem in sample surveys requires modeling. In my view, the distrust of modeling strategy, limited to a certain period, is largely due to a misconstrued threat to probability sample designs. The historical account and contributions of Hansen have deeper implications for future development in sample survey research and in training, ultimately integrating modeling and design principles for survey inferences. [ABSTRACT FROM AUTHOR]

Morris Hansen made seminal contributions to the early development of sampling theory, including convincing government survey administrators to use probability sampling as opposed to nonprobability (NP) methods like quota sampling. He codified many of the early results in design-based sampling theory in his 1953 two-volume set co-authored with Hurwitz and Madow. Since those developments, the explicit use of models has proliferated in sampling for use in basic point estimation, nonresponse and noncoverage adjustment, imputation, and a variety of other areas. This paper summarizes some of the early developments, controversies in the design-based versus model-based debate, and uses of models for inference from probability and NP samples. [ABSTRACT FROM AUTHOR]

The availability of both survey and non-survey data sources, such as administrative data, social media data, and digital trace data, has grown rapidly over the past decade. With this expansion in data, the statistical, methodological, computational, and ethical challenges around integrating multiple data sources have also grown. This special issue addresses these challenges by highlighting recent innovations and applications in data integration and related topics. [ABSTRACT FROM AUTHOR]

Entity resolution (record linkage or deduplication) is the process of identifying and linking duplicate records in databases. In this paper, we propose a Bayesian graphical approach for entity resolution that links records to latent entities, where the prior representation on the linkage structure is exchangeable. First, we adopt a flexible and tractable set of priors for the linkage structure, which corresponds to a special class of random partition models. Second, we propose a more realistic distortion model for categorical/discrete record attributes, which corrects a logical inconsistency with the standard hit-miss model. Third, we incorporate hyperpriors to improve flexibility. Fourth, we employ a partially collapsed Gibbs sampler for inferential speedups. Using a selection of private and nonprivate data sets, we investigate the impact of our modeling contributions and compare our model with two alternative Bayesian models. In addition, we conduct a simulation study for household survey data, where we vary distortion, duplication rates and data set size. We find that our model performs more consistently than the alternatives across a variety of scenarios and typically achieves the highest entity resolution accuracy (F1 score). Open source software is available for our proposed methodology, and we provide a discussion regarding our work and future directions. [ABSTRACT FROM AUTHOR]

Responsive survey design (RSD) aims to increase the efficiency of survey data collection via live monitoring of paradata and the introduction of protocol changes when survey errors and increased costs seem imminent. Daily predictions of response propensity for all active sampled cases are among the most important quantities for live monitoring of data collection outcomes, making sound predictions of these propensities essential for the success of RSD. Because it relies on real-time updates of prior beliefs about key design quantities, such as predicted response propensities, RSD stands to benefit from Bayesian approaches. However, empirical evidence of the merits of these approaches is lacking in the literature, and the derivation of informative prior distributions is required for these approaches to be effective. In this paper, we evaluate the ability of two approaches to deriving prior distributions for the coefficients defining daily response propensity models to improve predictions of daily response propensity in a real data collection employing RSD. The first approach involves analyses of historical data from the same survey, and the second approach involves literature review. We find that Bayesian methods based on these two approaches result in higher-quality predictions of response propensity than more standard approaches ignoring prior information. This is especially true during the early-to-middle periods of data collection, when survey managers using RSD often consider interventions. [ABSTRACT FROM AUTHOR]

Mail communications remain a prominent aspect of survey research, especially due to the increasing adoption of web-push surveys throughout the world. However, there is a lack of research on the effectiveness of comprehensive communication strategies for increasing response rates. This paper tests the influence of two communication theories—social exchange theory and pre-suasion theory—on response rates in a survey of residents of southern West Virginia. Social exchange theory has long been used as a strategy for encouraging response, but the theory of pre-suasion offers a new approach of tailoring and sequencing written and visual communications. We operationalized pre-suasion and social exchange communications techniques in the cover letter and questionnaire, then conducted a 2-by-2 experimental design to identify differences in response and data quality. The results show significant differences in response based on the communications in the letter, but no differences based on the questionnaire. The versions with social exchange letters received a response rate 5.5 percent higher than pre-suasion letters. Further, communication differences had no statistically significant effect on measures of data quality. These findings suggest future avenues for research on broad communication strategies in mail survey materials. [ABSTRACT FROM AUTHOR]

Nonparametric and machine learning methods are flexible methods for obtaining accurate predictions. Nowadays, data sets with a large number of predictors and complex structures are fairly common. In the presence of item nonresponse, nonparametric and machine learning procedures may thus provide a useful alternative to traditional imputation procedures for deriving a set of imputed values used next for the estimation of study parameters defined as solution of population estimating equation. In this paper, we conduct an extensive empirical investigation that compares a number of imputation procedures in terms of bias and efficiency in a wide variety of settings, including high-dimensional data sets. The results suggest that a number of machine learning procedures perform very well in terms of bias and efficiency. [ABSTRACT FROM AUTHOR]

JSSAMSPECIALISSUEONDISABILITY MEASUREMENTANDANALYSIS:PREFACE Health, like disability, is a multidimensional concept having many different meanings. The authors recommend that researchers take great care when using compiled cross-national census data to study disability and always consider how disability is defined and measured within surveys. The identification of the group "with disability" and the resulting prevalence of disability depend on the selection of cut points along the functioning continuums. [Extracted from the article]

Despite its importance in terms of survey participation, the literature is sparse on how face-to-face interviewers differentially affect specific groups of sample units. This paper demonstrates how an alternative parametrization of the random components in multilevel models, so-called separate coding, delivers valuable insights into differential interviewer effects for specific groups of sample members. In the example of a face-to-face recruitment interview for a probability-based online panel, we detect small interviewer effects regarding survey participation for non-Internet households, whereas we find sizable interviewer effects for Internet households. We derive practical guidance for survey practitioners to address differential interviewer effects based on the proposed variance decomposition. [ABSTRACT FROM AUTHOR]

Skip patterns, in which certain questions should only be answered by subsets of respondents, are particularly challenging for mail surveys because respondents must navigate them themselves with no assistance from a computer or interviewer. Incorrect navigation of skip patterns can lead to (1) errors of omission, which is when respondents fail to answer follow-up questions they should answer; (2) errors of commission, which is when respondents answer follow-up questions they should skip; or (3) filter item nonresponse, which is when respondents fail to answer the initial filter question. Using experimental data from two general population surveys of Nebraskans, this paper examines three visual design elements—common region, indentation, and subnumbering—aimed at decreasing skip errors in mail surveys. Each design element is intended to create stronger grouping and subgrouping among items within skip patterns, thus clarifying the navigational path and decreasing skip errors. We compare error rates across treatments overall and by respondent age, education, and literacy to assess whether visual cues can provide additional aid to those who might struggle with reading text cues. We find that using separate enclosures or subnumbering of the follow-up questions significantly reduces commission error rates and using indentation significantly increases omission error rates. In addition, none of the tested visual designs were more effective for older, less educated, or lower literacy respondents than for their counterparts. [ABSTRACT FROM AUTHOR]

In the context of the current "replication crisis" across the sciences, failures to reproduce a finding are often viewed as discrediting it. This paper shows how such a conclusion can be incorrect. In 1981, Schuman and Presser showed that including the word "freedom" in a survey question significantly increased approval of allowing a speech against religion in the USA. New experiments in probability sample surveys (n  = 23,370) in the USA and 10 other countries showed that the wording effect replicated in the USA and appeared in four other countries (Canada, Germany, Taiwan, and the Netherlands) but not in the remaining countries. The effect appeared only in countries in which the value of freedom is especially salient and endorsed. Thus, public support for a proposition was enhanced by portraying it as embodying a salient principle of a nation's culture. Instead of questioning initial findings, inconsistent results across countries signal limits on generalizability and identify an important moderator. [ABSTRACT FROM AUTHOR]

Motivated misreporting occurs when respondents give incorrect responses to survey questions to shorten the interview; studies have detected this behavior across many modes, topics, and countries. This paper tests whether motivated misreporting affects responses in a large survey of household purchases, the US Consumer Expenditure Interview Survey. The data from this survey inform the calculation of the official measure of inflation, among other uses. Using a parallel web survey and multiple imputation, this article estimates the size of the misreporting effect without experimentally manipulating questions in the survey itself. Results suggest that household purchases are underreported by approximately five percentage points in three sections of the first wave of the survey. The approach used here, involving a web survey built to mimic the expenditure survey, could be applied in other large surveys where budget or logistical constraints prevent experimentation. [ABSTRACT FROM AUTHOR]

Response burden has been a concern in survey research for some time. One area of concern is the negative impact that response burden can have on response rates. In an effort to mitigate negative impacts on response rates, survey research organizations try to minimize the burden respondents are exposed to and maximize the likelihood of response. Many organizations also try to be mindful of the role burden may play in respondents' likelihood to participate in future surveys by implementing rest periods or survey holidays. Recently, new evidence from a study of cross-sectional household surveys provided an interesting lens to examine burden. The evidence demonstrated that those sampled in two independent surveys are more likely to respond to the second survey if the first survey was more difficult to complete, and that this effect was not significantly influenced by the rest period in between the two surveys. These findings are compelling, and since the mechanisms influencing response in household and establishment surveys differ in important ways, a similar examination in an establishment survey context is warranted. To accomplish this, data are used from the National Agricultural Statistics Service. Overall, our research finds that prior survey features such as questionnaire complexity (or burden), prior response disposition and rest period are significantly associated with response to subsequent surveys. We also find that sample units first receiving a more complex questionnaire have significantly higher probabilities of response to a subsequent survey than do those receiving a simpler questionnaire first. The findings in this paper have implications for nonresponse adjustments and identification of subgroups for adaptive design data collection. [ABSTRACT FROM AUTHOR]

This paper demonstrates that the sample allocation that takes the expected response rates (ERRs) into account has certain advantages over other approaches in terms of reducing the variances of the estimates. The performance of the ERR allocation is assessed within the framework of stratified sampling by comparing the resulting variances with those obtained using the classical procedure of proportional-to-stratum size (PS) allocation and then applying poststratification. The main theoretical tool is asymptotic calculations using the δ -method, which are complemented with extensive finite sample evaluations using various combinations of specific population parameters. The main finding was that within a stratified sample design, ERR allocation leads to lower variances than PS allocation, not only when the response rates are correctly specified but also under a wide range of conditions where the response rates can only be approximately specified in advance. [ABSTRACT FROM AUTHOR]

One of the main components of survey costs is interviewer call attempts associated with making contact. This paper describes an experiment conducted on a U.K. household panel study, which sought to evaluate whether an “early-bird” approach whereby participants are encouraged to contact their interviewer before fieldwork began in order to set up an appointment could increase fieldwork efficiency by reducing the number of calls required. This approach has been successfully used on the National Longitudinal Study of Youth 1979 cohort for some time but has not been evaluated experimentally. Our experiment involved two treatment groups: one group was promised a modest financial incentive (£5 per participant) to take up the “early-bird” offer and complete an interview, and the other received an appeal to their altruistic tendencies that emphasized that being an “early bird” would make their interviewer's life easier. A parallel experiment sought to evaluate the impact of differential standard incentives on response. The early-bird take-up rate was higher for the incentivized group (10 percent compared with 6 percent for the non-incentivized group) and was highest when combined with the higher standard incentive rate (17 percent). Offering both an early-bird incentive and the higher standard incentive did increase overall fieldwork efficiency, as measured by calls required per completed case, but the modest take-up rates meant that the overall impact was fairly minimal. The paper also finds indicative evidence that the early-bird offer, if sufficiently incentivized, could potentially have a beneficial impact on response rates. [ABSTRACT FROM AUTHOR]

This paper provides a discussion of the Tourangeau (2019) Morris Hansen Lecture paper. I address issues related to compounding errors in web surveys and the relationship between nonresponse and measurement errors. I provide a potential model for understanding when error sources in nonprobability web surveys may compound or counteract one other. I also provide three conceptual models that help explicate the joint relationship between nonresponse and measurement errors. [ABSTRACT FROM AUTHOR]

Nonprobability samples, for example observational studies, online opt-in surveys, or register data, do not come from a sampling design and therefore may suffer from selection bias. To correct for selection bias, Elliott and Valliant (EV) proposed a pseudo-weight estimation method that applies a two-sample setup for a probability sample and a nonprobability sample drawn from the same population, sharing some common auxiliary variables. By estimating the propensities of inclusion in the nonprobability sample given the two samples, we may correct the selection bias by (pseudo) design-based approaches. This paper expands the original method, allowing for large sampling fractions in either sample or for high expected overlap between selected units in each sample, conditions often present in administrative data sets and more frequently occurring with Big Data. [ABSTRACT FROM AUTHOR]

A prior study found that mailing prepaid incentives with $5 cash visible from outside the envelope increased the response rate to a mail survey by 4 percentage points compared to cash that was not externally visible. This "visible cash effect" suggests opportunities to improve survey response at little or no cost, but many unknowns remain. Among them: Does the visible cash effect generalize to different survey modes, respondent burdens, and cash amounts? Does it differ between fresh samples and reinterview samples? Does it affect data quality or survey costs? This article examines these questions using two linked studies where incentive visibility was randomized in a large probability sample for the American National Election Studies. The first study used $10 incentives with invitations to a long web questionnaire (median 71 minutes, n  =   17,849). Visible cash increased response rates in a fresh sample for both screener and extended interview response (by 6.7 and 4.8 percentage points, respectively). Visible cash did not increase the response rate in a reinterview sample where the baseline reinterview response rate was very high (72 percent). The second study used $5 incentives with invitations to a mail-back paper questionnaire (n  =   8,000). Visible cash increased the response rate in a sample of prior nonrespondents by 4.0 percentage points (from 31.5 to 35.5), but it did not increase the response rate in a reinterview sample where the baseline reinterview rate was very high (84 percent). In the two studies, several aspects of data quality were investigated, including speeding, non-differentiation, item nonresponse, nonserious responses, noncredible responses, sample composition, and predictive validity; no adverse effects of visible cash were detected, and sample composition improved marginally. Effects on survey costs were either negligible or resulted in net savings. Accumulated evidence now shows that visible cash can increase incentives' effectiveness in several circumstances. [ABSTRACT FROM AUTHOR]

Adaptive survey designs are increasingly used by survey practitioners to counteract ongoing declines in household survey response rates and manage rising fieldwork costs. This paper reports findings from an evaluation of an early-bird incentive (EBI) experiment targeting high-effort respondents who participate in the 2019 wave of the US Panel Study of Income Dynamics. We identified a subgroup of high-effort respondents at risk of nonresponse based on their prior wave fieldwork effort and randomized them to a treatment offering an extra time-delimited monetary incentive for completing their interview within the first month of data collection (treatment group; N  = 800) or the standard study incentive (control group; N  = 400). In recent waves, we have found that the costs of the protracted fieldwork needed to complete interviews with high-effort cases in the form of interviewer contact attempts plus an increased incentive near the close of data collection are extremely high. By incentivizing early participation and reducing the number of interviewer contact attempts and fieldwork days to complete the interview, our goal was to manage both nonresponse and survey costs. We found that the EBI treatment increased response rates and reduced fieldwork effort and costs compared to a control group. We review several key findings and limitations, discuss their implications, and identify the next steps for future research. [ABSTRACT FROM AUTHOR]

Smartphones present many interesting opportunities for survey research, particularly through the use of mobile data collection applications (apps). There is still much to learn, however, about how to integrate apps in general population surveys. Recent studies investigating hypothetical willingness to complete mobile data collection tasks via an app suggest there may be substantial resistance, in particular, due to concerns around data privacy. There is not much evidence about how privacy concerns influence actual decisions to participate in app-based surveys. Theoretical approaches to understanding privacy concerns and survey participation decisions would suggest that the influence of the former over the latter is likely to vary situationally. In this paper, we present results from a methodological experiment conducted in the context of a three-wave probability-based online panel survey of the general population as part of the 2019 Swiss Election Study ("Selects") testing different ways of recruiting participants to an app. Questions included at wave 1 about online data privacy concerns and comfort sharing different types of data with academic researchers allow us to assess their impact on both hypothetical willingness to download a survey app for completing questionnaires, to take and share photos, and to share the smartphone's GPS location and actual completion of these tasks. Our findings confirm that general concerns about online data privacy do influence hypothetical willingness to complete mobile data collection tasks, but may be overridden by how comfortable people feel about sharing specific types of data with researchers. When it comes to actual compliance with task requests, however, neither privacy concerns nor comfort sharing data seem to matter. We conclude with recommendations for exploring these relationships further in future app-based studies. [ABSTRACT FROM AUTHOR]

Recent research in differential privacy demonstrated that (sub)sampling can amplify the level of protection. For example, for ϵ -differential privacy and simple random sampling with sampling rate r , the actual privacy guarantee is approximately r ϵ ⁠ , if a value of ϵ is used to protect the output from the sample. In this paper, we study whether these amplification effects can be exploited systematically to improve the accuracy of the privatized estimate. Specifically, assuming the agency has information for the full population, we ask under which circumstances accuracy gains could be expected, if the privatized estimate would be computed on a random sample instead of the full population. We find that accuracy gains can be achieved for certain regimes. However, gains can typically only be expected, if the sensitivity of the output with respect to small changes in the database does not depend too strongly on the size of the database. We only focus on algorithms that achieve differential privacy by adding noise to the final output and illustrate the accuracy implications for two commonly used statistics: the mean and the median. We see our research as a first step toward understanding the conditions required for accuracy gains in practice and we hope that these findings will stimulate further research broadening the scope of differential privacy algorithms and outputs considered. [ABSTRACT FROM AUTHOR]

Federal surveys often pledge that personally identifiable information (PII) will be used for statistical purposes only and that the willful disclosure of PII by employees who have been sworn to confidentiality is subject to penalties. In 2015, the passing of the Cybersecurity Enhancement Act required the Department of Homeland Security (DHS) to monitor federal computers for cyber threats. Since DHS personnel are not bound to the confidentiality pledges used by the statistical agencies, the question became how to inform respondents about these changes without alarming them and harming response rates. This paper reports on an experiment conducted in a survey of public-school principals with a simple random sample (n  = 4,122). The experiment examined differences in response rates across three versions of pledges disclosing the cybersecurity monitor: one pledge specified explicitly the role of DHS, one referred to the monitors as "Federal employees and contractors," and a control version referred implicitly to DHS monitors via a legal citation. Half of the pledges included the penalty language, and the other half did not. There were no statistically significant differences between any of the experimental comparisons. We conclude that, of the pledges studied, the DHS version of the pledge with penalty language provides respondents with the most accurate information for deciding whether to participate in the survey without harming response rates. [ABSTRACT FROM AUTHOR]

Social science researchers rely on study participants to provide information about themselves, and pledging to protect their identities is generally standard practice. This is usually specified through professional associations or policies and laws governing research agencies and institutions. Conventional methods of protecting participants' identities in quantitative research are undergoing a major overhaul in response to relatively new threats brought about by the sheer volume of large publicly available datasets along with powerful and affordable computing capabilities that enable data linkages and respondent re-identification. These developments have raised new questions about qualitative research products that previously may have been considered less vulnerable to re-identification. In this paper, we first summarize a range of conventional methods of protecting respondents' identities used in both quantitative and qualitative research. We then discuss the relatively new threats to confidentiality brought about by the internet, computing, and social media age. Next, we bring the focus to qualitative research specifically and discuss the development of a novel approach to a systematic method of disclosure avoidance for at least a subset of qualitative research products: typical research aimed at pretesting and evaluation of survey questions, data collection instruments, and related materials used for household surveys in the federal statistical system. We frame the discussion in terms of risk and mitigation. That is, we aim to articulate the nature of qualitative methods and data, the risks posed by the dissemination of qualitative information products, and consider how these risks might reasonably be mitigated while maximizing utility. Finally, we pull back the lens and discuss how the method could be applied to research outside the context of the federal statistical system if certain criteria are met. [ABSTRACT FROM AUTHOR]

Many agencies are currently investigating whether releasing synthetic microdata could be a viable dissemination strategy for highly sensitive data, such as business data, for which disclosure avoidance regulations would otherwise prohibit the release of public use microdata. The U.S. Census Bureau has identified the Economic Census as a candidate program and has been developing synthetic data generators. The synthetic data should account for skewed and irregular distributions, satisfy predetermined edit constraints, and preserve selected privacy features. Previous research on these generators was confined to businesses that were in operation for the full year, ignoring the special features of births and deaths in the models. These generators preserve multivariate relationships and yield marginal totals that closely correspond to the published official statistics. However, these synthetic data consequently do not reflect the state of economic expansion or contraction. This missing information is a severe deficiency for the targeted data users comprising economists, policymakers, and methodologists, especially since the global pandemic of 2020. This paper introduces an approach that addresses this deficiency, producing partially synthetic data with high utility and privacy protection. We provide preliminary results using selected industry data from the 2012 Economic Census. [ABSTRACT FROM AUTHOR]

Respondent-driven sampling (RDS) is a popular method of conducting surveys in hard to reach populations where strong assumptions are required in order to make valid statistical inferences. In this paper we investigate the assumption that network degrees are measured accurately by the RDS survey and find that there is likely significant measurement error present in typical studies. We prove that most RDS estimators remain consistent under an imperfect measurement model with little to no added bias, though the variance of the estimators does increase. [ABSTRACT FROM AUTHOR]

Rare populations, such as endangered animals and plants, drug users and individuals with rare diseases, tend to cluster in regions. Adaptive cluster sampling is generally applied to obtain information from clustered and sparse populations since it increases survey effort in areas where the individuals of interest are observed. This work proposes a unit-level model which assumes that counts are related to auxiliary variables, improving the sampling process by assigning different weights to the cells, besides referring to them spatially. The proposed model fits rare and grouped populations arranged on a regular grid in a Bayesian framework. The approach is compared to alternative methods using simulated data and a real experiment in which adaptive samples were drawn from an African buffalo population in a 24,108 square kilometer area in East Africa. Simulation studies show that the model is efficient in several settings, validating the method proposed in this paper for practical situations. [ABSTRACT FROM AUTHOR]

Low rates of survey response are a problem that survey organizations increasingly experience across the world. Understanding local socio-demographic reasons for low response, and projecting future response, can inform targeted outreach and messaging to communities. Using the US Census as a case study, we study mail response behavior both cross-sectionally and over time. The cross-sectional part of this study (2010) helps inform the selection of explanatory variables for a two-panel difference model (2000–2010). In turn, the two-panel difference model offers an in-depth understanding of which demographic indicators shape response across time. Increases in response between 2000 and 2010 were found to be associated with growth in the elderly and in the Dominican populations, while growth in blacks, renters, and those with low educational attainment were associated with decreases in response. Coefficients from the two-panel difference model are then applied to demographic changes from 2008–2012 to 2013–2017, to project response rates for the most recent period. While much of our findings are in line with prior survey research, we demonstrate the importance of projecting response behavior using a local perspective, as national-level results do not always apply to smaller jurisdictions. In a diverse country such as the United States, a hyper-diverse city such as New York often carves its own path. To our knowledge, this is the first paper that connects shifts in socio-demographic neighborhood characteristics with changes in response rates. As racial and ethnic diversity increases in cities across western societies, our methodology allows jurisdictions to project response rates using shifts in socio-demographic neighborhood characteristics that are tied to self-response. Such a tailored approach can be replicated across different kinds of surveys, countries, and time points, which will allow for the identification of local factors that drive changes in survey response and help establish a precisely customized outreach plan. [ABSTRACT FROM AUTHOR]

Several questionnaire design texts emphasize a dual role of question wording: the wording needs to express what is being measured and tell respondents how to answer. Researchers tend to focus heavily on the first of these goals, but sometimes overlook the second, resulting in question wording that does not match the response options provided (i.e. mismatches). Common examples are yes/no questions with ordinal or nominal response options, open-ended questions with closed-ended response options, and check-all-that apply questions with forced-choice response options. A slightly different type of mismatch utilizes a question stem that can be read as asking for two different types of answers with no indication of which type should be provided. In this paper, we report the results of twenty-two experimental comparisons of data quality indicators (i.e. item nonresponse and response time) and response distributions across matched and mismatched versions of questions from a postal mail survey and a telephone survey. We find that mismatched items generally have lower data quality than matched items and that substantive results differ significantly across matched and mismatched designs, especially in the telephone survey. The results suggest that researchers should be wary of mismatches and should strive for holistic design. [ABSTRACT FROM AUTHOR]

State-based telephone surveys are often designed to make estimates at substate levels, such as county or county group. Under a traditional random-digit-dial design, the telephone exchange of a landline number could be used to accurately identify the county for which the associated household resides. However, initially, no good analogous data methods existed for the cellphone frame. This required survey methodologists to draw random samples of cellphone numbers from the entire state, making it difficult to target areas within a state. To overcome this shortcoming, sample vendors have used a cellphone number's rate center (where the number was activated) as a proxy estimate for the county where the cellphone owner resides. Our paper shows that county assignations that are based on rate center data may have classification error rates as high as 30%. These high classification error rates make it difficult to accurately devise a cellphone frame sample allocation using the rate center data. This paper proposes a new method—the Rate Center Plus method—which uses rate centers and an estimate of the classification probabilities to stratify and allocate the desired respondent sample to counties. The new method uses Bayes' rule to distribute a desired county-level sample allocation across rate center counties. We demonstrate how the Rate Center Plus method was applied to the 2015 Ohio Medicaid Assessment Survey and the resulting efficacy of the method. Finally, we evaluate whether the new approach is more efficient than the traditional statewide sample method. In addition, we look at four approaches to estimating the necessary classification probabilities. We found that the Rate Center Plus method can be more cost efficient than the statewide sample method when the classification probabilities are reasonably estimated, reducing data collection costs as much as 12.8%. [ABSTRACT FROM AUTHOR]

Although most survey researchers agree that reliability is a critical requirement for survey data, there have not been many efforts to assess the reliability of responses in national surveys. In addition, there are quite different approaches to studying the reliability of survey responses. In the first section of the Lecture, I contrast a psychological theory of over-time consistency with three statistical models that use reinterview data, multi-trait multi-method experiments, and three-wave panel data to estimate reliability. The more sophisticated statistical models reflect concerns about memory effects and the impact of method factors in reinterview studies. In the following section of the Lecture, I examine some of the major findings from the literature on reliability. Despite the differences across methods for exploring reliability, the findings mostly converge, identifying similar respondent and question characteristics as major determinants of reliability. The next section of the paper looks at the correlations among estimates of reliability derived from the different methods; it finds some support for the validity of the measures from traditional reinterview studies. The empirical claims motivating the more sophisticated methods for estimating reliability are not strongly supported in the literature. Reliability is, in my judgment, a neglected topic among survey researchers, and I hope the Lecture spurs further studies of the reliability of survey questions. [ABSTRACT FROM AUTHOR]

As the 2020 US Census approaches, the preparations include tests of new methodologies for enumeration that have the potential to reduce cost and improve quality. The 2015 Census Test in Savannah, GA, included tests of Internet and mail response modes and of online delivery of social marketing communications focused on persuading the public to respond by Internet and mail. Merging data from the 2015 Census Test with external third-party lifestyle segments and the Census Bureau's new Low Response Score (LRS) produces a dataset suitable for studying relationships between census response, LRSs, and lifestyle segments. This paper uses the merged dataset to examine whether lifestyle segments can provide insight to hard-to-survey populations, their response behavior, and interactions with social marketing communications. The article also includes analyses with nationwide data that support the broader application of using segmentation variables in self-response propensity models and a discussion of potential applications of segment lifestyle information in tailored and targeted survey designs for hard-to-survey populations. [ABSTRACT FROM AUTHOR]

There is some evidence that questionnaire design (scrolling or paging) and invitation mode (SMS or e-mail) have an impact on response rates in web surveys completed on personal computers (PCs). This paper examines whether these findings can be generalized to mobile web surveys. First, we explore the effect of scrolling versus paging design on the breakoff rate, item nonresponse, and completion time in mobile web surveys. Second, we investigate which type of invitation and reminder mode (SMS or e-mail) is more effective in terms of producing higher participation rates and maximizing the percentage of respondents who complete the survey via a mobile device rather than a PC. The paper summarizes the results of an experiment conducted among members of a volunteer online access panel in Russia, who were asked to complete the survey using a mobile device. We find that the scrolling design leads to significantly faster completion times, lower (though not significantly lower) breakoff rates, fewer technical problems, and higher subjective ratings of the questionnaire. We also find that SMS invitations are more effective than e-mail invitations in mobile web surveys. [ABSTRACT FROM AUTHOR]

Telephone surveys have been a ubiquitous method of collecting survey data, but the environment for telephone surveys is changing. Many surveys are transitioning from telephone to self-administration or combinations of modes for both recruitment and survey administration. Survey organizations are conducting these transitions from telephone to mixed modes with only limited guidance from existing empirical literature and best practices. This article summarizes findings by an AAPOR Task Force on how these transitions have occurred for surveys and research organizations in general. We find that transitions from a telephone to a self-administered or mixed-mode survey are motivated by a desire to control costs, to maintain or improve data quality, or both. The most common mode to recruit respondents when transitioning is mail, but recent mixed-mode studies use only web or mail and web together as survey administration modes. Although early studies found that telephone response rates met or exceeded response rates to the self-administered or mixed modes, after about 2013, response rates to the self-administered or mixed modes tended to exceed those for the telephone mode, largely because of a decline in the telephone mode response rates. Transitioning offers opportunities related to improved frame coverage and geographic targeting, delivery of incentives, visual design of an instrument, and cost savings, but challenges exist related to selecting a respondent within a household, length of a questionnaire, differences across modes in use of computerization to facilitate skip patterns and other questionnaire design features, and lack of an interviewer for respondent motivation and clarification. Other challenges related to surveying youth, conducting surveys in multiple languages, collecting nonsurvey data such as biomeasures or consent to link to administrative data, and estimation with multiple modes are also prominent. [ABSTRACT FROM AUTHOR]