Search

Many in the survey research community have expressed concern at the growing popularity of nonprobability surveys. The absence of random selection prompts justified concerns about self-selection producing biased results and means that traditional, design-based estimation is inappropriate. This paper seeks to provide insight into the conditions under which nonprobability surveys can be expected to provide estimates free of selection bias. In fields such as epidemiology and economics that routinely work with observational data, researchers have identified the necessary conditions for unbiased estimation of causal effects when treatments are not assigned randomly. Similar conditions apply to survey estimates when respondents are not randomly selected. Drawing on this body of research, we propose a framework composed of three elements that determine the level of selection bias in survey estimates. In this paper, we first provide a general overview of these components and demonstrate the link between causal inference and survey inference in the probability-based setting. Second, we give simplified examples to demonstrate how each of the components can contribute to bias in survey estimates. Finally, we review current practices in the area of nonprobability data collection and estimation, and specify how these methods relate to the elements identified here. [ABSTRACT FROM AUTHOR]

This paper addresses speeding, that is, "too fast" responses, in web surveys. Relying on the response process model, we argue that very short response times indicate low data quality, stemming from a lack of attention on the part of respondents. To identify speeding, prior research employed case-wise procedures. Using data from nine online surveys, we demonstrate that response behavior of individual respondents varies considerably during a survey. Thus, we use case- and page-wise procedures to capture speeding behavior that taps different, although related, phenomena. Moreover, page-specific speeding measures capture aspects of data quality that traditional quality measures do not cover. Employing both page-specific and case-wise speeding measures, we examine whether removing speeders makes a difference in substantive findings. The evidence indicates that removing "too fast" responses does not alter marginal distributions, irrespective of which speeder-correction technique is employed. Moreover, explanatory models yield, by and large, negligible coefficient differences (on average about one standard error). Only in exceptional cases differences exceed two standard errors. Our findings suggest that speeding primarily adds some random noise to the data and attenuate correlations, if it makes a difference at all. The paper concludes by discussing implications and limitations. [ABSTRACT FROM AUTHOR]

For decades, sample surveys have provided information of value to sponsors in government, academia, business, and the public. That value proposition is threatened now by declining survey participation and the advent of competition from alternative data sources. In this environment, some developments in survey practice include new thinking about how to recruit respondents, new methods for applying communication technology, and new approaches to blending survey and non-survey data. Going forward, survey data may increasingly be one component of information products, formed from various sources, including administrative records and unstructured ("big") data. The papers in this special issue of Public Opinion Quarterly will contribute to discussions on how future surveys should be conducted and their place in a new information order. [ABSTRACT FROM AUTHOR]

The measurement of household relationships has a long history at the U.S. Census Bureau, but recent challenges include accurately measuring the number of same-sex couples. In 2010, the American Community Survey estimated that there were 593,000 same-sex households. In the same year, the census estimate for the number of same-sex couples was 902,000 (O’Connell and Feliz 2011). Same-sex couples are classified through two questionnaire items: relationship and sex. In an effort to understand the measurement error that might be causing the estimates to differ, we compared questionnaire design formats and modes and conducted a sex-by-name analysis, focus groups, and cognitive interviews. The paper demonstrates that misreporting of sex rather than relationship was responsible for the overreporting of same-sex couples. [ABSTRACT FROM PUBLISHER]

For survey researchers, it is common practice to use the check-all question format in Web and mail surveys but to convert to the forced-choice question format in telephone surveys. The assumption underlying this practice is that respondents will answer the two formats similarly. In this research note we report results from 16 experimental comparisons in two Web surveys and a paper survey conducted in 2002 and 2003 that test whether the check-all and forced-choice formats produce similar results. In all 16 comparisons, we find that the two question formats do not perform similarly; respondents endorse more options and take longer to answer in the forced-choice format than in the check-all format. These findings suggest that the forced-choice question format encourages deeper processing of response options and, as such, is preferable to the check-all format, which may encourage a weak satisficing response strategy. Additional analyses show that neither acquiescence bias nor item nonresponse seem to pose substantial problems for use of the forced-choice question format in Web surveys. [ABSTRACT FROM AUTHOR]

A probability sample of households frequently must be translated into a sample of persons representing characteristics of the adult population. Researchers have developed at least 14 within-household respondent selection procedures or their variants to accomplish this task. Although probability methods are preferable, they often increase nonresponse. Quasi-probability and nonprobability techniques have been devised to increase cooperation and decrease costs, although they sacrifice the advantages of randomness. The purpose of this paper is to analyze the results of 16 studies that have compared within-household respondent selection procedures, with respect to demographic representativeness, cooperation or response rates, and costs. Although there is an increasing need to hold down costs yet maintain quality and accuracy as response rates are falling, little systematic, accessible evidence exists to guide choice of respondent selection method. [ABSTRACT FROM AUTHOR]

This article presents information on sampling in random digit dial surveys. The study proposes a new and simple method of within-household sampling that is minimally intrusive and, thus, should not have a deleterious effect on response rates. The method results in a probability sample of eligible persons within households. The method is especially simple to implement in computer-assisted telephone interview random digit dial surveys. The study focus on sampling one adult from each household as this is the primary application, but we also describe how the method can be extended to sample more than one person per household. This paper provides an introduction to the method and results from one study. Experimental comparisons of this approach with other methods of within-household sampling would be a useful next step. The author has presented the method as a way of sampling one adult from all adults in the household, but it can be applied in any situation in which the screener respondent is eligible by first determining if the screener respondent is sampled. If the screener respondent is not eligible, a variant of the method can be used.

This article focuses on the person-level/household-level component of the 1999 Questionnaire Design Experimental Research Survey (QDERS) experiment. The QDERS was a split-sample controlled experiment, using paper-and-pencil questionnaires in a telephone interview with a single household respondent. Due in great measure to its significantly higher yield of employer or union-based health plan coverage, the person-level approach produced more insured persons overall than the household-level approach and, thus, lower estimates of the uninsured. The absence of information about the true characteristics of QDERS household prevents definitive conclusions about data quality differences. The QDERS experiment included a response variance reinterview or approximately five hundred households in each treatment to permit a comparison of the reliability of the data produced by the person-level and household screener interview designs. Item nonresponse in general was trivial, and differences in item nonresponse are for the most part nonexistent. Timing data from a small sample of tape-recorded interviews confirm the increased efficiency of the household-level interview. Interviews evaluated both questionnaire treatments via a debriefing questionnaire.