Search

There is an ongoing debate in the survey research literature about whether and when probability and nonprobability sample surveys produce accurate estimates of a larger population. Statistical theory provides a justification for confidence in probability sampling as a function of the survey design, whereas inferences based on nonprobability sampling are entirely dependent on models for validity. This article reviews the current debate about probability and nonprobability sample surveys. We describe the conditions under which nonprobability sample surveys may provide accurate results in theory and discuss empirical evidence on which types of samples produce the highest accuracy in practice. From these theoretical and empirical considerations, we derive best-practice recommendations and outline paths for future research.

For more than three decades, declining response rates have been of concern to both survey methodologists and practitioners. Still, international comparative studies have been scarce. In one of the first international trend analyses for the period 1980–1997, De Leeuw and De Heer (2002) describe that response rates declined over the years and that countries differed in response rates and nonresponse trends. In this article, we continued where De Leeuw and De Heer (2002) stopped, and present trend data for the next period 1998–2015 from National Statistical Institutes. When we looked at trends over time in this new data set, we found that response rates are still declining over the years. Furthermore, nonresponse trends do differ over countries, but not over surveys. Some countries show a steeper decline in response than others, but all types of surveys show the same downward trend. The differences in (non)response trends over countries can be partly explained by differences in survey design between the countries. Finally, for some countries cost indicators were available, these showed that costs increased over the years and are negatively correlated with noncontact rates.

Declining response rates worldwide have stimulated interest in understanding what may be influencing this decline and how it varies across countries and survey populations. In this paper, we describe the development and validation of a short 9-item survey attitude scale that measures three important constructs, thought by many scholars to be related to decisions to participate in surveys, that is, survey enjoyment, survey value, and survey burden. The survey attitude scale is based on a literature review of earlier work by multiple authors. Our overarching goal with this study is to develop and validate a concise and effective measure of how individuals feel about responding to surveys that can be implemented in surveys and panels to understand the willingness to participate in surveys and improve survey effectiveness. The research questions relate to factor structure, measurement equivalence, reliability, and predictive validity of the survey attitude scale. The data came from three probability-based panels: the German GESIS and PPSM panels and the Dutch LISS panel. The survey attitude scale proved to have a replicable three-dimensional factor structure (survey enjoyment, survey value, and survey burden). Partial scalar measurement equivalence was established across three panels that employed two languages (German and Dutch) and three measurement modes (web, telephone, and paper mail). For all three dimensions of the survey attitude scale, the reliability of the corresponding subscales (enjoyment, value, and burden) was satisfactory. Furthermore, the scales correlated with survey response in the expected directions, indicating predictive validity.

Household survey nonresponse is a matter of concern in many countries. In one of the first international trend analyses, de Leeuw and de Heer (2002) found that response rates declined over the years, and that countries differed in response rates and nonresponse trends. Their analyses cover longitudinal data on the Labour Force Survey from National Statistical Institutes for the period 1980 to 1997. We added a new data set, covering the period 1998 -2015, and analysed nonresponse data over time and countries. In these analyses we differentiated between voluntary and mandatory surveys. The trends visible in de Leeuw and de Heer (2002) continue with possibly a small deceleration in refusal rates.

Mixed-mode surveys have been around since the late 1980s. In the past thirty years, major changes in technology and society influenced and changed data collection and survey methodology. However, in those years, mixed-mode strategies remained part of the daily survey practice, although the type of mix implemented followed the changes in technology and data collection methods. In this paper, I summarize the state of the art in traditional mixed-mode surveys and discuss implications for mixed device surveys.

In an optimal design for mixed‐mode surveys, be it within a country or across countries, there are three phases that need attention. The first phase is the design phase: it is important to prevent mode measurement effects when designing the study. A richer source of additional data will help to achieve a better estimation of mode effects. After the data are collected, the next two phases are estimating the potential mode measurement effect due to different modes and adjusting for any such bias. Survey modes differ on several dimensions, for instance, interviewer‐administered versus self‐administered questionnaires and information transmission and communication. The use of well‐established and validated constructs and multiquestion scales across countries and modes helps achieving measurement equivalence countries and modes. Mixed‐mode designs and mode changes are often implemented for sound methodological reasons. Using a mixed‐mode design implies a careful trade‐off among coverage, nonresponse, measurement errors, and cost.

Online surveys are one of the most prominent data collection methods in Europe and the USA. Not only are they fast and cheap, data quality in well-designed online surveys is high, especially when sensitive questions are asked. Disadvantages are the threat of undercoverage, as not everyone has Internet access, and high nonresponse. In order to overcome these disadvantages, mixed-mode designs are used in which multiple data collection methods are combined. The strength of mixed-mode surveys is their potential to reduce coverage and nonresponse error at affordable costs. However, survey modes may differ in the effect they have on measurement error, and one critical question is how data from different modes may be combined. A special form of mixed mode design is a mixed-device survey. Web surveys are increasingly completed on a range of different devices. Mobile phones, tablets, and even smart watches are being used in addition to regular desktop PCs. The question arises whether or not answers obtained via smartphones or tablets are comparable to answers via regular desktop PCs. Screen sizes are very different and also the mode of response entry varies between devices. Survey software increasingly adapts to mobile survey responding via responsive survey design. The software detects the device used for completing the survey and adapts the format accordingly. This chapter discusses the most common designs for mixed mode surveys and summarizes the empirical evidence for reducing coverage and nonresponse error and then focuses on measurement error. We pay special attention to careful design and provide rules for doing web surveys that are in fact mixed-device surveys.

This study investigates the impact of different modes and tailor-made response enhancing measures (TMREM)—such as bilingual interviewers with a shared ethnic background and translated questionnaires—on the measurement of substantive variables in surveys among minority ethnic groups in the Netherlands. It also provides insight into the ability to detect mode measurement effects of a recently developed method for disentangling mode measurement and mode selection effects, as well as into the tenability of the assumptions underlying this method. The data used in this study come from a large-scale survey design experiment among the four largest non-Western minority ethnic groups in the Netherlands comparing single-mode computer-assisted personal interviewing (CAPI) and sequential computer-assisted web interviewing, computer-assisted telephone interviewing, and CAPI-MM. The number and intensity of the TMREM varied among the four ethnic groups. The results show that mode measurement effects occur among all ethnic groups and are the result of a combination of the presence or absence of an interviewer and TMREM. Mode measurement effects occur more often on sociocultural questions, but also, on occasion, on more sociostructural or background questions. The method used to disentangle mode measurement and mode selection effects can be applied to detect mode measurement effects, but one should be cautious in interpreting them. Implausible mode measurement effects can be caused by the violation of the assumptions underlying this method.

Different types of nonresponse threaten the validity of longitudinal studies: first, initial nonresponse during the recruitment in the baseline survey and second, successive dropout at each time point. Dropout in longitudinal surveys has three separate sources: failure to locate research participants, failure to contact participants, and failure to achieve cooperation. In this entry, specific attention is given to existing knowledge on the correlates of dropout and strategies to limit the problem. To limit nonresponse, a total design approach is advocated with specific attention to each source. That is to limit both noncontact (i.e., failure to locate and subsequent failure to contact a located research participant) and noncooperation.

One of the methods for evaluating online panels in terms of data quality is comparing the estimates that the panels provide with benchmark sources. For probability-based online panels, high-quality surveys or government statistics can be used as references. If differences among the benchmark and the online panel estimates are found, these can have several causes. First, the question wordings can differ between the sources, which can lead to differences in measurement. Second, the reference and the online panel may not be comparable in terms of sample composition. Finally, since the reference estimates are usually collected face-to-face or by telephone, mode effects might be expected. In this article, we investigate mode system effects, an alternative to mode effects that does not focus solely on measurement differences between the modes, but also incorporates survey design features into the comparison. The data from a probability-based offline-recruited online panel is compared to the data from two face-to-face surveys with almost identical recruitment protocols. In the analysis, the distinction is made between factual and attitudinal questions. We report both effect sizes of the differences and significances. The results show that the online panel differs from face-to-face surveys in both attitudinal and factual measures. However, the reference surveys only differ in attitudinal measures and show no significant differences for factual questions. We attribute this to the instability of attitudes and thus show the importance of triangulation and using two surveys of the same mode for comparison.