Search

Surveys are well known to contain response errors of different types, including acquiescence, social desirability, common method variance and random error simultaneously. Nevertheless, a single error source at a time is all that most methods developed to estimate and correct for such errors consider in practice. Consequently, estimation of response errors is inefficient, their relative importance is unknown and the optimal question format may not be discoverable. To remedy this situation, we demonstrate how multiple types of errors can be estimated concurrently with the recently introduced ‘multitrait-multierror’ (MTME) approach. MTME combines the theory of design of experiments with latent variable modelling to estimate response error variances of different error types simultaneously. This allows researchers to evaluate which errors are most impactful, and aids in the discovery of optimal question formats. We apply this approach using representative data from the United Kingdom to six survey items measuring attitudes towards immigrants that are commonly used across public opinion studies.

Hidden Markov models (HMMs) offer an attractive way of accounting and correcting for measurement error in longitudinal data as they do not require the use of a ‘gold standard’ data source as a benchmark. However, while the standard HMM assumes the errors to be independent or random, some common situations in survey and register data cause measurement error to be systematic. HMMs can correct for systematic error as well if the local independence assumption is relaxed. In this chapter, we present several (mixed) HMMs that relax this assumption with the use of two independent indicators for the variable of interest. Finally, we illustrate the results of some of these HMMs with the use of an example of employment mobility. For this purpose, we use linked survey-register data from the Netherlands.

Recently, an increasing amount of research has focused on methods to assess and account for fairness criteria when predicting ground truth targets in supervised learning. However, recent literature has shown that prediction unfairness can potentially arise due to measurement error when target labels are error prone. In this study we demonstrate that existing methods to assess and calibrate fairness criteria do not extend to the true target variable of interest, when an error-prone proxy target is used. As a solution to this problem, we suggest a framework that combines two existing fields of research: fair ML methods, such as those found in the counterfactual fairness literature and measurement models found in the statistical literature. Firstly, we discuss these approaches and how they can be combined to form our framework. We also show that, in a healthcare decision problem, a latent variable model to account for measurement error removes the unfairness detected previously.

Social science commonly studies relationships among variables by employing survey questions. Answers to these questions will contain some degree of measurement error, distorting the relationships of interest. Such distortions can be removed by standard statistical methods, when these are provided knowledge of a question’s measurement error variance. However, acquiring this information routinely necessitates additional experimentation, which is infeasible in practice. We use three decades’ worth of survey experiments combined with machine learning methods to show that survey measurement error variance can be predicted from the way a question was asked. By predicting experimentally obtained estimates of survey measurement error variance from question characteristics, we enable researchers to obtain estimates of the extent of measurement error in a survey question without requiring additional data collection. Our results suggest only some commonly accepted best practices in survey design have a noticeable impact on study quality, and that predicting measurement error variance is a useful approach to removing this impact in future social surveys. The authors thank Willem Saris for support and comments; the American Association for Public Opinion Research (AAPOR) for its support of a previous version of this work; and Wiebke Weber, Melanie Revilla and Diana Zavala-Rojas for comments on an earlier version of this manuscript.

Fair inference in supervised learning is an important and active area of research, yielding a range of useful methods to assess and account for fairness criteria when predicting ground truth targets. As shown in recent work, however, when target labels are error-prone, potential prediction unfairness can arise from measurement error. In this paper, we show that, when an error-prone proxy target is used, existing methods to assess and calibrate fairness criteria do not extend to the true target variable of interest. To remedy this problem, we suggest a framework resulting from the combination of two existing literatures: fair ML methods, such as those found in the counterfactual fairness literature on the one hand, and, on the other, measurement models found in the statistical literature. We discuss these approaches and their connection resulting in our framework. In a healthcare decision problem, we find that using a latent variable model to account for measurement error removes the unfairness detected previously., Comment: Online supplementary code is available at https://dx.doi.org/10.5281/zenodo.3708150

Longitudinal surveys often rely on dependent interviewing (DI) to lower the levels of random measurement error in survey data and reduce the incidence of spurious change. DI refers to a data collection technique that incorporates information from prior interview rounds into subsequent waves. While this method is considered an eective remedy for random measurement error, it can also introduce more systematic errors, in particular when respondents are rst reminded of their previously provided answer and then asked about their current status. The aim of this paper is to assess the impact of DI on measurement error in employment mobility. We take advantage of a unique experimental situation that was created by the roll-out of dependent interviewing in the Dutch Labour Force Survey (LFS). We apply Hidden Markov Modeling (HMM) to linked LFS and Employment Register (ER) data that cover a period before and after dependent interviewing was abolished, which in turn enables the modeling of systematic errors in the LFS data. Our results indicate that DI lowered the probability of obtaining random measurement error but had no signicant eect on the systematic component of the error. The lack of a signicant eect might be partially due to the fact that the probability of repeating the same error was extremely high at baseline (i.e when using standard, independent interviewing); therefore the use of DI could not increase this probability any further., Survey Research Methods, Vol 15 No 2 (2021)

Some textbooks on questionnaire design claim it is an art. That would make the criterion for a “good” question entirely subjective—a worrying conclusion given that surveys are often used to discover important facts about people. Are our discoveries about people also entirely subjective? This chapter shows that it is possible to study what a “good” or a “bad” question is by experimentation. There is already a body of scientific evidence on questionnaire design that can be taken into account when designing a questionnaire. The chapter reviews some of this evidence and shows how it can be used to the advantage of the survey researcher. Questionnaire science is far from complete. On the one hand, this means that some of our conclusions may still be more art than science. On the other, it means that we can agree on one aspect of questionnaire science: more of it is needed.