Search

In the context of the current “replication crisis” across the sciences, failures to reproduce a finding are often viewed as discrediting it. This paper shows how such a conclusion can be incorrect. In 1981, Schuman and Presser showed that including the word “freedom” in a survey question significantly increased approval of allowing a speech against religion in the USA. New experiments in probability sample surveys (n = 23,370) in the USA and 10 other countries showed that the wording effect replicated in the USA and appeared in four other countries (Canada, Germany, Taiwan, and the Netherlands) but not in the remaining countries. The effect appeared only in countries in which the value of freedom is especially salient and endorsed. Thus, public support for a proposition was enhanced by portraying it as embodying a salient principle of a nation’s culture. Instead of questioning initial findings, inconsistent results across countries signal limits on generalizability and identify an important moderator.

The article examines two important aspects of data quality in self-completion surveys of young people, taking advantage of a unique data source: Understanding Society: the United Kingdom Household Longitudinal Study. Young persons aged 10–15 are asked to complete a self-administered paper questionnaire at annual intervals. The number of completed interviews varies over waves from 4,049 to 5,020. Data are also collected from parents, providing important explanatory covariates for our analysis. Stronger parent–child relationship and higher mother’s involvement in education were associated with lower item nonresponse rate and lower inconsistency throughout waves. We also found some evidence for a negative panel conditioning effect with an increase of social desirability bias and measurement errors in the subsequent waves. There was a higher level of inconsistent responses and a higher probability of social desirability bias throughout waves in more sensitive items.

Prospect theory states that the influential power of avoiding negative outcomes is stronger than that of achieving positive outcomes. In a survey context, this theory has been tested with respect to not only participation in a CATI survey, but also giving consent to data linkage in CATI surveys. No study, however, has tested the theory with respect to participation in a CAPI or web survey. This study does so in a mixed-mode panel context; it also tests the moderating effects of time-in-panel, response history, and mode protocol. Results show that the framing of the survey participation request influences participation propensity in a way consistent with prospect theory, but only for relatively recent panel entrants. The opposite effect is found for long-term panel participants. No difference is found between mode protocols.

Background Improving response rates in epidemiologic studies is important for the generalizability of the outcome. The aim of this study was to examine whether it can be advantageous for participation to target different versions of the cover letters to different sample subgroups. Methods A randomized trial was incorporated in a cross-sectional health survey in Denmark (n = 25,000) where a motivational sentence in the cover letter intended to heighten perceptions of relevance of the survey was varied among 11 sample subgroups (treatment groups). Ten different versions of a sentence outlining questionnaire themes were tested: each mentioned three out of five themes: stress, alcohol, sex, sleep problems, and contact with family and friends. An eleventh group, the control group, omitted this sentence. Results On average, the additional motivational sentence resulted in a significantly lower response rate overall compared to the control group. However, the additional motivational sentence was found to have heterogeneous effects on survey response. Furthermore, the nature of the heterogeneity differed between the versions of the sentence. Specifically, the additional sentence tended to produce a higher response rate among the youngest age group and a lower response rate in the oldest age group compared to the generic letter. The use of alcohol in the motivational sentence tended to have a positive effect on response in the age group 16–24 years, and stress tended to have a positive effect in the age group ≥65 years. On the contrary, sex tended to have a negative effect in the age groups 45–64 years and ≥ 65 years. However, a significant interaction was only found between the use of stress and age group (p =

Previous research into survey invitation letters and prenotification letters has sought to identify optimal features of a standard letter that can be sent to all sample members. In the spirit of adaptive design, this study seeks to establish whether it can be advantageous to target different versions of a letter to different sample subgroups. Specifically, a paragraph intended to heighten perceptions of relevance of the survey is varied among six subgroups in a panel survey. Random allocation to control and treatment groups is crossed with variants of two other design features: time in sample and data collection mode. This enables analysis of the effects of the targeted letter in different survey contexts. The targeted version of the letter improved response rates for two operationally important low-response propensity groups, but only in certain survey design contexts.

Web questionnaires, including those used in mixed-mode surveys, generally produce higher levels of item nonresponse than interviewer-administered questionnaires. Item nonresponse is generally seen as having a detrimental impact on data quality. The current study examines using motivational statements to reduce item nonresponse in a web survey component of a mixed-mode design. The effects of alternative implementations are compared, both for web surveys and for mixed-mode surveys. In addition, the mixed-mode results are compared to a face-to-face survey. The current study adds to the literature on the use of motivational statements by using a unique large-scale randomized experiment to examine the impact of the timing of the motivational statement, and to compare with the same survey in an interviewer-administered context. Findings show that a motivational statement following immediately after an item is left unanswered greatly outperforms either the control or a motivational statement at a later point in the survey. Using this immediate reactive prompt reduces item nonresponse to levels equivalent to a face-to-face version. Conversely, the control (no statement) and later placed motivational statement lead to significantly greater item nonresponse. Point estimates for the tested variables are not affected by the additional responses obtained. The results suggest practical design implications to reduce item nonresponse when using a web design, specifically the use of a reactive motivational prompt in a planned way.

Over time, some respondents have been recruited to participate in longitudinal research may stop responding to survey requests. This phenomenon is known as panel attrition and it occurs when a longitudinal panel fails to retain all recruited members. While there are many potential causes of this attrition, there are a number of concrete steps that researchers that manage panels can take to tackle the problem of panel attrition. This chapter provides an overview of the causes and consequences of panel attrition. It also highlights strategies that may be effective in addressing this problem by reviewing innovations and best practices for minimizing panel attrition and also reducing the statistical impacts of attrition when it does occur.

We examine the effects on survey estimates of extended interviewer efforts to gain survey response, including refusal conversion attempts and attempts to make contact with difficult-to-contact sample members. Previous research on this topic has identified that extended efforts do appear to affect estimates, and in ways that seem consistent with bias reduction. We extend the previous research in three ways. First, we provide the first study of changes over time in the effects of extended efforts on estimates. We study change in the UK over a ten-year period. Second, we use a more precise measure of the difficulty of contact and third, we assess the effects of extended efforts conditional on weight adjustments for non-response estimates as well as on unweighted sample statistics.

This book chapter describes the main issues to be considered when a sample of people (including groups of people such as households, families, etc) is designed for research purposes. Much of the discussion is about sampling for interview surveys, though important sampling issues for other data collection methods are also mentioned.

SummarySurvey organizations often attempt to ‘convert’ sample members who refuse to take part in a survey. Persuasive techniques are used in an effort to change the refusers’ minds and to agree to an interview. This is done to improve the response rate and, possibly, to reduce non-response bias. However, refusal conversion attempts are expensive and must be justified. Previous studies of the effects of refusal conversion attempts are few and have been restricted to cross-sectional surveys. The criteria for ‘success’ of a refusal conversion attempt are different in a longitudinal survey, where for many purposes the researcher requires complete data over multiple waves. The paper uses data from the British Household Panel Survey from 1994 to 2003 to assess the long-term effectiveness of refusal conversion procedures in terms of sample sizes, sample composition and data quality.

The effects of unit non-response on survey errors are of great concern to researchers.However, direct assessment of non-response bias in survey estimates is rarely possible.Attempts are often made to adjust for the effects of non-response by weighting, but thisusually relies on the use of frame data or external population data, which are at bestmodestly correlated with the survey variables. This paper reports the development ofa method to collect limited survey data from non-respondents to personal interviewsurveys and a large-scale field test of the method on the British Crime Survey (BCS).The method is shown to be acceptable and low cost, to provide valid data, and to haveno detrimental effect on the main survey. The use of the resultant data to estimatenon-response bias is illustrated and some substantive conclusions are drawn for the BCS.

Survey researchers often assume that respondent burden is an important determinant of survey participation propensity and that interview length is a good indicator of burden. However, there is surprisingly little evidence of the effect of length of a completed interview on subsequent participation propensity, particularly in the case of face-to-face surveys. This article presents results from a large-scale randomized experiment in which respondents experienced interviews of different lengths at wave 1 of a panel survey. Subsequently, respondents were asked to complete a self-completion questionnaire and then to take part in further waves of the survey. For each of these subsequent tasks, the study compares completion rates between those administered the shorter and those administered the longer version of the wave 1 interview. No evidence is found that wave 1 interview length affects subsequent participation propensity.

In this journal, Elliott and Ellingworth (1997)reported their attempts to assess the impact of certain sources of survey error on the British Crime Survey. This article attempts to correct some flaws in their article, to place their results in a wider context, and to provide some further - arguably more robust - estimates of the impact of non-response bias.

If you talk to a keen angler, he or she will be replete with stories about favourite locations for angling, special bait, or a particularly effective way of casting; in short, those aspects of their craft that increase the probability of a catch. With a modest stretch of the imagination, angling can be thought of as an analogy for gaining survey co-operation. Survey researchers and interviewers are replete with their own stock of practices that they believe increase the probability of a sample member taking part in their survey. Both the angling enthusiast and the survey researcher may be able to draw on some evidence regarding the extent to which particular methods tend to be successful and the mechanisms through which this success is achieved. But both will also be influenced by subjective personal preferences and beliefs, often with little or no basis in evidence. The angling enthusiast is unlikely, however, to

Mobile phone survey interviewing is becoming increasingly prevalent, so it is important to understand how it affects measurement error. We link cognitive response process theory (Tourangeau et al 2000) to theories of mode effects to build a comprehensive framework that identifies ways in which the different context of mobile interviews and differences inherent in the technology can influence survey responses. We evaluate elements of the framework using data from a randomised experiment in which respondents were assigned to be interviewed either by mobile or fixed phone and in which relevant measures of interview context were gathered. Our findings address causes of survey error resulting from mode of data collection, both in general and in the specific context of mobile phones. We also outline needed future research.

Imagine a survey question that asks, “How many hours do you spend watching TV in a typical week?” A respondent may answer “16”. Whether that is the ‘correct’ answer depends on what exactly the researcher intended the question to measure. The respondent may have come up with his answer by thinking along the following lines: “The TV is on for about two hours each evening, so that’s 14 hours, plus I usually watch the sport for a couple of hours on Saturday afternoon, so that makes about 16 in total.” But maybe the researcher only had in mind time spent actually paying attention to the content of a TV broadcast, so having the TV turned on while devoting one’s attention to something different should not be counted. If, during the two hours each evening that the respondent’s TV is on, he is also eating dinner and talking to his children, then he may only be paying attention to the TV for half of that time. So, in the researcher’s terms the ‘correct’ answer would have been nine hours. There may be many people in the survey sample who, like this respondent, have included in their response to this question time that the TV is switched on even if they are not really paying attention to it. This will result in the researcher over-estimating the time people spend attentively watching TV. This error in estimation could have important consequences.

We conduct two large-scale randomised experiments on the British Household Panel Survey (BHPS) to study methods of reducing panel attrition. The first experiment compares different strategies for tracing and maintaining contact with sample members. We find that change-of-address cards conditional on moving are most effective both in increasing response rates and in reducing fieldwork costs. The second experiment substitutes the standard between-wave Respondent Report with reports tailored for specific categories of respondents. We find that tailored reports have a positive though small effect on the number of young people completing a face-to-face interview and increase cooperation for busy people when telephone interviews are included. Keywords: attrition, non-response, tailoring, respondent incentives, longitudinal surveys, experiments JEL codes: C81, C83, C90 Acknowledgements This research was funded by the UK Economic and Social Research Council (ESRC) Survey Design and Measurement Initiative as part of the project, “Understanding non response and reducing non-response bias” (principal investigator Peter Lynn, award number RES-175-25-0005) and forms part of the research programme of the UK Longitudinal Studies Centre, an ESRC-funded Resource Centre based at ISER. We are grateful to Mick Couper and to participants at the 2009 European Survey Research Association conference in Warsaw for useful comments and suggestions; and to our ISER colleagues Sandra Jones, Colette Lo, Mike Merrett and Jon Burton for administering the experiment and recording the outcomes.

Despite the increased interest in assessing the effects of dependent interviewing (DI) on data quality, the impact of DI on interviewer burden and interviewer—respondent interaction remains currently unexplored and undocumented. Drawing on Japec’s work, this article attempts to evaluate the impact of DI on interviewer burden and describes the mechanisms by which DI affects perceived interviewer burden and ultimately data quality. We use a diverse set of data including qualitative data collected from a survey carried out in 2006 on the British Household Panel Survey (BHPS) interviewers. Three main findings stand out: DI is usually welcomed by interviewers; under condition of no change in respondent circumstances, DI eases interviewer burden; the mechanisms by which DI eases interviewer burden are complex as interviewer burden factors are often strictly related to respondent burden; and there are strong interrelationships among the different factors that constitute interviewer burden.

Preface. 1. Methods for Longitudinal Surveys (Peter Lynn). 1.1 Introduction,. 1.2 Types of Longitudinal Surveys,. 1.3 Strengths of Longitudinal Surveys. 1.4 Weaknesses of Longitudinal Surveys. 1.5 Design Features Specific to Longitudinal Surveys. 1.6 Quality in Longitudinal Surveys. 1.7 Conclusions. References. 2. Sample Design for Longitudinal Surveys (Paul Smith, Peter Lynn and Dave Elliot). 2.1 Introduction. 2.2 Types of Longitudinal Sample Design. 2.3 Fundamental Aspects of Sample Design. 2.4 Other Aspects of Design and Implementation. 2.5 Conclusion. References. 3. Ethical Issues in Longitudinal Surveys (Carli Lessof). 3.1 Introduction. 3.2 History of Research Ethics. 3.3 Informed Consent. 3.4 Free Choice Regarding Participation. 3.5 Avoiding Harm. 3.6 Participant Confidentiality and Data Protection. 3.7 Independent Ethical Overview and Participant Involvement. Acknowledgements. References. 4. Enhancing Longitudinal Surveys by Linking to Administrative Data (Lisa Calderwood and Carli Lessof). 4.1 Introduction. 4.2 Administrative Data as a Research Resource. 4.3 Record Linkage Methodology. 4.4 Linking Survey Data with Administrative Data at Individual Level. 4.5 Ethical and Legal Issues. 4.6 Conclusion. References. 5. Tackling Seam Bias Through Questionnaire Design (Jeffrey Moore, Nancy Bates, Joanne Pascale and Aniekan Okon). 5.1 Introduction. 5.2 Previous Research on Seam Bias. 5.3 SIPP and its Dependent Interviewing Procedures. 5.4 Seam Bias Comparison - SIPP 2001 and SIPP 2004. 5.5 Conclusions and Discussion. References. 6. Dependent Interviewing: A Framework and Application. to Current Research (Annette Jackle) . 6.1 Introduction. 6.2 Dependent Interviewing - What and Why? 6.3 Design Options and their Effects. 6.4 Empirical Evidence. 6.5 Effects of Dependent Interviewing on Data Quality Across Surveys. 6.6 Open Issues. References. 7. Attitudes Over Time: The Psychology of Panel Conditioning (Patrick Sturgis, Nick Allum and Ian Brunton-Smith). 7.1 Introduction. 7.2 Panel Conditioning. 7.3 The Cognitive Stimulus Hypothesis. 7.4 Data and Measures. 7.5 Analysis. 7.6 Discussion. References. 8. Some Consequences of Survey Mode Changes in Longitudinal Surveys (Don A. Dillman). 8.1 Introduction. 8.2 Why Change Survey Modes in Longitudinal Surveys? 8.3 Why Changing Survey Mode Presents a Problem. 8.4 Conclusions. References. 9. Using Auxiliary Data for Adjustment in Longitudinal Research (Dirk Sikkel, Joop Hox and Edith de Leeuw). 9.1 Introduction. 9.2 Missing Data. 9.3 Calibration. 9.4 Calibrating Multiple Waves. 9.5 Differences Between Waves. 9.6 Single Imputation. 9.7 Multiple Imputation. 9.8 Conclusion and Discussion. References. 10. Identifying Factors Affecting Longitudinal Survey Response (Nicole Watson and Mark Wooden). 10.1 Introduction. 10.2 Factors Affecting Response and Attrition. 10.3 Predicting Response in the HILDA Survey. 10.4 Conclusion. References. 11. Keeping in Contact with Mobile Sample Members (Mick P. Couper and Mary Beth Ofstedal). 11.1 Introduction. 11.2 The Location Problem in Panel Surveys. 11.3 Case Study 1: Panel Study of Income Dynamics. 11.4 Case Study 2: Health and Retirement Study. 11.5 Discussion. References. 12. The Use of Respondent Incentives on Longitudinal Surveys (Heather Laurie and Peter Lynn). 12.1 Introduction. 12.2 Respondent Incentives on Cross-Sectional Surveys. 12.3 Respondent Incentives on Longitudinal Surveys. 12.4 Current Practice on Longitudinal Surveys. 12.5 Experimental Evidence on Longitudinal Surveys. 12.6 Conclusion. Acknowledgements. References. 13. Attrition in Consumer Panels (Robert D. Tortora). 13.1 Introduction. 13.2 The Gallup Poll Panel. 13.3 Attrition on the Gallup Poll Panel. 13.4 Summary. References. 14. Joint Treatment of Nonignorable Dropout and Informative Sampling. for Longitudinal Survey Data (Abdulhakeem A. H. Eideh and Gad Nathan) . 14.1 Introduction. 14.2 Population Model. 14.3 Sampling Design and Sample Distribution. 14.4 Sample Distribution Under Informative Sampling and Informative Dropout. 14.5 Sample Likelihood and Estimation. 14.6 Empirical Example - British Labour Force Survey. 14.7 Conclusions. References. 15. Weighting and Calibration for Household Panels (Ulrich Rendtel and Torsten Harms). 15.1 Introduction. 15.2 Follow-up Rules. 15.3 Design-Based Estimation. 15.4 Calibration, 274. 15.5 Nonresponse and Attrition. 15.6 Summary. References. 16. Statistical Modelling for Structured Longitudinal Designs (Ian Plewis). 16.1 Introduction. 16.2 Methodological Framework. 16.3 The Data. 16.4 Modelling One Response from One Cohort. 16.5 Modelling One Response from More Than One Cohort. 16.6 Modelling More Than One Response from One Cohort. 16.7 Modelling Variation Between Generations. 16.8 Conclusion. References. 17. Using Longitudinal Surveys to Evaluate Interventions (Andrea Piesse, David Judkins and Graham Kalton). 17.1 Introduction. 17.2 Interventions, Outcomes and Longitudinal Data. 17.3 Youth Media Campaign Longitudinal Survey. 17.4 National Survey of Parents and Youth. 17.5 Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP). 17.6 Concluding Remarks. References. 18. Robust Likelihood-Based Analysis of Longitudinal Survey Data with Missing Values (Roderick Little and Guangyu Zhang). 18.1 Introduction. 18.2 Multiple Imputation for Repeated-Measures Data. 18.3 Robust MAR Inference with a Single Missing Outcome. 18.4 Extensions of PSPP to Monotone and General Patterns. 18.5 Extensions to Inferences Other than Means. 18.6 Example. 18.7 Discussion. Acknowledgements. References. 19. Assessing the Temporal Association of Events Using Longitudinal Complex Survey Data (Norberto Pantoja-Galicia, Mary E. Thompson and Milorad). S. Kovacevic. 19.1 Introduction. 19.2 Temporal Order. 19.3 Nonparametric Density Estimation. 19.4 Survey Weights. 19.5 Application: The National Population Health Survey. 19.6 Application: The Survey of Labour and Income Dynamics. 19.7 Discussion. References. 20. Using Marginal Mean Models for Data from Longitudinal Surveys with a Complex Design: Some Advances in Methods (Georgia Roberts, Qunshu Ren and J.N.K. Rao). 20.1 Introduction. 20.2 Survey-Weighted GEE and Odds Ratio Approach. 20.3 Variance Estimation: One-Step EF-Bootstrap. 20.4 Goodness-of-Fit Tests. 20.5 Illustration Using NPHS Data. 20.6 Summary. References. 21. A Latent Class Approach for Estimating Gross Flows in the Presence of Correlated Classification Errors (Francesca Bassi and Ugo Trivellato). 21.1 Introduction. 21.2 Correlated Classification Errors and Latent Class Modelling. 21.3 The Data and Preliminary Evidence from Them. 21.4 A Model for Correlated Classification Errors in Retrospective Surveys. 21.5 Concluding Remarks. References. 22. A Comparison of Graphical Models and Structural Equation Models. for the Analysis of Longitudinal Survey Data )Peter W. F. Smith, Ann Berrington and Patrick Sturgis) . 22.1 Introduction. 22.2 Conceptual Framework. 22.3 Graphical Chain Modelling Approach. 22.4 Structural Equation Modelling Approach. 22.5 Model Fitting. 22.6 Results. 22.7 Conclusions. References. Index.

SummaryWe analyse patterns of consent and consent bias in the context of a large general household survey, the ‘Improving survey measurement of income and employment’ survey, also addressing issues that arise when there are multiple consent questions. A multivariate probit regression model for four binary outcomes with two incidental truncations is used. We show that there are biases in consent to data linkage with benefit and tax credit administrative records that are held by the Department for Work and Pensions, and with wage and employment data held by employers. There are also biases in respondents’ willingness and ability to supply their national insurance number. The biases differ according to the question that is considered. We also show that modelling questions on consent independently rather than jointly may lead to misleading inferences about consent bias. A positive correlation between unobservable individual factors affecting consent to Department for Work and Pensions record linkage and consent to employer record linkage is suggestive of a latent individual consent propensity.

Refusal by sample units and failure to contact sample units can both contribute to survey non-response bias. However, the nature of the contribution can be rather different in the two cases. Extended field efforts may reduce either or both components of non-response bias. We examine data from several UK surveys on extended field efforts and the effect that they appear to have on non-response bias and its components. Some consistent patterns are found. We also explore the sensitivity of such analyses to the operational definition of extended efforts. The findings provide some evidence of the relative roles of refusal conversion and repeated contact attempts in reducing non-response bias. The study also has implications for the development of appropriate field strategies to combat non-response bias.