Search

This paper attempts to aid the process of accumulating the necessary information for making more informed judgments about the effects of nonresponse under different conditions. Two measures, which permit quantifiable nonsubjective assessment of the effects of nonresponse on sample estimates, are introduced and are used to examine the effects of respondent refusals in a random-digit-dialed general population telephone survey of over 1,200 households as the response rate is increased from 74.5 percent to 86.8 percent. By applying these measures under a wide range of conditions, the adequacy of various response rates may be assessed and more rational decisions made about the costs and benefits of devoting extraordinary resources to minimizing nonresponse. Michael J. O'Neil is a postdoctoral fellow at the Survey Research Center of the Institute for Social Research and Lecturer in Sociology at the University of Michigan. This research was supported in part by a grant (EPP75-08963) from the National Science Foundation for which the author was principal investigator. Drafts of this paper have been read by Norman Bradburn, Don Dillman, Fredric DuBow, Andrew Gordon, Marilyn Johnson, Richard McCleary, Stanley Presser, Kent Smith, and Seymour Sudman. An earlier version of this paper was presented at the American Association for Public Opinion Research Annual Meetings, Roanoke, Virginia, June 1978. Public Opinion Quarterly ? 1979 by The Trustees of Columbia University Published by Elsevier North-Holland, Inc. 0033-362X/79/0043-0218/$1.75 This content downloaded from 157.55.39.116 on Thu, 02 Mar 2017 05:51:54 UTC All use subject to http://about.jstor.org/terms NONRESPONSE BIAS IN TELEPHONE SURVEYS 219 techniques tend not to be systematically applied, and when they are there is almost always residual nonresponse. Proposals for dealing with nonresponse have generally involved either (1) adjusting for nonresponse bias ex post facto, or (2) estimating the maximum potential distortion it can cause. Daniel (1975) reviewed several classic adjustment procedures.2 Recently suggested adjustments have included extrapolating observed differences between easy-to-reach and difficult-to-reach respondents (Filion, 1975) and weighting respondents according to known demographic population parameters (Mandell, 1974). While the particular assumptions differ, all these adjustments are problematic in that they assume precisely what is at issue; that is, the nature of the differences between respondents and nonrespondents. Cochran (1963) suggested an alternative approach for surveys with low nonresponse which requires no such assumptions. By assuming extreme values for nonrespondents, the maximum bias that could be caused by nonresponse under varying conditions can be specified. Fuller (1974) and Wayne (1975) used Cochran's formulas to estimate the size of these expanded "confidence intervals" under a variety of response rates, sample sizes, and estimated proportions. Looking at these under a range of realistic conditions, however, we find that the calculated "confidence intervals" are almost always far too wide to permit most substantively meaningful inferences. Furthermore, this approach also tells us nothing of the probable direction of the bias, only how large it could be. To reduce the width of these "confidence intervals" to the point where they might be usable or to specify the direction of the impact of nonresponse, empirical data are required on tion, cover letters, anonymity, and cash incentives. In addition, Kanuk and Berenson (1975) dealt with the effectiveness of including return envelopes, various forms of questionnaire reproduction, and specifying deadline dates. Among the most promising techniques in terms of the results obtained have been the methods developed by Dillman and his associates (Dillman, 1972; Dillman et al., 1974). A similar literature on increasing response rates for telephone surveys is almost nonexistent. The absence of a comparable body of knowledge on which to draw may help explain why attempts to increase response rates in telephone surveys have often been less successful (Dillman et al., 1976). The comparative absence of research on increasing response rates over the phone is probably due to the fact that mail surveys have traditionally been more extensively used and have usually experienced higher nonresponse rates. Even if more "tailored" methods for phone surveys were developed, however, there would inevitably be some nonresponse and little means of assessing its impact. None of these methods in any way address the question of the extent of the bias induced by various levels and sources of nonresponse. 2 Specifically, Daniel reviews the Hansen and Hurwitz (1946) proposal to subsample nonrespondents, the Politz-Simmons (1949, 1950) times-at-home procedure, the Kish and Hess (1959) replacement procedure, the Bartholomew (1961) two-call technique, Hendricks' (1956) extrapolation method, Ericson's (1967) Bayesian approach, and Dalenius' (1955, 1957) variable selection probability method for selection within the household. This content downloaded from 157.55.39.116 on Thu, 02 Mar 2017 05:51:54 UTC All use subject to http://about.jstor.org/terms 220 MICHAEL J. O'NEIL what actually does (not just what could) happen to various estimates under differing conditions with less than complete response. Most of the empirical examinations of the actual effects of nonresponse have almost invariably been based on specialized samples not easily generalizable: migratory duck hunters (Filion, 1975), elderly males in Iowa (Goudy, 1976), college alumni from a particular institution (Reuss, 1943), science talent search participants (Edgarton et al., 1947), and contributors to the National Committee for an Effective Congress (Bachrack and Scoble, 1967). It is not, however, only the samples that have been specialized. Almost all empirical examinations of nonresponse bias (including all the above) have been based on mail surveys. Not only was Dunkelberg and Day's (1973) examination of the effects of increasing numbers of callbacks in personal interviews a rare exception to this rule, it was one of the few empirical examinations of nonresponse to employ a general population sample. The objectives of this paper are twofold: first, to present comparable data for telephone surveys within the context of a general population survey; second, because there is considerable evidence that the characteristics of refusers and "not-at-homes" differ (Colombotos, 1962; Mayer and Pratt, 1966; Wilcox, 1977), to isolate the effects of refusals. Data Collection Procedures The data analyzed in this paper are taken from a general population telephone survey of 1,209 Chicago households. Several measures were employed to insure maintenance of equal selection probabilities. First, random-digit-dialing was used to include households with unlisted telephones. Second, verification of the number reached was required to discover misdials and account for the telephone company switching system's erratic disposition of nonworking numbers dialed. Third, exhaustive callbacks of persons not at home (up to 20 calls at staggered times) were undertaken to insure that all households were reached and to allow for separation of the effects of "refusals" and "unavailables." Finally, persons who refused to be interviewed on the first contact3 were mailed a "persuasion letter"4 and recontacted. How effective were these procedures in minimizing nonresponse? 3When it was unclear whether a respondent's request to "call me back later" represented a disguised refusal or a genuine request to be contacted at a more convenient time, respondents were taken literally and were not considered refusals. Only when it became obvious that this was really a refusal was it treated as such. For this reason, the "second contact" referred to in this discussion had been preceded in some cases by more than a single phone contact. 4 Because addresses could not be obtained for the unlisted telephone numbers, these letters could only be sent to the portion of the sample with listed telephones. Both listed and unlisted households, however, were recontacted by phone. This content downloaded from 157.55.39.116 on Thu, 02 Mar 2017 05:51:54 UTC All use subject to http://about.jstor.org/terms NONRESPONSE BIAS IN TELEPHONE SURVEYS 221 Results are presented in Figure 1. From 1,392 eligible households 5 there were 1,037 completed interviews6 after the first contact and 1,209 after the second. These represent completion rates of 74.5 percent and 86.8 percent, respectively. Ideally, one would like to compare groups C and D in Figure 1. Unfortunately, very little is known about group D except that they refused, at least once, to be interviewed. We attempt to approximate this comparison by comparing group C with group E, which was interviewed after having refused on the first attempt.7 We cannot be sure how groups E and F differ, Figure 1. Interview Attempt Outcomes

The characteristics of a very small group of people who refused to participate in a personal interview panel study are investigated in this article. Comparisons are made between characteristics of respondents and refusers in the initial interview and subsequent interviews. Details of the refusal situation and reasons given by refusers for their noncooperation are also discussed. Theresa J. DeMaio is a survey statistician at the U.S. Bureau of the Census. The author wishes to acknowledge the assistance of her coworkers at the Bureau, in particular Naomi D. Rothwell, who designed and made arrangements for the reports on which this paper is based, and who critically reviewed drafts of it. Public Opinion Quarterly ? 1980 by The Trustees of Columbia University Published by Elsevier North Holland, Inc. 0033-362X/80/0044-223/$1.75 This content downloaded from 207.46.13.157 on Thu, 09 Jun 2016 06:53:38 UTC All use subject to http://about.jstor.org/terms 224 THERESA J. DE MAIO sons are more likely to refuse than their counterparts. On the other hand, he found no substantial difference on the basis of either sex or income in a one-time survey conducted by the University of Michigan. Concern at the Census Bureau about potential nonresponse bias has led to studies of nonrespondent characteristics and the effect of their exclusion on survey results. Unlike previous research, however, these attempts are successful in obtaining information about a national cross-section of nonrespondents. Until now, investigation of the refusal component of nonresponse has been limited to those noncooperative households for which some information has already been obtained-i.e., respondents in panel surveys who refused after having been previously cooperative. This research (Palmer, 1966) showed that, although the characteristics of nonrespondents as a whole biased the survey estimates, there were no differences between the small group of refusers (who had previously cooperated) and cooperative respondents in the Current Population Survey. However, Palmer described her inability to provide information about an appreciable proportion of refusals as a major limitation of the findings. Studies of refusals are important not only to determine what biases exist but also to see whether knowledge about refusers might suggest appeals and procedures to reduce their numbers. Accordingly, a project was undertaken which had two main purposes: to learn about characteristics of refusers in households which have not been interviewed before, and to explore the role of the Privacy Act with regard to refusal rates (which have increased from 1 percent in the 1960s to the present 3 percent in the Current Populafion Survey). The Privacy Act requires explicit notification of potential respondents that participation in the survey is voluntary, and that no penalties are involved in refusal. In order to assess the prominence of the Privacy Act as a factor in refusals, information was collected from refusers about their reasons for noncooperation. To date, little systematic research has been conducted on this important topic.

Valid predictions for the direction of nonresponse bias were obtained from subjective estimates and extrapolations in an analysis of mail survey data from published studies. For estimates of the magnitude of bias, the use of extrapolations led to substantial improvements over a strategy of not using extrapolations.

Methods for estimating nonresponse bias are reviewed and several methods are tried on the 1980 GSS. The results indicate that various estimating procedures are inappropriate and that even the more promising techniques can provide faulty estimates of nonresponse bias. By its nature, nonresponse bias is very difficult to assess accurately and no simple, certain method exists. Tom W. Smith is Senior Study Director, National Opinion Research Center, University of Chicago. This research was done for the General Social Survey Project directed by James A. Davis. The project is supported by the National Science Foundation, Grant No. SOC77-03279. This is an abridged version of GSS Technical Report No. 25 published by NORC, 1981. The author wishes to thank James A. Davis, Howard Schuman, and Stanley Presser for their comments. Public Opinion Quarterly Vol. 47:386-404 ? 1983 by the Trustees of Columbia University Published by Elsevier Science Publishing Co., Inc. 0033-362X/83/0047-386/$2.50 This content downloaded from 157.55.39.138 on Sun, 26 Jun 2016 06:02:31 UTC All use subject to http://about.jstor.org/terms NONRESPONSE ON THE 1980 GSS 387 ponse rate, we do not know the nonresponse mean since we have no measure of Y among nonrespondents. Two alternatives are usually presented in discussing nonresponse-how to minimize nonresponse and how to estimate and correct for differences between the respondents and nonrespondents. In this paper we ignore the first alternative, accepting that a nonresponse rate of .25 is typical for good, state-of-the-art surveys (Smith, 1978; Davis et al., 1980; and Groves and Kahn, 1979). Instead, we will review the various existing approaches to estimating the characteristics of nonrespondents and then apply several of the proposed approaches to nonresponse on the 1980 GSS. Measuring Nonrespondents and Assessing Nonresponse Bias Numerous methods have been proposed to estimate the attributes of nonrespondents (Daniel, 1975). Some are appropriate for certain types of surveys (e.g., list samples only) while others can be used with modification across various methods of administration with various sample frames (e.g., from mail lists to RDD telephone). Attention will focus primarily on methods that are appropriate, or at least have been offered as appropriate, for face-to-face, national surveys. Among other things, this eliminates list samples where information about the respondent is known prior to the survey. Our review of nonresponse studies found nine major approaches to assess and adjust for nonresponse: 1. External population checks 2. Geographic/aggregate level data 3. Interviewer estimates 4. Interviewing nonrespondents about nonresponse 5. Subsampling of nonrespondents 6. Substitution for nonrespondents 7. Politz-Simmons adjustment 8. Extrapolation based on difficulty 9. Conversion adjustments Probably the simplest check is to compare sample estimates (usually distributions) to some universe figures or preferred sample estimates such as the U.S. Census or the Current Population Survey (Crossley and Fink, 1951; Stephen and McCarthy, 1958: Smith, 1979; and Presser, 1981). Strictly speaking, when using such a criterion comparison, one is not checking how much difference comes from nonresponse but how much comes from nonresponse and all other This content downloaded from 157.55.39.138 on Sun, 26 Jun 2016 06:02:31 UTC All use subject to http://about.jstor.org/terms

A household budget survey often suffers from a high nonresponse rate and a selective response. The bias that may be introduced in the estimation of budget shares because of this nonresponse can affect the estimate of a consumer price index, which is a weighted sum of partial price index numbers (weighted with the estimated budget shares). The bias is especially important when related to the standard error of the estimate. Because of the impossibility of subsampling nonrespondents to the budget survey, no exact information on the bias can be obtained. To evaluate the nonresponse bias, bounds for this bias are calculated using linear programming methods for several assumptions. The impact on a price index of a high nonresponse rate among people with a high income can also be assessed by using the elasticity with respect to total expenditure. Attention is also given to the possible nonresponse bias in a time series of price index numbers. The possible nonresponse bias is much larger than the standard error o...

Summary Theoretical properties of nonresponse adjustments based on adjustment cells are studied, for estimates of means for the whole population and in subclasses that cut across adjustment cells. Three forms of adjustment are considered: weighting by the inverse response rate within cells, post-stratification on known population cell counts, and mean imputation within adjustment cells. Two dimensions of covariate information x are distinguished as particularly useful for reducing nonresponse bias: the response propensity f(x) and the conditional mean ^(x) of the outcome variable y given x. Weighting within adjustment cells based on ^f(x) controls bias, but not necessarily variance. Imputation within adjustment cells based on ^(x) controls bias and variance. Post-stratification yields some gains in efficiency for overall population means, and smaller gains for means in subclasses of the population. A simulation study similar to that of Holt & Smith (1979) is described which explores the mean squared error properties of the estimators. Finally, some modifications of response propensity weighting to control variance are suggested.

THIS ARTICLE recommends a statistical hypothesis test to investigate nonresponse bias. Even in a well-designed survey, the unknown opinions of individuals who fail to respond cannot contribute to estimates of population preferences. Nonresponse bias occurs if there is a difference between the preferences of these nonrespondents and those of the responders on whom estimates are based. The magnitude of this difference times the proportion of nonrespondents gives the expected size of the bias. Thus the problem is particularly acute for mail surveys which, though popular for being inexpensive, sometimes have response rates well below 50 percent. Interestingly, the problem of nonresponse bias in mail surveys was noted in the statistical literature as early as 1838 (Porter, 1838). Researchers who conduct mail surveys try to reduce the size of the nonresponse bias in a variety of ways, most notably through efforts to stimulate higher participation (Linsky, 1975). For a given level of participation, they may attempt to adjust their estimates through the application of reasonable sample weights. One method gives greater weights to respondents who took longer or were more difficult to bring into the sample on the assumption that they more closely resemble the nonrespondents (Pace, 1939; Politz and Simmons, 1949). A second

A callback policy must balance the reduced yield (and higher cost) from additional callbacks against the benefits of a more representative final sample. This study provides evidence on the rate at which sample values converge on their estimated population distribution as the number of callbacks increases.

Survey researchers are necessarily concerned about possible bias in their survey data resulting from inability to interview all persons selected by the sampling plan. The proportion of persons in the sample with a particular socioeconomic characteristic, for example, may be underor overrepresented among those interviewed. Measuring bias caused by nonresponse is the responsibility of the statistician. The staff of the Master Sample Survey in the Washington Heights Health District undertook to study nonresponse bias by investigating the characteristics of a "hard-to-obtain" subsample.

ONE of the problems of survey research is the bias that results when the attribute being tested influences the willingness to respond. When this bias is present, the respondent group is not represe...