Search

Participant selection bias is of concern to researchers conducting surveys of all types. For momentary data capture studies, such as Ecological Momentary Assessment, the level of burden associated with these techniques and the possibility of low uptake rates makes the concerns especially salient. This study invited 3,000 individuals to participate in a study of health and mood and recorded the uptake rates at various points in the process. Respondents expressing interest in participating in general were randomized into a one-time survey, a low-burden momentary study, or a high-burden momentary study. Overall, 85.9% of the sample did not respond to the study invitation (including confirming non-interest); 6.9% of the sample expressed interest in the study by completing a brief survey; 2.1% agreed to participate in the study when the protocol specifics were described (none of the study protocols were actually run). Whites were more likely to complete the survey. Of those completing the survey, individuals who reported higher income, a more “open” personality, better typing skills, better computer skills, who viewed the research topic as important, and who expressed interest in research on daily feelings more likely consented to being enrolled in the experiment. The number of prior surveys taken had an inverted-U shaped association with participation in this study. Finally, all individuals randomized to the one-time survey group agreed to participate compared to two-thirds of individuals in the momentary groups. These results suggest that participant selection bias may affect both one-time survey and momentary data capture studies, with the caveat that the degree of such bias will be related to a study’s hypotheses.

In push-to-web surveys that use postal mail to contact sampled cases, participation is contingent on the mail being opened and the survey invitations being delivered. The design of the mailings is crucial to the success of the survey. We address the question of how to design invitation mailings that can grab potential respondents’ attention and sway them to be interested in the survey in a short window of time. In the household screening stage of a national survey, the American Family Health Study, we experimentally tested three mailing design techniques for recruiting respondents: (1) a visible cash incentive in the initial mailing, (2) a second incentive for initial nonrespondents, and (3) use of Priority Mail in the nonresponse follow-up mailing. We evaluated the three techniques’ overall effects on response rates as well as how they differentially attracted respondents with different characteristics. We found that all three techniques were useful in increasing the screening response rates, but there was little evidence that they had differential effects on sample subgroups that could help to reduce nonresponse biases.

Purpose To measure the rate of recall of study participation and study attrition in survivors of acute respiratory distress syndrome(ARDS). Materials/Methods In this ancillary study of the Re-evaluation of Systemic Early neuromuscular blockade(ROSE) trial, we measured the rate of study participation recall 3 months following discharge and subsequent study attrition at 6 months. We compared patient and hospital characteristics, and long-term outcomes by recall. As surrogate decision-makers provided initial consent, we measured the rate of patient reconsent and its association with study recall. Results Of 487 patients evaluated, recall status was determined in 386(82.7%). Among these, 287(74.4%) patients recalled participation in the ROSE trial, while 99(25.6%) did not. There was no significant difference in 6-month attrition among patients who recalled study participation (9.1%) and those who did not (12.1%) (p = 0.38). Patient characteristics were similar between groups, except SOFA scores, ventilator-free days, and length of stay. 330(68%) were reconsented. Compared to those not reconsented, significantly more patients who were reconsented recalled study participation(78% vs. 66%;p = 0.01). Conclusions One in 4 ARDS survivors do not recall their participation in a clinical trial during hospitalization 3 months following hospital discharge, which did not influence 6-month attrition. However, more patients recall study participation if reconsent is obtained.

Objective The All of Us Research Program collects data from multiple information sources, including health surveys, to build a national longitudinal research repository that researchers can use to advance precision medicine. Missing survey responses pose challenges to study conclusions. We describe missingness in All of Us baseline surveys. Study design and setting We extracted survey responses between May 31, 2017, to September 30, 2020. Missing percentages for groups historically underrepresented in biomedical research were compared to represented groups. Associations of missing percentages with age, health literacy score, and survey completion date were evaluated. We used negative binomial regression to evaluate participant characteristics on the number of missed questions out of the total eligible questions for each participant. Results The dataset analyzed contained data for 334,183 participants who submitted at least one baseline survey. Almost all (97.0%) of the participants completed all baseline surveys, and only 541 (0.2%) participants skipped all questions in at least one of the baseline surveys. The median skip rate was 5.0% of the questions, with an interquartile range (IQR) of 2.5% to 7.9%. Historically underrepresented groups were associated with higher missingness (incidence rate ratio (IRR) [95% CI]: 1.26 [1.25, 1.27] for Black/African American compared to White). Missing percentages were similar by survey completion date, participant age, and health literacy score. Skipping specific questions were associated with higher missingness (IRRs [95% CI]: 1.39 [1.38, 1.40] for skipping income, 1.92 [1.89, 1.95] for skipping education, 2.19 [2.09–2.30] for skipping sexual and gender questions). Conclusion Surveys in the All of Us Research Program will form an essential component of the data researchers can use to perform their analyses. Missingness was low in All of Us baseline surveys, but group differences exist. Additional statistical methods and careful analysis of surveys could help mitigate challenges to the validity of conclusions.

Longitudinal surveys traditionally conducted by interviewers are facing increasing pressures to explore alternatives such as sequential mixed-mode designs, which start with a cheaper self-administered mode (online) then follow up using more expensive methods such as telephone or face-to-face interviewing. Using a designed experiment conducted as part of the 2018 wave of the Health and Retirement Study (HRS) in the US, we compare a sequential mixed-mode design (web then telephone) with the standard telephone-only protocol. Using an intent-to-treat analysis, we focus on response quality and response distributions for several domains key to HRS: physical and psychological health, financial status, expectations and family composition. Respondents assigned to the sequential mixed-mode (web) had slightly higher missing data rates and more focal responses than those assigned to telephone-only. However, we find no evidence of differential quality in verifying and updating roster information. We find slightly lower rates of asset ownership reported by those assigned to the web mode. Conditional on ownership, we find no detectable mode effects on the value of assets. We find more negative (pessimistic) expectations for those assigned to the web mode. We find little evidence of poorer health reported by those assigned to the web mode. We find that effects of mode assignment on measurement are present, but for most indicators the effects are small. Finding ways to remediate the differences in item-missing data and focal values should help reduce mode effects in mixed-mode surveys or those transitioning from interviewer- to self-administration.

Passive data from a tracking application (or “meter”) installed on participants’ devices to register the URLs visited have great potential for studying people’s online activities. However, given privacy concerns, obtaining cooperation installing a meter can be difficult and lead to selection bias. Therefore, in this article, we address three research questions: (1) To what extent are panelists willing to install a meter? (2) On which devices do they install the meter? (3) How do panelists who installed the meter differ from those who were invited but did not install it? Using data from online non-probability opt-in panels in nine countries, we found that the proportions of panelists installing the meter usually vary from 20% to 42%. Moreover, 20–25% of participants installed the meter on three or more devices. Finally, those who were invited but did not install the meter differ from those who did.

Internet-enabled smartphones and wireless communication technologies are opening new ways to conduct web-based self-administered data collection for academic or nonacademic research. Considering the relative advantages of self-administration such as the low cost, overall convenience, and collection of better data about sensitive topics, survey researchers are eager to explore conducting national web surveys of the general population via smartphones, especially if they can use probability-based random-digit-dialing (RDD) sampling methods. But questions about the feasibility of such surveys remain. We conducted an experiment using national samples drawn from an RDD wireless sampling frame to compare two administration methods: a smartphone web survey using SMS (text messages) invitations and a cell phone (smartphone or feature phone) survey through computer-assisted telephone interviewing (CATI). This study was conducted using the National Survey of Smoking and Health in South Korea, a country with a very high rate of smartphone ownership. The geographic and demographic representativeness of respondents in the smartphone web mode (self-administered mode) was similar to that of the cell phone CATI survey (interviewer-administered mode), although the completion rate in the former was nearly half that of the latter. A majority of demographic variables and measures of experiences, attitudes, and perceptions of cigarettes or smoking showed significant mode differences for both unweighted and weighted estimates. The total cost of the web survey was about one fifth that of the telephone survey. The results demonstrate the potential of a smartphone web survey as a stand-alone or primary mode of data collection, if carefully designed and implemented.

Background Patient satisfaction is a proxy for quality clinical care. Understanding the factors that drive patient satisfaction scores is important because they are publicly reported, may be used in determining hospital and physician compensation, and may allow patients to preselect physicians. Objective This single-center survey study of adult patients at the Michigan Medicine outpatient dermatology clinics aimed to investigate how patients respond differently to theoretical dermatologic scenarios with varying dermatologist gender. Methods Each questionnaire contained one of four clinical scenarios illustrating overall positive or negative encounters with a male or female dermatologist, followed by questions derived from the Press Ganey survey to assess patient satisfaction. Results A total of 452 completed questionnaires were collected. There were statistically significant differences in overall patient satisfaction scores between positive versus negative female and positive versus negative male dermatologists, but there were no differences in scores between positive female and positive male dermatologists or between negative female and negative male dermatologists. There were also no differences in overall scores after controlling for patient demographic characteristics or patient–dermatologist gender concordance. Conclusion Previous studies have suggested that male physicians receive better patient satisfaction scores compared to female physicians. However, our study found that, in response to hypothetical scenarios of positive and negative dermatology encounters, dermatologist gender did not affect any domain of patient satisfaction scores. Limitations include the use of hypothetical patient–dermatologist encounters and possible lack of generalizability because the study was conducted at one academic center in southeast Michigan with a predominantly Caucasian patient population.

Longitudinal survey and experimental research is essential for understanding psychological and economic processes, but attrition is a concern. We examine attrition in a 2016 longitudinal randomized controlled trial of more than 13,000 prospective U.S. homeowners, in which participants were invited to up to seven online surveys delivered every two weeks. Our research has two aims. First, we document overall attrition, finding that 60% of participants completed the study and attrition primarily occurred soon after enrollment. Second, we examine correlates of attrition, examining not only relationships with demographic and study characteristics (enrolling for text message reminders, survey delivery day) but also considering under-studied economic and psychological characteristics (time preferences, subjective numeracy, and financial literacy). Overall, we find that many variables, including some psychological characteristics, predict attrition. Our results suggest specific variables for researchers to measure in adjusting for attrition and identifying interventions to reduce attrition.

Offering participants in mobile app studies personalized feedback on the data they report seems an obvious thing to do: Participants might expect an app to provide feedback given their experiences with commercial apps, feedback might motivate more people to participate in the study, and participants might be more motivated to provide accurate data so that the feedback is more useful to them. However, personalized feedback might lead participants to change the behavior that is being measured with the app, is costly to implement, and also constrains other design decisions for the data collection. In this article, we report on an experimental study that tested the effects of providing personalized feedback in a 1-month mobile app–based spending study. Based on the app paradata and responses to a debrief survey, it seems that participants reacted positively to the feedback. The feedback did not have the potential negative effect of altering the spending participants reported in the app. However, the feedback also did not have the intended effect of increasing initial participation or ongoing adherence to the study protocol.

It is increasingly common for researchers to link survey data to administrative data. If several administrative data sources are of interest, respondents are required to give consent to each of them, meaning that multiple consent questions have to be included in one survey. Existing literature suggests that individual consent varies widely between data sources and over time, but little is known about how respondents process multiple consent requests in a single survey. Using an online access panel in Great Britain, we conducted a set of experiments in two surveys to explore multiple consent requests (covering five domains or data sources). In the first study, we experimentally varied the format of the request, testing three versions: (1) a sequence of pages (with one response per domain), (2) all five requests on the same page (with one response per domain), and (3) a single request (with one joint request covering all five domains). We also varied the order of the domains. We find that average consent rates do not differ by format, but asking a less-sensitive or easier-to-comply request first yields slightly higher average consent rates than asking a more sensitive request first. We repeated the order experiment in a second study, using an independent sample from the same panel, and adding two more order conditions. We find average consent rates are not affected much by order, but the consent to individual domains is affected by order. However, we fail to replicate the pattern of consents found in the first study. We conclude that the order in which multiple consent requests are asked does matter, but in complicated ways that depend on the particular outcomes in which one is interested. Objective knowledge and subjective comprehension of the consent process, and confidence in the decision are largely unaffected by format or order.

BACKGROUND: The All of Us Research Program is building a national longitudinal cohort and collecting data from multiple information sources (e.g., biospecimens, electronic health records (EHRs), and mobile/wearable technologies) to advance precision medicine. Participant-provided information, collected via surveys, will complement and augment these information sources. We report the process used to develop and refine the initial three surveys for this program. METHODS: The All of Us survey development process included: (1) prioritization of domains for scientific needs, (2) examination of existing validated instruments, (3) content creation, (4) evaluation and refinement via cognitive interviews and online testing, (5) content review by key stakeholders, and (6) launch in the All of Us electronic participant portal. All content was translated into Spanish. RESULTS: We conducted cognitive interviews in English and Spanish with 169 participants, and 573 individuals completed online testing. Feedback led to over 40 item content changes. Lessons learned included: (1) validated survey instruments performed well in diverse populations reflective of All of Us; (2) parallel evaluation of multiple languages can ensure optimal survey deployment; (3) recruitment challenges in diverse populations required multiple strategies; and (4) key stakeholders improved integration of surveys into larger Program context. CONCLUSIONS: This efficient, iterative process led to successful testing, refinement, and launch of three All of Us surveys. Reuse of All of Us surveys, available at http://researchallofus.org, may facilitate large consortia targeting diverse populations in English and Spanish to capture participant-provided information to supplement other data, such as genetic, physical measurements, or data from EHRs.

This study examines the two-year follow up (data collected in 2016 at modal age 21/22) of an original mixed-mode longitudinal survey experiment (data collected at modal age 19/20 in 2014). The study compares participant retention in the experimental conditions to retention in the standard Monitoring the Future (MTF) control condition (participants who completed an in-school baseline survey in 12(th) grade in 2012 or 2013 and were selected to participate in the first follow-up survey by mail in 2014, N=2,451). A supplementary sample who completed the 12(th) grade baseline survey in 2012 or 2013 but were not selected to participate in the main MTF follow-up (N=4,950) were recruited and randomly assigned to one of three experimental conditions: 1: Mail Push, 2: Web Push, 3: Web Push + Email in 2014 and again in 2016. Results from the first experiment indicated that Condition 3 (Web Push + Email) was promising based on similar response rates and lower costs (Patrick et al. 2018). The current study examines how experimental condition and type of 2014 response were associated with response in 2016, the extent to which response mode and device type changed from 2014 to 2016, and cumulative cost comparisons across conditions. Results indicated that responding via web in 2014 was associated with greater odds of participation again in 2016 regardless of condition; respondents tended to respond in the same mode although the “push” condition did move respondents toward web over paper; device type varied between waves; and the cumulative cost savings of Web Push + Email grew larger compared to the MTF Control. The web push strategy is therefore promising for maintaining respondent engagement while reducing cost.

This research note compares sequential and concurrent web-mail mixed-mode approaches and incentives for a survey on US consumers’ use of financial products, especially when their finances are tight. The sample (n = 2,000) was drawn from credit bureau data. We examine the effects on response rates, survey costs, and possible nonresponse bias in an experiment varying two factors in a 2x2 design: (1) using concurrent or sequential web and then mail survey modes, and (2) different incentive amounts given to initial survey nonrespondents ($5 versus $10). The sequential (web-first) design had a significantly lower response rate (3.9 percent) at week five—before the paper questionnaire was mailed—than the concurrent group (11.0 percent). This difference was nearly fully eliminated by the end of the field period. The higher incentive brought in slightly more respondents in the concurrent arm and slightly fewer respondents in the sequential arm, but neither difference is statistically significant. Compared with the sample frame, respondents in both groups were generally older and had many of the characteristics that come from being older: higher credit scores, more open credit cards, lower credit card utilization, greater likelihood of having a mortgage, and lower likelihood of being delinquent on credit card payments. Given the lower initial response rate and the need for more follow-up mailing, the sequential mixed-mode approach resulted in a higher cost per complete survey.

The grid question refers to a table layout for a series of survey question items (i.e., sub-questions) with the same introduction and identical response categories. Because of their complexity, concerns have already been raised about grids in web surveys on PCs, and these concerns have heightened regarding mobile devices. Some studies suggest decomposing grids into item-by-item layouts, while others argue that this is unnecessary. To address this challenge, this paper provides a comprehensive evaluation of the grid layout and four item-by-item alternatives, using 10 response quality indicators and 20 survey estimates. Results from the experimental web survey ( n = 4644) suggest that item-by-item layouts (unfolding or scrolling) should be used instead of grids, not only on mobile devices but also on PCs. While the former justifies the already increasing use of item-by-item layouts on mobile devices in survey practice, the latter implies that the prevailing routine of using grids on PCs should be reconsidered.

Given the promise of the web push plus e-mail survey design for providing cost-effective and high-quality data (Patrick et al. 2018, 2019) as an alternative to a paper-and-pencil mailed survey design for the longitudinal Monitoring the Future (MTF) study, the current study sought to further enhance the web push condition. The MTF sample is based on US nationally representative samples of 12th grade students surveyed annually. The MTF control group for the current study included participants who completed the in-school baseline survey in the 12th grade and were selected to participate in their first follow-up survey in 2017 via mailed surveys (N = 1,222). A supplementary sample (N = ∼2,450) was assigned to one of the two sequential mixed-mode conditions. Those in condition 1 (N = 1,198), or mail push, were invited to complete mailed surveys and later given a web survey option. Those in condition 2 (N = 1,173), or enhanced web push, were invited to complete a web survey (the same as in the 2014 study, but with the addition of text messages and quick response (QR) codes and the web survey was optimized for mobile devices) and then later given a mailed survey option. Research aims were to examine response rates across conditions, as well as how responses were distributed across mode (paper, web), devices (computer, smartphone, table), and method of accessing the web survey (hand-entered URL, QR code, e-mail link, SMS link). Response rates differed significantly: the MTF control group was 34.2 percent, mail push was 35.4 percent, and enhanced web push was 42.05 percent. The higher response rate in the enhanced web push condition suggests that the additional strategies were effective at bringing in more respondents. Key estimates produced by the enhanced web push condition did not differ from those of the MTF control group.

Background and aims Increasing numbers of school-based drug surveys are transitioning data collection to electronic tablets from paper-and-pencil, which may produce a survey mode effect and consequent discontinuity in time trends for population estimates of drug prevalence. This study tested whether (a) overall, self-reported drug use prevalence is higher on electronic tablets versus paper-and-pencil surveys, (b) socio-demographics moderate survey mode effects and (c) levels of missing data are lower for electronic tablet versus paper-and-pencil modes. Design A randomized controlled experiment. Setting Results are nationally representative of students in the contiguous United States. Participants A total of 41 866 8th, 10th and 12th grade students who participated in the 2019 Monitoring the Future school-based survey administration. Intervention and comparator Surveys were administered to students in a randomly selected half of schools with electronic tablets (intervention) and with paper-and-pencil format (comparator) for the other half. Measurements Primary outcome was the total number of positive drug use responses. Secondary outcomes were the percentage of respondents completing all drug questions, percentage of drug questions unanswered and mean number of missing drug items. Findings The relative risk (RR) for total number of positive drug use responses for electronic tablets versus paper-and-pencil surveys were small and their 95% confidence intervals (CI) included the value of one for reporting intervals of life-time (RR = 1.03; 95% CI, 0.93-1.14), past 12 months (RR = 1.01; 95% CI, 0.91-1.11), past 30 days (RR = 1.05; 95% CI, 0.93-1.20) and for heavy use (RR = 1.10; 95% CI, 0.93-1.29). Multiplicative interaction tests indicated no moderation of these relative risks by race (white versus non-white), population density, census region, public/private school, year of school participation, survey version or non-complete drug responses. Levels of missing data were significantly lower for electronic tablets versus paper-and-pencil surveys. Conclusions Adolescent drug prevalence estimates in the United States differed little across electronic tablet versus paper-and-pencil survey modes, and showed little to no effect modification by socio-demographics. Levels of missing data were lower for electronic tablets.

We implemented an experiment within a smartphone web survey to explore the feasibility of using voice input (VI) options. Based on device used, participants were randomly assigned to a treatment or control group. Respondents in the iPhone operating system (iOS) treatment group were asked to use the dictation button, in which the voice was translated automatically into text by the device. Respondents with Android devices were asked to use a VI button which recorded the voice and transmitted the audio file. Both control groups were asked to answer open-ended questions using standard text entry. We found that the use of VI still presents a number of challenges for respondents. Voice recording (Android) led to substantially higher nonresponse, whereas dictation (iOS) led to slightly higher nonresponse, relative to text input. However, completion time was significantly reduced using VI. Among those who provided an answer, when dictation was used, we found fewer valid answers and less information provided, whereas for voice recording, longer and more elaborated answers were obtained. Voice recording (Android) led to significantly lower survey evaluations, but not dictation (iOS).

We studied the impact of different layouts for rank order questions on respondent effort, data quality, and substantive results among PC and smartphone respondents, in an experiment in an opt-in on...