Search

In ecological momentary assessment (EMA), respondents answer brief questionnaires about their current behaviours or experiences several times per day across multiple days. The frequent measurement enables a thorough grasp of the dynamics inherent in psychological constructs, but it also increases respondent burden. To lower this burden, respondents may engage in careless and insufficient effort responding (C/IER), leaving data contaminated with responses that do not reflect what researchers want to measure. We introduce a novel approach to investigating C/IER in EMA data. Our approach combines a confirmatory mixture item response theory model separating C/IER from attentive behaviour with latent Markov factor analysis. This enables gauging the occurrence of C/IER and studying transitions among states of different response behaviours including their contextual correlates. The approach can be implemented using R packages. An empirical application showcases the approach's efficacy in pinpointing C/IER instances and gaining insights into their underlying causes. We showcase that the approach identifies various C/IER response patterns but requires heterogeneous and negatively worded items to detect straightlining. In a simulation investigating robustness against unaccounted for changes in measurement models underlying attentive responses, the approach proved robust against heterogeneity in loading patterns but not against heterogeneity in factor structures. Extensions to accommodate the latter are discussed. [ABSTRACT FROM AUTHOR]

To screen for careless responding, researchers have a choice between several direct measures (i.e., bogus items, requiring the respondent to choose a specific answer) and indirect measures (i.e., unobtrusive post hoc indices). Given the dearth of research in the area, we examined how well direct and indirect indices perform relative to each other. In five experimental studies, we investigated whether the detection rates of the measures are affected by contextual factors: severity of the careless response pattern, type of item keying, and type of item presentation. We fully controlled the information environment by experimentally inducing careless response sets under a variety of contextual conditions. In Studies 1 and 2, participants rated the personality of an actor that presented himself in a 5-min-long videotaped speech. In Studies 3, 4, and 5, participants had to rate their own personality across two measurements. With the exception of maximum longstring, intra-individual response variability, and individual contribution to model misfit, all examined indirect indices performed better than chance in most of the examined conditions. Moreover, indirect indices had detection rates as good as and, in many cases, better than the detection rates of direct measures. We therefore encourage researchers to use indirect indices, especially within-person consistency indices, instead of direct measures. [ABSTRACT FROM AUTHOR]

Careless responding measures are important for several purposes, whether it's screening for careless responding or for research centered on careless responding as a substantive variable. One such approach for assessing carelessness in surveys is the use of an instructional manipulation check. Despite its apparent popularity, little is known about the construct validity of instructional manipulation checks as measures of careless responding. Initial results are inconclusive, and no study has thoroughly evaluated the validity of the instructional manipulation check as a measure of careless responding. Across 2 samples (N = 762), we evaluated the construct validity of the instructional manipulation check under a nomological network. We found that the instructional manipulation check converged poorly with other measures of careless responding, weakly predicted participant inability to recognize study content, and did not display incremental validity over existing measures of careless responding. Additional analyses revealed that instructional manipulation checks performed poorly compared to single scores of other alternative careless responding measures and that screening data with alternative measures of careless responding produced greater or similar gains in data quality to instructional manipulation checks. Based on the results of our studies, we do not recommend using instructional manipulation checks to assess or screen for careless responding to surveys. [ABSTRACT FROM AUTHOR]

The multidimensional forced-choice (MFC) format is an alternative to rating scales in which participants rank items according to how well the items describe them. Currently, little is known about how to detect careless responding in MFC data. The aim of this study was to adapt a number of indices used for rating scales to the MFC format and additionally develop several new indices that are unique to the MFC format. We applied these indices to a data set from an online survey (N = 1,169) that included a series of personality questionnaires in the MFC format. The correlations among the careless responding indices were somewhat lower than those published for rating scales. Results from a latent profile analysis suggested that the majority of the sample (about 76–84%) did not respond carelessly, although the ones who did were characterized by different levels of careless responding. In a simulation study, we simulated different careless responding patterns and varied the overall proportion of carelessness in the samples. With one exception, the indices worked as intended conceptually. Taken together, the results suggest that careless responding also plays an important role in the MFC format. Recommendations on how it can be addressed are discussed. [ABSTRACT FROM AUTHOR]

Insufficient effort responding (IER) to surveys, which occurs when participants provide responses in a haphazard, careless, or random fashion, has been identified as a threat to data quality in survey research because it can inflate observed relationships between self-reported measures. Building on this discovery, we propose two mechanisms that lead to IER exerting an unexpected confounding effect between self-reported and informant-rated measures. First, IER can contaminate self-report measures when the means of attentive and inattentive responses differ. Second, IER may share variance with some informant-rated measures, particularly supervisor ratings of participants' job performance. These two mechanisms operating in tandem would suggest that IER can act as a "third variable" that inflates observed relationships between self-reported predictor scores and informant-rated criteria. We tested this possibility using a multisource dataset (N = 398) that included incumbent self-reports of five-factor model personality traits and supervisor-ratings of three job performance dimensions—task performance, organizational citizenship behavior (OCB), and counterproductive work behavior (CWB). We observed that the strength of the relationships between self-reported personality traits and supervisor-rated performance significantly decreased after IER was controlled: Across the five personality traits, the average reduction of magnitude from the zero-order to partial correlations was |.08| for task performance, |.07| for OCB, and |.14| for CWB. Because organizational practices are often driven by research linking incumbent-reported predictors to supervisor-rated criteria (e.g., validation of predictors used in various organizational contexts), our findings have important implications for research and practice. [ABSTRACT FROM AUTHOR]

Abstract Common indicator-based approaches to identifying careless and insufficient effort responding (C/IER) in survey data scan response vectors or timing data for aberrances, such as patterns signaling straight lining, multivariate outliers, or signals that respondents rushed through the administered items. Each of these approaches is susceptible to unique types of misidentifications. We developed a C/IER indicator that requires agreement on C/IER identification from multiple behavioral sources, thereby alleviating the effect of each source’s standalone C/IER misidentifications and increasing the robustness of C/IER identification. To this end, we combined a response-pattern-based multiple-hurdle approach with a recently developed screen-time-based mixture decomposition approach. In an application of the proposed multiple-source indicator to PISA 2022 field trial data we (a) showcase how the indicator hedges against (presumed) C/IER overidentification of its constituting components, (b) replicate associations with commonly reported external correlates of C/IER, namely agreement with self-reported effort and C/IER position effects, and (c) employ the indicator to study the effects of changes of scale characteristics on C/IER occurrence. To this end, we leverage a large-scale survey experiment implemented in the PISA 2022 field trial and investigate the effects of using frequency instead of agreement scales as well as approximate instead of abstract frequency scale labels. We conclude that neither scale format manipulation has the potential to curb C/IER occurrence.

This article explores the issue of careless responding in self-report measures and its impact on data quality. The study compares different methods for detecting careless responders in a cross-cultural context and finds that the rate of careless responding was higher in Singapore compared to Turkey. Removing careless responders from the dataset resulted in increased scores related to enjoyment of reading, perception of competence, and perception of difficulty. The study highlights the importance of screening data for careless responding in future research. The document provides an overview of careless responding in surveys and various methods to detect it. It discusses a study that used response time, long-string analysis, and outlier analysis to identify careless responses among Turkish and Singaporean students. The findings suggest that response time and long-string analysis are effective in identifying careless responders and improving model fit. The document also explores the differences in descriptive statistics and reliability coefficients after excluding flagged careless responders. The study recommends that researchers screen for careless responding in future studies and offers suggestions for addressing this issue. However, the results are limited to the specific sample and measures used in the study, and further research is needed to explore different identification methods and personal characteristics that may affect careless responding. [Extracted from the article]

Introduction: This study explores the stability of scores on psychometrically validated trait questionnaires over time. We illustrate potential pitfalls through a larger study that used the Ruminative Response Scale (RRS) to categorize participants prior to study inclusion into two groups based on their habitual tendency to ruminate. Surprisingly, when we re-administered the RRS at the start of an experimental session, significant score changes occurred, resulting in participants shifting between the two groups. Methods: To address this, we modified our recruitment process, aiming to reduce careless responses, including an online RRS assessment a week before the lab appointment. We analyzed the different samples prior to and after changing the recruitment procedure, as well as the total sample regarding the psychometric properties of the RRS. We also explored various indices to identify and predict score changes due to careless responding; however, only a subgroup of participants was successfully identified. Results: Our findings suggest that Mahalanobis distances are effective for identifying substantial score changes, with baseline state rumination emerging as a marginally significant predictor. Discussion: We discuss the importance of conducting manipulation checks and offer practical implications for research involving psychometrically validated trait questionnaires. [ABSTRACT FROM AUTHOR]

Common indicator-based approaches to identifying careless and insufficient effort responding (C/IER) in survey data scan response vectors or timing data for aberrances, such as patterns signaling straight lining, multivariate outliers, or signals that respondents rushed through the administered items. Each of these approaches is susceptible to unique types of misidentifications. We developed a C/IER indicator that requires agreement on C/IER identification from multiple behavioral sources, thereby alleviating the effect of each source's standalone C/IER misidentifications and increasing the robustness of C/IER identification. To this end, we combined a response-pattern-based multiple-hurdle approach with a recently developed screen-time-based mixture decomposition approach. In an application of the proposed multiple-source indicator to PISA 2022 field trial data we (a) showcase how the indicator hedges against (presumed) C/IER overidentification of its constituting components, (b) replicate associations with commonly reported external correlates of C/IER, namely agreement with self-reported effort and C/IER position effects, and (c) employ the indicator to study the effects of changes of scale characteristics on C/IER occurrence. To this end, we leverage a large-scale survey experiment implemented in the PISA 2022 field trial and investigate the effects of using frequency instead of agreement scales as well as approximate instead of abstract frequency scale labels. We conclude that neither scale format manipulation has the potential to curb C/IER occurrence. [ABSTRACT FROM AUTHOR]

It is common for some participants in self-report surveys to be careless, inattentive, or lacking in effort. Data quality can be severely compromised by responses that are not based on item content (non-content-based [nCB] responses), leading to strong biases in the results of data analysis and misinterpretation of individual scores. In this study, we propose a specification of factor mixture analysis (FMA) to detect nCB responses. We investigated the usefulness and effectiveness of the FMA model in detecting nCB responses using both simulated data (Study 1) and real data (Study 2). In the first study, FMA showed reasonably robust sensitivity (.60 to.86) and excellent specificity (.96 to.99) on mixed-worded scales, suggesting that FMA had superior properties as a screening tool under different sample conditions. However, FMA performance was poor on scales composed of only positive items because of the difficulty in distinguishing acquiescent patterns from valid responses representing high levels of the trait. In Study 2 (real data), FMA detected a minority of cases (6.5%) with highly anomalous response patterns. Removing these cases resulted in a large increase in the fit of the unidimensional model and a substantial reduction in spurious multidimensionality. [ABSTRACT FROM AUTHOR]

Questionnaires are by far the most common tool for measuring noncognitive constructs in psychology and educational sciences. Response bias may pose an additional source of variation between respondents that threatens validity of conclusions drawn from questionnaire data. We present a mixture modeling approach that leverages response time data from computer-administered questionnaires for the joint identification and modeling of two commonly encountered response bias that, so far, have only been modeled separately—careless and insufficient effort responding and response styles (RS) in attentive answering. Using empirical data from the Programme for International Student Assessment 2015 background questionnaire and the case of extreme RS as an example, we illustrate how the proposed approach supports gaining a more nuanced understanding of response behavior as well as how neglecting either type of response bias may impact conclusions on respondents' content trait levels as well as on their displayed response behavior. We further contrast the proposed approach against a more heuristic two-step procedure that first eliminates presumed careless respondents from the data and subsequently applies model-based approaches accommodating RS. To investigate the trustworthiness of results obtained in the empirical application, we conduct a parameter recovery study. [ABSTRACT FROM AUTHOR]

IntroductionThis study explores the stability of scores on psychometrically validated trait questionnaires over time. We illustrate potential pitfalls through a larger study that used the Ruminative Response Scale (RRS) to categorize participants prior to study inclusion into two groups based on their habitual tendency to ruminate. Surprisingly, when we re-administered the RRS at the start of an experimental session, significant score changes occurred, resulting in participants shifting between the two groups.MethodsTo address this, we modified our recruitment process, aiming to reduce careless responses, including an online RRS assessment a week before the lab appointment. We analyzed the different samples prior to and after changing the recruitment procedure, as well as the total sample regarding the psychometric properties of the RRS. We also explored various indices to identify and predict score changes due to careless responding; however, only a subgroup of participants was successfully identified.ResultsOur findings suggest that Mahalanobis distances are effective for identifying substantial score changes, with baseline state rumination emerging as a marginally significant predictor.DiscussionWe discuss the importance of conducting manipulation checks and offer practical implications for research involving psychometrically validated trait questionnaires.

Powerful methods for identifying careless respondents in survey data are not just important to ensure the validity of subsequent data analyses, they are also instrumental for studying the psychological processes that drive humans to respond carelessly. Conversely, a deeper understanding of the phenomenon of careless responding enables the development of improved methods for the identification of careless respondents. While machine learning has gained substantial attention and popularity in many scientific fields, it is largely unexplored for the detection of careless responding. On the one hand, machine learning algorithms can be highly powerful tools due to their flexibility. On the other hand, science based on machine learning has been criticized in the literature for a lack of reproducibility. We assess the potential and the pitfalls of machine learning approaches for identifying careless respondents from an open science perspective. In particular, we discuss possible sources of reproducibility issues when applying machine learning in the context of careless responding, and we give practical guidelines on how to avoid them. Furthermore, we illustrate the high potential of an unsupervised machine learning method for the identification of careless respondents in a proof‐of‐concept simulation experiment. Finally, we stress the necessity of building an open data repository with labeled benchmark data sets, which would enable the evaluation of methods in a more realistic setting and make it possible to train supervised learning methods. Without such a data repository, the true potential of machine learning for the identification of careless responding may fail to be unlocked. [ABSTRACT FROM AUTHOR]

It is common to model responses to surveys within latent variable frameworks (e.g., item response theory [IRT], confirmatory factor analysis [CFA]) and use model fit indices to evaluate model–data congruence. Unfortunately, research shows that people occasionally engage in careless responding (CR) when completing online surveys. While CR has the potential to negatively impact model fit, this issue has not been systematically explored. To better understand the CR–fit linkage, two studies were conducted. In study 1, participants' response behaviors were experimentally shaped and used to embed aspects of a comprehensive simulation (study 2) with empirically informed data. For this simulation, 144 unique conditions (which varied the sample size, number of items, CR prevalence, CR severity, and CR type), two latent variable models (IRT, CFA), and six model fit indices (χ2, RMSEA, SRMSR [CFA] and M2, RMSEA, SRMSR [IRT]), were examined. The results indicated that CR deteriorates model fit under most circumstances, though these effects are nuanced, variable, and contingent on many factors. These findings can be leveraged by researchers and practitioners to improve survey methods, obtain more accurate survey results, develop more precise theories, and enable more justifiable data-driven decisions. [ABSTRACT FROM AUTHOR]

Objective: The prevalence of research conducted online in the addiction field has increased rapidly over the past decade. However, little focus has been given to careless responding in these online studies, despite the issues it may cause for statistical inference and generalizability. Our aim was to examine whether alcohol use is associated with careless responses. Method: Raw data were requested from online studies examining alcohol use and related problems which also addressed careless responding. We obtained 13 data sets of 12,237 participants (Mage = 42.16, SD = 15.65, 50.5% female). The sample had an average Alcohol Use Disorders Identification Test (AUDIT) score of 10.88 (SD = 7.77). Predictors included demographic information (age, gender) and AUDIT total scores. The primary outcome was whether an individual was classed as a careless responder, for example, by failing an explicit attention check question. Results: AUDIT total scores were associated with careless responding (OR = 1.07, 95% CI [1.06, 1.08], p < .001). Hazardous drinking or worse was associated with 2.21 greater odds (OR = 2.21, 95% CI [1.81, 2.71] of careless responding, whereas harmful drinking or worse was associated with 3.43 greater odds (OR = 3.43, 95% CI [2.83, 4.17]) and probable dependence was associated with 3.63 greater odds (OR = 3.63, 95% CI [2.95, 4.48]). Conclusions: Alcohol use and related problems are positively associated with careless responding in online research. Removal of individuals identified as careless responders may lead to issues of generalizability, and more care should be taken to identify and handle careless responder data. Public Health Significance Statement This mega-analysis demonstrates a robust relationship between alcohol use (defined by the Alcohol Use Disorders Identification Test) and careless responding in online studies. The findings suggest that the typical removal of careless responders from analyses in online alcohol studies is insufficient at best and at worst leads to issues with statistical inference and generalizability. [ABSTRACT FROM AUTHOR]

This article explores the analysis and interpretation of wording effects associated with using direct and reverse items in psychological assessment. Previous research using bifactor models has suggested a substantive nature of this effect. The present study uses mixture modeling to systematically examine an alternative hypothesis and surpass recognized limitations in the bifactor modeling approach. In preliminary supplemental Studies S1 and S2 , we examined the presence of participants who exhibited wording effects and evaluated their impact on the dimensionality of Rosenberg's Self-Esteem and the Revised Life Orientation Test, confirming the ubiquity of wording effects in scales containing direct and reverse items. Then, after analyzing the data for both scales (n = 5,953), we found that, despite a significant association between wording factors (Study 1), a low proportion of participants simultaneously exhibited asymmetric responses in both scales (Study 2). Similarly, despite finding both longitudinal invariance and temporal stability of this effect in three waves (n = 3,712, Study 3), a small proportion of participants was identified with asymmetric responses over time (Study 4), reflected in lower transition parameters compared to the other patterns of profiles examined. In both cases, we illustrate how bifactor models capitalize on the responses of individuals who do not even exhibit wording effects, yielding spurious correlations suggesting a substantive nature of the wording effect. These findings support the notion of an ephemeral nature underlying wording effects. The discussion focuses on alternative hypotheses to understand these findings and emphasizes the utility of including reverse items in psychological assessment. [ABSTRACT FROM AUTHOR]

Inconsistent or careless responding is a significant threat to the validity of self-reported personality data. Using archival samples of undergraduate and community participants, we developed an inconsistent responding scale using items that appear on both the 60- and 100-item versions of the HEXACO Personality Inventory-Revised—two widely used measures of the HEXACO model of personality trait structure. We identified pairs of correlated HEXACO items in Sample 1 and created a total inconsistent responding score by summing absolute differences between each item pair. The Brief Response Inconsistency Evaluation (BRIE) for the HEXACO effectively differentiated between genuine and randomly generated responses across samples. The BRIE also correlated as expected with other measures of careless responding and relevant personality traits (e.g., conscientiousness). Tentative cut scores for the BRIE that appear to provide a reasonable balance between sensitivity and specificity in Sample 1 were investigated. Future research should examine the BRIE with different populations and translations of the HEXACO inventories and further investigate the effectiveness of the recommended cut scores. [ABSTRACT FROM AUTHOR]

Orientation: Presently, carelessly responding (CR) individuals are omitted in terms of several individual indices, including consistency-type indices (that compare performance on only a limited number of matched item pairs), and subsequently, the effectiveness of such screening is evaluated in terms of, among others, the group mean item interrelatedness (IIR) (based on all J[J – 1] item pairs). Research purpose: This research aims to develop individualised versions of the group IIR measures to render them applicable during the screening phase as substitutes for the presently used consistency-type indices. Motivation for the study: Such individual consistency indices may be used together with other CR indices to jointly determine the eventual evaluation results. Research approach/design and method: To develop the intended CR indices mathematical-statistical principles were applied on the product moment correlation and coefficient alpha formulae. Main findings: Three individual IIR indices have been developed which show individual respondents’ respective contributions to the mean item inter-correlation and to coefficient alpha, as measures of group mean IIR. Practical/managerial implications: These indices may be used during screening in lieu of the existing restrictive consistency indices. Contribution/value-add: Carelessly responding respondents who previously may have survived screening because of less-inclusive consistency-type IIR indices and consequently may have negatively affected the eventual evaluation results, are now screened out.

Methods to identify carelessness in survey research can be valuable tools in reducing bias during survey development, validation, and use. Because carelessness may take multiple forms, researchers typically use multiple indices when identifying carelessness. In the current study, we extend the literature on careless response identification by examining the usefulness of three item-response theory-based person-fit indices for both random and overconsistent careless response identification: infit MSE outfit MSE, and the polytomous l z statistic. We compared these statistics with traditional careless response indices using both empirical data and simulated data. The empirical data included 2,049 high school student surveys of teaching effectiveness from the Network for Educator Effectiveness. In the simulated data, we manipulated type of carelessness (random response or overconsistency) and percent of carelessness present (0%, 5%, 10%, 20%). Results suggest that infit and outfit MSE and the l z statistic may provide complementary information to traditional indices such as LongString, Mahalanobis Distance, Validity Items, and Completion Time. Receiver operating characteristic curves suggested that the person-fit indices showed good sensitivity and specificity for classifying both over-consistent and under-consistent careless patterns, thus functioning in a bidirectional manner. Carelessness classifications based on low fit values correlated with carelessness classifications from LongString and completion time, and classifications based on high fit values correlated with classifications from Mahalanobis Distance. We consider implications for research and practice. [ABSTRACT FROM AUTHOR]

IntroductionSelf-report questionnaires are widely used in high schools and universities to gain insights into students’ learning strategies and enhance the quality of their education. However, it is important to acknowledge the possibility of respondents being inattentive when completing these questionnaires. While reliability analyses are typically performed at the group level, when providing individual feedback, it is crucial that each respondent’s results are reliable. This study aimed to evaluate the prevalence of careless response behaviour in a questionnaire concerning student learning.MethodsData analysis encompassed a substantial sample of 12,000+ students in their final two years of secondary education, averaging around 17 years of age. Employing five complementary detection techniques, the study identified instances of careless responding present in the questionnaire data.ResultsOur results underscore a notable prevalence of careless response behaviour among the surveyed students. Application of the five detection techniques revealed a substantial number of instances indicating inattentive responding. Furthermore, the questionnaire’s measurement scales were evaluated for reliability. The study noted the presence of carelessness but observed minimal impact on group-level results.DiscussionThe outcomes of this study hold important implications for using self-report questionnaires in education. The prevalence of careless responding emphasizes the need for scrutinizing individual responses. Despite careless responses, their influence on overall group-level data integrity seems restricted. Nonetheless, the study underscores the importance of cautiously interpreting individual-level outcomes, particularly when using these results for individual feedback.

Several recent studies have examined the prevention, causes, and consequences of insufficient effort responding (IER) to surveys. Scientific progress in this area, however, rests on the availability of construct-valid IER measures. In the current paper we describe the potential merits of the page time index, which is computed by counting the number of questionnaire pages to which a participant has responded more quickly than two seconds per item (see Huang et al., 2012). We conducted three studies (total N = 1,056) to examine the page time index's construct validity. Across these studies, we found that page time converged highly with other IER indices, that it was sensitive to an experimental manipulation warning participants to respond carefully, and that it predicted the extent to which participants were unable to recognize item content. We also found that page time's validity was superior to that of total completion time and that the two-seconds-per-item rule yielded a construct-valid page time score for items of various word lengths. Given its apparent validity, we provide practical recommendations for the use of the page time index. [ABSTRACT FROM AUTHOR]

Surveys administered online have several benefits, but they are particularly prone to careless responding, which occurs when respondents fail to read item content or give sufficient attention, resulting in raw data that may not accurately reflect respondents' true levels of the constructs being measured. Careless responding can lead to various psychometric issues, potentially impacting any area of psychology that uses self-reported surveys and assessments. This review synthesizes the careless responding literature to provide a comprehensive understanding of careless responding and ways to prevent, identify, report, and clean careless responding from data sets. Further, we include recommendations for different levels of screening for careless responses. Finally, we highlight some of the most promising areas for future work on careless responding. [ABSTRACT FROM AUTHOR]

A crucial step in analysing text data is the detection and removal of invalid texts (e.g., texts with meaningless or irrelevant content). To date, research topics that rely heavily on analysis of text data, such as autobiographical memory, have lacked methods of detecting invalid texts that are both effective and practical. Although researchers have suggested many data quality indicators that might identify invalid responses (e.g., response time, character/word count), few of these methods have been empirically validated with text responses. In the current study, we propose and implement a supervised machine learning approach that can mimic the accuracy of human coding, but without the need to hand-code entire text datasets. Our approach (a) trains, validates, and tests on a subset of texts manually labelled as valid or invalid, (b) calculates performance metrics to help select the best model, and (c) predicts whether unlabelled texts are valid or invalid based on the text alone. Model validation and evaluation using autobiographical memory texts indicated that machine learning accurately detected invalid texts with performance near human coding, significantly outperforming existing data quality indicators. Our openly available code and instructions enable new methods of improving data quality for researchers using text as data. [ABSTRACT FROM AUTHOR]

The effects of different response option orders on survey responses have been studied extensively. The typical research design involves examining the differences in response characteristics between conditions with the same item stems and response option orders that differ in valence—either incrementally arranged (e.g., strongly disagree to strongly agree) or decrementally arranged (e.g., strongly agree to strongly disagree). The present study added two additional experimental conditions—randomly incremental or decremental and completely randomized. All items were presented in an item-by-item format. We also extended previous studies by including an examination of response option order effects on: careless responding, correlations between focal predictors and criteria, and participant reactions, all the while controlling for false discovery rate and focusing on the size of effects. In a sample of 1,198 university students, we found little to no response option order effects on a recognized personality assessment vis-à-vis measurement equivalence, scale mean differences, item-level distributions, or participant reactions. However, the completely randomized response option order condition differed on several careless responding indices suggesting avenues for future research. [ABSTRACT FROM AUTHOR]

Research questionnaires frequently include dozens—if not hundreds—of self-report items. Lengthy questionnaires, however, are often a necessity. In some cases, they are needed to assess the many variables found in a complex model; in other cases, they are the result of the inclusion of a single lengthy measure. This raises an important question: Do participants provide accurate responses to measures positioned at the end of a lengthy questionnaire? One possibility is that participants experience fatigue during questionnaire completion, leading them to engage in careless responding, and thus compromising the accuracy of their responses. Another possibility is that even the longest research questionnaires are generally too short to evoke participant fatigue. This latter possibility suggests that participants are largely able to maintain their attention while completing most questionnaires. Given the lack of clarity on this issue, we conducted two experiments (Study 1 N = 244; Study 2 N = 461) in which we randomly assigned each participant to complete a block of target scales at either the beginning or the end of a lengthy (> 300-item) questionnaire. Each participant also recruited an informant who provided reports of the participant's personality, attitudes, and behaviors. These informant data allowed us to examine the effects of the experimental manipulation on the target scales' convergent and criterion-related validity. The findings of both studies indicated that the target scales performed similarly across the two conditions. Given the ubiquity of lengthy questionnaires, these findings have far-reaching practical implications. [ABSTRACT FROM AUTHOR]

There is extensive literature on identifying careless responses in survey data and acknowledging their negative impact on accurate data analysis (Goodman, Cryder, & Cheema, 2013; Huang et al., 2015a, b; Meade and Craig, 2012). However, there are very limited findings on how researchers can help prevent participants from responding carelessly in the first place. The current study manipulated the placement of a quality check item (i.e., early placement versus late placement) and showed that participants were less likely to carelessly respond when the quality check items were placed later in the survey. Quality check items were also able to identify careless responses more than other approaches (i.e., LongString index, completion time index, and self-reported indicators). When using quality check items, it appears to be best when used toward the end of the survey. There are plenty of opportunities for future studies to expand on this research to uncover ways that can help deter careless responding. [ABSTRACT FROM AUTHOR]

Careless and inconsistent responding is a common issue with self-reported instruments, observed in approximately 10% of respondents. The purpose of this study was to develop an Inconsistency Responding Scale (IRS) for the Durand Adaptive Psychopathic Traits Questionnaire (DAPTQ). First, a derivation sample (N = 525) was used to develop the IRS-DAPTQ, a scale generating a total score from the absolute difference of 12 pairs of items with high intra-correlation. Second, the validity of the IRS-DAPTQ was assessed using human (N = 943) and randomly generated (N = 1,000) data. The IRS-DAPTQ successfully differentiated human data from randomly generated data. The IRS-DAPTQ scores showed a negative association with conscientiousness and had higher specificity for a sample of the general population recruited online than for a sample of undergraduate students. These findings support the usefulness of the IRS-DAPTQ for identifying individuals displaying an inconsistent responding style.

Experience sampling methodology (ESM) requires participants to repeatedly rate various psychological states in everyday life. This can be burdensome, leading to frequently skipped or missed assessments that undermine the ecological validity of collected data. Financially compensating participants may counter suboptimal response rates, but monetary rewards are not always feasible or ethical, and are unfitting to counter non-response in the moment. Here, we investigated the potential of a gamified momentary reward strategy to encourage survey completion. In a between-person experiment, we randomly assigned 193 participants to take part in a standard (i.e., no momentary incentives) or gamified ESM protocol. Upon completing a survey in the gamified condition, participants immediately received some virtual coins to purchase various in-app rewards (fun facts or personalized graphs). Gamifying momentary survey completion resulted in higher response rates and shorter latencies, but only for participants who frequently bought rewards in the coin store. Paradoxically, momentary gamification also caused participants' responses to be slightly more unreliable. Completion times did not differ. The reward schemes of future ESM studies should focus on both response quantity and quality. To further explore the potential of momentary incentive strategies, we provide open access to customizable gamification building blocks within our ESM platform, m-Path. • Real-time game-based rewards for survey completion may lead to higher response rates in experience sampling (ESM) studies. • However, gamification also paradoxically undermines the reliability of participants' momentary self-reported responses. • We should include both measures of response quantity and quality in the reward schemes of future ESM studies. [ABSTRACT FROM AUTHOR]

Background and Objectives It is widely recognized that survey satisficing, inattentive, or careless responding in questionnaires reduce the quality of self-report data. In this study, we propose that such low-quality responding (LQR) can carry substantive meaning at older ages. Completing questionnaires is a cognitively demanding task and LQR among older adults may reflect early signals of cognitive deficits and pathological aging. We hypothesized that older people displaying greater LQR would show faster cognitive decline and greater mortality risk. Research Design and Methods We analyzed data from 9, 288 adults 65 years or older in the Health and Retirement Study. Indicators of LQR were derived from participants' response patterns in 102 psychosocial questionnaire items administered in 2006–2008. Latent growth models examined whether LQR predicted initial status and change in cognitive functioning, assessed with the modified Telephone Interview for Cognitive Status, over the subsequent 10 years. Discrete-time survival models examined whether LQR was associated with mortality risk over the 10 years. We also examined evidence for indirect (mediated) effects in which LQR predicts mortality via cognitive trajectories. Results After adjusting for age, gender, race, marital status, education, health conditions, smoking status, physical activity, and depressive symptoms, greater LQR was cross-sectionally associated with poorer cognitive functioning, and prospectively associated with faster cognitive decline over the follow-up period. Furthermore, greater LQR was associated with increased mortality risk during follow-up, and this effect was partially accounted for by the associations between LQR and cognitive functioning. Discussion and Implications Self-report questionnaires are not formally designed as cognitive tasks, but this study shows that LQR indicators derived from self-report measures provide objective, performance-based information about individuals' cognitive functioning and survival. Self-report surveys are ubiquitous in social science, and indicators of LQR may be of broad relevance as predictors of cognitive and health trajectories in older people. [ABSTRACT FROM AUTHOR]

It is often assumed that survey results reflect only the quality of the sample and the underlying measuring instruments used in the survey. However, various phenomena can affect the results, but these influences are often neglected when conducting surveys. This study aimed to test the influences of various method effects on survey results. We tested the influences of the following method effects: item wording, confirmatory bias, careless responding, and acquiescence bias. Using a split-ballot survey design with online questionnaires, we collected data from 791 participants. We tested if these method effects had an influence on mean values, item correlations, construct correlations, model fits, and construct measurement invariance. The instruments used to test these influences were from the domain of personality and gender inequality, and their items were adapted based on the method effect tested. All tested method effects, except careless responding, had a statistically significant effect on at least one component of the analysis. Item wording and confirmatory bias affected mean values, model fit, and measurement invariance. Controlling for acquiescence bias improved the fit of the model. This paper confirms that the tested method effects should be carefully considered when using surveys in research, and suggests some guidelines on how to do so.

Much psychological research depends on participants' diligence in filling out materials such as surveys. However, not all participants are motivated to respond attentively, which leads to unintended issues with data quality, known as careless responding. Our question is: how do different modes of data collection—paper/pencil, computer/web-based, and smartphone—affect participants' diligence vs. "careless responding" tendencies and, thus, data quality? Results from prior studies suggest that different data collection modes produce a comparable prevalence of careless responding tendencies. However, as technology develops and data are collected with increasingly diversified populations, this question needs to be readdressed and taken further. The present research examined the effect of survey mode on careless responding in a repeated-measures design with data from three different samples. First, in a sample of working adults from China, we found that participants were slightly more careless when completing computer/web-based survey materials than in paper/pencil mode. Next, in a German student sample, participants were slightly more careless when completing the paper/pencil mode compared to the smartphone mode. Finally, in a sample of Chinese-speaking students, we found no difference between modes. Overall, in a meta-analysis of the findings, we found minimal difference between modes across cultures. Theoretical and practical implications are discussed. [ABSTRACT FROM AUTHOR]

While self-report data is a staple of modern psychological studies, they rely on participants accurately self-reporting. Two constructs that impede accurate results are insufficient effort responding (IER) and response styles. These constructs share conceptual underpinnings and both utilized to reduce cognitive effort when responding to self-report scales. Little research has extensively explored the relationship of the two constructs. The current study explored the relationship of the two constructs across even-point and odd-point scales, as well as before and after data cleaning procedures. We utilized IRTrees, a statistical method for modeling response styles, to examine the relationship between IER and response styles. To capture the wide range of IER metrics available, we employed several forms of IER assessment in our analyses and generated IER factors based on the type of IER being detected. Our results indicated an overall modest relationship between IER and response styles, which varied depending on the type of IER metric being considered or type of scale being evaluated. As expected, data cleaning also changed the relationships of some of the variables. We posit the difference between the constructs may be the degree of cognitive effort participants are willing to expend. Future research and applications are discussed.

Careless responding is a bias in survey responses that disregards the actual item content, constituting a threat to the factor structure, reliability, and validity of psychological measurements. Different approaches have been proposed to detect aberrant responses such as probing questions that directly assess test-taking behavior (e.g., bogus items), auxiliary or paradata (e.g., response times), or data-driven statistical techniques (e.g., Mahalanobis distance). In the present study, gradient boosted trees, a state-of-the-art machine learning technique, are introduced to identify careless respondents. The performance of the approach was compared with established techniques previously described in the literature (e.g., statistical outlier methods, consistency analyses, and response pattern functions) using simulated data and empirical data from a web-based study, in which diligent versus careless response behavior was experimentally induced. In the simulation study, gradient boosting machines outperformed traditional detection mechanisms in flagging aberrant responses. However, this advantage did not transfer to the empirical study. In terms of precision, the results of both traditional and the novel detection mechanisms were unsatisfactory, although the latter incorporated response times as additional information. The comparison between the results of the simulation and the online study showed that responses in real-world settings seem to be much more erratic than can be expected from the simulation studies. We critically discuss the generalizability of currently available detection methods and provide an outlook on future research on the detection of aberrant response patterns in survey research. [ABSTRACT FROM AUTHOR]

While self-report data is a staple of modern psychological studies, they rely on participants accurately self-reporting. Two constructs that impede accurate results are insufficient effort responding (IER) and response styles. These constructs share conceptual underpinnings and both utilized to reduce cognitive effort when responding to self-report scales. Little research has extensively explored the relationship of the two constructs. The current study explored the relationship of the two constructs across even-point and odd-point scales, as well as before and after data cleaning procedures. We utilized IRTrees, a statistical method for modeling response styles, to examine the relationship between IER and response styles. To capture the wide range of IER metrics available, we employed several forms of IER assessment in our analyses and generated IER factors based on the type of IER being detected. Our results indicated an overall modest relationship between IER and response styles, which varied depending on the type of IER metric being considered or type of scale being evaluated. As expected, data cleaning also changed the relationships of some of the variables. We posit the difference between the constructs may be the degree of cognitive effort participants are willing to expend. Future research and applications are discussed. [ABSTRACT FROM AUTHOR]

Following research that demonstrates insufficient effort responding (IER) may confound survey measures and inflate observed correlations (Huang, Liu, & Bowling, 2015c), a question emerges as to whether and when IER can act as a confound between objective tests and surveys. Using data (N = 243) originally designed to examine training and transfer, study 1 demonstrates that (a) IER is negatively related to performance on tests, and (b) IER's influence on surveys depends on the sample means of these measures. As a result, IER could inflate a test's association with other tests and surveys. Study 2 investigates the impact of two parameters—within-person consistency of IER and percentage of IER cases in the sample—by randomly replacing bootstrapped attentive responses (10,000 bootstrapped samples of 200 cases identified from study 1). When predicting the confounding effects of IER, within-person consistency has positive linear and quadratic effects, percentage of IER cases has a positive linear effect, and consistency and percentage have a positive interactive effect. Research and practical implications for the design and evaluation of surveys and tests are discussed. [ABSTRACT FROM AUTHOR]

Is there a point within a self-report questionnaire where participants will start responding carelessly? If so, then after how many items do participants reach that point? And what can researchers do to encourage participants to remain careful throughout the entirety of a questionnaire? We conducted two studies (Study 1 N = 358; Study 2 N = 129) to address these questions. Our results found (a) consistent evidence that participants responded more carelessly as they progressed further into a questionnaire, (b) mixed evidence that participants who were warned that carelessness would be punished displayed smaller increases in carelessness, and (c) mixed evidence that increases in carelessness were greater within an unproctored online study (Study 1) than within a proctored laboratory study (Study 2). These findings help address when and why careless responding is likely to occur, and they suggest effective preventive strategies. [ABSTRACT FROM AUTHOR]

The response entropy (RE) index calculates the balance of proportions of values endorsed by participants on Likert-scaled surveys, and is proposed as a new method for flagging careless response patterns. In the first study, performance of the RE index was compared to other commonly used post hoc indices for detecting careless responding (CR) such as the Mahalanobis distance (MD) and the psychometric synonym (PS) index. Three different types of Bogus Scores (BS) were generated: 1) uniform random values produced by computer (n = 100); 2) normally distributed random values produced by computer (n = 100); and 3) purposefully careless responses produced by human participants (n = 100). The BS data were then implanted in a true, cleaned social science dataset (n = 500). Multinomial logistic regression determined that the RE index made independent contributions from other indices to the prediction of BS. Latent variable analyses suggest that the variability type RE index may be tapping distinct constructs from regression type indices such as the PS index. In study 2, potential cultural bias in CR indices was examined with a true social science dataset (n = 302) comprised of racially diverse participants. Unlike other post hoc indices of CR, the RE index was unrelated to participant race. Further analyses demonstrated that racial differences on other indices of CR could be accounted for by culturally different styles of survey responding. For example, Asian participants’ higher MD scores relative to White participants’ was mediated by a culturally specific acquiescent survey response style.

Copyright of Revija za Sociologiju is the property of Revija za Sociologiju and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Background: Intensive longitudinal data (ILD) collection methods have gained popularity in social and behavioral research as a tool to better understand behavior and experiences over time with reduced recall bias. Engaging participants in these studies over multiple months and ensuring high data quality are crucial but challenging due to the potential burden of repeated measurements. It is suspected that participants may engage in inattentive responding (IR) behavior to combat burden, but the processes underlying this behavior are unclear as previous studies have focused on the barriers to compliance rather than the barriers to providing high-quality data., Objective: This study aims to broaden researchers' knowledge about IR during ILD studies using qualitative analysis and uncover the underlying IR processes to aid future hypothesis generation., Methods: We explored the process of IR by conducting semistructured qualitative exit interviews with 31 young adult participants (aged 18-29 years) who completed a 12-month ILD health behavior study with daily evening smartphone-based ecological momentary assessment (EMA) surveys and 4-day waves of hourly EMA surveys. The interviews assessed participants' motivations, the impact of time-varying contexts, changes in motivation and response patterns over time, and perceptions of attention check questions (ACQs) to understand participants' response patterns and potential factors leading to IR., Results: Thematic analysis revealed 5 overarching themes on factors that influence participant engagement: (1) friends and family also had to tolerate the frequent surveys, (2) participants tried to respond to surveys quickly, (3) the repetitive nature of surveys led to neutral responses, (4) ACQs within the surveys helped to combat overly consistent response patterns, and (5) different motivations for answering the surveys may have led to different levels of data quality., Conclusions: This study aimed to examine participants' perceptions of the quality of data provided in an ILD study to contribute to the field's understanding of engagement. These findings provide insights into the complex process of IR and participant engagement in ILD studies with EMA. The study identified 5 factors influencing IR that could guide future research to improve EMA survey design. The identified themes offer practical implications for researchers and study designers, including the importance of considering social context, the consideration of dynamic motivations, and the potential benefit of including ACQs as a technique to reduce IR and leveraging the intrinsic motivators of participants. By incorporating these insights, researchers might maximize the scientific value of their multimonth ILD studies through better data collection protocols., International Registered Report Identifier (irrid): RR2-10.2196/36666., (©Shirlene D Wang, Lori Hatzinger, Jeremy Morales, Micaela Hewus, Stephen Intille, Genevieve F Dunton. Originally published in JMIR Formative Research (https://formative.jmir.org), 02.08.2024.)

Despite recent concerns about data quality, various academic fields rely increasingly on crowdsourced samples. Thus, the goal of this study was to systematically assess carelessness in a crowdsourced sample (N = 394) by applying various measures and detection methods. A Latent Profile Analysis revealed that 45.9% of the participants showed some form of careless behavior. Excluding these participants increased the effect size in an experiment included in the survey. Based on our findings, several recommendations of easy to apply measures for assessing data quality are given.

Participant attentiveness may represent a major concern for all researchers using online self-report survey data, as findings from non-diligent participants add noise and can significantly decrease results reliability. Therefore, attention checks have become a popular method in survey design across social sciences to capture careless or insufficient-effort of respondents, thus increasing quality of samples and the internal validity of the research. The aim of this note is to offer an overview and categorization of the different techniques adopted to flag inattentive respondents and present the potential drawbacks of not considering the issue in social sciences research.

In this paper, we examine rates of careless responding and reactions towards detection methods (i.e., attention check item and instruction) in an experimental setting based on two different samples. First, we use a quota sample (with monetary incentive), a central data source for internet-based surveys in sociological and political research. Second, we include a voluntary opt-in panel (without monetary incentive) well suited to conduct survey experiments (e.g., factorial surveys). Respondents’ reactions towards the detection items are analyzed by objective, non-reactive indicators (i.e., break-off, item nonresponse, and measurement quality), and two self-report scales. Our reaction analyses reveal that the detection methods that we applied are not only well suited to identify careless respondents but also exert a motivational rather than demotivating influence on respondents’ answer behavior and, hence, contribute to data quality.

In self-report surveys, it is common that some individuals do not pay enough attention and effort to give valid responses. Our aim was to investigate the extent to which careless and insufficient effort responding contributes to the biasing of data. We performed analyses of dimensionality, internal structure, and data reliability of four personality scales (extroversion, conscientiousness, stability, and dispositional optimism) in two independent samples. In order to identify careless/insufficient effort (C/IE) respondents, we used a factor mixture model (FMM) designed to detect inconsistencies of response to items with different semantic polarity. The FMM identified between 4.4% and 10% of C/IE cases, depending on the scale and the sample examined. In the complete samples, all the theoretical models obtained an unacceptable fit, forcing the rejection of the starting hypothesis and making additional wording factors necessary. In the clean samples, all the theoretical models fitted satisfactorily, and the wording factors practically disappeared. Trait estimates in the clean samples were between 4.5% and 11.8% more accurate than in the complete samples. These results show that a limited amount of C/IE data can lead to a drastic deterioration in the fit of the theoretical model, produce large amounts of spurious variance, raise serious doubts about the dimensionality and internal structure of the data, and reduce the reliability with which the trait scores of all surveyed are estimated. Identifying and filtering C/IE responses is necessary to ensure the validity of research results. [ABSTRACT FROM AUTHOR]

While it is easy to assume that university students who wait until the last minute to complete surveys for their class research requirements provide low‐quality data, this issue has not been empirically examined. The goal of the present study was to examine the relation between student research procrastination and two important data quality issues—careless responding and measurement noninvariance. Data gathered from university students across two semesters tentatively indicated that procrastination is related to low‐quality survey data. Procrastination was slightly more problematic for certain data quality issues (measurement noninvariance) than others (careless responding). These relations, however, were small and contingent on how procrastination and careless responding were measured. Accordingly, it seems more beneficial for researchers to select a sampling window that supports their research goals and statistical power requirements rather than select a sampling window that attempts to minimize careless survey responding or other measurement issues. [ABSTRACT FROM AUTHOR]