Search

This paper addresses insufficient effort responding (IER), a significant issue in survey research affecting data quality. We focus on IER prediction through nonreactive measures and gauge its prevalence in a pupil population, leveraging a sizable online survey of adolescents. The analysis uncovers IER as a nonmarginal issue that varies considerably by gender, migration status, and school type. Utilizing Random Forest models, we evaluate nonreactive measures’ predictive power for IER, notably response time, intra-individual response variability, and Mahalanobis distance. The findings highlight the future research value of these measures, emphasizing the strong influence of response time. We also explore the relationship between predictors and IER and find that shorter response times and less response variability correspond to a greater likelihood of IER. This study illustrates the potential of nonreactive measures and advanced machine learning techniques for predicting IER and highlights the necessity for further research.

This project studies the changes in participants' responding styles in time.

A preregistration of a study using an existing data set

Crowdsourcing research participants online has grown in prevalence in the previous decade. It involves drawing on a large pool of people to provide 'labour', in this case participation in research studies. This methodology for recruitment has considerable benefits, including access to diverse populations without the geographical restrictions of lab-based research, and increasing the scale and speed of research (often with reduced costs) (Wazny, 2017: J Glob health, 020602). There are also potential negatives to large-scale crowdsourced research. Given the 'distance' between researchers and participants and the lack of controlled environment, there is the increased possibility of distractions and 'careless responding'. These responses can lead to poor quality or missing data, which in turn can have serious consequences such as failed-replications or false positives (Bruhlmann et al, 2020: Methods in Psychology, 100022). The aim of this systematic review/meta-analysis is to examine the prevalence of carelessness responding in crowdsourced research with a particular focus on alcohol use. By capturing the prevalence of careless /non-effortful responding researchers can be made aware of potential issues and how they may be overcome.

While self-report data is a staple of modern psychological studies, they rely on participants accurately self-reporting. Two constructs that impede accurate results are insufficient effort responding (IER) and response styles. These constructs share conceptual underpinnings and both utilized to reduce cognitive effort when responding to self-report scales. Little research has extensively explored the relationship of the two constructs. The current study explored the relationship of the two constructs across even-point and odd-point scales, as well as before and after data cleaning procedures. We utilized IRTrees, a statistical method for modeling response styles, to examine the relationship between IER and response styles. To capture the wide range of IER metrics available, we employed several forms of IER assessment in our analyses and generated IER factors based on the type of IER being detected. Our results indicated an overall modest relationship between IER and response styles, which varied depending on the type of IER metric being considered or type of scale being evaluated. As expected, data cleaning also changed the relationships of some of the variables. We posit the difference between the constructs may be the degree of cognitive effort participants are willing to expend. Future research and applications are discussed.

Careless responding is a bias in survey responses that disregards the actual item content, constituting a threat to the factor structure, reliability, and validity of psychological measurements. Different approaches have been proposed to detect aberrant responses such as probing questions that directly assess test-taking behavior (e.g., bogus items), auxiliary or paradata (e.g., response times), or data-driven statistical techniques (e.g., Mahalanobis distance). In the present study, gradient boosted trees, a state-of-the-art machine learning technique, are introduced to identify careless respondents. The performance of the approach was compared with established techniques previously described in the literature (e.g., statistical outlier methods, consistency analyses, and response pattern functions) using simulated data and empirical data from a web-based study, in which diligent versus careless response behavior was experimentally induced. In the simulation study, gradient boosting machines outperformed traditional detection mechanisms in flagging aberrant responses. However, this advantage did not transfer to the empirical study. In terms of precision, the results of both traditional and the novel detection mechanisms were unsatisfactory, although the latter incorporated response times as additional information. The comparison between the results of the simulation and the online study showed that responses in real-world settings seem to be much more erratic than can be expected from the simulation studies. We critically discuss the generalizability of currently available detection methods and provide an outlook on future research on the detection of aberrant response patterns in survey research.

The crosswise model is a popular sensitive question technique often considered more accurate than direct questioning. When this technique is used, a sensitive question is paired with a nonsensitive question that has a known prevalence and respondents are asked to give a joint answer to the pair of questions. Recent research has shown that prevalence estimates based on the crosswise model are biased towards 50% when respondents answer randomly, and that random answers are frequent. I develop methods to adjust the crosswise model for self-reported random answers. Results from an exploratory online survey (n = 103) show that (i) fewer respondents report random answers than might be expected given unadjusted results, (ii) results differ considerably between questions, and (iii) one of three questions yields an estimate that is substantially and significantly above the true value even after adjusting for random answers., methods, data, analyses, Vol 13, No 2 (2019)

Because careless responding to questionnaire items can be quite common, and even low rates of careless responding can substantially impact some analyzes, researchers have been advised to remove careless respondents before data analysis. However, identifying these respondents is a non-trivial task. In the context of multi-item, unidimensional scales, it has been suggested that response variance may hold information about careless responding. This notion was tested in the current research with two studies. Data collected from respondents indicates that people responding carelessly display more variance in their responses than people responding with more effort. Given this finding, a procedure which uses measures of response variance and cluster analysis to identify careless respondents was developed. The effectiveness of the procedure with different specifications was tested with simulated data and validated with data from actual respondents. On the basis of the results, we advocate using the procedure with specifications that conservatively identify careless respondents. Such an approach will identify the most extreme careless respondents, while maximizing the retention of careful, honest respondents. We discuss the advantages the developed procedure has over existing procedures for identifying careless respondents. © 2015 Macmillan Publishers Ltd.

Сугоняев Константин Владимирович, кандидат технических наук, доцент, старший научный сотрудник Научно-практического центра, Военная академия Генерального штаба ВС РФ (Москва), skv-354@yandex.ru K.V. Sugonyaev, skv-354@yandex.ru Military Academy of General Staff, Moscow, Russian Federation Некооперативное тестовое поведение (НКТП) представляет собой серьезную, но часто недооцениваемую угрозу достоверности тестовых оценок и валидности их интерпретации.Одним из методов выявления НКТП может быть включение в состав многошкальных опросников шкал атипичности ответов (АТ), построенных на учете частоты подтверждения тех или иных опций ответа. Учитывая практическую важность задачи различения реалистических данных от проявлений НКТП при проведении массовых обследований военнослужащих, осуществлена разработка и включение в компьютерные версии ряда многошкальных опросников шкал АТ в сочетании с настройкой алгоритмов выявления хаотического реагирования, как одного из наиболее частых и деструктивных стилей НКТП. Проверка дифференцирующей способности шкал АТ осуществлялась путем сопоставления эмпирических данных, полученных в результате обследования восьми независимых выборок (N = 406–5689) с помощью шести опросников с разным форматом ответов, и множествами программно сгенерированных протоколов со случайным выбором ответов. Показано, что в среднем около 95 % сгенерированных таким образом протоколов идентифицируются как недостоверные, в том числе, 65 % из них опознаны как хаотические. Практика применения оснащенных шкалами АТ опросников в сочетании с результатами программного моделирования позволяют предположить, что включение подобных шкал в известные и вновразрабатываемые опросники может существенно снизить риски ошибочных диагностических выводов. Non-cooperative testing behavior (NCTB) during questionnaire completion is serious threat to the valid interpretations of test scores. There are a number of methods recommended for use to identify protocols with NCTB features in order to enhance the credibility of personality testing results (Karabatsos, 2003; Desimone, 2015), but not all of them are proper and accessible for practitioners. One of the earliest methods of aberrant responding detection is based on incorporation of validity scales, including those composed of response options with rare endorsement. Assumptions and considerations regarding design (without adding supplementary items) and appropriate use of “atypical responses” (AR) scales are discussed. Such scales have been designed and implemented in some known and newly designed questionnaires. Findings suggest that the AR scale cutoff that is optimally efficient for detecting NCTB in high-stake testing may be calculated on randomly generated data as MR – 2*SDR. Validation of AR cutoff scores was accomplished by means of comparison of eight empirical (N = 406–5689) and randomly generated data for six questionnaires with different number of response options. Established sensitivity (0.92–0.99) is provided evidence for acceptable discriminating power of AR scores. This figure is comparable with person-fit methods based on IRT. Overall, the results provide support for the AR scores as indexes of NCTB. Military personnel testing practice suggest that taking into account AR scores enables to enhance the credibility of testing results and to diminish risks of diagnostic failures.

We investigated the effects that careless responding and acquiescence response bias have on analyses conducted to assess construct dimensionality. Using job satisfaction/dissatisfaction as the focal construct, we measured and controlled for careless responding and acquiescence bias in data obtained from an online survey of employees ( N = 666) from different organizations and occupational groups. We found that the negative correlation between factors defined by job satisfaction and dissatisfaction items, respectively, was attenuated by careless responding and acquiescence bias, and was not significantly different from –1.0 when both were controlled. Moreover, the correlations between the two factors and measures of other constructs were similar when careless responding and acquiescence were controlled. These findings challenge recent research purporting to demonstrate that job satisfaction and dissatisfaction are distinct constructs. Recommendations for future investigations of construct dimensionality are provided.