Search

Cliff’s Delta is a non-parametric effect size that is based on data observations. In this paper, we introduce an adaptation of Cliff’s Delta in order to compare behaviour profiles. Behaviour profiles are density distributions in which survey answer behaviour is summarized for specific groups of respondents or items. Such profiles are useful, as they take into account the varying number of survey items that is filled out per respondent due to filter questions. By the adapted Cliff’s Delta, two subgroups of respondents (for instance higher and lower educated respondents) can be compared on the occurrence of specific answer behaviour (for instance giving ‘don’t know’-answers). By means of simulations, we show that the adapted profile-based Cliff’s Delta converges towards the original observation-based Cliff’s Delta as the number of items that is filled out by respondents increases. We conclude that the profile-based Cliff’s Delta is a solid and conservative statistic that is both useful and advantageous to compare behaviour profiles. We close with various data examples to illustrate its broad usefulness and by discussing potential difficulty in using the profile-based Cliff’s Delta., Survey Research Methods, Vol. 16 No. 3 (2022)

Answers to survey questions may deviate from the truth. We name suboptimal or non-substantive answers ‘undesirable answer behaviour’. The extent to which responses deviate from the ‘true’ value is referred to as ‘measurement error’. The purpose of this thesis was to detect respondent characteristics that are structurally related to measurement error. First, we investigated a large selection of answer behaviours across ten population surveys from CentERdata and Statistics Netherlands on their occurrences and variances by means of multilevel models. Based on the outcomes, we chose to switch to a method that summarizes and visualizes answer behaviour for each survey and behaviour in detail: The construction of ‘answer behaviour profiles’. Second, we defined the answer behaviour profile as a density distribution that summarizes answer behaviour for a group of respondents (the ‘respondent profile’) or a group of items (the ‘item profile’). By means of simulations, we examined the statistical properties of the behaviour profiles. We showed that a respondent profile becomes more informative and precise as each respondent fills out more items. And we showed that a respondent profile becomes more stable and certain as the group consists of a larger number of respondents. When a respondent profile is both precise and certain, the respondent profile is accurate. The very same applies to item profiles, with the concepts of ‘respondent’ and ‘item’ interchanged. We used real survey data to illustrate the properties of respondent profiles, item profiles, and smoothing in practice. Third, we examined the potentially consistent relation between respondent characteristics and undesirable answer behaviour across ten population surveys. We constructed respondent profiles for subgroups of respondents that are based on the categories of several respondent characteristics. These characteristics were investigated on their relation to eight specific undesirable answer behaviours. The profile for each characteristics’ category was compared to the profile for all other categories of that characteristic taken together. These comparisons were executed by an adaptation of the robust effect size measure Cliff’s Delta. Although various categories of various characteristics appeared to be related to various behaviours, the occurrence and direction of these relations were substantially dependent on the specific characteristic, survey, and behaviour. This means that a subgroup of respondents frequently showed more of some behaviour than the other subgroups for one survey, while less of that behaviour than the other subgroups for another survey. Thus, we did not find consistent relations between any respondent characteristic and any answer behaviour. We consider it likely that item characteristics have a more substantial and consistent influence on answer behaviour than respondent characteristics. Therefore, we recommend the use of item profiles to investigate this influence to detect item characteristics that may lead to measurement error. Hence, we anticipated on examining the association between item characteristics and undesirable answer behaviour in follow-up research. We investigated to what degree item characteristics could be determined reliably. The results showed that intercoder reliability was low for various item characteristics. We tried to explain low intercoder reliability and made several suggestions to cope with low intercoder reliability.

The Cliff's Delta statistic is a non-parametric effect size measure that quantifies the amount of difference between two groups of observations beyond p-values interpretation. This measure can be understood as a useful complementary analysis for the corresponding hypothesis testing. During the last two decades the use of effect size measures has been strongly encouraged by methodologists and leading institutions of behavioral sciences. The aim of this contribution is to introduce the Cliff's Delta Calculator software that performs such analysis and offers some interpretation tips. Differences and similarities with the parametric case are analysed and illustrated. The implementation of this free program is fully described and compared with other calculators. Alternative algorithmic approaches are mathematically analysed and a basic linear algebra proof of its equivalence is formally presented. Two worked examples in cognitive psychology are commented. A visual interpretation of Cliff's Delta is suggested. Availability, installation and applications of the program are presented and discussed. El estadístico Delta de Cliff permite cuantificar la magnitud de la diferencia entre dos grupos de observaciones que resultan incompatibles con el presupuesto de normalidad. El análisis de esta cuantificación complementa la interpretación del p-valor asociado a la correspondiente prueba de hipótesis. La aplicación de medidas del tamaño del efecto ha sido promovida, durante las últimas décadas, tanto por metodólogos, como por instituciones líderes en las ciencias del comportamiento. El propósito de la presente contribución es presentar un programa denominado Cliff's Delta Calculator con capacidad para calcular y graficar el valor del tamaño del efecto no paramétrico para dos grupos de observaciones. Se ofrece una comparación de este programa con otras calculadoras comerciales y no comerciales actualmente disponibles. Se describe su funcionamiento, sus fundamentos matemáticos y se presentan y discuten dos ejemplos de aplicación en la investigación psicológica. Se concluye que este programa presenta algunas ventajas, en comparación con otras calculadoras comerciales disponibles.