Search

The sentences “More than half of the students passed the exam” and “Fewer than half of the students failed the exam” describe the same set of situations, and yet the former results in shorter reaction times in verification tasks. The two-step model explains this result by postulating that negative quantifiers contain hidden negation, which involves an extra processing stage. To test this theory, we applied a novel EEG analysis technique focused on detecting cognitive stages (HsMM-MVPA) to data from a picture-sentence verification task. We estimated the number of processing stages during reading and verification of quantified sentences (e.g. “Fewer than half of the dots are blue”) that followed the presentation of pictures containing coloured geometric shapes. We did not find evidence for an extra step during the verification of sentences with fewer than half. We provide an alternative interpretation of our results in line with an expectation-based pragmatic account.

For the foreseeable future, automated vehicles (AVs) will coexist on the roads with human drivers. To avoid accidents, AVs will require knowledge on how human drivers typically make high-stakes and time-sensitive decisions (e.g., whether or not to brake). Providing such insights could be statistical models designed to explain human information processing and decision making. This paper attempts to address a roadblock that prevents one class of such "cognitive models", evidence accumulation models (EAMs), from being widely applied in the design of AV systems: their high demands for data. Specifically, we investigate whether Bayesian hierarchical modeling can be used to determine a person's characteristics, if we only have limited data about their behavior but extensive data on other (comparable) people's behaviors. Leveraging a simulation study and a reanalysis of experimental data, we find that most parameters of Decision Diffusion Models (a class of EAMs) – representing information processing components – can be adequately estimated with as few as 20 observations, if prior information regarding the decision-making processes of the population is incorporated. Subsequently, we discuss the implications of our findings for the modeling of traffic situations.

When two cognitive processes contribute to a behavioral output—each process producing a specific distribution of the behavioral variable of interest—and when the mixture proportion of these two processes varies as a function of an experimental condition, a common density point should be present in the observed distributions of the data across said conditions. In principle, one can statistically test for the presence (or absence) of a fixed point in experimental data to provide evidence in favor of (or against) the presence of a mixture of processes, whose proportions are affected by an experimental manipulation. In this paper, we provide an empirical diagnostic of this test to detect a mixture of processes. We do so using resampling of real experimental data under different scenarios, which mimic variations in the experimental design suspected to affect the sensitivity and specificity of the fixed-point test (i.e., mixture proportion, time on task, and sample size). Resampling such scenarios with real data allows us to preserve important features of data which are typically observed in real experiments while maintaining tight control over the properties of the resampled scenarios. This is of particular relevance considering such stringent assumptions underlying the fixed-point test. With this paper, we ultimately aim at validating the fixed-point property of binary mixture data and at providing some performance metrics to researchers aiming at testing the fixed-point property on their experimental data.

The Implicit Association Test (IAT, Greenwald et al., J. Pers. Soc. Psychol., 74, 1998, 1464) is a popular instrument for measuring attitudes and (stereotypical) biases. Greenwald et al. (Behav. Res. Methods, 54, 2021, 1161) proposed a concrete method for validating IAT stimuli: appropriate stimuli should be familiar and easy to classify – translating to rapid (response times <800 ms) and accurate (error < 10%) participant responses. We conducted three analyses to explore the theoretical and practical utility of these proposed validation criteria. We first applied the proposed validation criteria to the data of 15 IATs that were available via Project Implicit. A bootstrap approach with 10,000 ‘experiments’ of 100 participants showed that 5.85% of stimuli were reliably valid (i.e., we are more than 95% confident that a stimulus will also be valid in a new sample of 18- to 25–year-old US participants). Most stimuli (78.44%) could not be reliably validated, indicating a less than 5% certainty in the outcome of stimulus (in)validity for a new sample of participants. We then explored how stimulus validity differs across IATs. Results show that only some stimuli are consistently (in)valid. Most stimuli show between-IAT variances, which indicate that stimulus validity differs across IAT contexts. In the final analysis, we explored the effect of stimulus type (images, nouns, names, adjectives) on stimulus validity. Stimulus type was a significant predictor of stimulus validity. Although images attain the highest stimulus validity, raw data show large differences within stimulus types. Together, the results indicate a need for revised validation criteria. We finish with practical recommendations for stimulus selection and (post-hoc) stimulus validation.

As Large Language Models and Natural Language Processing (NLP) technology rapidly develop and spread into daily life, it becomes crucial to anticipate how their use could harm people. One problem that has received a lot of attention in recent years is that this technology has displayed harmful biases, from generating derogatory stereotypes to producing disparate outcomes for different social groups. Although a lot of effort has been invested in assessing and mitigating these biases, our methods of measuring the biases of NLP models have serious problems and it is often unclear what they actually measure. In this paper, we provide an interdisciplinary approach to discussing the issue of NLP model bias by adopting the lens of psychometrics — a field specialized in the measurement of concepts like bias that are not directly observable. In particular, we will explore two central notions from psychometrics, the construct validity and the reliability of measurement tools, and discuss how they can be applied in the context of measuring model bias. Our goal is to provide NLP practitioners with methodological tools for designing better bias measures, and to inspire them more generally to explore tools from psychometrics when working on bias measurement tools. This article appears in the AI & Society track.

Artificial intelligence (AI) plays an important role in modern society. AI applications are omnipresent and assist many decisions we make in daily life. A common and important feature of such AI applications are user models. These models allow an AI application to adapt to a specific user. Here, we argue that user models in AI can be optimized by modeling these user models more closely to models of human cognition. We identify three levels at which insights from human cognition can be—and have been—integrated in user models. Such integration can be very loose with user models only being inspired by general knowledge of human cognition or very tight with user models implementing specific cognitive processes. Using AI-based applications in the context of education as a case study, we demonstrate that user models that are more deeply rooted in models of cognition offer more valid and more fine-grained adaptations to an individual user. We propose that such user models can also advance the development of explainable AI.

When doing a cognitive task, people can employ different cognitive strategies. A strategy consists of different cognitive processing stages, which is how different strategies can be differentiated. A novel machine learning method developed by Anderson et al., 2016 is able to model cognitive processing stages in EEG, MEG, and fMRI as a hidden semi-Markov model, calling it Hidden semi-Markov Model Multivariate Pattern Analysis (HsMM-MVPA). This method works across subjects, so among other things it seems to be able to deal with the inter-subject variability of EEG data. This leads to the hypothesis that HsMM-MVPA could potentially be used to predict what cognitive strategy someone used in new, unseen data. To test this hypothesis, EEG data collected from a group of subjects who performed a multiplication task with self-reported cognitive strategies was used. Subjects reported either knowing the answer to a multiplication problem from memory ("retrieval"), or had to compute the answer ("procedural"). We estimated hidden semi-Markov models on some of the subjects and tested how well these models could predict what strategy was used on the other subjects. The models are able to correctly identify retrieval-strategies, but tend to be less sensitive to the procedural-class. This seems to be because the retrieval-strategy is more consistent. HsMM-MVPA can be used for classification, but might fare better with more consistent cognitive strategies.

Due to COVID-19, governments have issued strict rules to mandate social distancing and virus-avoiding behavior in order to slow the spread of the virus. However, many factors influence people to either follow these rules or to disregard them. In this paper, we will investigate the theoretical background of the persuasive effect of a personalized simulated environment to visualize the spread of COVID-19. We propose an experiment where participants first answer questions about their virus-avoiding behavior, and their answers become the settings of the behavior of a simulated environment where agents can catch COVID-19. After one week, we ask about their virus-avoiding behavior again to see if it has changed. We conclude that the literary background suggests that personalization and persuasive technology could have a beneficial effect on virus-avoiding behavior.

Recently, sampling algorithms have demonstrated their ability in simulating ubiquitous cognitive characteristics, such as autocorrelations and levy-like distributions, found in foraging behaviours. However, this effect has been observed independently in individual tasks. Exploring the co-occurrence of such phenomena has yet to be investigated. This paper explores three main questions: Whether two foraging characteristics co-occur, whether cognitive load impacts said foraging characteristics and to what extent can sampling algorithms explain these cognitive characteristics. Seven participants were given two sequential tasks (Random number generation (RNGT) and a metronome tapping task) separately as well as together. The findings suggest that foraging does not co-occur; that autocorrelations are present in the tapping task but not RNGT and heavy tailed distributions are present in RNGT but not in the tapping task. Cognitive load only plays a role in the tapping task. Further analysis explored task-specific cognitive characteristics outlining potential strategies and patterns participants used (including randomness, run length, pattern type and jump length) and the findings were compared to four sampling algorithms: A Direct Sampler (DS), Markov Chain Monte Carlo (MCMC), Metropolis-Coupled Markov Chain Monte Carlo (MC3) and a No-U turn Hamiltonian Monte Carlo (HMC). The DS, MC3 and HMC were able to produce similar results to human behaviour but identifies that a hybrid approach of these three samplers might simulate the metrics produced by humans better.

Most data analyses rely on models. To complement statistical models, psychologists have developed cognitive models, which translate observed variables into psychologically interesting constructs. Response time models, in particular, assume that response time and accuracy are the observed expression of latent variables including 1) ease of processing, 2) response caution, 3) response bias, and 4) non-decision time. Inferences about these psychological factors hinge upon the validity of the models' parameters. Here, we use a blinded, collaborative approach to assess the validity of such model-based inferences. Seventeen teams of researchers analyzed the same 14 data sets. In each of these two-condition data sets, we manipulated properties of participants' behavior in a two-alternative forced choice task. The contributing teams were blind to the manipulations, and had to infer what aspect of behavior was changed using their method of choice. The contributors chose to employ a variety of models, estimation methods, and inference procedures. Our results show that, although conclusions were similar across different methods, these "modeler's degrees of freedom" did affect their inferences. Interestingly, many of the simpler approaches yielded as robust and accurate inferences as the more complex methods. We recommend that, in general, cognitive models become a typical analysis tool for response time data. In particular, we argue that the simpler models and procedures are sufficient for standard experimental designs. We finish by outlining situations in which more complicated models and methods may be necessary, and discuss potential pitfalls when interpreting the output from response time models.

Most data analyses rely on models. To complement statistical models, psychologists have developed cognitive models, which translate observed variables into psychologically interesting constructs. Response time models, in particular, assume that response time and accuracy are the observed expression of latent variables including 1) ease of processing, 2) response caution, 3) response bias, and 4) non-decision time. Inferences about these psychological factors, hinge upon the validity of the models’ parameters. Here, we use a blinded, collaborative approach to assess the validity of such model-based inferences. Seventeen teams of researchers analyzed the same 14 data sets. In each of these two-condition data sets, we manipulated properties of participants’ behavior in a two-alternative forced choice task. The contributing teams were blind to the manipulations, and had to infer what aspect of behavior was changed using their method of choice. The contributors chose to employ a variety of models, estimation methods, and inference procedures. Our results show that, although conclusions were similar across different methods, these "modeler’s degrees of freedom" did affect their inferences. Interestingly, many of the simpler approaches yielded as robust and accurate inferences as the more complex methods. We recommend that, in general, cognitive models become a typical analysis tool for response time data. In particular, we argue that the simpler models and procedures are sufficient for standard experimental designs. We finish by outlining situations in which more complicated models and methods may be necessary, and discuss potential pitfalls when interpreting the output from response time models.

Piéron's Law, the power relation between mean RT and stimulus intensity or discriminability, has historically been understood to reflect a non-linear scaling between objective intensity and perception. More recently, Piéron's Law was demonstrated to arise out of the architecture of rise-to-threshold decision-making models (Stafford and Gurney, 2004). Here we explicitly tested whether such an explanation would suffice to fit human data, or whether additional assumptions about the nature of perceptual processing are required. We fitted a simple rise-to-threshold model to full RT distributions and choice probabilities from three data sets that show Piéron's Law. The model assumed that accumulation rate was linearly related to perceptual processing, leaving only the architecture of the model to produce Piéron's Law. For two data sets, this linear rate model is unable to account for the data, suggesting that Piéron's Law sometimes reflects additional perceptual scaling information. For the third data set, however, Piéron's Law does appear to simply arise out of the rise-to-threshold architecture of decision-making models. Our results suggest that it is important to fit models to data in order to draw inference about the causes underlying Piéron's Law.