Search

The propensity for people to avoid mentally demanding tasks in the absence of reward is well documented. As a result, humans are often described as cognitive misers. This characterisation, while consistent with the psychological literature, contradicts everyday instances of effort being sought: reading, board games, and brain-teasing puzzles. Such examples however are markedly different from the types of tasks typically used in the mental effort literature (e.g., working memory tasks, demand selection tasks). The current set of experiments assessed whether the type of task (i.e., N-Back, Number Sequence Problems [NSP], or Anagrams) affects people’s aversion to, or desire for, increased effort. On average, across 3 experiments, participants showed an aversion to effort regardless of whether the effort required was more attentional (N-Back) or cognitive (NSP and anagrams) in nature, and were willing to forgo financial reward in order to avoid more difficult tasks. A minority of participants, however, sought more effortful tasks for equal or lesser reward.

Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

The target article on robust modeling (Lee et al.) generated a lot of commentary. In this reply, we discuss some of the common themes in the commentaries; some are simple points of agreement while others are extensions of a practical or abstract nature. We also address a small number of disagreements or confusions.

Most data analyses rely on models. To complement statistical models, psychologists have developed cognitive models, which translate observed variables into psychologically interesting constructs. Response time models, in particular, assume that response time and accuracy are the observed expression of latent variables including 1) ease of processing, 2) response caution, 3) response bias, and 4) non-decision time. Inferences about these psychological factors, hinge upon the validity of the models' parameters. Here, we use a blinded, collaborative approach to assess the validity of such model-based inferences. Seventeen teams of researchers analyzed the same 14 data sets. In each of these two-condition data sets, we manipulated properties of participants' behavior in a two-alternative forced choice task. The contributing teams were blind to the manipulations, and had to infer what aspect of behavior was changed using their method of choice. The contributors chose to employ a variety of models, estimation methods, and inference procedures. Our results show that, although conclusions were similar across different methods, these "modeler's degrees of freedom" did affect their inferences. Interestingly, many of the simpler approaches yielded as robust and accurate inferences as the more complex methods. We recommend that, in general, cognitive models become a typical analysis tool for response time data. In particular, we argue that the simpler models and procedures are sufficient for standard experimental designs. We finish by outlining situations in which more complicated models and methods may be necessary, and discuss potential pitfalls when interpreting the output from response time models.

Computational models of visual working memory (VWM)generally fall into two categories: slots-based models andresources-based models. Slots-based models theorise that thecapacity of memory is defined by a finite number of items.Each slot can only contain one item and once an item is inmemory it is remembered accurately. If an item is not inmemory, however, there is no memory of the item at all. Bycontrast, resources-based models claim that all items, ratherthan just a few enter memory. However, unlike the slots modelthey are not necessarily remembered accurately. On thesurface, these models appear to make distinct predictions.However, as these models have been developed and expandedto capture empirical data, they have begun to mimic each other.Further complicating matters, Donkin, Kary, Tahir and Taylor(2016) proposed that observers were capable of using eitherslot- or resource-based encoding strategies. In the currentexperiment, we aimed to test the claim that observers adapttheir encoding strategies depending on the task environment byobserving how participants move their eyes in a VWMexperiment. We ran participants on a standard colour recall task(Zhang and Luck, 2008) while tracking their eye movements.All participants were asked to remember either 3 or 6 items ina given trial, and we manipulated whether the number of itemswas held constant for a block of trials, or varied randomly. Weexpected to see participants use more resource-like encodingwhen the number of items to remember was predictable.Contrary to these expectations, we observed no differencebetween blocked and unblocked conditions. Further, the eyegaze data was only very weakly related to behaviour in the task.We conclude that caution should be taken in interpreting eyegaze data in VWM experiments.

Interruptions are an inevitable part of every day life. Previousresearch suggests that interruptions can decrease performanceand increase errors and response time. Additionally, there isevidence that providing a lag time prior to an interruption canmitigate some of the interruption costs. The goal of this pa-per is to investigate the effects of interruptions and interrup-tion lags and explore possible strategies to attenuate interrup-tion costs. A novel sequential decision-making paradigm wasused, where the difficulty of the task and type of interruptionwere the two experimental manipulations. The results indicatethat there is a potential benefit to including a lag time whenpresented with interruptions.

We describe and demonstrate an empirical strategy useful for discovering and replicating empirical effects in psychological science. The method involves the design of a metastudy, in which many independent experimental variables-that may be moderators of an empirical effect-are indiscriminately randomized. Radical randomization yields rich datasets that can be used to test the robustness of an empirical claim to some of the vagaries and idiosyncrasies of experimental protocols and enhances the generalizability of these claims. The strategy is made feasible by advances in hierarchical Bayesian modeling that allow for the pooling of information across unlike experiments and designs and is proposed here as a gold standard for replication research and exploratory research. The practical feasibility of the strategy is demonstrated with a replication of a study on subliminal priming.

Over the last four decades, sequential accumulation models for choice response times have spread through cognitive psychology like wildfire. The most popular style of accumulator model is the diffusion model (Ratcliff Psychological Review, 85, 59-108, 1978), which has been shown to account for data from a wide range of paradigms, including perceptual discrimination, letter identification, lexical decision, recognition memory, and signal detection. Since its original inception, the model has become increasingly complex in order to account for subtle, but reliable, data patterns. The additional complexity of the diffusion model renders it a tool that is only for experts. In response, Wagenmakers et al. (Psychonomic Bulletin & Review, 14, 3-22, 2007) proposed that researchers could use a more basic version of the diffusion model, the EZ diffusion. Here, we simulate experimental effects on data generated from the full diffusion model and compare the power of the full diffusion model and EZ diffusion to detect those effects. We show that the EZ diffusion model, by virtue of its relative simplicity, will be sometimes better able to detect experimental effects than the data-generating full diffusion model.

How does the process of information transmission affect thecultural products that emerge from that process? This questionis often studied experimentally and computationally via iter-ated learning, in which participants learn from previous partic-ipants in a chain. Much research in this area builds on math-ematical analyses suggesting that iterated learning chains con-verge to people’s priors. We present three simulation studiessuggesting that when the population of learners is heteroge-neous, the behavior of the chain is systematically distorted bythe learners with the most extreme biases. We discuss implica-tions for the use of iterated learning as a methodological tooland for the processes that might have shaped cultural productsin the real world.

Past research indicates that individuals respond adaptively to contextual factors in multiattribute choice tasks. Yet it remains unclear how this adaptation is cognitively governed. In this article, empirically testable implementations of two prominent competing theoretical frameworks are developed and compared across two multiattribute choice experiments: the adaptive toolbox framework assuming discrete choice strategies and the adjustable spanner framework assuming one comprehensive adaptive strategy. Results from two experiments indicate that in the environments we tested, in which all cue information was presented openly, the toolbox makes better predictions than the adjustable spanner both in- and out-of-sample. Follow-up simulation studies indicate that it is difficult to discriminate the models based on choice outcomes alone but allowed the identification of a small subset of cases where the predictions of both models diverged. Our results suggest that people adapt their decision strategies by flexibly switching between using as little information as possible and use of all of the available information.

Does the similarity between two items change over time? Previousstudies (Goldstone & Medin, 1994; Gentner & Brem,1999) have found suggestive results but have relied on interpretingcomplex interaction effects from “deadline” decisiontasks in which the decision making process is not well understood(Luce, 1986). Using a self-paced simple decisiontask in which the similarity between two items can be isolatedfrom strategic decision processes using computational modelingtechniques (Ratcliff, 1978), we show strong evidence thatthe similarity between two items changes over time and shiftsin systematic ways. The change in similarity from early tolate processing in Experiment 1 is consistent with the theoryof structural alignment (Gentner, 1983; Goldstone & Medin,1994), and Experiment 2 demonstrates evidence for a strongerinfluence of thematic knowledge than taxonomic knowledge inearly processing of word associations (Lin & Murphy, 2001).

Utility based models are common in both the risky and intertemporalchoice literatures. Recently there have been efforts toformulate models of choices which involve both risks andtime delays. An important question then is whether theconcept of utility is the same for risky and inter-temporalchoices. We address this question by fitting versions of twopopular utility based models, Cumulative Prospect Theory forrisky choice, and Hyperbolic Discounting for inter-temporalchoice, to data from three experiments which involved bothchoice types. The models were fit assuming either the sameconcept of utility for both, by way of a common valuefunction, or different utilities with separate value functions.Our results show that while many participants seem to requirethe flexibility of different value functions, an approximatelyequal number do not suggesting they may have a singleconcept of utility. Furthermore for both choice types valuefunctions were concave

How does the process of information transmission affect the cultural or linguistic products that emerge? This question is often studied experimentally and computationally via iterated learning, a procedure in which participants learn from previous participants in a chain. Iterated learning is a powerful tool because, when all participants share the same priors, the stationary distributions of the iterated learning chains reveal those priors. In many situations, however, it is unreasonable to assume that all participants share the same prior beliefs. We present four simulation studies and one experiment demonstrating that when the population of learners is heterogeneous, the behavior of an iterated learning chain can be unpredictable and is often systematically distorted by the learners with the most extreme biases. This results in group-level outcomes that reflect neither the behavior of any individuals within the population nor the overall population average. We discuss implications for the use of iterated learning as a methodological tool as well as for the processes that might have shaped cultural and linguistic evolution in the real world.

We report an experiment designed to provide a qualitative contrast between knowledge-limited versions of mixed-state and variable-resources (VR) models of visual change detection. The key data pattern is that observers often respond "same" on big-change trials, while simultaneously being able to discriminate between same and small-change trials. The mixed-state model provides a natural account of this data pattern: With some probability, the observer is in a zero-memory state and is forced to guess. Thus, even on big-change trials, there is a significant probability that the observer will respond "same." On other trials, the observer retains memory for the probed study item, and these memory-based responses allow the observer to show above-chance discrimination between same and small-change trials. By contrast, we show that important versions of the VR models that we refer to as "knowledge-limited" models are stymied by this simple pattern of results. In agreement with Keshvari, van den Berg, and Ma (2012, 2013), alternative "knowledge-rich" VR models that employ ideal-observer decision rules provide a significant improvement over the knowledge-limited VR models; however, extant versions of the knowledge-rich VR models still fall short quantitatively compared to the descriptive mixed-state model. We discuss implications of the knowledge-rich assumptions that are posited in current versions of the VR models that have been used to fit change-detection data.

Reasoning and inference are well-studied aspects of basic cognition that have been explained as statistically optimal Bayesian inference. Using a simplified experimental design, we conducted quantitative comparisons between Bayesian inference and human inference at the level of individuals. In 3 experiments, with more than 13,000 participants, we asked people for prior and posterior inferences about the probability that 1 of 2 coins would generate certain outcomes. Most participants' inferences were inconsistent with Bayes' rule. Only in the simplest version of the task did the majority of participants adhere to Bayes' rule, but even in that case, there was a significant proportion that failed to do so. The current results highlight the importance of close quantitative comparisons between Bayesian inference and human data at the individual-subject level when evaluating models of cognition.

The strength of conclusions about the adoption of different categorization strategies--and their implications for theories about the cognitive and neural bases of category learning--depend heavily on the techniques for identifying strategy use. We examine performance in an often-used "information-integration" category structure and demonstrate that strategy identification is affected markedly by the range of models under consideration, the type of data collected, and model-selection techniques. We use a set of 27 potential models that represent alternative rule-based and information-integration categorization strategies. Our experimental paradigm includes the presentation of nonreinforced transfer stimuli that improve one's ability to discriminate among the predictions of alternative models. Our model-selection techniques incorporate uncertainty in the identification of individuals as either rule-based or information-integration strategy users. Based on this analysis we identify 48% of participants as unequivocally using an information-integration strategy. However, adopting the standard practice of using a restricted set of models, restricted data, and ignoring the degree of support for a particular strategy, we would typically conclude that 89% of participants used an information-integration strategy. We discuss the implications of potentially erroneous strategy identification for the security of conclusions about the categorization capabilities of various participant and patient groups.

Decision-makers effortlessly balance the need for urgency against the need for caution. Theoretical and neurophysiological accounts have explained this tradeoff solely in terms of the "quantity" of evidence required to trigger a decision (the "threshold"). This explanation has also been used as a benchmark test for evaluating new models of decision making, but the explanation itself has not been carefully tested against data. We rigorously test the assumption that emphasizing decision speed versus decision accuracy selectively influences only decision thresholds. In data from a new brightness discrimination experiment we found that emphasizing decision speed over decision accuracy not only decreases the amount of evidence required for a decision but also decreases the quality of information being accumulated during the decision process. This result was consistent for 2 leading decision-making models and in a model-free test. We also found the same model-based results in archival data from a lexical decision task (reported by Wagenmakers, Ratcliff, Gomez, & McKoon, 2008) and new data from a recognition memory task. We discuss implications for theoretical development and applications.