Search

In 2015, the Australian Government's Excellence in Research for Australia (ERA) assessment of research quality declined to rate 1.5 per cent of submissions from universities. The public debate focused on practices of gaming or "coding errors" within university submissions as the reason for this outcome. The issue was about the in/appropriate allocation of research activities to Fields of Research. This paper argues that such practices are only part of the explanation. With the support of statistical modelling, unrated outcomes are shown to have also arisen from particular evaluation practices within the discipline of Psychology and the associated Medical and Health Sciences Research Evaluation Committee. Given the high stakes nature of unrated outcomes and that the evaluation process breaches public administration principles by being not appealable nor appropriately transparent, the paper concludes with recommendations for the strengthening ERA policy and procedures to enhance trust in future ERA processes.

Theories of perceptual decision making have been dominated by the idea that evidence accumulates in favor of different alternatives until some fixed threshold amount is reached, which triggers a decision. Recent theories have suggested that these thresholds may not be fixed during each decision but change as time passes. These collapsing thresholds can improve performance in particular decision environments, but reviews of data from typical decision-making paradigms have failed to support collapsing thresholds. We designed three experiments to test collapsing threshold assumptions in decision environments specifically tailored to make them optimal. An emphasis on decision speed encouraged the adoption of collapsing thresholds--most strongly through the use of response deadlines but also through instruction to a lesser extent--but setting an explicit goal of reward rate optimality through both instructions and task design did not. Our results suggest that collapsing thresholds models of decision-making are inconsistent with human behaviour even in some situations where there are normative motivations for these models.

How does the process of information transmission affect the cultural or linguistic products that emerge? This question is often studied experimentally and computationally via iterated learning, a procedure in which participants learn from previous participants in a chain. Iterated learning is a powerful tool because, when all participants share the same priors, the stationary distributions of the iterated learning chains reveal those priors. In many situations, however, it is unreasonable to assume that all participants share the same prior beliefs. We present four simulation studies and one experiment demonstrating that when the population of learners is heterogeneous, the behavior of an iterated learning chain can be unpredictable and is often systematically distorted by the learners with the most extreme biases. This results in group-level outcomes that reflect neither the behavior of any individuals within the population nor the overall population average. We discuss implications for the use of iterated learning as a methodological tool as well as for the processes that might have shaped cultural and linguistic evolution in the real world.

Understanding individual differences in cognitive performance is an important part of understanding how variations in underlying cognitive processes can result in variations in task performance. However, the exploration of individual differences in the components of the decision process--such as cognitive processing speed, response caution, and motor execution speed--in previous research has been limited. Here, we assess the heritability of the components of the decision process, with heritability having been a common aspect of individual differences research within other areas of cognition. Importantly, a limitation of previous work on cognitive heritability is the underlying assumption that variability in response times solely reflects variability in the speed of cognitive processing. This assumption has been problematic in other domains, due to the confounding effects of caution and motor execution speed on observed response times. We extend a cognitive model of decision-making to account for relatedness structure in a twin study paradigm. This approach can separately quantify different contributions to the heritability of response time. Using data from the Human Connectome Project, we find strong evidence for the heritability of response caution, and more ambiguous evidence for the heritability of cognitive processing speed and motor execution speed. Our study suggests that the assumption made in previous studies--that the heritability of cognitive ability is based on cognitive processing speed--may be incorrect. More generally, our methodology provides a useful avenue for future research in complex data that aims to analyze cognitive traits across different sources of related data, whether the relation is between people, tasks, experimental phases, or methods of measurement.

Model comparison is the cornerstone of theoretical progress in psychological research. Common practice overwhelmingly relies on tools that evaluate competing models by balancing in-sample descriptive adequacy against model flexibility, with modern approaches advocating the use of marginal likelihood for hierarchical cognitive models. Cross-validation is another popular approach but its implementation remains out of reach for cognitive models evaluated in a Bayesian hierarchical framework, with the major hurdle being its prohibitive computational cost. To address this issue, we develop novel algorithms that make variational Bayes (VB) inference for hierarchical models feasible and computationally efficient for complex cognitive models of substantive theoretical interest. It is well known that VB produces good estimates of the first moments of the parameters, which gives good predictive densities estimates. We thus develop a novel VB algorithm with Bayesian prediction as a tool to perform model comparison by cross-validation, which we refer to as CVVB. In particular, CVVB can be used as a model screening device that quickly identifies bad models. We demonstrate the utility of CVVB by revisiting a classic question in decision making research: what latent components of processing drive the ubiquitous speed-accuracy tradeoff? We demonstrate that CVVB strongly agrees with model comparison via marginal likelihood, yet achieves the outcome in much less time. Our approach brings cross-validation within reach of theoretically important psychological models, making it feasible to compare much larger families of hierarchically specified cognitive models than has previously been possible. To enhance the applicability of the algorithm, we provide Matlab code together with a user manual so users can easily implement VB and/or CVVB for the models considered in this article and their variants. Progress in psychological science can be made by choosing between competing theories: Does sleep deprivation cause attentional lapses? Does alcohol impair the speed of information processing or reduce cautiousness, or both? When these theories are quantitative models that can be estimated from observed data, the problem is known as statistical model selection. Cross-validation is a desirable method for model selection as it directly addresses the question that scientists are often interested in: how well will this model predict new data? However, cross-validation is not used very widely in psychological science due to its computational cost. Modern psychological theories often include random effects, which are necessary to account for individual differences, but these can mean that Bayesian evaluation of the model can take days of computer time. This makes repeated evaluation for cross-validation impractical. Another practical limitation arises when comparing a very large number of competing hypotheses, and this can amplify issues of researcher bias: researchers are forced to focus on a limited subset of models, which they identify by idiosyncratic means. We propose crossvalidation with variational Bayes (CVVB) to tackle these issues. CVVB allows researchers to address larger sets of model-based hypotheses, even using modern multilevel theories. As a demonstration, we use CVVB to investigate an important question about strategies for trading speed versus caution in simple decision-making. [ABSTRACT FROM AUTHOR]

For decisions between many alternatives, the benchmark result is Hick's Law: that response time increases log-linearly with the number of choice alternatives. Even when Hick's Law is observed for response times, divergent results have been observed for error rates--sometimes error rates increase with the number of choice alternatives, and sometimes they are constant. We provide evidence from two experiments that error rates are mostly independent of the number of choice alternatives, unless context effects induce participants to trade speed for accuracy across conditions. Error rate data have previously been used to discriminate between competing theoretical accounts of Hick's Law, and our results question the validity of those conclusions. We show that a previously dismissed optimal observer model might provide a parsimonious account of both response time and error rate data. The model suggests that people approximate Bayesian inference in multi-alternative choice, except for some perceptual limitations. (Contains 7 figures and 1 note.)

In most of the long history of the study of absolute identification--since Miller's (1956) seminal article--a severe limit on performance has been observed, and this limit has resisted improvement even by extensive practice. In a startling result, Rouder, Morey, Cowan, and Pfaltz (2004) found substantially improved performance with practice in the absolute identification of line lengths, albeit for only 3 participants and in a somewhat atypical paradigm. We investigated the limits of this effect and found that it also occurs in more typical paradigms, is not limited to a few virtuoso participants or due to relative judgment strategies, and generalizes to some (e.g., line inclination and tone frequency) but not other (e.g., tone loudness) dimensions. We also observed, apart from differences between dimensions, 2 unusual aspects of improvement with practice: (a) a positive correlation between initial performance and the effect of practice and (b) a large reduction in a characteristic trial-to-trial decision bias with practice. (Contains 12 figures, 4 tables and 2 footnotes.)

When required to predict sequential events, such as random coin tosses or basketball free throws, people reliably use inappropriate strategies, such as inferring temporal structure when none is present. We investigate the ability of observers to predict sequential events in dynamically changing environments, where there is an opportunity to detect true temporal structure. In two experiments we demonstrate that participants often make correct statistical decisions when asked to infer the hidden state of the data generating process. However, when asked to make predictions about future outcomes, accuracy decreased even though normatively correct responses in the two tasks were identical. A particle filter model accounts for all data, describing performance in terms of a plausible psychological process. By varying the number of particles, and the prior belief about the probability of a change occurring in the data generating process, we were able to model most of the observed individual differences. (Contains 10 figures.)

We propose a linear ballistic accumulator (LBA) model of decision making and reaction time. The LBA is simpler than other models of choice response time, with independent accumulators that race towards a common response threshold. Activity in the accumulators increases in a linear and deterministic manner. The simplicity of the model allows complete analytic solutions for choices between any number of alternatives. These solutions (and freely-available computer code) make the model easy to apply to both binary and multiple choice situations. Using data from five previously published experiments, we demonstrate that the LBA model successfully accommodates empirical phenomena from binary and multiple choice tasks that have proven difficult for other theoretical accounts. Our results are encouraging in a field beset by the tradeoff between complexity and completeness. (Contains 10 figures and 2 tables.)

Recent theoretical developments in the field of absolute identification have stressed differences between relative and absolute processes, that is, whether stimulus magnitudes are judged relative to a shorter term context provided by recently presented stimuli or a longer term context provided by the entire set of stimuli. The authors developed a model (SAMBA: selective attention, mapping, and ballistic accumulation) that integrates shorter and longer term memory processes and accounts for both the choices made and the associated response time distributions, including sequential effects in each. The model's predictions arise as a consequence of its architecture and require estimation of only a few parameters with values that are consistent across numerous data sets. The authors show that SAMBA provides a quantitative account of benchmark choice phenomena in classical absolute identification experiments and in contemporary data involving both choice and response time. (Contains 19 figures, 3 tables and 11 footnotes.)