Reliable Idiographic Parameters From Noisy Behavioral Data: The Case ofIndividual Differences in a Reinforcement Learning Task

Behavioral data, though has been an influential index oncognitive processes, is under scrutiny for having poorreliability as a result of noise or lacking replications ofreliable effects. Here, we argue that cognitive modeling canbe used to enhance the test-retest reliability of the behavioralmeasures by recovering individual-level parameters frombehavioral data. We tested this empirically with theProbabilistic Stimulus Selection (PSS) task, which is used tomeasure a participant’s sensitivity to positive or negativereinforcement. An analysis of 400,000 simulations from anAdaptive Control of Thought - Rational (ACT-R) model ofthis task showed that the poor reliability of the task is due tothe instability of the end-estimates: because of the way thetask works, the same participants might sometimes end uphaving apparently opposite scores. To recover the underlyinginterpretable parameters and enhance reliability, we used aBayesian Maximum A Posteriori (MAP) procedure. We wereable to obtain reliable parameters across sessions (IntraclassCorrelation Coefficient ~ 0.5), and showed that this approachcan further be used to provide superior measures in terms ofreliability, and bring greater insights into individualdifferences.