Search

Abstract Many species learn temporal regularities in their visual environment, demonstrating visual statistical learning. In this study, we explored the sensitivity of macaque inferior temporal (IT) cortical neurons to transition probabilities of sequentially presented visual images, presented at different locations in the visual field. We exposed monkeys to sequences of two images, where the first image was presented either foveally or peripherally, and the second image was consistently presented foveally. Following several weeks of exposure, we recorded IT responses to assess differences between the exposed (Fixed) and new, Deviant sequences, where the identity of the first image in a sequence differed from the exposure phase. While enhanced responses to Deviant sequences were observed when both images of a pair were foveally presented during exposure, no such deviant responses were present when the first image was presented peripherally. This finding challenges the notion that mere exposure to image sequences always leads to deviant responses in macaque IT. The results highlight the complexity of the mechanisms underlying statistical learning in primates, particularly in the context of peripheral image presentations, emphasizing the need for further investigation into the origins of these responses in the IT cortex.

Summary: The temporal cortex represents social stimuli, including bodies. We examine and compare the contributions of dynamic and static features to the single-unit responses to moving monkey bodies in and between a patch in the anterior dorsal bank of the superior temporal sulcus (dorsal patch [DP]) and patches in the anterior inferotemporal cortex (ventral patch [VP]), using fMRI guidance in macaques. The response to dynamics varies within both regions, being higher in DP. The dynamic body selectivity of VP neurons correlates with static features derived from convolutional neural networks and motion. DP neurons’ dynamic body selectivity is not predicted by static features but is dominated by motion. Whereas these data support the dominance of motion in the newly proposed “dynamic social perception” stream, they challenge the traditional view that distinguishes DP and VP processing in terms of motion versus static features, underscoring the role of inferotemporal neurons in representing body dynamics.

Humans and other animals can learn and exploit repeating patterns that occur within their environments. These learned patterns can be used to form expectations about future sensory events. Several influential predictive coding models have been proposed to explain how learned expectations influence the activity of stimulus-selective neurons in the visual system. These models specify reductions in neural response measures when expectations are fulfilled (termed expectation suppression) and increases following surprising sensory events. However, there is currently scant evidence for expectation suppression in the visual system when confounding factors are taken into account. Effects of surprise have been observed in blood oxygen level dependent (BOLD) signals, but not when using electrophysiological measures. To provide a strong test for expectation suppression and surprise effects we performed a predictive cueing experiment while recording electroencephalographic (EEG) data. Participants (n=48) learned cue-face associations during a training session and were then exposed to these cue-face pairs in a subsequent experiment. Using univariate analyses of face-evoked event-related potentials (ERPs) we did not observe any differences across expected (90% probability), neutral (50%) and surprising (10%) face conditions. Across these comparisons, Bayes factors consistently favoured the null hypothesis throughout the time-course of the stimulus-evoked response. When using multivariate pattern analysis we did not observe above-chance classification of expected and surprising face-evoked ERPs. By contrast, we found robust within- and across-trial stimulus repetition effects. Our findings do not support predictive coding-based accounts that specify reduced prediction error signalling when perceptual expectations are fulfilled. They instead highlight the utility of other types of predictive processing models that describe expectation-related phenomena in the visual system without recourse to prediction error signalling.

Neuroimaging and single cell recordings have demonstrated the presence of STS body category-selective regions (body patches) containing neurons responding to presentation of static bodies and body parts. To date, it remains unclear if these body patches and additional STS regions respond during observation of different categories of dynamic actions and to what extent categorization learning influences representations of observed actions in the STS. In the present study, we trained monkeys to discriminate videos depicting three different actions categories (grasping, touching and reaching) with a forced-choice action categorization task. Before and after categorization training, we performed fMRI recordings while monkeys passively observed the same action videos. At the behavioral level, after categorization training, monkeys generalized to untrained action exemplars, in particular for grasping actions. Before training, uni- and/or multivariate fMRI analyses suggest a broad representation of dynamic action categories in particular in posterior and middle STS. Univariate analysis further suggested action category specific training effects in middle and anterior body patches, face patch ML and posterior STS region MT and FST. Overall, our fMRI experiments suggest a widespread representation of observed dynamic bodily actions in the STS that can be modulated by visual learning, supporting its proposed role in action recognition.

In primates, faces and bodies activate distinct regions in the inferior temporal (IT) cortex and are typically studied separately. Yet, primates interact with whole agents and not with random concatenations of faces and bodies. Despite its social importance, it is still poorly understood how faces and bodies interact in IT. Here, we addressed this gap by measuring fMRI activations to whole agents and to unnatural face-body configurations in which the head was mislocated with respect to the body, and examined how these relate to the sum of the activations to their corresponding faces and bodies. First, we mapped patches in the IT of awake macaques that were activated more by images of whole monkeys compared to objects and found that these mostly overlapped with body and face patches. In a second fMRI experiment, we obtained no evidence for superadditive responses in these “monkey patches”, with the activation to the monkeys being less or equal to the summed face-body activations. However, monkey patches in the anterior IT were activated more by natural compared to unnatural configurations. The stronger activations to natural configurations could not be explained by the summed face-body activations. These univariate results were supported by regression analyses in which we modeled the activations to both configurations as a weighted linear combination of the activations to the faces and bodies, showing higher regression coefficients for the natural compared to the unnatural configurations. Deeper layers of trained convolutional neural networks also contained units that responded more to natural compared to unnatural monkey configurations. Unlike the monkey fMRI patches, these units showed substantial superadditive responses to the natural configurations. Our monkey fMRI data suggest configuration-sensitive face-body interactions in anterior IT, adding to the evidence for an integrated face-body processing in the primate ventral visual stream, and open the way for mechanistic studies using single unit recordings in these patches.

In order to examine underlying mechanisms of repetition suppression, Stam et al conducted an fMRI study in which participants were presented with face and house stimuli followed by an immediate and 48 h-delayed memory task. Their data demonstrated that adapter-related and familiarity account modelling are more likely to be more accurate for visual memory than previous predictive coding models.

Animals of several species, including primates, learn the statistical regularities of their environment. In particular, they learn the temporal regularities that occur in streams of visual images. Previous human neuroimaging studies reported discrepant effects of such statistical learning, ranging from stronger occipito-temporal activations for sequences in which image order was fixed, compared with sequences of randomly ordered images, to weaker activations for fixed-order sequences compared with sequences that violated the learned order. Several single-unit studies in macaque monkeys reported that after statistical learning of temporal regularities, inferior temporal (IT) neurons show reduced responses to learned fixed-order sequences of visual images compared with random or mispredicted sequences. However, it is unknown how other macaque brain areas respond to such temporal statistical regularities. To address this gap, we exposed rhesus monkeys (Macaca mulatta) to two types of sequences of complex images. The “regular” sequences consisted of a continuous stream of quartets, and within each quartet, the image order was fixed. The quartets themselves were displayed, uninterrupted, in a random order. The same monkeys were exposed to sequences of other images having a pseudorandomized order (“random” sequence). After exposure, both monkeys were scanned with functional MRI (fMRI) using a block design with three conditions: regular sequence, random sequence, and fixation-only blocks. A whole-brain analysis showed a reduced activation in mainly the occipito-temporal cortex for the regular compared to the random sequences. Marked response reductions for the regular sequence were observed in early extrastriate visual cortical areas, including area V2, despite the use of rather complex images of animals. These data suggest that statistical learning signals are already present in early visual areas of monkeys, even for complex visual images. These monkey fMRI data are in line with recent human fMRI studies that showed a reduced activation in early visual areas for predicted compared with mispredicted complex images.

Abstract Many inferior temporal (IT) cortical neurons reduce their response when a stimulus is repeated. Proposed mechanisms underlying this repetition suppression range from “fatigue” to top-down expectations of repetition. Here we examine a prediction from simple fatigue-based models of adaptation: prolonging adapter duration will increase the adapter response, leading to more repetition suppression. To test this, we varied adapter duration from 300 to 3000 ms, keeping the test stimulus duration constant. We observed no effect of adapter duration on repetition suppression when averaging responses across the test presentation. This was not because of a ceiling effect, since repeated presentations of a short adapter increased repetition suppression. Examination of test stimulus responses showed increased repetition suppression with longer adapter durations during the initial response phase, which reversed at a later phase. Across neurons, we found that the degree of repetition suppression covaried with the ratio of the response during the initial transient and later sustained phase of the response during the long adapter presentation, suggesting overlapping mechanisms that underlie adaptation during the adapter and the delayed test. We propose a fatigue-based account in which fatigue increases non-linearly with adapter duration to explain these unexpected findings.

Recent studies suggest that deep Convolutional Neural Network (CNN) models show higher representational similarity, compared to any other existing object recognition models, with macaque inferior temporal (IT) cortical responses, human ventral stream fMRI activations and human object recognition. These studies employed natural images of objects. A long research tradition employed abstract shapes to probe the selectivity of IT neurons. If CNN models provide a realistic model of IT responses, then they should capture the IT selectivity for such shapes. Here, we compare the activations of CNN units to a stimulus set of 2D regular and irregular shapes with the response selectivity of macaque IT neurons and with human similarity judgements. The shape set consisted of regular shapes that differed in nonaccidental properties, and irregular, asymmetrical shapes with curved or straight boundaries. We found that deep CNNs (Alexnet, VGG-16 and VGG-19) that were trained to classify natural images show response modulations to these shapes that were similar to those of IT neurons. Untrained CNNs with the same architecture than trained CNNs, but with random weights, demonstrated a poorer similarity than CNNs trained in classification. The difference between the trained and untrained CNNs emerged at the deep convolutional layers, where the similarity between the shape-related response modulations of IT neurons and the trained CNNs was high. Unlike IT neurons, human similarity judgements of the same shapes correlated best with the last layers of the trained CNNs. In particular, these deepest layers showed an enhanced sensitivity for straight versus curved irregular shapes, similar to that shown in human shape judgments. In conclusion, the representations of abstract shape similarity are highly comparable between macaque IT neurons and deep convolutional layers of CNNs that were trained to classify natural images, while human shape similarity judgments correlate better with the deepest layers.

According to a recent study, semantic similarity between concrete entities correlates with the similarity of activity patterns in left middle IPS during category naming. We examined the replicability of this effect under passive viewing conditions, the potential role of visuoperceptual similarity, where the effect is situated compared to regions that have been previously implicated in visuospatial attention, and how it compares to effects of object identity and location. Forty-six subjects participated. Subjects passively viewed pictures from two categories, musical instruments and vehicles. Semantic similarity between entities was estimated based on a concept-feature matrix obtained in more than 1,000 subjects. Visuoperceptual similarity was modeled based on the HMAX model, the AlexNet deep convolutional learning model, and thirdly, based on subjective visuoperceptual similarity ratings. Among the IPS regions examined, only left middle IPS showed a semantic similarity effect. The effect was significant in hIP1, hIP2, and hIP3. Visuoperceptual similarity did not correlate with similarity of activity patterns in left middle IPS. The semantic similarity effect in left middle IPS was significantly stronger than in the right middle IPS and also stronger than in the left or right posterior IPS. The semantic similarity effect was similar to that seen in the angular gyrus. Object identity effects were much more widespread across nearly all parietal areas examined. Location effects were relatively specific for posterior IPS and area 7 bilaterally. To conclude, the current findings replicate the semantic similarity effect in left middle IPS under passive viewing conditions, and demonstrate its anatomical specificity within a cytoarchitectonic reference frame. We propose that the semantic similarity effect in left middle IPS reflects the transient uploading of semantic representations in working memory.

The anterior intraparietal area (AIP) of rhesus monkeys is part of the dorsal visual stream and contains neurons whose visual response properties are commensurate with a role in three-dimensional (3D) shape perception. Neuronal responses in AIP signal the depth structure of disparity-defined 3D shapes, reflect the choices of monkeys while they categorize 3D shapes, and mirror the behavioral variability across different stimulus conditions during 3D-shape categorization. However, direct evidence for a role of AIP in 3D-shape perception has been lacking. We trained rhesus monkeys to categorize disparity-defined 3D shapes and examined AIP's contribution to 3D-shape categorization by microstimulating in clusters of 3D-shape selective AIP neurons during task performance. We find that microstimulation effects on choices (monkey M1) and reaction times (monkey M1 and M2) depend on the 3D-shape preference of the stimulated site. Moreover, electrical stimulation of the same cells, during either the 3D-shape-categorization task or a saccade task, could affect behavior differently. Interestingly, in one monkey we observed a strong correlation between the strength of choice-related AIP activity (choice probabilities) and the influence of microstimulation on 3D-shape-categorization behavior (choices and reaction time). These findings propose AIP as part of the network responsible for 3D-shape perception. The results also show that the anterior intraparietal cortex contains cells with different tuning properties, i.e. 3D-shape- or saccade-related, that can be dynamically read out depending on the requirements of the task at hand.

Stimulus repetition produces a decrease of the response and affects neuronal synchronization of macaque inferior temporal (IT) neurons. Previously we showed that such stimulus-specific adaptation results in a decreased accuracy by which IT neurons encode repeated compared to non-repeated objects. Not only spiking activity, but also local field potentials (LFPs) are affected by repetition. Here we ask how the repetition-induced changes in IT LFPs affect object decoding accuracy. To answer this, we recorded local field potentials using a laminar microelectrode in macaque IT. We presented two familiar stimuli each for 500 ms successively with an inter-stimulus interval of 500 ms. Trials consisted either of a repetition of the same stimulus or of their alternation. Machine learning-based classifier was employed to decode stimulus identity from the LFP power in different frequency bands of each penetration. We found that the object classification accuracy depended strongly on spectral frequency, with frequencies below 30 Hz (alpha and beta) producing greater accuracies than gamma bands. However, the effect of repetition on classification accuracy was stronger at the gamma frequencies, showing a decrease in classification accuracy for repeated stimuli and a tendency for an improved object encoding when the stimulus was preceded by a different stimulus. The present results demonstrate that due to adapting input, stimulus encoding in IT (1) can be more accurate for stimuli that differ from recently preceding ones while being impaired for stimuli that are repeated, and (2) these effects are more pronounced at high spectral frequencies of the LFP.

Previous reports have described that neural activities in midbrain dopamine areas are sensitive to unexpected reward delivery and omission. These activities are correlated with reward prediction error in reinforcement learning models, the difference between predicted reward values and the obtained reward outcome. These findings suggest that the reward prediction error signal in the brain updates reward prediction through stimulus-reward experiences. It remains unknown, however, how sensory processing of reward-predicting stimuli contributes to the computation of reward prediction error. To elucidate this issue, we examined the relation between stimulus discriminability of the reward-predicting stimuli and the reward prediction error signal in the brain using functional magnetic resonance imaging (fMRI). Before main experiments, subjects learned an association between the orientation of a perceptually salient (high-contrast) Gabor patch and a juice reward. The subjects were then presented with lower-contrast Gabor patch stimuli to predict a reward. We calculated the correlation between fMRI signals and reward prediction error in two reinforcement learning models: a model including the modulation of reward prediction by stimulus discriminability and a model excluding this modulation. Results showed that fMRI signals in the midbrain are more highly correlated with reward prediction error in the model that includes stimulus discriminability than in the model that excludes stimulus discriminability. No regions showed higher correlation with the model that excludes stimulus discriminability. Moreover, results show that the difference in correlation between the two models was significant from the first session of the experiment, suggesting that the reward computation in the midbrain was modulated based on stimulus discriminability before learning a new contingency between perceptually ambiguous stimuli and a reward. These results suggest that the human reward system can incorporate the level of the stimulus discriminability flexibly into reward computations by modulating previously acquired reward values for a typical stimulus.