Search

Multiples studies have investigated the role of familiarity, emotion and novelty in the detection and recognition of human faces (see Natu & O'Toole, 2011 for a review). Gobbini & Haxby (2007) have suggested a model of familiar face recognition that describes how prior knowledge of a person's identity modulates the visual systems involved in processing personally familiar faces. Building on this proposed model, the present study sought to investigate the neural representation of personally familiar (friends) and unfamiliar (matched for age and gender strangers) faces. Thirty-three participants underwent functional MRI while performing an oddball detection task in which they were presented with faces of friends and strangers. Univariate GLM and MVPA classification searchlight results agreed with previous findings of greater activity for the processing of familiar faces within some core visual modules and emotion-related regions (e.g. amygdala and insular cortex) suggested by the model. In addition, MVPA exclusively detected differential involvement of regions of the prefrontal cortex and the anterior temporal lobe providing further evidence of distributed representations of person knowledge. Furthermore, MVPA results for identity classification identified a cluster in the right middle frontal gyrus which was sensitive to identity (across friends and strangers combined). This suggests that familiarity differentially modulates brain regions implicated previously in carrying biographic knowledge. The same analysis also identified right inferior frontal gyrus which was also not found to be modulated by familiarity. Future directions are the investigation of functional connectivity between identified regions and subject-specific traits associated with individual stimuli.

We explored the neural correlates of learning about people when the affective value of both facial appearance and behavioral information is manipulated. Participants were presented with faces that were either rated as high or low on trustworthiness. Subsequently, we paired these faces with positive, negative, or no behavioral information. Prior to forming face-behavior associations, a cluster in the right amygdala responded more strongly to untrustworthy than to trustworthy faces. During learning, a cluster in the dorsomedial prefrontal cortex (dmPFC) responded more strongly to faces paired with behaviors than faces not paired with behaviors. We also observed that the activity in the dmPFC was correlated with behavioral learning performance assessed after scanning. Interestingly, individual differences in the initial amygdala response to face trustworthiness prior to learning modulated the relationship between dmPFC activity and learning. This finding suggests that the activity of the amygdala can affect the interaction between dmPFC activity and learning.

We present an fMRI investigation of object representation in the human ventral vision pathway highlighting three aspects of similarity analysis that make it especially useful for illuminating the representational content underlying neural activation patterns. First, similarity structures allow for an abstract depiction of representational content in a given brain region. This is demonstrated using hierarchical clustering and multidimensional scaling (MDS) of the dissimilarity matrices defined by our stimulus categories—female and male human faces, dog faces, monkey faces, chairs, shoes, and houses. For example, in ventral temporal (VT) cortex the similarity space was neatly divided into face and non-face regions. Within the face region of the MDS space, male and female human faces were closest to each other, and dog faces were closer to human faces than monkey faces. Within the non-face region of the abstract space, the smaller objects—shoes and chairs—were closer to each other than they were to houses. Second, similarity structures are independent of the data source. Dissimilarities among stimulus categories can be derived from behavioral measures, from stimulus models, or from neural activity patterns in different brain regions and different subjects. The similarity structures from these diverse sources all have the same dimensionality. This source independence allowed for the direct comparison of similarity structures across subjects (n = 16) and across three brain regions representing early, middle, and late stages of the object vision pathway. Finally, similarity structures can change shape in well-ordered ways as the source of the dissimilarities changes—helping to illuminate how representational content is transformed along a neural pathway. By comparing similarity spaces from three regions along the ventral visual pathway, we demonstrate how the similarity structure transforms from an organization based on low-level visual features—as reflected by patterns in early visual cortex—to a more categorical representation in late object vision cortex with intermediate organization at the middle stage.

Face perception plays a central role in social communication and is, arguably, one of the most sophisticated visual perceptual skills in humans. Consequently, face perception has been the subject of intensive investigation and theorizing in both visual and social neuroscience. The organization of neural systems for face perception has stimulated intense debate. Much of this debate has focused on models that posit the existence of a module that is specialized for face perception versus models that propose that face perception is mediated by distributed processing. In our work, we have proposed that face perception is mediated by distributed systems, both in terms of the involvement of multiple brain areas and in terms of locally distributed population codes within these areas. Specifically, we proposed a model for the distributed neural system for face perception that has a Core System of visual extrastriate areas for visual analysis of faces and an Extended System that consists of additional neural systems that work in concert with the Core System to extract various types of information from faces. We also have shown that in visual extrastriate cortices, information that distinguishes faces from other categories of animate and inanimate objects is not restricted to regions that respond maximally to faces, i.e. the fusiform and occipital face areas.

In a functional magnetic resonance imaging experiment, we explored whether affective person knowledge based on memories formed from minimal information is spontaneously retrieved in face perception. In the first stage of the experiment, participants were presented with 120 unfamiliar faces. Each face was presented with a description of one of four types of behaviors: aggressive, disgusting, neutral, and nice. In the second stage, participants were scanned while engaged in a one-back recognition task in which they saw the faces that were associated with behaviors and 30 novel faces. Although this task is a simple perceptual task that neither demands person evaluation nor retrieval of person knowledge, neural responses to faces differed as a function of the behaviors. Faces associated with behaviors evoked stronger activity than did novel faces in regions implicated in social cognition-anterior paracingulate cortex and superior temporal sulcus. Explicit memory for the behaviors enhanced the neural response in these regions. Faces associated with disgusting behaviors evoked stronger activity in left anterior insula than did faces associated with aggressive behaviors. This effect was equally strong for faces associated with explicitly recalled behaviors and faces associated with non-recalled behaviors. The findings suggest that affective person knowledge acquired from minimal information is spontaneously retrieved in face perception, engaging neural systems for analysis of social cognition and emotions.

Immediate access to information about people that we encounter is an essential requirement for effective social interactions. In this manuscript we briefly review our work and work of others on familiar face recognition and propose a modified version of our model of neural systems for face perception with a special emphasis on processes associated with recognition of familiar faces. We argue that visual appearance is only one component of successful recognition of familiar individuals. Other fundamental aspects include the retrieval of "person knowledge" - the representation of the personal traits, intentions, and outlook of someone we know - and the emotional response we experience when seeing a familiar individual. Specifically, we hypothesize that the "theory of mind" areas, that have been implicated in social and cognitive functions other than face perception, play an essential role in the spontaneous activation of person knowledge associated with the recognition of familiar individuals. The amygdala and the insula, structures that are involved in the representation of emotion, also are part of the distributed network of areas that are modulated by familiarity, reflecting the role of emotion in face recognition.

Recognizing personally familiar faces is the result of a spatially distributed process that involves visual perceptual areas and areas that play an essential role in other cognitive and social functions, such as the anterior paracingulate cortex, the precuneus and the amygdala [M.I. Gobbini, E. Leibenluft, N. Santiago, J.V. Haxby, Social and emotional attachment in the neural representation of faces, Neuroimage 22 (2004) 1628–1635; M.I. Gobbini, J.V. Haxby, Neural systems for recognition of familiar faces, Neuropsychologia, in press; E. Leibenluft, M.I. Gobbini, T. Harrison, J.V. Haxby, Mothers’ neural activation in response to pictures of their, and other, children, Biol. Psychiatry 56 (2004) 225–232]. In order to isolate the role of visual familiarity in face recognition, we used fMRI to measure the response to faces characterized by experimentally induced visual familiarity that carried no biographical information or emotional content. The fMRI results showed a stronger response in the precuneus to the visually familiar faces consistent with studies that implicate this region in the retrieval of information from long term memory and imagery. Moreover, this finding supports the hypothesis of a key role of the precuneus in the acquisition of familiarity with faces [H. Kosaka, M. Omori, T. Iidaka, T. Murata, T. Shimoyama, T. Okada, N. Sadato, Y. Yonekura, Y. Wada, Neural substrates participating in acquisition of facial familiarity: an fMRI study, Neuroimage 20 (2003) 1734–1742]. By contrast, the visually familiar faces evoked a weaker response in the fusiform gyrus, which may reflect the development of a sparser encoding, or a reduced attentional load when processing stimuli that are familiar. The visually familiar faces evoked also a weaker response in the amygdala, supporting the proposed role of this structure in mediating the guarded attitude when meeting someone new.

Numerous imaging studies have reported a consistent activation in the fusiform gyrus (in a portion localized between the occipital and temporal lobes) when contrasting the response to faces to other categories of objects. This region has been called “Fusiform Face Area”, FFA (Kanwisher et al. 1997). Since this pivotal finding, further work has investigated the role of the fusiform gyrus in the perception of object categories other than faces. With this purpose, many experiments have contrasted the response to different types of objects and other areas have been identified by their maximal response to specific categories. For example, an area in the parahippocampal gyrus responds maximally to interior spaces, buildings or landscapes, the so-called parahippocampal place area (PPA) (Epstein and Kanwisher 1998), and an area localized in the inferior temporal gyrus responds maximally to body parts other than faces, the so-called extrastriate body part (EBA) (Downing et al. 2001). This preferential activation in specific regions in response to specific categories has contributed to a controversy about how the brain represents and differentiates among categories of objects. One line of thought proposes a modular theory according to which the brain is organized in ‘modules’ each dedicated to the representation of a specific type of information (Kanwisher J Neurosci 1997). Another line of thought proposes, in contrast to previous hypothesis, that the representation of different categories of objects in the brain is shaped by experience. According to this view, the “fusiform face area” is specialized for the representation of ‘visual expertise’ – which refers to the capacity for finer-grained discrimination among individuals within category – not just for face perception (Gauthier et al Nat Neurosci 2000). A third line of thought argues that the organization of object-responsive cortex can be explained by a coarse retinotopy that biases the representation of object categories according to how consistently those categories are perceived in central or peripheral vision (Malach et al. 2002). We have proposed that the representations of faces and objects in ventral temporal cortex are distributed and overlapping (Haxby et al. 2001). We showed that the distinctiveness of the response to a given category is not due simply to the regions that respond maximally to that category by demonstrating that the category been viewed can still be identified on the basis of the pattern of response when those regions were excluded from the analysis. Thus, representations of faces and objects are distributed locally within ventral temporal cortex. Faces and other categories of objects also evoke neural activity in cortical areas outside the ventral object vision pathway. These responses indicate the spontaneous activation of other types of information that are associated with objects, such as the emotion associated with a facial expression, the direction of attention indicated by eye gaze, or the motion associated with tool use. Thus, the representation of a visually-presented object appears to be distributed not only locally but also across multiple cortical areas. Finally, the temporal course for the development of face and object representations can be studied with electro- and magnetoencephalography (EEG and MEG). Although early neural responses (< 200 ms) can differentiate among some categories, only later responses show the effect of familiarity or reflect finer discriminations within a category (Bentin et al. 1998; Sugase et al. 1999; Eimer 2000). In this chapter, we review the functional neuroanatomy of face and object recognition with emphasis on explicating how representation are distributed in space and time.

We investigated whether the topographically organized, category-related patterns of neural response in the ventral visual pathway are a representation of sensory images or a more abstract representation of object form that is not dependent on sensory modality. We used functional MRI to measure patterns of response evoked during visual and tactile recognition of faces and manmade objects in sighted subjects and during tactile recognition in blind subjects. Results showed that visual and tactile recognition evoked category-related patterns of response in a ventral extrastriate visual area in the inferior temporal gyrus that were correlated across modality for manmade objects. Blind subjects also demonstrated category-related patterns of response in this "visual" area, and in more ventral cortical regions in the fusiform gyrus, indicating that these patterns are not due to visual imagery and, furthermore, that visual experience is not necessary for category-related representations to develop in these cortices. These results demonstrate that the representation of objects in the ventral visual pathway is not simply a representation of visual images but, rather, is a representation of more abstract features of object form.

Common or folk knowledge about animals is dominated by three dimensions: (1) level of cognitive complexity or “animacy;” (2) dangerousness or “predacity;” and (3) size. We investigated the neural basis of the perceived dangerousness or aggressiveness of animals, which we refer to more generally as “perception of threat.” Using functional magnetic resonance imaging (fMRI), we analyzed neural activity evoked by viewing images of animal categories that spanned the dissociable semantic dimensions of threat and taxonomic class. The results reveal a distributed network for perception of threat extending along the right superior temporal sulcus. We compared neural representational spaces with target representational spaces based on behavioral judgments and a computational model of early vision and found a processing pathway in which perceived threat emerges as a dominant dimension: whereas visual features predominate in early visual cortex and taxonomy in lateral occipital and ventral temporal cortices, these dimensions fall away progressively from posterior to anterior temporal cortices, leaving threat as the dominant explanatory variable. Our results suggest that the perception of threat in the human brain is associated with neural structures that underlie perception and cognition of social actions and intentions, suggesting a broader role for these regions than has been thought previously, one that includes the perception of potential threat from agents independent of their biological class.SIGNIFICANCE STATEMENTFor centuries, philosophers have wondered how the human mind organizes the world into meaningful categories and concepts. Today this question is at the core of cognitive science, but our focus has shifted to understanding how knowledge manifests in dynamic activity of neural systems in the human brain. This study advances the young field of empirical neuroepistemology by characterizing the neural systems engaged by an important dimension in our cognitive representation of the animal kingdom ontological subdomain: how the brain represents the perceived threat, dangerousness, or “predacity” of animals. Our findings reveal how activity for domain-specific knowledge of animals overlaps the social perception networks of the brain, suggesting domain-general mechanisms underlying the representation of conspecifics and other animals.

Measurements of response time (RT) have long been used to infer neural processes underlying various cognitive functions such as working memory, attention, and decision making. However, it is currently unknown if RT is also informative about various stages of value-based choice, particularly how reward values are constructed. To investigate these questions, we analyzed the pattern of RT during a set of multi-dimensional learning and decision-making tasks that can prompt subjects to adopt different learning strategies. In our experiments, subjects could use reward feedback to directly learn reward values associated with possible choice options (object-based learning). Alternatively, they could learn reward values of options’ features (e.g. color, shape) and combine these values to estimate reward values for individual options (feature-based learning). We found that RT was slower when the difference between subjects’ estimates of reward probabilities for the two alternative objects on a given trial was smaller. Moreover, RT was overall faster when the preceding trial was rewarded or when the previously selected object was present. These effects, however, were mediated by an interaction between these factors such that subjects were faster when the previously selected object was present rather than absent but only after unrewarded trials. Finally, RT reflected the learning strategy (i.e. object-based or feature-based approach) adopted by the subject on a trial-by-trial basis, indicating an overall faster construction of reward value and/or value comparison during object-based learning. Altogether, these results demonstrate that the pattern of RT can be informative about how reward values are learned and constructed during complex value-based learning and decision making.

The face processing network consists of a core system, involved in the processing of visual appearance of faces, and an extended system, involved in the retrieval of person-knowledge and other nonvisual information associated with faces. Behavioral evidence shows that personally familiar faces are processed in a more robust and efficient way than unfamiliar faces. Personally familiar faces evoke a stronger activation in the extended system as compared to faces of strangers. This increased activation of the extended system may reflect aspects of familiarity that are shared by all familiar identities or enhanced representations that distinguish individual familiar identities. Here we distinguished between these two aspects of familiarity, and investigated the organization of the core and extended systems derived from similarities in neural representations. We applied multivariate pattern classification and network-level representational similarity analysis to fMRI data collected while 33 subjects viewed personally familiar faces and unfamiliar faces. Both face identity and identity-independent familiarity information could be decoded from patterns of response to faces in extended system areas. Analysis of the representational similarities in the face processing system revealed a clear distinction between core and extended systems, and a subdivision of the core system into ventral, dorsal and anterior components. This study provides evidence that patterns of activity in the extended system carry information about both individual identities and personal familiarity, while clarifying and extending the organization of the core system for face perception.

Eye gaze is a powerful cue that indicates where another person’s attention is directed in the environment. Seeing another person’s eye gaze shift spontaneously and reflexively elicits a shift of one’s own attention to the same region in space. Here, we investigated whether reallocation of attention in the direction of eye gaze is modulated by personal familiarity with faces. On the one hand, the eye gaze of a close friend should be more effective in redirecting our attention as compared to the eye gaze of a stranger. On the other hand, the social relevance of a familiar face might itself hold attention and, thereby, slow lateral shifts of attention. To distinguish between these possibilities, we measured the efficacy of the eye gaze of personally familiar and unfamiliar faces as directional attention cues using adapted versions of the Posner paradigm with saccadic and manual responses. We found that attention shifts were slower when elicited by a perceived change in the eye gaze of a familiar individual as compared to attention shifts elicited by unfamiliar faces at short latencies (100 ms). We also measured simple detection of change in direction of gaze in personally familiar and unfamiliar faces to test whether slower attention shifts were due to slower detection. Participants detected changes in eye gaze faster for familiar faces than for unfamiliar faces. Our results suggest that personally familiar faces briefly hold attention due to their social relevance, thereby slowing shifts of attention, even though the direction of eye movements are detected faster in familiar faces.

Chang and Tsao recently reported that the monkey face patch system encodes facial identity in a space of facial features as opposed to exemplars. Here, we discuss how such coding might contribute to face recognition, emphasizing the critical role of learning and interactions with other brain areas for optimizing the recognition of familiar faces.

Recognition of personally familiar faces is remarkably efficient, effortless and robust. We asked if feature-based face processing drives facilitated detection of familiar faces by testing the effect of face inversion on a visual search task for familiar and unfamiliar faces. Because face inversion disrupts configural and holistic face processing, we hypothesized that inversion would diminish the familiarity advantage to the extent that it is mediated by such processes. Subjects detected personally familiar and stranger target faces in arrays of two, four, or six face images. Subjects showed significant facilitation of personally familiar face detection for both upright and inverted faces, in terms of reaction times for target present trials and both reaction times and search rate for target absent trials. The effect of familiarity even on target absent trials suggests that familiarity facilitates rejection of unfamiliar distractors as well as detection of familiar targets. The preserved familiarity effect for inverted faces suggests that facilitation of face detection afforded by familiarity reflects mostly feature-based processes.