Background Face recognition is a key skill of social cognition. Congenital prosopagnosia (CP), i.e. the inability to recognize faces in the absence of any other underlying pathology, is a model system for the understanding of face processing and affects about 2% of the population. Controversy exists whether activation of the fusiform face area (FFA), a core region within the face processing network, is preserved in CP. Here we scrutinized the global activity of the FFA in a working memory paradigm and specifically probed neural representations with Representational Similarity Analysis (RSA). Methods A total of 13 CP and 12 healthy control participants were included. Diagnosis was based on a semi-structured interview (Kennerknecht et al., 2008). CP subjects showed poorer performance in the Cambridge Face Memory Test (Duchaine et al., 2006), an independent diagnostic tool for CP. A modified Sternberg working memory paradigm presented faces with three levels of difficulty: low load condition (1 face, 3 scrambled pictures), medium load condition (2 faces, 2 scrambled pictures), and high load memory condition (4 faces). After a maintenance phase, a probe picture was shown. Working memory processing in the right fusiform face area (FFA), individually determined by a functional localizer, was analyzed with a modified RSA approach (Kriegeskorte et al., 2008). RSA assesses the correlation of voxel-wise activity across trials, represented in individual correlation matrices. Mean correlation coefficients were computed for each condition across groups and a group comparison of individual correlation matrices was performed with a random walk mapping and Donsker’s theorem (Donsker, 1951). Results CP participants showed significantly lower working memory accuracy, most notably in the high load condition. For FFA activation in the encoding phase, an ANOVA yielded a significant effect of group (F = 5.82, p = 0.024), and condition (low, medium, high memory load) (F = 38.02, p 0.001 ). When comparing mean correlation coefficients of the RSA in an ANOVA, we found an effect of condition ( p 0.001 ) and a trend for the factor group (p = 0.058) with CP participants showing lower mean correlation of voxel activity as compared to control participants. Individual matrix comparison based on Donsker’s theorem allowed to confirm the group difference in all memory load conditions. Conclusion Our study on working memory processing in CP revealed both a global reduction of FFA activity and altered neural representations as assessed by representational similarity analysis. We suggest that the latter effect contributes to deficits in face recognition.