Search

The “Narratives” collection aggregates a variety of functional MRI datasets collected while human subjects listened to naturalistic spoken stories. The current release includes 345 subjects, 891 functional scans, and 27 diverse stories of varying duration totaling ~4.6 hours of unique stimuli (~43,000 words). This data collection is well-suited for naturalistic neuroimaging analysis, and is intended to serve as a benchmark for models of language and narrative comprehension. We provide standardized MRI data accompanied by rich metadata, preprocessed versions of the data ready for immediate use, and the spoken story stimuli with time-stamped phoneme- and word-level transcripts. All code and data are publicly available with full provenance in keeping with current best practices in transparent and reproducible neuroimaging., Measurement(s)functional brain measurementTechnology Type(s)functional magnetic resonance imagingFactor Type(s)story • native languageSample Characteristic - OrganismHomo sapiens Machine-accessible metadata file describing the reported data: 10.6084/m9.figshare.14818587

Participant-specific, functionally-defined brain areas are usually mapped with functional localizers and estimated by making contrasts between responses to single categories of input. Naturalistic stimuli engage multiple brain systems in parallel, provide more ecologically plausible estimates of real-world statistics, and are friendly to special populations. The current study shows that cortical functional topographies in individual participants can be estimated with high fidelity from naturalistic stimuli. Importantly, we demonstrate that robust, individualized estimates can be obtained even when participants watched different movies, were scanned with different parameters/scanners, and were sampled from different institutes across the world. Our results create a foundation for future studies that allow researchers to estimate a broad range of functional topographies based on naturalistic movies and a normative database, making it possible to integrate high-level cognitive functions across datasets from laboratories worldwide.

We use language to organize and communicate our thoughts to others. Natural language, however, is fundamentally contextual, making it unamenable to experimental control. In this talk, I argue that the large language models emerging from the machine learning community are an important step toward a more ecological human neuroscience of language. These self-supervised models learn the rich, context-sensitive structure of language from real-world examples, allowing us to quantify contextual meaning in a way that was previously inaccessible. I provide two examples of how these models can provide novel insights into the neural machinery for human language and communication. First, I show how we can deconstruct the internal circuits of these models to directly access the computations—or “transformations”—that sculpt the unique meaning of words in context. I show that these transformations provide a window onto linguistic computations in the human brain using fMRI data recorded while subjects listened to spoken stories. Critically, we can also disassemble these transformations into functionally-specialized sub-computations and map these sub-computations onto the cortical language network. Second, I show how we can position these models as linguistic intermediaries for translating neural activity from one brain to another. In ongoing work, we use contextual embeddings to capture the linguistic content of brain-to-brain coupling during spontaneous, interactive conversations between dyadic pairs of ECoG patients. We show that linguistic content in the speaker’s brain prior to word articulation (re)emerges in the listener’s brain shortly after articulation—word by word in natural conversations. These findings suggest that large language models key into the same context-sensitive linguistic space that human speakers use to transmit their thoughts to others.

Functional magnetic resonance imaging (fMRI) offers a rich source of data for studying the neural basis of cognition. Here, we describe the Brain Imaging Analysis Kit (BrainIAK), an open-source, free Python package that provides computationally optimized solutions to key problems in advanced fMRI analysis. A variety of techniques are presently included in BrainIAK: intersubject correlation (ISC) and intersubject functional connectivity (ISFC), functional alignment via the shared response model (SRM), full correlation matrix analysis (FCMA), a Bayesian version of representational similarity analysis (BRSA), event segmentation using hidden Markov models, topographic factor analysis (TFA), inverted encoding models (IEMs), an fMRI data simulator that uses noise characteristics from real data (fmrisim), and some emerging methods. These techniques have been optimized to leverage the efficiencies of high-performance compute (HPC) clusters, and the same code can be seamlessly transferred from a laptop to a cluster. For each of the aforementioned techniques, we describe the data analysis problem that the technique is meant to solve and how it solves that problem; we also include an example Jupyter notebook for each technique and an annotated bibliography of papers that have used and/or described that technique. In addition to the sections describing various analysis techniques in BrainIAK, we have included sections describing the future applications of BrainIAK to real-time fMRI, tutorials that we have developed and shared online to facilitate learning the techniques in BrainIAK, computational innovations in BrainIAK, and how to contribute to BrainIAK. We hope that this manuscript helps readers to understand how BrainIAK might be useful in their research.

Deep language models (DLMs) provide a novel computational paradigm for how the brain processes natural language. Unlike symbolic, rule-based models described in psycholinguistics, DLMs encode words and their context as continuous numerical vectors. These “embeddings” are constructed by a sequence of computations organized in “layers” to ultimately capture surprisingly sophisticated representations of linguistic structures. How does this layered hierarchy map onto the human brain during natural language comprehension? In this study, we used electrocorticography (ECoG) to record neural activity in language areas along the superior temporal gyrus and inferior frontal gyrus while human participants listened to a 30-minute spoken narrative. We supplied this same narrative to a high-performing DLM (GPT2-XL) and extracted the contextual embeddings for each word in the story across all 48 layers of the model. We next trained a set of linear encoding models to predict the temporally-evolving neural activity from the embeddings at each layer. We found a striking correspondence between the layer-by-layer sequence of embeddings from GPT2-XL and the temporal sequence of neural activity in language areas. In addition, we found evidence for the gradual accumulation of recurrent information along the linguistic processing hierarchy. However, we also noticed additional neural processes in the brain, but not in DLMs, during the processing of surprising (unpredictable) words. These findings point to a connection between human language processing and DLMs where the layer-by-layer accumulation of contextual information in DLM embeddings matches the temporal dynamics of neural activity in high-order language areas.

Piecing together the meaning of a narrative requires understanding both individual words and the intricate relationships between them. How does the brain construct this kind of rich, contextual meaning? Recently, a new class of artificial neural networks—based on the Transformer architecture—has revolutionized the field of language modeling. Transformers integrate information across words via multiple layers of structured circuit computations, forming increasingly contextualized representations of linguistic content. In this paper, we deconstruct these circuit computations and analyze the associated “transformations” (alongside the more commonly studied “embeddings”) to provide a fine-grained window onto linguistic computations in the human brain. Using functional MRI data acquired while participants listened to naturalistic spoken stories, we find that these transformations capture a hierarchy of linguistic computations across cortex, with transformations at later layers in the model mapping onto higher-level language areas in the brain. We then decompose these transformations into individual, functionally-specialized “attention heads” and demonstrate that the emergent syntactic computations performed by individual heads correlate with predictions of brain activity in specific cortical regions. These heads fall along gradients corresponding to different layers, contextual distances, and syntactic dependencies in a low-dimensional cortical space. Our findings indicate that large language models and the cortical language network converge on similar trends of computational specialization for processing natural language.

Recent fMRI studies of event segmentation have found that default mode regions represent high-level event structure during movie watching. In these regions, neural patterns are relatively stable during events and shift at event boundaries. Music, like narratives, contains hierarchical event structure (e.g., sections are composed of phrases). Here, we tested the hypothesis that brain activity patterns in default mode regions reflect the high-level event structure of music. We used fMRI to record brain activity from 25 participants (male and female) as they listened to a continuous playlist of 16 musical excerpts, and additionally collected annotations for these excerpts by asking a separate group of participants to mark when meaningful changes occurred in each one. We then identified temporal boundaries between stable patterns of brain activity using a hidden Markov model and compared the location of the model boundaries to the location of the human annotations. We identified multiple brain regions with significant matches to the observer-identified boundaries, including auditory cortex, medial prefrontal cortex, parietal cortex, and angular gyrus. From these results, we conclude that both higher-order and sensory areas contain information relating to the high-level event structure of music. Moreover, the higher-order areas in this study overlap with areas found in previous studies of event perception in movies and audio narratives, including regions in the default mode network.Significance StatementListening to music requires the brain to track dynamics at multiple hierarchical timescales. In our study, we had fMRI participants listen to real-world music (classical and jazz pieces) and then used an unsupervised learning algorithm (a hidden Markov model) to model the high-level event structure of music within participants’ brain data. This approach revealed that default mode brain regions involved in representing the high-level event structure of narratives are also involved in representing the high-level event structure of music. These findings provide converging support for the hypothesis that these regions play a domain-general role in processing events occurring over long timescales.

Contextual embeddings, derived from deep language models (DLMs), provide a continuous vectorial representation of language. This embedding space differs fundamentally from the symbolic representations posited by traditional psycholinguistics. Do language areas in the human brain, similar to DLMs, rely on a continuous embedding space to represent language? To test this hypothesis, we densely recorded the neural activity in the Inferior Frontal Gyrus (IFG, also known as Broca’s area) of three participants using dense intracranial arrays while they listened to a 30-minute podcast. From these fine-grained spatiotemporal neural recordings, we derived for each patient a continuous vectorial representation for each word (i.e., a brain embedding). Using stringent, zero-shot mapping, we demonstrated that brain embeddings in the IFG and the DLM contextual embedding space have strikingly similar geometry. This shared geometry allows us to precisely triangulate the position of unseen words in both the brain embedding space (zero-shot encoding) and the DLM contextual embedding space (zero-shot decoding). The continuous brain embedding space provides an alternative computational framework for how natural language is represented in cortical language areas.

Quantifying how brain functional architecture differs from person to person is a key challenge in human neuroscience. Current individualized models of brain functional organization are based on brain regions and networks, limiting their use in studying fine-grained vertex-level differences. In this work, we present the Individualized Neural Tuning (INT) model, a fine-grained individualized model of brain functional organization. The INT model is designed to have vertex-level granularity, to capture both representational and topographic differences, and to model stimulus-general neural tuning. Through a series of analyses, we demonstrate that (a) our INT model provides a reliable individualized measure of fine-grained brain functional organization, (b) it accurately predicts individualized brain response patterns to new stimuli, and (c) it requires only 10–20 minutes of data for good performance. The high reliability, specificity, precision, and generalizability of our INT model affords new opportunities for building brain-based biomarkers based on naturalistic neuroimaging paradigms.

Deep convolutional neural networks (DCNNs) trained for face identification can rival and even exceed human-level performance. The relationships between internal representations learned by DCNNs and those of the primate face processing system are not well understood, especially in naturalistic settings. We developed the largest naturalistic dynamic face stimulus set in human neuroimaging research (700+ naturalistic video clips of unfamiliar faces) and used representational similarity analysis to investigate how well the representations learned by high-performing DCNNs match human brain representations across the entire distributed face processing system. DCNN representational geometries were strikingly consistent across diverse architectures and captured meaningful variance among faces. Similarly, representational geometries throughout the human face network were highly consistent across subjects. Nonetheless, correlations between DCNN and neural representations were very weak overall--DCNNs captured 3% of variance in the neural representational geometries at best. Intermediate DCNN layers better matched visual and face-selective cortices than the final fully-connected layers. Behavioral ratings of face similarity were highly correlated with intermediate layers of DCNNs, but also failed to capture representational geometry in the human brain. Our results suggest that the correspondence between intermediate DCNN layers and neural representations of naturalistic human face processing is weak at best, and diverges even further in the later fully-connected layers. This poor correspondence can be attributed, at least in part, to the dynamic and cognitive information that plays an essential role in human face processing but is not modeled by DCNNs. These mismatches indicate that current DCNNs have limited validity as in silico models of dynamic, naturalistic face processing in humans.

Our lives revolve around sharing emotional stories (i.e., happy and sad stories) with other people. Such emotional communication enhances the similarity of story comprehension and neural across speaker-listener pairs. The Social Information Model (EASI) suggests that such emotional communication may influence interpersonal closeness. However, few studies have examined speaker-listener interpersonal brain synchronization during emotional communication and whether it is associated with meaningful aspects of the speaker-listener interpersonal relationship. Here, one speaker watched emotional videos and communicated the content of the videos to thirty-two people as listeners (happy/sad/neutral group). Both speaker and listeners’ neural activities were recorded using EEG. After listening, we assessed the interpersonal closeness between the speaker and listeners. Compared to the sad group, sharing happy stories showed a better recall quality and a higher rating of interpersonal closeness. The happy group showed higher IBS in the frontal cortex and left temporoparietal cortex than the sad group. The relationship between frontal IBS and interpersonal closeness was moderated by sharing happy/sad stories. Exploratory analysis using support vector regression showed that the IBS could also predict the ratings of interpersonal closeness. These results suggest that frontal IBS could serve as an indicator of whether sharing emotional stories facilitate interpersonal closeness. These findings improve our understanding of emotional communication among individuals that guides behaviors during interpersonal interactions. Significance Statement Despite extensive research on interpersonal communication, little is known about emotional communication (happy/sad) between speaker and listener and whether these two types of emotional communication involve differential neurocognitive mechanisms from a brain-to-brain perspective. We address these questions from the perspective of the brain-to-brain approach and suggest that these two types of emotional communication are associated with differential interpersonal brain synchronization, in particular, subserved by the prefrontal region. Our findings shed light on the effect of sharing emotional (happy/sad) stories on interpersonal closeness and suggest that frontal IBS could serve as an indicator of whether sharing emotional (happy/sad) stories facilitate interpersonal closeness.

The brain actively reshapes our understanding of past events in light of new incoming information. In the current study, we ask how the brain supports this updating process during the encoding and recall of naturalistic stimuli. One group of participants watched a movie (“The Sixth Sense”) with a cinematic “twist” at the end that dramatically changed the interpretation of previous events. Next, participants were asked to verbally recall the movie events, taking into account the new “twist” information. Most participants updated their recall to incorporate the twist. Two additional groups recalled the movie without having to update their memories during recall: one group never saw the twist; another group was exposed to the twist prior to the beginning of the movie, and thus the twist information was incorporated both during encoding and recall. We found that providing participants with information about the twist beforehand altered neural response patterns during movie-viewing in the default mode network (DMN). Moreover, presenting participants with the twist at the end of the movie changed the neural representation of the previously-encoded information during recall in a subset of DMN regions. Further evidence for this transformation was obtained by comparing the neural activation patterns during encoding and recall and correlating them with behavioral signatures of memory updating. Our results demonstrate that neural representations of past events encoded in the DMN are dynamically integrated with new information that reshapes our understanding in natural contexts.

As the global health crisis unfolded throughout the world, many academic conferences moved online in 2020. This move has been hailed as a positive step towards inclusivity in its attenuation of economic, physical and legal barriers and effectively enabled many individuals who have traditionally been underrepresented to join and participate. A number of studies have outlined how moving online made it possible to gather a more global community and has increased opportunities for individuals with various constraints, e.g. caregiving responsibilities. Yet, the mere existence of online conferences is unfortunately no guarantee that everyone can attend and participate meaningfully. In fact, many elements of an online conference are still significant barriers to truly diverse participation: the tools used can be inaccessible for some individuals; the scheduling choices can favour some geographical locations; the setup of the conference can provide more visibility to well-established researchers and reduce opportunities for early career researchers. While acknowledging the benefits of an online setting, especially for individuals who have traditionally been underrepresented or excluded, we recognize that fostering social justice requires inclusivity to actively be centered in every aspect of online conference design.Here, we draw from the literature and from our own experiences to identify practices that purposefully encourage a diverse community to: attend, participate in, and lead online conferences. Reflecting on how to design more inclusive online events is especially important as multiple scientific organizations have announced that they will continue offering an online version of their event when in-person conferences can resume.

Invited talk presented at University of Michigan, Ann Arbor, MI.

Our lives revolve around sharing stories with others. Expressing emotion (i.e., happy and sad) is an essential characteristic of sharing stories and could enhance the similarity of story comprehension across speaker–listener pairs. The Emotions as Social Information Model (EASI) suggests that emotional communication may influence interpersonal closeness, but the effect of sharing emotional (happy/sad) stories on interpersonal closeness remains poorly understood. Here, one speaker watched emotional videos and communicated the content of the videos to thirty-two listeners (happy/sad/neutral group). Both speaker and listeners’ neural activities were recorded using EEG. After listening, we assessed the interpersonal closeness between the speaker and listeners. Compared to the sad group, sharing happy stories showed a better recall quality and a higher rating of interpersonal closeness.Meanwhile, the happy group showed higher interpersonal brain synchronization (IBS) in the prefrontal cortex (PFC) and temporal cortex (specifically TPJ) than the sad group. Moreover, such IBS mediated the relationship between the quality of sharing stories and interpersonal closeness, and happy emotion moderated this mediation model. The magnitude of IBS differentiated high interpersonal closeness from low interpersonal closeness. Exploratory analysis using support vector regression showed that the IBS could also predict the ratings of interpersonal closeness in left-out subjects. These results suggest that IBS could serve as an indicator of whether sharing emotional stories facilitate interpersonal closeness. These findings improve our understanding of sharing emotional information among individuals that guide behaviors during interpersonal interactions.

Departing from traditional linguistic models, advances in deep learning have resulted in a new type of predictive (autoregressive) deep language models (DLMs). Using a self-supervised next-word prediction task, these models generate appropriate linguistic responses in a given context. In the current study, nine participants listened to a 30-min podcast while their brain responses were recorded using electrocorticography (ECoG). We provide empirical evidence that the human brain and autoregressive DLMs share three fundamental computational principles as they process the same natural narrative: (1) both are engaged in continuous next-word prediction before word onset; (2) both match their pre-onset predictions to the incoming word to calculate post-onset surprise; (3) both rely on contextual embeddings to represent words in natural contexts. Together, our findings suggest that autoregressive DLMs provide a new and biologically feasible computational framework for studying the neural basis of language.

The “Narratives” collection aggregates a variety of functional MRI datasets collected while human subjects listened to naturalistic spoken stories. The current release includes 345 subjects, 891 functional scans, and 27 diverse stories of varying duration totaling ~4.6 hours of unique stimuli (~43,000 words). This data collection is well-suited for naturalistic neuroimaging analysis, and is intended to serve as a benchmark for models of language and narrative comprehension. We provide standardized MRI data accompanied by rich metadata, preprocessed versions of the data ready for immediate use, and the spoken story stimuli with time-stamped phoneme- and word-level transcripts. All code and data are publicly available with full provenance in keeping with current best practices in transparent and reproducible neuroimaging.

Shared information content is represented across brains in idiosyncratic functional topographies. Hyperalignment addresses these idiosyncrasies by using neural responses to project individuals’ brain data into a common model space while maintaining the geometric relationships between distinct activity patterns. The dimensions of this common model can encode any kind of functional profiles shared across individuals, such as cortical response profiles collected during a common time-locked stimulus presentation (e.g. movie viewing) or functional connectivity profiles. Performing hyperalignment with either response-based or connectivity-based input data derives transformations to project individuals’ neural data from anatomical space into the common model such that functional information is optimally aligned across brains. Previously, only response or connectivity profiles were used in the derivation of these transformations. In this study, we used three separate data sets collected while participants watched feature films to derive transformations representing both response-based and connectivity-based information with a single algorithm. Our new method, hybrid hyperalignment, aligns response-based information as well as or better than response hyperalignment while simultaneously aligning connectivity-based information better than connectivity hyperalignment, all in one information space. These results suggest that a single common information space could encode both shared cortical response and functional connectivity profiles across individuals.

Cognitive neuroscientists are trained to use clever experimental manipulations to isolate simple, interpretable relationships between brain, behavior, and environment. This approach relies on the assumption that these relations can be recomposed into a satisfying understanding of ecological brain function. Drawing parallels with evolutionary theory and machine learning, we challenge this assumption and argue that ecological considerations should play a more central role in cognitive neuroscience. With this in mind, we present the “Narratives” data collection—an fMRI benchmark for developing and evaluating models of naturalistic language comprehension. Although sharing the “long tail” of naturalistic neuroimaging data has clear benefits for the community, it also introduces challenges in data harmonization. We develop a novel hyperalignment algorithm that leverages shared connectivity to estimate a common response space across heterogeneous naturalistic datasets. This connectivity-based shared response model yields a consensus space across stimuli and dramatically increases the dimensionality of shared information across subjects.

Naturalistic, dynamic movies evoke strong, consistent, and information-rich patterns of activity over a broad expanse of cortex and engage multiple perceptual and cognitive systems in parallel. The use of naturalistic stimuli enables functional brain imaging research to explore cognitive domains that are poorly sampled in highly-controlled experiments. These domains include perception and understanding of agentic action, which plays a larger role in visual representation than was appreciated from experiments using static, controlled stimuli.

These slides on BIDS and BIDS Apps were presented as part of the TrainTrack for OHBM Brainhack 2020.

Information that is shared across brains is encoded in idiosyncratic fine-scale functional topographies. Hyperalignment captures shared information by projecting pattern vectors for neural responses and connectivities into a common, high-dimensional information space, rather than by aligning topographies in a canonical anatomical space. Individual transformation matrices project information from individual anatomical spaces into the common model information space, preserving the geometry of pairwise dissimilarities between pattern vectors, and model cortical topography as mixtures of overlapping, individual-specific topographic basis functions, rather than as contiguous functional areas. The fundamental property of brain function that is preserved across brains is information content, rather than the functional properties of local features that support that content. In this Perspective, we present the conceptual framework that motivates hyperalignment, its computational underpinnings for joint modeling of a common information space and idiosyncratic cortical topographies, and discuss implications for understanding the structure of cortical functional architecture.

Over many generations, evolution constructs organisms that are marvelously adapted to their environment through a simple, “blind” fitting process. Does the brain use similar brute-force fitting processes to learn how to perceive and act upon the world? Modern artificial neural networks have revealed the power in optimizing millions of parameters over millions of observations to solve real-world tasks. However, these models do not seem to learn simple, human-interpretable representations of the world. In this talk, we draw evolutionary theory and ecological psychology as a lens for understanding biological and artificial neural networks. We argue that this new family of direct-fit models presents a radical challenge to many of the theoretical assumptions in psychology and neuroscience.

Human communication is remarkably versatile, enabling teachers to share highly abstracted and novel information with their students. What neural processes enable such transfer of information across brains during naturalistic teaching and learning? Here, we show that during lectures, wherein information transmission is unidirectional and flows from the teacher to the student, the student’s brain mirrors the teacher’s brain and that this neural coupling is correlated with learning outcomes. A teacher was scanned in fMRI giving an oral lecture with slides on a scientific topic followed by a review lecture. Students were then scanned watching either the intact lecture and review (N = 20) or a temporally scrambled version of the lecture (N = 20). Using intersubject correlation (ISC), we observed widespread teacher-student neural coupling spanning sensory cortex and language regions along the superior temporal sulcus as well as higher-level regions including posterior medial cortex (PMC), superior parietal lobule (SPL), and dorsolateral and dorsomedial prefrontal cortex. Teacher-student alignment in higher-level areas was not observed when learning was disrupted by temporally scrambling the lecture. Moreover, teacher-student coupling in PMC was significantly correlated with learning outcomes: the more closely the student’s brain mirrored the teacher’s brain, the more the student improved between behavioral pre-learning and post-learning assessments. Together, these results suggest that the alignment of neural responses between teacher and students may underlie effective communication of complex information across brains in classroom settings.Significance statementHow is technical, non-narrative information communicated from one brain to another during teaching and learning? In this fMRI study, we show that the DMN activity of teachers and students are coupled during naturalistic teaching. This teacher-student neural coupling emerges only during intact learning and is correlated with learning outcomes. Together, these findings suggest that teacher-student neural alignment underlies effective communication during teaching.

People organize and convey their thoughts according to narratives. However, neuroscientists are often reluctant to incorporate narrative stimuli into their experiments. We argue that narratives deserve wider adoption in human neuroscience because they tap into the brain's native machinery for representing the world and provide rich variability for testing hypotheses.

Naturalistic experimental paradigms in neuroimaging arose from a pressure to test the validity of models we derive from highly-controlled experiments in real-world contexts. In many cases, however, such efforts led to the realization that models developed under particular experimental manipulations failed to capture much variance outside the context of that manipulation. The critique of non-naturalistic experiments is not a recent development; it echoes a persistent and subversive thread in the history of modern psychology. The brain has evolved to guide behavior in a multidimensional world with many interacting variables. The assumption that artificially decoupling and manipulating these variables will lead to a satisfactory understanding of the brain may be untenable. We develop an argument for the primacy of naturalistic paradigms, and point to recent developments in machine learning as an example of the transformative power of relinquishing control. Naturalistic paradigms should not be deployed as an afterthought if we hope to build models of brain and behavior that extend beyond the laboratory into the real world.

Available online 30 April 2021. Brainhack is an innovative meeting format that promotes scientific collaboration and education in an open, inclusive environment. This NeuroView describes the myriad benefits for participants and the research community and how Brainhacks complement conventional formats to augment scientific progress. The present manuscript is part of a growing community effort to collate Brainhack-related insights and expertise into a Jupyter Book (http://brainhack.org/brainhack_jupyter_book/) that will serve as a centralized set of resources for the community; we acknowledge all the individuals who contributed and will make ongoing contributions to these resources. A pre-print version of the present manuscript is available as part of the Jupyter Book. Moreover, we would like to acknowledge all Brainhack organizers, supporters, presenters, and participants for their contribution to growing and maintaining this community. The benefits described in this manuscript would not be possible without them. We also thank all institutions, labs, and organizations who have helped this community grow, meet in stimulating environments, and add an excellent educational resource pool and agenda. With an expanding community, Brainhack’s support network keeps growing, and we thank all labs and individual researchers for their dedication and expertise offered to this community (see http://brainhack.org/brainhack_jupyter_book/acknowledgments.html for a full list of individual acknowledgments; an updated list will be maintained in the Jupyter Book). Grants and funding bodies: This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement no. 867458 awarded to Julia M. Huntenburg; from ANR-19-DATA-0025 NeuroWebLab for Katja Heuer, Roberto Toro, and Nicholas Traut; and from NIH K00MH122372 for Stephanie Noble. The Brainhack Community member list and contributions of the different authors are detailed at http://brainhack.org/brainhack_jupyter_book/contributors.html. Our crediting system is described here: http://brainhack.org/brainhack_jupyter_book/neuroview_authorship-agreement.html.

Humans prioritize different semantic qualities of a complex stimulus depending on their behavioral goals. These semantic features are encoded in distributed neural populations, yet it is unclear how attention might operate across these distributed representations. To address this, we presented participants with naturalistic video clips of animals behaving in their natural environments while the participants attended to either behavior or taxonomy. We used models of representational geometry to investigate how attentional allocation affects the distributed neural representation of animal behavior and taxonomy. Attending to animal behavior transiently increased the discriminability of distributed population codes for observed actions in anterior intraparietal, pericentral, and ventral temporal cortices. Attending to animal taxonomy while viewing the same stimuli increased the discriminability of distributed animal category representations in ventral temporal cortex. For both tasks, attention selectively enhanced the discriminability of response patterns along behaviorally relevant dimensions. These findings suggest that behavioral goals alter how the brain extracts semantic features from the visual world. Attention effectively disentangles population responses for downstream read-out by sculpting representational geometry in late-stage perceptual areas.

How does the human brain construct narratives from a sequence of spoken words? Here we present a benchmark fMRI dataset for evaluating neural models for naturalistic speech perception and narrative comprehension. The ‘Narratives’ collection comprises over 500 scanning runs across more than 300 subjects. The story stimuli comprise over 20 spoken narratives ranging from 3 minutes to ~1 hour in duration, for ~4.5 hours of unique stimuli. Overall, this yields over 300,000 TRs across all subjects and stories, or ~5 days of fMRI data. All data and materials have been standardized and staged for public release. The stories span a variety of media, including commercially-produced radio and internet broadcasts, authors reading written works, professional storytellers performing in front of live audiences, and experimental subjects verbally recalling previous events. Alongside the stimuli, we provide time-stamped word-level transcripts created using a semi-supervised forced-alignment algorithm. MRI data have been organized according to the machine-readable Brain Imaging Data Structure (BIDS) with exhaustive metadata. Anonymized subject labels are linked across stories and include demographic and behavioral variables including age, gender, group or condition, and comprehension score (where available). MRI data are provided with various levels of preprocessing, including volumetric and surface-based spatial normalization, spatial smoothing, and temporal filtering with confound regressors. To more effectively aggregate data across subjects and stories, we developed a variant of hyperalignment that capitalizes on intersubject functional correlation (ISFC) analysis to derive a single connectivity-based shared response space across disjoint datasets. To validate the quality of the data, we applied intersubject correlation (ISC) analysis to each dataset. First, we measured ISCs in early auditory cortex to evaluate the temporal alignment of responses and identify outliers. Next, we performed whole-brain ISC analysis, revealing consistent responses throughout a network of cortical areas supporting language and event representation. Well-curated, naturalistic data have tremendous potential for re-use, and we hope the community will benefit from these data in disentangling the neural mechanisms of language comprehension.

Evolution is a blind fitting process by which organisms, over generations, adapt to the niches of an ever-changing environment. Does the mammalian brain use similar brute-force fitting processes to learn how to perceive and act upon the world? Recent advances in training deep neural networks has exposed the power of optimizing millions of synaptic weights to map millions of observations along ecologically relevant objective functions. This class of models has dramatically outstripped simpler, more intuitive models, operating robustly in real-life contexts spanning perception, language, and action coordination. These models do not learn an explicit, human-interpretable representation of the underlying structure of the data; rather, they use local computations to interpolate over task-relevant manifolds in a high-dimensional parameter space. Furthermore, counterintuitively, over-parameterized models, similarly to evolutionary processes, can be simple and parsimonious as they provide a versatile, robust solution for learning a diverse set of functions. In contrast to traditional scientific models, where the ultimate goal is interpretability, over-parameterized models eschew interpretability in favor of solving real-life problems or tasks. We contend that over-parameterized blind fitting presents a radical challenge to many of the underlying assumptions and practices in computational neuroscience and cognitive psychology. At the same time, this shift in perspective informs longstanding debates and establishes unexpected links with evolution, ecological psychology, and artificial life.

Summary Evolution is a blind fitting process by which organisms become adapted to their environment. Does the brain use similar brute-force fitting processes to learn how to perceive and act upon the world? Recent advances in artificial neural networks have exposed the power of optimizing millions of synaptic weights over millions of observations to operate robustly in real-world contexts. These models do not learn simple, human-interpretable rules or representations of the world; rather, they use local computations to interpolate over task-relevant manifolds in a high-dimensional parameter space. Counterintuitively, similar to evolutionary processes, over-parameterized models can be simple and parsimonious, as they provide a versatile, robust solution for learning a diverse set of functions. This new family of direct-fit models present a radical challenge to many of the theoretical assumptions in psychology and neuroscience. At the same time, this shift in perspective establishes unexpected links with developmental and ecological psychology.

Connectivity hyperalignment can be used to estimate a single shared response space across disjoint datasets. We describe a variant of connectivity hyperalignment—the connectivity-based shared response model—that factorizes aggregated fMRI datasets into a single reduced-dimension shared connectivity space and subject-specific topographic transformations. These transformations resolve idiosyncratic functional topographies and can be used to project response time series into shared space. We benchmark this algorithm on a large, heterogeneous collection of story-listening functional MRI datasets assembled over the course of approximately seven years. This data collection comprises 10 unique auditory story stimuli across 300 scans with 160 unique subjects. Projecting subject data into shared space dramatically improves between-subject story time-segment classification and increases the dimensionality of shared information across subjects. This improvement generalizes to subjects and stories excluded when estimating the shared space. We demonstrate that estimating a simple semantic encoding model in shared space improves between-subject forward encoding and inverted encoding model performance. The shared space estimated across all datasets is distinct from the shared space derived from any particular constituent dataset; the algorithm leverages shared connectivity to yield a consensus shared space conjoining diverse story stimuli.

Our capacity to jointly represent information about the world underpins our social experience. By leveraging one individual’s brain activity to model another’s, we can measure shared information across brains—even in dynamic, naturalistic scenarios where an explicit response model may be unobtainable. Introducing experimental manipulations allows us to measure, for example, shared responses between speakers and listeners, or between perception and recall. In this tutorial, we develop the logic of intersubject correlation (ISC) analysis and discuss the family of neuroscientific questions that stem from this approach. We also extend this logic to spatially distributed response patterns and functional network estimation. We provide a thorough and accessible treatment of methodological considerations specific to ISC analysis, and outline best practices.

Common or folk knowledge about animals is dominated by three dimensions: (1) level of cognitive complexity or “animacy;” (2) dangerousness or “predacity;” and (3) size. We investigated the neural basis of the perceived dangerousness or aggressiveness of animals, which we refer to more generally as “perception of threat.” Using functional magnetic resonance imaging (fMRI), we analyzed neural activity evoked by viewing images of animal categories that spanned the dissociable semantic dimensions of threat and taxonomic class. The results reveal a distributed network for perception of threat extending along the right superior temporal sulcus. We compared neural representational spaces with target representational spaces based on behavioral judgments and a computational model of early vision and found a processing pathway in which perceived threat emerges as a dominant dimension: whereas visual features predominate in early visual cortex and taxonomy in lateral occipital and ventral temporal cortices, these dimensions fall away progressively from posterior to anterior temporal cortices, leaving threat as the dominant explanatory variable. Our results suggest that the perception of threat in the human brain is associated with neural structures that underlie perception and cognition of social actions and intentions, suggesting a broader role for these regions than has been thought previously, one that includes the perception of potential threat from agents independent of their biological class.SIGNIFICANCE STATEMENTFor centuries, philosophers have wondered how the human mind organizes the world into meaningful categories and concepts. Today this question is at the core of cognitive science, but our focus has shifted to understanding how knowledge manifests in dynamic activity of neural systems in the human brain. This study advances the young field of empirical neuroepistemology by characterizing the neural systems engaged by an important dimension in our cognitive representation of the animal kingdom ontological subdomain: how the brain represents the perceived threat, dangerousness, or “predacity” of animals. Our findings reveal how activity for domain-specific knowledge of animals overlaps the social perception networks of the brain, suggesting domain-general mechanisms underlying the representation of conspecifics and other animals.