Consensus-building conversation leads to neural alignment

Conversation is a primary means of social influence, but its effects on brain activity remain unknown. Previous work on conversation and social influence has emphasized public compliance, largely setting private beliefs aside. Here, we show that consensus-building conversation aligns future brain activity within groups, with alignment persisting through novel experiences participants did not discuss. Participants watched ambiguous movie clips during fMRI scanning, then conversed in groups with the goal of coming to a consensus about each clip's narrative. After conversation, participants' brains were scanned while viewing the clips again, along with novel clips from the same movies. Groups that reached consensus showed greater similarity of brain activity after conversation. Participants perceived as having high social status spoke more and signaled disbelief in others, and their groups had unequal turn-taking and lower neural alignment. By contrast, participants with central positions in their real-world social networks encouraged others to speak, facilitating greater group neural alignment. Socially central participants were also more likely to become neurally aligned to others in their groups.

Alignment of brain embeddings and artificial contextual embeddings in natural language points to common geometric patterns

Contextual embeddings, derived from deep language models (DLMs), provide a continuous vectorial representation of language. This embedding space differs fundamentally from the symbolic representations posited by traditional psycholinguistics. We hypothesize that language areas in the human brain, similar to DLMs, rely on a continuous embedding space to represent language. To test this hypothesis, we densely record the neural activity patterns in the inferior frontal gyrus (IFG) of three participants using dense intracranial arrays while they listened to a 30-minute podcast. From these fine-grained spatiotemporal neural recordings, we derive a continuous vectorial representation for each word (i.e., a brain embedding) in each patient. Using stringent zero-shot mapping we demonstrate that brain embeddings in the IFG and the DLM contextual embedding space have common geometric patterns. The common geometric patterns allow us to predict the brain embedding in IFG of a given left-out word based solely on its geometrical relationship to other non-overlapping words in the podcast. Furthermore, we show that contextual embeddings capture the geometry of IFG embeddings better than static word embeddings. The continuous brain embedding space exposes a vector-based neural code for natural language processing in the human brain.

How a speaker herds the audience: Multi-brain neural convergence over time during naturalistic storytelling

Storytelling-an ancient way for humans to share individual experiences with others-has been found to induce neural synchronization among listeners. In our exploration of the dynamic fluctuations in listener-listener (LL) coupling throughout stories, we uncover a significant correlation between LL and lag-speaker-listener (lag-SL) couplings over time. Using the analogy of neural pattern (dis)similarity as distances between participants, we term this phenomenon the "herding effect": like a shepherd guiding a group of sheep, the more closely listeners follow the speaker's prior brain activity patterns (higher lag-SL similarity), the more tightly they cluster together (higher LL similarity). This herding effect is particularly pronounced in brain regions where neural synchronization among listeners tracks with behavioral ratings of narrative engagement, highlighting the mediating role of narrative content in the observed multi-brain neural coupling dynamics. By integrating LL and SL neural couplings, this study illustrates how unfolding stories shape a dynamic multi-brain functional network and how the configuration of this network may be associated with moment-by-moment efficacy of communication.

Improved prediction of behavioral and neural similarity spaces using pruned DNNs

Deep Neural Networks (DNNs) have become an important tool for modeling brain and behavior. One key area of interest has been to apply these networks to model human similarity judgements. Several previous works have used the embeddings from the penultimate layer of vision DNNs and showed that a reweighting of these features improves the fit between human similarity judgments and DNNs. These studies underline the idea that these embeddings form a good basis set but lack the correct level of salience. Here we re-examined the grounds for this idea and on the contrary, we hypothesized that these embeddings, beyond forming a good basis set, also have the correct level of salience to account for similarity judgments. It is just that the huge dimensional embedding needs to be pruned to select those features relevant for the considered domain for which a similarity space is modeled. In Study 1 we supervised DNN pruning based on a subset of human similarity judgments. We found that pruning: i) improved out-of-sample prediction of human similarity judgments from DNN embeddings, ii) produced better alignment with WordNet hierarchy, and iii) retained much higher classification accuracy than reweighting. Study 2 showed that pruning by neurobiological data is highly effective in improving out-of-sample prediction of brain-derived representational dissimilarity matrices from DNN embeddings, at times fleshing out isomorphisms not otherwise observable. Using pruned DNNs, image-level heatmaps can be produced to identify image sections whose features load on dimensions coded by a brain area. Pruning supervised by human brain/behavior therefore effectively identifies alignable dimensions of knowledge between DNNs and humans and constitutes an effective method for understanding the organization of knowledge in neural networks.

Bayesian Surprise Predicts Human Event Segmentation in Story Listening

Event segmentation theory posits that people segment continuous experience into discrete events and that event boundaries occur when there are large transient increases in prediction error. Here, we set out to test this theory in the context of story listening, by using a deep learning language model (GPT-2) to compute the predicted probability distribution of the next word, at each point in the story. For three stories, we used the probability distributions generated by GPT-2 to compute the time series of prediction error. We also asked participants to listen to these stories while marking event boundaries. We used regression models to relate the GPT-2 measures to the human segmentation data. We found that event boundaries are associated with transient increases in Bayesian surprise but not with a simpler measure of prediction error (surprisal) that tracks, for each word in the story, how strongly that word was predicted at the previous time point. These results support the hypothesis that prediction error serves as a control mechanism governing event segmentation and point to important differences between operational definitions of prediction error.

A shared linguistic space for transmitting our thoughts from brain to brain in natural conversations

Effective communication hinges on a mutual understanding of word meaning in different contexts. The embedding space learned by large language models can serve as an explicit model of the shared, context-rich meaning space humans use to communicate their thoughts. We recorded brain activity using electrocorticography during spontaneous, face-to-face conversations in five pairs of epilepsy patients. We demonstrate that the linguistic embedding space can capture the linguistic content of word-by-word neural alignment between speaker and listener. Linguistic content emerged in the speaker's brain before word articulation, and the same linguistic content rapidly reemerged in the listener's brain after word articulation. These findings establish a computational framework to study how human brains transmit their thoughts to one another in real-world contexts.

Perspective changes in human listeners are aligned with the contextual transformation of the word embedding space

Word embedding representations have been shown to be effective in predicting human neural responses to lingual stimuli. While these representations are sensitive to the textual context, they lack the extratextual sources of context such as prior knowledge, thoughts, and beliefs, all of which constitute the listener's perspective. In this study, we propose conceptualizing the listeners' perspective as a source that induces changes in the embedding space. We relied on functional magnetic resonance imaging data collected by Yeshurun Y, Swanson S, Simony E, Chen J, Lazaridi C, Honey CJ, Hasson U. Same story, different story: the neural representation of interpretive frameworks. Psychol Sci. 2017:28(3):307-319, in which two groups of human listeners (n = 40) were listening to the same story but with different perspectives. Using a dedicated fine-tuning process, we created two modified versions of a word embedding space, corresponding to the two groups of listeners. We found that each transformed space was better fitted with neural responses of the corresponding group, and that the spatial distances between these spaces reflect both interpretational differences between the perspectives and the group-level neural differences. Together, our results demonstrate how aligning a continuous embedding space to a specific context can provide a novel way of modeling listeners' intrinsic perspectives.

Evidence That Event Boundaries Are Access Points for Memory Retrieval

When recalling memories, we often scan information-rich continuous episodes, for example, to find our keys. How does our brain access and search through those memories? We suggest that high-level structure, marked by event boundaries, guides us through this process: In our computational model, memory scanning is sped up by skipping ahead to the next event boundary upon reaching a decision threshold. In adult Mechanical Turk workers from the United States, we used a movie (normed for event boundaries; Study 1, N = 203) to prompt memory scanning of movie segments for answers (Study 2, N = 298) and mental simulation (Study 3, N = 100) of these segments. Confirming model predictions, we found that memory-scanning times varied as a function of the number of event boundaries within a segment and the distance of the search target to the previous boundary (the key diagnostic parameter). Mental simulation times were also described by a skipping process with a higher skipping threshold than memory scanning. These findings identify event boundaries as access points to memory.

Lipreading a naturalistic narrative in a female population: Neural characteristics shared with listening and reading

INTRODUCTION

Few of us are skilled lipreaders while most struggle with the task. Neural substrates that enable comprehension of connected natural speech via lipreading are not yet well understood.

METHODS

We used a data-driven approach to identify brain areas underlying the lipreading of an 8-min narrative with participants whose lipreading skills varied extensively (range 6-100%, mean = 50.7%). The participants also listened to and read the same narrative. The similarity between individual participants' brain activity during the whole narrative, within and between conditions, was estimated by a voxel-wise comparison of the Blood Oxygenation Level Dependent (BOLD) signal time courses.

RESULTS

Inter-subject correlation (ISC) of the time courses revealed that lipreading, listening to, and reading the narrative were largely supported by the same brain areas in the temporal, parietal and frontal cortices, precuneus, and cerebellum. Additionally, listening to and reading connected naturalistic speech particularly activated higher-level linguistic processing in the parietal and frontal cortices more consistently than lipreading, probably paralleling the limited understanding obtained via lip-reading. Importantly, higher lipreading test score and subjective estimate of comprehension of the lipread narrative was associated with activity in the superior and middle temporal cortex.

CONCLUSIONS

Our new data illustrates that findings from prior studies using well-controlled repetitive speech stimuli and stimulus-driven data analyses are also valid for naturalistic connected speech. Our results might suggest an efficient use of brain areas dealing with phonological processing in skilled lipreaders.

A functional neuroimaging dataset acquired during naturalistic movie watching and narrated recall of a series of short cinematic films

Whole-brain functional magnetic resonance imaging (fMRI) data from twenty healthy human participants were collected during naturalistic movie watching and free spoken recall tasks. Participants watched ten short (approximately 2 - 8 min) audiovisual movies and then verbally described what they remembered about the movies in their own words. Participants' verbal responses were audio recorded using an MR-compatible microphone. The audio recordings were transcribed and timestamped by independent coders. The neural and behavioral data were organized in the Brain Imaging Data Structure (BIDS) format and made publicly available via OpenNeuro.org. The dataset can be used to explore the neural bases of naturalistic memory and other cognitive functions including but not limited to visual/auditory perception, language comprehension, and speech generation.

Real-time neurofeedback to alter interpretations of a naturalistic narrative

We explored the potential of using real-time fMRI (rt-fMRI) neurofeedback training to bias interpretations of naturalistic narrative stimuli. Participants were randomly assigned to one of two possible conditions, each corresponding to a different interpretation of an ambiguous spoken story. While participants listened to the story in the scanner, neurofeedback was used to reward neural activity corresponding to the assigned interpretation. After scanning, final interpretations were assessed. While neurofeedback did not change story interpretations on average, participants with higher levels of decoding accuracy during the neurofeedback procedure were more likely to adopt the assigned interpretation; additional control conditions are needed to establish the role of individualized feedback in driving this result. While naturalistic stimuli introduce a unique set of challenges in providing effective and individualized neurofeedback, we believe that this technique holds promise for individualized cognitive therapy.

Shared computational principles for language processing in humans and deep language models

Departing from traditional linguistic models, advances in deep learning have resulted in a new type of predictive (autoregressive) deep language models (DLMs). Using a self-supervised next-word prediction task, these models generate appropriate linguistic responses in a given context. In the current study, nine participants listened to a 30-min podcast while their brain responses were recorded using electrocorticography (ECoG). We provide empirical evidence that the human brain and autoregressive DLMs share three fundamental computational principles as they process the same natural narrative: (1) both are engaged in continuous next-word prediction before word onset; (2) both match their pre-onset predictions to the incoming word to calculate post-onset surprise; (3) both rely on contextual embeddings to represent words in natural contexts. Together, our findings suggest that autoregressive DLMs provide a new and biologically feasible computational framework for studying the neural basis of language.

Deep language models have revolutionized natural language processing. The paper discovers three computational principles shared between deep language models and the human brain, which can transform our understanding of the neural basis of language.

BrainIAK: The Brain Imaging Analysis Kit

Functional magnetic resonance imaging (fMRI) offers a rich source of data for studying the neural basis of cognition. Here, we describe the Brain Imaging Analysis Kit (BrainIAK), an open-source, free Python package that provides computationally optimized solutions to key problems in advanced fMRI analysis. A variety of techniques are presently included in BrainIAK: intersubject correlation (ISC) and intersubject functional connectivity (ISFC), functional alignment via the shared response model (SRM), full correlation matrix analysis (FCMA), a Bayesian version of representational similarity analysis (BRSA), event segmentation using hidden Markov models, topographic factor analysis (TFA), inverted encoding models (IEMs), an fMRI data simulator that uses noise characteristics from real data (fmrisim), and some emerging methods. These techniques have been optimized to leverage the efficiencies of high-performance compute (HPC) clusters, and the same code can be seamlessly transferred from a laptop to a cluster. For each of the aforementioned techniques, we describe the data analysis problem that the technique is meant to solve and how it solves that problem; we also include an example Jupyter notebook for each technique and an annotated bibliography of papers that have used and/or described that technique. In addition to the sections describing various analysis techniques in BrainIAK, we have included sections describing the future applications of BrainIAK to real-time fMRI, tutorials that we have developed and shared online to facilitate learning the techniques in BrainIAK, computational innovations in BrainIAK, and how to contribute to BrainIAK. We hope that this manuscript helps readers to understand how BrainIAK might be useful in their research.

A neural network model of when to retrieve and encode episodic memories

Recent human behavioral and neuroimaging results suggest that people are selective in when they encode and retrieve episodic memories. To explain these findings, we trained a memory-augmented neural network to use its episodic memory to support prediction of upcoming states in an environment where past situations sometimes reoccur. We found that the network learned to retrieve selectively as a function of several factors, including its uncertainty about the upcoming state. Additionally, we found that selectively encoding episodic memories at the end of an event (but not mid-event) led to better subsequent prediction performance. In all of these cases, the benefits of selective retrieval and encoding can be explained in terms of reducing the risk of retrieving irrelevant memories. Overall, these modeling results provide a resource-rational account of why episodic retrieval and encoding should be selective and lead to several testable predictions.

High-Order Areas and Auditory Cortex Both Represent the High-Level Event Structure of Music

Recent fMRI studies of event segmentation have found that default mode regions represent high-level event structure during movie watching. In these regions, neural patterns are relatively stable during events and shift at event boundaries. Music, like narratives, contains hierarchical event structure (e.g., sections are composed of phrases). Here, we tested the hypothesis that brain activity patterns in default mode regions reflect the high-level event structure of music. We used fMRI to record brain activity from 25 participants (male and female) as they listened to a continuous playlist of 16 musical excerpts and additionally collected annotations for these excerpts by asking a separate group of participants to mark when meaningful changes occurred in each one. We then identified temporal boundaries between stable patterns of brain activity using a hidden Markov model and compared the location of the model boundaries to the location of the human annotations. We identified multiple brain regions with significant matches to the observer-identified boundaries, including auditory cortex, medial pFC, parietal cortex, and angular gyrus. From these results, we conclude that both higher-order and sensory areas contain information relating to the high-level event structure of music. Moreover, the higher-order areas in this study overlap with areas found in previous studies of event perception in movies and audio narratives, including regions in the default mode network.

Neural representations of naturalistic events are updated as our understanding of the past changes

The brain actively reshapes our understanding of past events in light of new incoming information. In the current study, we ask how the brain supports this updating process during the encoding and recall of naturalistic stimuli. One group of participants watched a movie ('The Sixth Sense') with a cinematic 'twist' at the end that dramatically changed the interpretation of previous events. Next, participants were asked to verbally recall the movie events, taking into account the new 'twist' information. Most participants updated their recall to incorporate the twist. Two additional groups recalled the movie without having to update their memories during recall: one group never saw the twist; another group was exposed to the twist prior to the beginning of the movie, and thus the twist information was incorporated both during encoding and recall. We found that providing participants with information about the twist beforehand altered neural response patterns during movie-viewing in the default mode network (DMN). Moreover, presenting participants with the twist at the end of the movie changed the neural representation of the previously-encoded information during recall in a subset of DMN regions. Further evidence for this transformation was obtained by comparing the neural activation patterns during encoding and recall and correlating them with behavioral signatures of memory updating. Our results demonstrate that neural representations of past events encoded in the DMN are dynamically integrated with new information that reshapes our understanding in natural contexts.

Changes in Cortical Coherence Supporting Complex Visual and Social Processing in Adolescence

Despite our differences, there is much about the natural visual world that most observers perceive in common. Across adults, approximately 30% of the brain is activated in a consistent fashion while viewing naturalistic input. At what stage of development is this consistency of neural profile across individuals present? Here, we focused specifically on whether this mature profile is present in adolescence, a key developmental period that bridges childhood and adulthood, and in which new cognitive and social challenges are at play. We acquired fMRI data evoked by a movie shown twice to younger (9-14 years old) and older adolescents (15-19 years old) and to adults, and conducted three key analyses. First, we characterized the consistency of the neural response within individuals (across separate runs of the movie), then within individuals of the same age group, and, last, between age groups. The neural consistency within individuals was similar across age groups with reliable activation in largely overlapping but slightly different cortical regions. In contrast, somewhat differing regions exhibited higher within-age correlations in both groups of adolescents than in the adults. Last, across the whole cortex, we identified regions evincing different patterns of maturation across age. Together, these findings provide a fine-grained characterization of functional neural development in adolescence and uncover signatures of widespread change in cortical coherence that supports the emerging mature stereotypical responses to naturalistic stimuli. These results also offer a more nuanced account of development that obeys neither a rigid linear progression nor a large qualitative change over time.

The "Narratives" fMRI dataset for evaluating models of naturalistic language comprehension

The "Narratives" collection aggregates a variety of functional MRI datasets collected while human subjects listened to naturalistic spoken stories. The current release includes 345 subjects, 891 functional scans, and 27 diverse stories of varying duration totaling ~4.6 hours of unique stimuli (~43,000 words). This data collection is well-suited for naturalistic neuroimaging analysis, and is intended to serve as a benchmark for models of language and narrative comprehension. We provide standardized MRI data accompanied by rich metadata, preprocessed versions of the data ready for immediate use, and the spoken story stimuli with time-stamped phoneme- and word-level transcripts. All code and data are publicly available with full provenance in keeping with current best practices in transparent and reproducible neuroimaging.

Moment-by-moment tracking of naturalistic learning and its underlying hippocampo-cortical interactions

Humans form lasting memories of stimuli that were only encountered once. This naturally occurs when listening to a story, however it remains unclear how and when memories are stored and retrieved during story-listening. Here, we first confirm in behavioral experiments that participants can learn about the structure of a story after a single exposure and are able to recall upcoming words when the story is presented again. We then track mnemonic information in high frequency activity (70–200 Hz) as patients undergoing electrocorticographic recordings listen twice to the same story. We demonstrate predictive recall of upcoming information through neural responses in auditory processing regions. This neural measure correlates with behavioral measures of event segmentation and learning. Event boundaries are linked to information flow from cortex to hippocampus. When listening for a second time, information flow from hippocampus to cortex precedes moments of predictive recall. These results provide insight on a fine-grained temporal scale into how episodic memory encoding and retrieval work under naturalistic conditions.

When listening to a story, humans learn about its structure and content. Here the authors reveal the neural processes behind episodic memory and predictive recall at a fine temporal scale in this naturalistic setting

Teacher-Student Neural Coupling During Teaching and Learning

Human communication is remarkably versatile, enabling teachers to share highly abstracted and novel information with their students. What neural processes enable such transfer of information across brains during naturalistic teaching and learning? Here, a teacher was scanned in functional magnetic resonance imaging while giving an oral lecture with slides on a scientific topic followed by a review lecture. Students were then scanned while watching either the intact Lecture and Review (N = 20) or a temporally scrambled version of the lecture (N = 20). Using intersubject correlation, we observed widespread Teacher–Student neural coupling spanning sensory cortex and language regions along the superior temporal sulcus as well as higher-level regions including posterior medial cortex (PMC), superior parietal lobule, and dorsolateral and dorsomedial prefrontal cortex. Teacher–student alignment in higher-level areas was not observed when learning was disrupted by temporally scrambling the lecture. Moreover, teacher–student coupling in PMC was significantly correlated with learning: the more closely the student’s brain mirrored the teacher’s brain, the more the student improved their learning score. Together, these results suggest that the alignment of neural responses between teacher and students may reflect effective communication of complex information across brains in classroom settings.

Spontaneous eye movements during eyes-open rest reduce resting-state-network modularity by increasing visual-sensorimotor connectivity

During wakeful rest, individuals make small eye movements during fixation. We examined how these endogenously driven oculomotor patterns impact topography and topology of functional brain networks. We used a dataset consisting of eyes-open resting-state (RS) fMRI data with simultaneous eye tracking. The eye-tracking data indicated minor movements during rest, which correlated modestly with RS BOLD data. However, eye-tracking data correlated well with echo-planar imaging time series sampled from the area of the eye-orbit (EO-EPI), which is a signal previously used to identify eye movements during exogenous saccades and movie viewing. Further analyses showed that EO-EPI data were correlated with activity in an extensive motor and sensorimotor network, including components of the dorsal attention network and the frontal eye fields. Partialling out variance related to EO-EPI from RS data reduced connectivity, primarily between sensorimotor and visual areas. It also produced networks with higher modularity, lower mean connectivity strength, and lower mean clustering coefficient. Our results highlight new aspects of endogenous eye movement control during wakeful rest. They show that oculomotor-related contributions form an important component of RS network topology, and that those should be considered in interpreting differences in network structure between populations or as a function of different experimental conditions.

We studied how subtle eye movements made during fixation, in absence of any other task, are related to resting-state connectivity measured using fMRI. We used a dataset for which eye tracking and BOLD resting-state were acquired simultaneously. We correlated brain activity with both eye-tracking metrics as well as time series sampled from the area of the eye orbits (EO-EPI). Eye-tracking data correlated well with the EO-EPI data. Furthermore, EO-EPI correlated with BOLD signal in sensorimotor and visual brain systems. Removing variance related to EO-EPI reduced connectivity between sensorimotor and visual areas and resulted in more modular resting-state networks. Our findings show that oculomotor-related contributions are an important component of resting-state network topology, and that they can be studied using EPI data from the eye orbits.

Relating the Past with the Present: Information Integration and Segregation during Ongoing Narrative Processing

This study examined how the brain dynamically updates event representations by integrating new information over multiple minutes while segregating irrelevant input. A professional writer custom-designed a narrative with two independent storylines, interleaving across minute-long segments (ABAB). In the last (C) part, characters from the two storylines meet and their shared history is revealed. Part C is designed to induce the spontaneous recall of past events, upon the recurrence of narrative motifs from A/B, and to shed new light on them. Our fMRI results showed storyline-specific neural patterns, which were reinstated (i.e., became more active) during storyline transitions. This effect increased along the processing timescale hierarchy, peaking in the default mode network. Similarly, the neural reinstatement of motifs was found during Part C. Furthermore, participants showing stronger motif reinstatement performed better in integrating A/B and C events, demonstrating the role of memory reactivation in information integration over intervening irrelevant events.

Neural alignment predicts learning outcomes in students taking an introduction to computer science course

Despite major advances in measuring human brain activity during and after educational experiences, it is unclear how learners internalize new content, especially in real-life and online settings. In this work, we introduce a neural approach to predicting and assessing learning outcomes in a real-life setting. Our approach hinges on the idea that successful learning involves forming the right set of neural representations, which are captured in canonical activity patterns shared across individuals. Specifically, we hypothesized that learning is mirrored in neural alignment: the degree to which an individual learner's neural representations match those of experts, as well as those of other learners. We tested this hypothesis in a longitudinal functional MRI study that regularly scanned college students enrolled in an introduction to computer science course. We additionally scanned graduate student experts in computer science. We show that alignment among students successfully predicts overall performance in a final exam. Furthermore, within individual students, we find better learning outcomes for concepts that evoke better alignment with experts and with other students, revealing neural patterns associated with specific learned concepts in individuals.

Semantically predictable input streams impede gaze-orientation to surprising locations

When available, people use prior knowledge to predict dimensions of future events such as their location and semantic features. However, few studies have examined how multi-dimensional predictions are implemented, and mechanistic accounts are absent. Using eye tracking, we evaluated whether predictions of target-location and target-category interact during the earliest stages of orientation. We presented stochastic series so that across four conditions, participants could predict either the location of the next target-image, its semantic category, both dimensions, or neither. Participants observed images in absence of any task involving their semantic content. We modeled saccade latencies using ELATER, a rise-to-threshold model that accounts for accumulation rate (AR), variance of AR over trials, and variance of decision baseline. The main findings were: 1) AR scaled with the degree of surprise associated with a target's location; 2) predictability of semantic-category hindered saccade latencies, suggesting a bottleneck in implementing joint predictions; 3) saccades to targets that satisfied semantic expectations were associated with greater AR-variance than saccades to semantically-surprising images, consistent with a richer repertoire of early evaluative processes for semantically-expected images. Predictability of target-category also impacted gaze pre-positioning prior to target presentation. The results indicate a strong interaction between foreknowledge of object location and semantics during stimulus-guided saccades, and suggest statistical regularities in an input stream can also impact anticipatory, non-stimulus-guided processes.

The default mode network: where the idiosyncratic self meets the shared social world

The default mode network (DMN) is classically considered an 'intrinsic' system, specializing in internally oriented cognitive processes such as daydreaming, reminiscing and future planning. In this Perspective, we suggest that the DMN is an active and dynamic 'sense-making' network that integrates incoming extrinsic information with prior intrinsic information to form rich, context-dependent models of situations as they unfold over time. We review studies that relied on naturalistic stimuli, such as stories and movies, to demonstrate how an individual's DMN neural responses are influenced both by external information accumulated as events unfold over time and by the individual's idiosyncratic past memories and knowledge. The integration of extrinsic and intrinsic information over long timescales provides a space for negotiating a shared neural code, which is necessary for establishing shared meaning, shared communication tools, shared narratives and, above all, shared communities and social networks.

Keep it real: rethinking the primacy of experimental control in cognitive neuroscience

Naturalistic experimental paradigms in neuroimaging arose from a pressure to test the validity of models we derive from highly-controlled experiments in real-world contexts. In many cases, however, such efforts led to the realization that models developed under particular experimental manipulations failed to capture much variance outside the context of that manipulation. The critique of non-naturalistic experiments is not a recent development; it echoes a persistent and subversive thread in the history of modern psychology. The brain has evolved to guide behavior in a multidimensional world with many interacting variables. The assumption that artificially decoupling and manipulating these variables will lead to a satisfactory understanding of the brain may be untenable. We develop an argument for the primacy of naturalistic paradigms, and point to recent developments in machine learning as an example of the transformative power of relinquishing control. Naturalistic paradigms should not be deployed as an afterthought if we hope to build models of brain and behavior that extend beyond the laboratory into the real world.

Propagation of Information Along the Cortical Hierarchy as a Function of Attention While Reading and Listening to Stories

How does attention route information from sensory to high-order areas as a function of task, within the relatively fixed topology of the brain? In this study, participants were simultaneously presented with 2 unrelated stories-one spoken and one written-and asked to attend one while ignoring the other. We used fMRI and a novel intersubject correlation analysis to track the spread of information along the processing hierarchy as a function of task. Processing the unattended spoken (written) information was confined to auditory (visual) cortices. In contrast, attending to the spoken (written) story enhanced the stimulus-selective responses in sensory regions and allowed it to spread into higher-order areas. Surprisingly, we found that the story-specific spoken (written) responses for the attended story also reached secondary visual (auditory) regions of the unattended sensory modality. These results demonstrate how attention enhances the processing of attended input and allows it to propagate across brain areas.

Leveraging shared connectivity to aggregate heterogeneous datasets into a common response space

Connectivity hyperalignment can be used to estimate a single shared response space across disjoint datasets. We develop a connectivity-based shared response model that factorizes aggregated fMRI datasets into a single reduced-dimension shared connectivity space and subject-specific topographic transformations. These transformations resolve idiosyncratic functional topographies and can be used to project response time series into shared space. We evaluate this algorithm on a large collection of heterogeneous, naturalistic fMRI datasets acquired while subjects listened to spoken stories. Projecting subject data into shared space dramatically improves between-subject story time-segment classification and increases the dimensionality of shared information across subjects. This improvement generalizes to subjects and stories excluded when estimating the shared space. We demonstrate that estimating a simple semantic encoding model in shared space improves between-subject forward encoding and inverted encoding model performance. The shared space estimated across all datasets is distinct from the shared space derived from any particular constituent dataset; the algorithm leverages shared connectivity to yield a consensus shared space conjoining diverse story stimuli.

Direct Fit to Nature: An Evolutionary Perspective on Biological and Artificial Neural Networks

Evolution is a blind fitting process by which organisms become adapted to their environment. Does the brain use similar brute-force fitting processes to learn how to perceive and act upon the world? Recent advances in artificial neural networks have exposed the power of optimizing millions of synaptic weights over millions of observations to operate robustly in real-world contexts. These models do not learn simple, human-interpretable rules or representations of the world; rather, they use local computations to interpolate over task-relevant manifolds in a high-dimensional parameter space. Counterintuitively, similar to evolutionary processes, over-parameterized models can be simple and parsimonious, as they provide a versatile, robust solution for learning a diverse set of functions. This new family of direct-fit models present a radical challenge to many of the theoretical assumptions in psychology and neuroscience. At the same time, this shift in perspective establishes unexpected links with developmental and ecological psychology.

Infant and Adult Brains Are Coupled to the Dynamics of Natural Communication

Infancy is the foundational period for learning from adults, and the dynamics of the social environment have long been considered central to children's development. Here, we reveal a novel, naturalistic approach for studying live interactions between infants and adults. Using functional near-infrared spectroscopy (fNIRS), we simultaneously and continuously measured the brains of infants (N = 18; 9-15 months of age) and an adult while they communicated and played with each other. We found that time-locked neural coupling within dyads was significantly greater when dyad members interacted with each other than with control individuals. In addition, we characterized the dynamic relationship between neural activation and the moment-to-moment fluctuations of mutual gaze, joint attention to objects, infant emotion, and adult speech prosody. This investigation advances what is currently known about how the brains and behaviors of infants both shape and reflect those of adults during real-life communication.

Measuring shared responses across subjects using intersubject correlation

Our capacity to jointly represent information about the world underpins our social experience. By leveraging one individual's brain activity to model another's, we can measure shared information across brains-even in dynamic, naturalistic scenarios where an explicit response model may be unobtainable. Introducing experimental manipulations allows us to measure, for example, shared responses between speakers and listeners or between perception and recall. In this tutorial, we develop the logic of intersubject correlation (ISC) analysis and discuss the family of neuroscientific questions that stem from this approach. We also extend this logic to spatially distributed response patterns and functional network estimation. We provide a thorough and accessible treatment of methodological considerations specific to ISC analysis and outline best practices.

Predictions as a window into learning: Anticipatory fixation offsets carry more information about environmental statistics than reactive stimulus-responses

A core question underlying neurobiological and computational models of behavior is how individuals learn environmental statistics and use them to make predictions. Most investigations of this issue have relied on reactive paradigms, in which inferences about predictive processes are derived by modeling responses to stimuli that vary in likelihood. Here we deployed a novel anticipatory oculomotor metric to determine how input statistics impact anticipatory behavior that is decoupled from target-driven-response. We implemented transition constraints between target locations, so that the probability of a target being presented on the same side as the previous trial was 70% in one condition (pret70) and 30% in the other (pret30). Rather than focus on responses to targets, we studied subtle endogenous anticipatory fixation offsets (AFOs) measured while participants fixated the screen center, awaiting a target. These AFOs were small (<0.4° from center on average), but strongly tracked global-level statistics. Speaking to learning dynamics, trial-by-trial fluctuations in AFO were well-described by a learning model, which identified a lower learning rate in pret70 than pret30, corroborating prior suggestions that pret70 is subjectively treated as more regular. Most importantly, direct comparisons with saccade latencies revealed that AFOs: (a) reflected similar temporal integration windows, (b) carried more information about the statistical context than did saccade latencies, and (c) accounted for most of the information that saccade latencies also contained about inputs statistics. Our work demonstrates how strictly predictive processes reflect learning dynamics, and presents a new direction for studying learning and prediction.

Emerging Opportunities for Advancing Cognitive Neuroscience

Cognitive neuroscience can be substantially advanced if structured mechanisms are created to increase its social impact and to develop synergies with some currently more distant disciplines that are developing relevant knowledge. We present such opportunities and argue that pursuing these can benefit from establishing a centralized coordinating organizational approach.

Elucidating relations between fMRI, ECoG, and EEG through a common natural stimulus

Human brain mapping relies heavily on fMRI, ECoG and EEG, which capture different physiological signals. Relationships between these signals have been established in the context of specific tasks or during resting state, often using spatially confined concurrent recordings in animals. But it is not certain whether these correlations generalize to other contexts relevant for human cognitive neuroscience. Here, we address the case of complex naturalistic stimuli and ask two basic questions. First, how reliable are the responses evoked by a naturalistic audio-visual stimulus in each of these imaging methods, and second, how similar are stimulus-related responses across methods? To this end, we investigated a wide range of brain regions and frequency bands. We presented the same movie clip twice to three different cohorts of subjects (NEEG = 45, NfMRI = 11, NECoG = 5) and assessed stimulus-driven correlations across viewings and between imaging methods, thereby ruling out task-irrelevant confounds. All three imaging methods had similar repeat-reliability across viewings when fMRI and EEG data were averaged across subjects, highlighting the potential to achieve large signal-to-noise ratio by leveraging large sample sizes. The fMRI signal correlated positively with high-frequency ECoG power across multiple task-related cortical structures but positively with low-frequency EEG and ECoG power. In contrast to previous studies, these correlations were as strong for low-frequency as for high frequency ECoG. We also observed links between fMRI and infra-slow EEG voltage fluctuations. These results extend previous findings to the case of natural stimulus processing.

Shared understanding of narratives is correlated with shared neural responses

Humans have a striking ability to infer meaning from even the sparsest and most abstract forms of narratives. At the same time, flexibility in the form of a narrative is matched by inherent ambiguity in its interpretation. How does the brain represent subtle, idiosyncratic differences in the interpretation of abstract and ambiguous narratives? In this fMRI study, subjects were scanned either watching a novel 7-min animation depicting a complex narrative through the movement of geometric shapes, or listening to a narration of the animation's social story. Using an intersubject representational similarity analysis that compared interpretation similarity and neural similarity across subjects, we found that the more similar two people's interpretations of the abstract shapes animation were, the more similar were their neural responses in regions of the default mode network (DMN) and fronto-parietal network. Moreover, these shared responses were modality invariant: the shapes movie and the verbal interpretation of the movie elicited shared responses in linguistic areas and a subset of the DMN when subjects shared interpretations. Together, these results suggest a network of high-level regions that are not only sensitive to subtle individual differences in narrative interpretation during naturalistic conditions, but also resilient to large differences in the modality of the narrative.

Grounding the neurobiology of language in first principles: The necessity of non-language-centric explanations for language comprehension

Recent decades have ushered in tremendous progress in understanding the neural basis of language. Most of our current knowledge on language and the brain, however, is derived from lab-based experiments that are far removed from everyday language use, and that are inspired by questions originating in linguistic and psycholinguistic contexts. In this paper we argue that in order to make progress, the field needs to shift its focus to understanding the neurobiology of naturalistic language comprehension. We present here a new conceptual framework for understanding the neurobiological organization of language comprehension. This framework is non-language-centered in the computational/neurobiological constructs it identifies, and focuses strongly on context. Our core arguments address three general issues: (i) the difficulty in extending language-centric explanations to discourse; (ii) the necessity of taking context as a serious topic of study, modeling it formally and acknowledging the limitations on external validity when studying language comprehension outside context; and (iii) the tenuous status of the language network as an explanatory construct. We argue that adopting this framework means that neurobiological studies of language will be less focused on identifying correlations between brain activity patterns and mechanisms postulated by psycholinguistic theories. Instead, they will be less self-referential and increasingly more inclined towards integration of language with other cognitive systems, ultimately doing more justice to the neurobiological organization of language and how it supports language as it is used in everyday life.

Learning Naturalistic Temporal Structure in the Posterior Medial Network

The posterior medial network is at the apex of a temporal integration hierarchy in the brain, integrating information over many seconds of viewing intact, but not scrambled, movies. This has been interpreted as an effect of temporal structure. Such structure in movies depends on preexisting event schemas, but temporal structure can also arise de novo from learning. Here, we examined the relative role of schema-consistent temporal structure and arbitrary but consistent temporal structure on the human posterior medial network. We tested whether, with repeated viewing, the network becomes engaged by scrambled movies with temporal structure. Replicating prior studies, activity in posterior medial regions was immediately locked to stimulus structure upon exposure to intact, but not scrambled, movies. However, for temporally structured scrambled movies, functional coupling within the network increased across stimulus repetitions, rising to the level of intact movies. Thus, temporal structure is a key determinant of network dynamics and function in the posterior medial network.

How We Transmit Memories to Other Brains: Constructing Shared Neural Representations Via Communication

Humans are able to mentally construct an episode when listening to another person's recollection, even though they themselves did not experience the events. However, it is unknown how strongly the neural patterns elicited by mental construction resemble those found in the brain of the individual who experienced the original events. Using fMRI and a verbal communication task, we traced how neural patterns associated with viewing specific scenes in a movie are encoded, recalled, and then transferred to a group of naïve listeners. By comparing neural patterns across the 3 conditions, we report, for the first time, that event-specific neural patterns observed in the default mode network are shared across the encoding, recall, and construction of the same real-life episode. This study uncovers the intimate correspondences between memory encoding and event construction, and highlights the essential role our common language plays in the process of transmitting one's memories to other brains.

Task-induced deactivation in diverse brain systems correlates with interindividual differences in distinct autonomic indices

Neuroimaging research has shown that different cognitive tasks induce relatively specific activation patterns, as well as less task-specific deactivation patterns. Here we examined whether individual differences in Autonomic Nervous System (ANS) activity during task performance correlate with the magnitude of task-induced deactivation. In an fMRI study, participants performed a continuous mental arithmetic task in a task/rest block design, while undergoing combined fMRI and heart/respiration rate acquisitions using photoplethysmograph and respiration belt. As expected, task performance increased heart-rate and reduced the RMSSD, a cardiac index related to vagal tone. Across participants, higher heart rate during task was linked to increased activation in fronto-parietal regions, as well as to stronger deactivation in ventromedial prefrontal regions. Respiration frequency during task was associated with similar patterns, but in different regions than those identified for heart-rate. Finally, in a large set of regions, almost exclusively limited to the Default Mode Network, lower RMSSD was associated with greater deactivation, and furthermore, the vast majority of these regions were task-deactivated at the group level. Together, our findings show that inter-individual differences in ANS activity are strongly linked to task-induced deactivation. Importantly, our findings suggest that deactivation is a multifaceted construct potentially linked to ANS control, because distinct ANS measures correlate with deactivation in different regions. We discuss the implications for current theories of cortical control of the ANS and for accounts of deactivation, with particular reference to studies documenting a "failure to deactivate" in multiple clinical states.

The Role of Working Memory in the Probabilistic Inference of Future Sensory Events

The ability to represent the emerging regularity of sensory information from the external environment has been thought to allow one to probabilistically infer future sensory occurrences and thus optimize behavior. However, the underlying neural implementation of this process is still not comprehensively understood. Through a convergence of behavioral and neurophysiological evidence, we establish that the probabilistic inference of future events is critically linked to people's ability to maintain the recent past in working memory. Magnetoencephalography recordings demonstrated that when visual stimuli occurring over an extended time series had a greater statistical regularity, individuals with higher working-memory capacity (WMC) displayed enhanced slow-wave neural oscillations in the θ frequency band (4-8 Hz.) prior to, but not during stimulus appearance. This prestimulus neural activity was specifically linked to contexts where information could be anticipated and influenced the preferential sensory processing for this visual information after its appearance. A separate behavioral study demonstrated that this process intrinsically emerges during continuous perception and underpins a realistic advantage for efficient behavioral responses. In this way, WMC optimizes the anticipation of higher level semantic concepts expected to occur in the near future.

A combinatorial framework to quantify peak/pit asymmetries in complex dynamics

We explore a combinatorial framework which efficiently quantifies the asymmetries between minima and maxima in local fluctuations of time series. We first showcase its performance by applying it to a battery of synthetic cases. We find rigorous results on some canonical dynamical models (stochastic processes with and without correlations, chaotic processes) complemented by extensive numerical simulations for a range of processes which indicate that the methodology correctly distinguishes different complex dynamics and outperforms state of the art metrics in several cases. Subsequently, we apply this methodology to real-world problems emerging across several disciplines including cases in neurobiology, finance and climate science. We conclude that differences between the statistics of local maxima and local minima in time series are highly informative of the complex underlying dynamics and a graph-theoretic extraction procedure allows to use these features for statistical learning purposes.

A probabilistic approach to discovering dynamic full-brain functional connectivity patterns

Recent research shows that the covariance structure of functional magnetic resonance imaging (fMRI) data - commonly described as functional connectivity - can change as a function of the participant's cognitive state (for review see Turk-Browne, 2013). Here we present a Bayesian hierarchical matrix factorization model, termed hierarchical topographic factor analysis (HTFA), for efficiently discovering full-brain networks in large multi-subject neuroimaging datasets. HTFA approximates each subject's network by first re-representing each brain image in terms of the activities of a set of localized nodes, and then computing the covariance of the activity time series of these nodes. The number of nodes, along with their locations, sizes, and activities (over time) are learned from the data. Because the number of nodes is typically substantially smaller than the number of fMRI voxels, HTFA can be orders of magnitude more efficient than traditional voxel-based functional connectivity approaches. In one case study, we show that HTFA recovers the known connectivity patterns underlying a collection of synthetic datasets. In a second case study, we illustrate how HTFA may be used to discover dynamic full-brain activity and connectivity patterns in real fMRI data, collected as participants listened to a story. In a third case study, we carried out a similar series of analyses on fMRI data collected as participants viewed an episode of a television show. In these latter case studies, we found that the HTFA-derived activity and connectivity patterns can be used to reliably decode which moments in the story or show the participants were experiencing. Further, we found that these two classes of patterns contained partially non-overlapping information, such that decoders trained on combinations of activity-based and dynamic connectivity-based features performed better than decoders trained on activity or connectivity patterns alone. We replicated this latter result with two additional (previously developed) methods for efficiently characterizing full-brain activity and connectivity patterns.

Method for retrospective estimation of natural head movement during structural MRI

BACKGROUND

Head motion during brain structural MRI scans biases brain morphometry measurements but quantitative retrospective methods estimating head motion from structural MRI have not been evaluated.

PURPOSE

To verify the hypothesis that two metrics retrospectively computed from MR images: 1) average edge strength (AES, reduced with image blurring) and 2) entropy (ENT, increased with blurring and ringing artifacts) could be sensitive to in-scanner head motion during acquisition of T₁ -weighted MR images.

STUDY TYPE

Retrospective.

POPULATION/SUBJECTS/PHANTOM/SPECIMEN/ANIMAL MODEL

In all, 83 healthy control (HC) and 120 Parkinson's disease (PD) patients.

FIELD STRENGTH/SEQUENCE

3D magnetization-prepared rapid gradient-echo (MPRAGE) images at 3T.

ASSESSMENT

We 1) compared AES and ENT distribution between HC and PD; 2) evaluated the correlation between tremor score (TS) and AES (or ENT) in PD; and 3) investigated cortical regions showing an association between AES (or ENT) and local and network-level covariance measures of cortical thickness (CT), gray to white matter contrast (GWC) and gray matter density maps (GMx).

STATISTICAL TESTS

1) Student's t-test. 2) Spearman's rank correlation. 3) General linear model and partial least square analysis.

RESULTS

AES, but not ENT, differentiated HC and PD (P = 0.02, HC median AES = 39.8, interquartile range = 9.8, PD median AES = 37.6, interquartile range = 8.1). In PD, AES correlated negatively with TS (ρ = -0.21, P = 0.02) and showed a significant relationship (|Z| >3, P < 0.001) with structural covariance of CT and GWC in 54 out of 68 cortical regions.

DATA CONCLUSION

In clinical populations prone to head motion, AES can provide a reliable retrospective index of motion during structural scans, identifying brain areas whose morphometric measures covary with motion.

LEVEL OF EVIDENCE

3 Technical Efficacy: Stage 2 J. Magn. Reson. Imaging 2018;48:927-937.

Altered topology of neural circuits in congenital prosopagnosia

Using a novel, fMRI-based inter-subject functional correlation (ISFC) approach, which isolates stimulus-locked inter-regional correlation patterns, we compared the cortical topology of the neural circuit for face processing in participants with an impairment in face recognition, congenital prosopagnosia (CP), and matched controls. Whereas the anterior temporal lobe served as the major network hub for face processing in controls, this was not the case for the CPs. Instead, this group evinced hyper-connectivity in posterior regions of the visual cortex, mostly associated with the lateral occipital and the inferior temporal cortices. Moreover, the extent of this hyper-connectivity was correlated with the face recognition deficit. These results offer new insights into the perturbed cortical topology in CP, which may serve as the underlying neural basis of the behavioral deficits typical of this disorder. The approach adopted here has the potential to uncover altered topologies in other neurodevelopmental disorders, as well.

DOI:http://dx.doi.org/10.7554/eLife.25069.001

Human babies prefer to look at faces and pictures of faces over any other object or pattern. A recent study found that even fetuses in the womb will turn their heads towards dots of light shone through the mother’s skin if the dots broadly resemble a face. Brain imaging studies show that face recognition depends on the coordinated activity of multiple brain regions. A core set of areas towards the back of the brain processes the visual features of faces, while regions elsewhere process more variable features such as emotional expressions.

Around 2% of people are born with difficulties in recognizing faces, a condition known as congenital prosopagnosia. These individuals have no obvious anatomical abnormalities in the brain, and brain scans reveal normal activity in core regions of the face processing network. So why do these people have difficulty with face recognition?

One possibility is that the condition reflects differences in the number of connections (or “connectivity”) between brain regions within the face processing network. To test this idea, Rosenthal et al. compared connectivity in individuals with congenital prosopagnosia with that in healthy volunteers. In the healthy volunteers, an area of the network called the anterior temporal cortex was highly connected to many other face processing regions: that is, it acted as a face processing hub. In individuals with congenital prosopagnosia, this hub-like connectivity was missing. Instead, a number of core regions involved in processing the basic visual features of faces, were more highly connected to one another. The greater this “hyperconnectivity”, the better the individual’s face processing abilities.

The findings of Rosenthal et al. pave the way for developing imaging-based tools to diagnose congenital prosopagnosia. The same approach could then be used to investigate the basis of other neurodevelopmental disorders that are thought to involve abnormal communication within brain networks, such as developmental dyslexia.

DOI:http://dx.doi.org/10.7554/eLife.25069.002

Computational approaches to fMRI analysis

Analysis methods in cognitive neuroscience have not always matched the richness of fMRI data. Early methods focused on estimating neural activity within individual voxels or regions, averaged over trials or blocks and modeled separately in each participant. This approach mostly neglected the distributed nature of neural representations over voxels, the continuous dynamics of neural activity during tasks, the statistical benefits of performing joint inference over multiple participants and the value of using predictive models to constrain analysis. Several recent exploratory and theory-driven methods have begun to pursue these opportunities. These methods highlight the importance of computational techniques in fMRI analysis, especially machine learning, algorithmic optimization and parallel computing. Adoption of these techniques is enabling a new generation of experiments and analyses that could transform our understanding of some of the most complex-and distinctly human-signals in the brain: acts of cognition such as thoughts, intentions and memories.

Structural neuroplasticity of the superior temporal plane in early and late blindness

Blindness is associated with well-documented changes to the morphometry and function of the occipital cortex. By comparison, its impact on the perisylvian regions in the superior temporal plane (STP) is poorly understood, with many studies reporting null findings on this issue. Here we re-approach this question using a morphometric analysis that relied on fine-scale, manual annotation of 13 sub-regions within the STP and that quantified both univariate and multivariate differences in morphometry. We applied these analyses to both cortical thickness (CT) and surface area (SA) data from congenitally and late blind, as compared to two matched sighted control groups. The univariate analyses indicated that for CT, no region differentiated blind from sighted, and for SA, two regions showed lower values for congenitally blind. Moreover, the multivariate analyses identified more robust signatures of plasticity in blindness. Specifically, pairwise regional correlations of CT values between contralateral regions were significantly higher for both blind groups as compared to sighted controls. A similar pattern for SA data was found for congenitally blind alone. Our findings indicate that blindness strongly impacts STP, resulting in a more coordinated pattern of interhemispheric morphometric development. We discuss implications for theories of language plasticity and models of neuroplasticity in the blind.

Mapping between fMRI responses to movies and their natural language annotations

Several research groups have shown how to map fMRI responses to the meanings of presented stimuli. This paper presents new methods for doing so when only a natural language annotation is available as the description of the stimulus. We study fMRI data gathered from subjects watching an episode of BBCs Sherlock (Chen et al., 2017), and learn bidirectional mappings between fMRI responses and natural language representations. By leveraging data from multiple subjects watching the same movie, we were able to perform scene classification with 72% accuracy (random guessing would give 4%) and scene ranking with average rank in the top 4% (random guessing would give 50%). The key ingredients underlying this high level of performance are (a) the use of the Shared Response Model (SRM) and its variant SRM-ICA (Chen et al., 2015; Zhang et al., 2016) to aggregate fMRI data from multiple subjects, both of which are shown to be superior to standard PCA in producing low-dimensional representations for the tasks in this paper; (b) a sentence embedding technique adapted from the natural language processing (NLP) literature (Arora et al., 2017) that produces semantic vector representation of the annotations; (c) using previous timestep information in the featurization of the predictor data. These optimizations in how we featurize the fMRI data and text annotations provide a substantial improvement in classification performance, relative to standard approaches.

The neurobiology of uncertainty: implications for statistical learning

The capacity for assessing the degree of uncertainty in the environment relies on estimating statistics of temporally unfolding inputs. This, in turn, allows calibration of predictive and bottom-up processing, and signalling changes in temporally unfolding environmental features. In the last decade, several studies have examined how the brain codes for and responds to input uncertainty. Initial neurobiological experiments implicated frontoparietal and hippocampal systems, based largely on paradigms that manipulated distributional features of visual stimuli. However, later work in the auditory domain pointed to different systems, whose activation profiles have interesting implications for computational and neurobiological models of statistical learning (SL). This review begins by briefly recapping the historical development of ideas pertaining to the sensitivity to uncertainty in temporally unfolding inputs. It then discusses several issues at the interface of studies of uncertainty and SL. Following, it presents several current treatments of the neurobiology of uncertainty and reviews recent findings that point to principles that serve as important constraints on future neurobiological theories of uncertainty, and relatedly, SL. This review suggests it may be useful to establish closer links between neurobiological research on uncertainty and SL, considering particularly mechanisms sensitive to local and global structure in inputs, the degree of input uncertainty, the complexity of the system generating the input, learning mechanisms that operate on different temporal scales and the use of learnt information for online prediction.This article is part of the themed issue 'New frontiers for statistical learning in the cognitive sciences'.