BrainIAK tutorials: User-friendly learning materials for advanced fMRI analysis

Suppressing the Maintenance of Information in Working Memory Alters Long-term Memory Traces

The sensory recruitment hypothesis conceptualizes information in working memory as being activated representations of information in long-term memory. Accordingly, changes made to an item in working memory would be expected to influence its subsequent retention. Here, we tested the hypothesis that suppressing information from working memory, which can reduce short-term access to that information, may also alter its long-term neural representation. We obtained fMRI data (n = 25; 13 female / 12 male participants) while participants completed a working memory removal task with scene images as stimuli, followed by a final surprise recognition test of the examined items. We applied a multivariate pattern analysis to the data to quantify the engagement of suppression on each trial, to track the contents of working memory during suppression, and to assess representational changes afterward. Our analysis confirms previous reports that suppression of information in working memory involves focused attention to target and remove unwanted information. Furthermore, our findings provide new evidence that even a single dose of suppression of an item in working memory can (if engaged with sufficient strength) produce lasting changes in its neural representation, particularly weakening the unique, item-specific features, which leads to forgetting. Our study sheds light on the underlying mechanisms that contribute to the suppression of unwanted thoughts and highlights the dynamic interplay between working memory and long-term memory.

Auditory areas are recruited for naturalistic visual meaning in early deaf people

Congenital deafness enhances responses of auditory cortices to non-auditory tasks, yet the nature of the reorganization is not well understood. Here, naturalistic stimuli are used to induce neural synchrony across early deaf and hearing individuals. Participants watch a silent animated film in an intact version and three versions with gradually distorted meaning. Differences between groups are observed in higher-order auditory cortices in all stimuli, with no statistically significant effects in the primary auditory cortex. Comparison between levels of scrambling revealed a heterogeneity of function in secondary auditory areas. Both hemispheres show greater synchrony in the deaf than in the hearing participants for the intact movie and high-level variants. However, only the right hemisphere shows an increased inter-subject synchrony in the deaf people for the low-level movie variants. An event segmentation validates these results: the dynamics of the right secondary auditory cortex in the deaf people consist of shorter-length events with more transitions than the left. Our results reveal how deaf individuals use their auditory cortex to process visual meaning.

Enhanced generalization and specialization of brain representations of semantic knowledge in healthy aging

Aging is often associated with a decrease in cognitive capacities. However, semantic memory appears relatively well preserved in healthy aging. Both behavioral and neuroimaging studies support the view that changes in brain networks contribute to this preservation of semantic cognition. However, little is known about the role of healthy aging in the brain representation of semantic categories. Here we used pattern classification analyses and computational models to examine the neural representations of living and non-living word concepts. The results demonstrate that brain representations of animacy in healthy aging exhibit increased similarity across categories, even across different task contexts. This pattern of results aligns with the neural dedifferentiation hypothesis that proposes that aging is associated with decreased specificity in brain activity patterns and less efficient neural resource allocation. However, the loss in neural specificity for different categories was accompanied by increased dissimilarity of item-based conceptual representations within each category. Taken together, the age-related patterns of increased generalization and specialization in the brain representations of semantic knowledge may reflect a compensatory mechanism that enables a more efficient coding scheme characterized by both compression and sparsity, thereby helping to optimize the limited neural resources and maintain semantic processing in the healthy aging brain.

An In-depth Exploration of the Interplay between fMRI Methods and Theory in Cognitive Neuroscience

Functional magnetic resonance imaging (fMRI) has been a cornerstone of cognitive neuroscience since its invention in the 1990s. The methods that we use for fMRI data analysis allow us to test different theories of the brain, thus different analyses can lead us to different conclusions about how the brain produces cognition. There has been a centuries-long debate about the nature of neural processing, with some theories arguing for functional specialization or localization (e.g., face and scene processing) while other theories suggest that cognition is implemented in distributed representations across many neurons and brain regions. Importantly, these theories have received support via different types of analyses; therefore, having students implement hands-on data analysis to explore the results of different fMRI analyses can allow them to take a firsthand approach to thinking about highly influential theories in cognitive neuroscience. Moreover, these explorations allow students to see that there are not clearcut "right" or "wrong" answers in cognitive neuroscience, rather we effectively instantiate assumptions within our analytical approaches that can lead us to different conclusions. Here, I provide Python code that uses freely available software and data to teach students how to analyze fMRI data using traditional activation analysis and machine-learning-based multivariate pattern analysis (MVPA). Altogether, these resources help teach students about the paramount importance of methodology in shaping our theories of the brain, and I believe they will be helpful for introductory undergraduate courses, graduate-level courses, and as a first analysis for people working in labs that use fMRI.

A machine learning approach towards the differentiation between interoceptive and exteroceptive attention

Interoception, the representation of the body's internal state, plays a central role in emotion, motivation and wellbeing. Interoceptive sensibility, the ability to engage in sustained interoceptive awareness, is particularly relevant for mental health but is exclusively measured via self-report, without methods for objective measurement. We used machine learning to classify interoceptive sensibility by contrasting using data from a randomized control trial of interoceptive training, with functional magnetic resonance imaging assessment before and after an 8-week intervention (N = 44 scans). The neuroimaging paradigm manipulated attention targets (breath vs. visual stimuli) and reporting demands (active reporting vs. passive monitoring). Machine learning achieved high accuracy in distinguishing between interoceptive and exteroceptive attention, both for within-session classification (~80% accuracy) and out-of-sample classification (~70% accuracy), revealing the reliability of the predictions. We then explored the classifier potential for 'reading out' mental states in a 3-min sustained interoceptive attention task. Participants were classified as actively engaged about half of the time, during which interoceptive training enhanced their ability to sustain interoceptive attention. These findings demonstrate that interoceptive and exteroceptive attention is distinguishable at the neural level; these classifiers may help to demarcate periods of interoceptive focus, with implications for developing an objective marker for interoceptive sensibility in mental health research.

Neither Enhanced Nor Lost: The Unique Role of Attention in Children's Neural Representations

A defining feature of children's cognition is the especially slow development of their attention. Despite a rich behavioral literature characterizing the development of attention, little is known about how developing attentional abilities modulate neural representations in children. This information is critical to understanding how attentional development shapes the way children process information. One possibility is that attention might be less likely to shape neural representations in children as compared with adults. In particular, representations of attended items may be less likely to be enhanced relative to unattended items. To investigate this possibility, we measured brain activity using fMRI while children (seven to nine years; male and female) and adults (21-31 years; male and female) performed a one-back task in which they were directed to attend to either motion direction or an object in a display where both were present. We used multivoxel pattern analysis to compare decoding accuracy of attended and unattended information. Consistent with attentional enhancement, we found higher decoding accuracy for task-relevant information (i.e., objects in the object-attended condition) than for task-irrelevant information (i.e., motion in the object-attended condition) in adults' visual cortices. However, in children's visual cortices, both task-relevant and task-irrelevant information were decoded equally well. What is more, whole-brain analysis showed that the children represented task-irrelevant information more than adults in multiple regions across the brain, including the prefrontal cortex. These findings show that (1) attention does not modulate neural representations in the child visual cortex, and (2) developing brains can, and do, represent more information than mature brains.SIGNIFICANCE STATEMENT Children have been shown to struggle with maintaining their attention to specific information, and at the same time, can show better learning of "distractors." While these are critical properties of childhood, their underlying neural mechanisms are unknown. To fill in this critical knowledge gap, we explored how attention shapes what is represented in children's and adults' brains using fMRI while both were asked to focus on just one of two things (objects and motion). We found that unlike adults, who prioritize the information they were asked to focus on, children represent both what they were asked to prioritize and what they were asked to ignore. This shows that attention has a fundamentally different impact on children's neural representations.

Detection of autism spectrum disorder using graph representation learning algorithms and deep neural network, based on fMRI signals

Introduction

Can we apply graph representation learning algorithms to identify autism spectrum disorder (ASD) patients within a large brain imaging dataset? ASD is mainly identified by brain functional connectivity patterns. Attempts to unveil the common neural patterns emerged in ASD are the essence of ASD classification. We claim that graph representation learning methods can appropriately extract the connectivity patterns of the brain, in such a way that the method can be generalized to every recording condition, and phenotypical information of subjects. These methods can capture the whole structure of the brain, both local and global properties.

Methods

The investigation is done for the worldwide brain imaging multi-site database known as ABIDE I and II (Autism Brain Imaging Data Exchange). Among different graph representation techniques, we used AWE, Node2vec, Struct2vec, multi node2vec, and Graph2Img. The best approach was Graph2Img, in which after extracting the feature vectors representative of the brain nodes, the PCA algorithm is applied to the matrix of feature vectors. The classifier adapted to the features embedded in graphs is an LeNet deep neural network.

Results and discussion

Although we could not outperform the previous accuracy of 10-fold cross-validation in the identification of ASD versus control patients in this dataset, for leave-one-site-out cross-validation, we could obtain better results (our accuracy: 80%). The result is that graph embedding methods can prepare the connectivity matrix more suitable for applying to a deep network.

Neural event segmentation of continuous experience in human infants

How infants experience the world is fundamental to understanding their cognition and development. A key principle of adult experience is that, despite receiving continuous sensory input, we perceive this input as discrete events. Here we investigate such event segmentation in infants and how it differs from adults. Research on event cognition in infants often uses simplified tasks in which (adult) experimenters help solve the segmentation problem for infants by defining event boundaries or presenting discrete actions/vignettes. This presupposes which events are experienced by infants and leaves open questions about the principles governing infant segmentation. We take a different, data-driven approach by studying infant event segmentation of continuous input. We collected whole-brain functional MRI (fMRI) data from awake infants (and adults, for comparison) watching a cartoon and used a hidden Markov model to identify event states in the brain. We quantified the existence, timescale, and organization of multiple-event representations across brain regions. The adult brain exhibited a known hierarchical gradient of event timescales, from shorter events in early visual regions to longer events in later visual and associative regions. In contrast, the infant brain represented only longer events, even in early visual regions, with no timescale hierarchy. The boundaries defining these infant events only partially overlapped with boundaries defined from adult brain activity and behavioral judgments. These findings suggest that events are organized differently in infants, with longer timescales and more stable neural patterns, even in sensory regions. This may indicate greater temporal integration and reduced temporal precision during dynamic, naturalistic perception.

Representations of Temporal Community Structure in Hippocampus and Precuneus Predict Inductive Reasoning Decisions

Our understanding of the world is shaped by inferences about underlying structure. For example, at the gym, you might notice that the same people tend to arrive around the same time and infer that they are friends that work out together. Consistent with this idea, after participants are presented with a temporal sequence of objects that follows an underlying community structure, they are biased to infer that objects from the same community share the same properties. Here, we used fMRI to measure neural representations of objects after temporal community structure learning and examine how these representations support inference about object relationships. We found that community structure learning affected inferred object similarity: When asked to spatially group items based on their experience, participants tended to group together objects from the same community. Neural representations in perirhinal cortex predicted individual differences in object grouping, suggesting that high-level object representations are affected by temporal community learning. Furthermore, participants were biased to infer that objects from the same community would share the same properties. Using computational modeling of temporal learning and inference decisions, we found that inductive reasoning is influenced by both detailed knowledge of temporal statistics and abstract knowledge of the temporal communities. The fidelity of temporal community representations in hippocampus and precuneus predicted the degree to which temporal community membership biased reasoning decisions. Our results suggest that temporal knowledge is represented at multiple levels of abstraction, and that perirhinal cortex, hippocampus, and precuneus may support inference based on this knowledge.

BrainIAK: The Brain Imaging Analysis Kit

Functional magnetic resonance imaging (fMRI) offers a rich source of data for studying the neural basis of cognition. Here, we describe the Brain Imaging Analysis Kit (BrainIAK), an open-source, free Python package that provides computationally optimized solutions to key problems in advanced fMRI analysis. A variety of techniques are presently included in BrainIAK: intersubject correlation (ISC) and intersubject functional connectivity (ISFC), functional alignment via the shared response model (SRM), full correlation matrix analysis (FCMA), a Bayesian version of representational similarity analysis (BRSA), event segmentation using hidden Markov models, topographic factor analysis (TFA), inverted encoding models (IEMs), an fMRI data simulator that uses noise characteristics from real data (fmrisim), and some emerging methods. These techniques have been optimized to leverage the efficiencies of high-performance compute (HPC) clusters, and the same code can be seamlessly transferred from a laptop to a cluster. For each of the aforementioned techniques, we describe the data analysis problem that the technique is meant to solve and how it solves that problem; we also include an example Jupyter notebook for each technique and an annotated bibliography of papers that have used and/or described that technique. In addition to the sections describing various analysis techniques in BrainIAK, we have included sections describing the future applications of BrainIAK to real-time fMRI, tutorials that we have developed and shared online to facilitate learning the techniques in BrainIAK, computational innovations in BrainIAK, and how to contribute to BrainIAK. We hope that this manuscript helps readers to understand how BrainIAK might be useful in their research.

A neural network model of when to retrieve and encode episodic memories

Recent human behavioral and neuroimaging results suggest that people are selective in when they encode and retrieve episodic memories. To explain these findings, we trained a memory-augmented neural network to use its episodic memory to support prediction of upcoming states in an environment where past situations sometimes reoccur. We found that the network learned to retrieve selectively as a function of several factors, including its uncertainty about the upcoming state. Additionally, we found that selectively encoding episodic memories at the end of an event (but not mid-event) led to better subsequent prediction performance. In all of these cases, the benefits of selective retrieval and encoding can be explained in terms of reducing the risk of retrieving irrelevant memories. Overall, these modeling results provide a resource-rational account of why episodic retrieval and encoding should be selective and lead to several testable predictions.

Real-Time Functional MRI in the Treatment of Mental Health Disorders

Multiple mental disorders have been associated with dysregulation of precise brain processes. However, few therapeutic approaches can correct such specific patterns of brain activity. Since the late 1960s and early 1970s, many researchers have hoped that this feat could be achieved by closed-loop brain imaging approaches, such as neurofeedback, that aim to modulate brain activity directly. However, neurofeedback never gained mainstream acceptance in mental health, in part due to methodological considerations. In this review, we argue that, when contemporary methodological guidelines are followed, neurofeedback is one of the few intervention methods in psychology that can be assessed in double-blind placebo-controlled trials. Furthermore, using new advances in machine learning and statistics, it is now possible to target very precise patterns of brain activity for therapeutic purposes. We review the recent literature in functional magnetic resonance imaging neurofeedback and discuss current and future applications to mental health.

Expected final online publication date for the Annual Review of Clinical Psychology, Volume 18 is May 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Predicting speech from a cortical hierarchy of event-based time scales

How do predictions in the brain incorporate the temporal unfolding of context in our natural environment? We here provide evidence for a neural coding scheme that sparsely updates contextual representations at the boundary of events. This yields a hierarchical, multilayered organization of predictive language comprehension. Training artificial neural networks to predict the next word in a story at five stacked time scales and then using model-based functional magnetic resonance imaging, we observe an event-based “surprisal hierarchy” evolving along a temporoparietal pathway. Along this hierarchy, surprisal at any given time scale gated bottom-up and top-down connectivity to neighboring time scales. In contrast, surprisal derived from continuously updated context influenced temporoparietal activity only at short time scales. Representing context in the form of increasingly coarse events constitutes a network architecture for making predictions that is both computationally efficient and contextually diverse.

Predicting speech from a cortical hierarchy of event-based time scales

How do predictions in the brain incorporate the temporal unfolding of context in our natural environment? We here provide evidence for a neural coding scheme that sparsely updates contextual representations at the boundary of events. This yields a hierarchical, multilayered organization of predictive language comprehension. Training artificial neural networks to predict the next word in a story at five stacked time scales and then using model-based functional magnetic resonance imaging, we observe an event-based “surprisal hierarchy” evolving along a temporoparietal pathway. Along this hierarchy, surprisal at any given time scale gated bottom-up and top-down connectivity to neighboring time scales. In contrast, surprisal derived from continuously updated context influenced temporoparietal activity only at short time scales. Representing context in the form of increasingly coarse events constitutes a network architecture for making predictions that is both computationally efficient and contextually diverse.

The "Narratives" fMRI dataset for evaluating models of naturalistic language comprehension

The "Narratives" collection aggregates a variety of functional MRI datasets collected while human subjects listened to naturalistic spoken stories. The current release includes 345 subjects, 891 functional scans, and 27 diverse stories of varying duration totaling ~4.6 hours of unique stimuli (~43,000 words). This data collection is well-suited for naturalistic neuroimaging analysis, and is intended to serve as a benchmark for models of language and narrative comprehension. We provide standardized MRI data accompanied by rich metadata, preprocessed versions of the data ready for immediate use, and the spoken story stimuli with time-stamped phoneme- and word-level transcripts. All code and data are publicly available with full provenance in keeping with current best practices in transparent and reproducible neuroimaging.

Conducting decoded neurofeedback studies

Closed-loop neurofeedback has sparked great interest since its inception in the late 1960s. However, the field has historically faced various methodological challenges. Decoded fMRI neurofeedback may provide solutions to some of these problems. Notably, thanks to the recent advancements of machine learning approaches, it is now possible to target unconscious occurrences of specific multivoxel representations. In this tools of the trade paper, we discuss how to implement these interventions in rigorous double-blind placebo-controlled experiments. We aim to provide a step-by-step guide to address some of the most common methodological and analytical considerations. We also discuss tools that can be used to facilitate the implementation of new experiments. We hope that this will encourage more researchers to try out this powerful new intervention method.

Unveiling functions of the visual cortex using task-specific deep neural networks

The human visual cortex enables visual perception through a cascade of hierarchical computations in cortical regions with distinct functionalities. Here, we introduce an AI-driven approach to discover the functional mapping of the visual cortex. We related human brain responses to scene images measured with functional MRI (fMRI) systematically to a diverse set of deep neural networks (DNNs) optimized to perform different scene perception tasks. We found a structured mapping between DNN tasks and brain regions along the ventral and dorsal visual streams. Low-level visual tasks mapped onto early brain regions, 3-dimensional scene perception tasks mapped onto the dorsal stream, and semantic tasks mapped onto the ventral stream. This mapping was of high fidelity, with more than 60% of the explainable variance in nine key regions being explained. Together, our results provide a novel functional mapping of the human visual cortex and demonstrate the power of the computational approach.

The Amsterdam Open MRI Collection, a set of multimodal MRI datasets for individual difference analyses

We present the Amsterdam Open MRI Collection (AOMIC): three datasets with multimodal (3 T) MRI data including structural (T1-weighted), diffusion-weighted, and (resting-state and task-based) functional BOLD MRI data, as well as detailed demographics and psychometric variables from a large set of healthy participants (N = 928, N = 226, and N = 216). Notably, task-based fMRI was collected during various robust paradigms (targeting naturalistic vision, emotion perception, working memory, face perception, cognitive conflict and control, and response inhibition) for which extensively annotated event-files are available. For each dataset and data modality, we provide the data in both raw and preprocessed form (both compliant with the Brain Imaging Data Structure), which were subjected to extensive (automated and manual) quality control. All data is publicly available from the OpenNeuro data sharing platform.

Changes to information in working memory depend on distinct removal operations

Holding information in working memory is essential for cognition, but removing unwanted thoughts is equally important. Here we use multivariate pattern analyses of brain activity to demonstrate the successful manipulation and removal of information from working memory using different strategies including suppressing a specific thought, replacing a thought with a different one, and clearing the mind of all thought. These strategies are supported by distinct brain regions and have differential consequences for allowing new information to be encoded.

Searching through functional space reveals distributed visual, auditory, and semantic coding in the human brain

The extent to which brain functions are localized or distributed is a foundational question in neuroscience. In the human brain, common fMRI methods such as cluster correction, atlas parcellation, and anatomical searchlight are biased by design toward finding localized representations. Here we introduce the functional searchlight approach as an alternative to anatomical searchlight analysis, the most commonly used exploratory multivariate fMRI technique. Functional searchlight removes any anatomical bias by grouping voxels based only on functional similarity and ignoring anatomical proximity. We report evidence that visual and auditory features from deep neural networks and semantic features from a natural language processing model, as well as object representations, are more widely distributed across the brain than previously acknowledged and that functional searchlight can improve model-based similarity and decoding accuracy. This approach provides a new way to evaluate and constrain computational models with brain activity and pushes our understanding of human brain function further along the spectrum from strict modularity toward distributed representation.

Emergence and organization of adult brain function throughout child development

Adult cognitive neuroscience has guided the study of human brain development by identifying regions associated with cognitive functions at maturity. The activity, connectivity, and structure of a region can be compared across ages to characterize the developmental trajectory of the corresponding function. However, developmental differences may reflect both the maturation of the function and also its organization across the brain. That is, a function may be present in children but supported by different brain regions, leading its maturity to be underestimated. Here we test the presence, maturity, and localization of adult functions in children using shared response modeling, a machine learning approach for functional alignment. After learning a lower-dimensional feature space from fMRI activity as adults watched a movie, we translated these shared features into the anatomical brain space of children 3–12 years old. To evaluate functional maturity, we correlated this reconstructed activity with children’s actual fMRI activity as they watched the same movie. We found reliable correlations throughout cortex, even in the youngest children. The strength of the correlation in the precuneus, inferior frontal gyrus, and lateral occipital cortex predicted chronological age. These age-related changes were driven by three types of developmental trajectories: emergence from absence to presence, consistency in anatomical expression, and reorganization from one anatomical region to another. We also found evidence that the processing of pain-related events in the movie underwent reorganization across childhood. This data-driven, naturalistic approach provides a new perspective on the development of functional neuroanatomy throughout childhood.

Behavioral, Physiological, and Neural Signatures of Surprise during Naturalistic Sports Viewing

Surprise signals a discrepancy between past and current beliefs. It is theorized to be linked to affective experiences, the creation of particularly resilient memories, and segmentation of the flow of experience into discrete perceived events. However, the ability to precisely measure naturalistic surprise has remained elusive. We used advanced basketball analytics to derive a quantitative measure of surprise and characterized its behavioral, physiological, and neural correlates in human subjects observing basketball games. We found that surprise was associated with segmentation of ongoing experiences, as reflected by subjectively perceived event boundaries and shifts in neocortical patterns underlying belief states. Interestingly, these effects differed by whether surprising moments contradicted or bolstered current predominant beliefs. Surprise also positively correlated with pupil dilation, activation in subcortical regions associated with dopamine, game enjoyment, and long-term memory. These investigations support key predictions from event segmentation theory and extend theoretical conceptualizations of surprise to real-world contexts.

Incorporating structured assumptions with probabilistic graphical models in fMRI data analysis

With the wide adoption of functional magnetic resonance imaging (fMRI) by cognitive neuroscience researchers, large volumes of brain imaging data have been accumulated in recent years. Aggregating these data to derive scientific insights often faces the challenge that fMRI data are high-dimensional, heterogeneous across people, and noisy. These challenges demand the development of computational tools that are tailored both for the neuroscience questions and for the properties of the data. We review a few recently developed algorithms in various domains of fMRI research: fMRI in naturalistic tasks, analyzing full-brain functional connectivity, pattern classification, inferring representational similarity and modeling structured residuals. These algorithms all tackle the challenges in fMRI similarly: they start by making clear statements of assumptions about neural data and existing domain knowledge, incorporate those assumptions and domain knowledge into probabilistic graphical models, and use those models to estimate properties of interest or latent structures in the data. Such approaches can avoid erroneous findings, reduce the impact of noise, better utilize known properties of the data, and better aggregate data across groups of subjects. With these successful cases, we advocate wider adoption of explicit model construction in cognitive neuroscience. Although we focus on fMRI, the principle illustrated here is generally applicable to brain data of other modalities.

Facilitating open-science with realistic fMRI simulation: validation and application

With advances in methods for collecting and analyzing fMRI data, there is a concurrent need to understand how to reliably evaluate and optimally use these methods. Simulations of fMRI data can aid in both the evaluation of complex designs and the analysis of data. We present fmrisim, a new Python package for standardized, realistic simulation of fMRI data. This package is part of BrainIAK: a recently released open-source Python toolbox for advanced neuroimaging analyses. We describe how to use fmrisim to extract noise properties from real fMRI data and then create a synthetic dataset with matched noise properties and a user-specified signal. We validate the noise generated by fmrisim to show that it can approximate the noise properties of real data. We further show how fmrisim can help researchers find the optimal design in terms of power. The fmrisim package holds promise for improving the design of fMRI experiments, which may facilitate both the pre-registration of such experiments as well as the analysis of fMRI data.