Decoding neural representational spaces using multivariate pattern analysis

Multiple Brain Networks Mediating Stimulus-Pain Relationships in Humans

The brain transforms nociceptive input into a complex pain experience comprised of sensory, affective, motivational, and cognitive components. However, it is still unclear how pain arises from nociceptive input and which brain networks coordinate to generate pain experiences. We introduce a new high-dimensional mediation analysis technique to estimate distributed, network-level patterns that formally mediate the relationship between stimulus intensity and pain. We applied the model to a large-scale analysis of functional magnetic resonance imaging data (N = 284), focusing on brain mediators of the relationship between noxious stimulus intensity and trial-to-trial variation in pain reports. We identify mediators in both traditional nociceptive pathways and in prefrontal, midbrain, striatal, and default-mode regions unrelated to nociception in standard analyses. The whole-brain mediators are specific for pain versus aversive sounds and are organized into five functional networks. Brain mediators predicted pain ratings better than previous brain measures, including the neurologic pain signature (Wager et al. 2013). Our results provide a broader view of the networks underlying pain experience, as well as novel brain targets for interventions.

Hyperalignment of motor cortical areas based on motor imagery during action observation

Multivariate Pattern Analysis (MVPA) has grown in importance due to its capacity to use both coarse and fine scale patterns of brain activity. However, a major limitation of multivariate analysis is the difficulty of aligning features across brains, which makes MVPA a subject specific analysis. Recent work by Haxby et al. (2011) introduced a method called Hyperalignment that explored neural activity in ventral temporal cortex during object recognition and demonstrated the ability to align individual patterns of brain activity into a common high dimensional space to facilitate Between Subject Classification (BSC). Here we examined BSC based on Hyperalignment of motor cortex during a task of motor imagery of three natural actions (lift, knock and throw). To achieve this we collected brain activity during the combined tasks of action observation and motor imagery to a parametric action space containing 25 stick-figure blends of the three natural actions. From these responses we derived Hyperalignment transformation parameters that were used to map subjects’ representational spaces of the motor imagery task in the motor cortex into a common model representational space. Results showed that BSC of the neural response patterns based on Hyperalignment exceeded both BSC based on anatomical alignment as well as a standard Within Subject Classification (WSC) approach. We also found that results were sensitive to the order in which participants entered the Hyperalignment algorithm. These results demonstrate the effectiveness of Hyperalignment to align neural responses across subject in motor cortex to enable BSC of motor imagery.

The function of the hippocampus and middle temporal gyrus in forming new associations and concepts during the processing of novelty and usefulness features in creative designs

Creative thought relies on the reorganization of existing knowledge to generate novel and useful concepts. However, how these new concepts are formed, especially through the processing of novelty and usefulness (which are usually regarded as the key properties of creativity), is not clear. Taking familiar and useful (FU) objects/designs as the starting point or fundamental baseline, we modified them into novel and useless (NS) objects/designs or novel and useful (NU) ones (i.e., truly creative ones) to investigate how the features of novelty and usefulness are processed (processing of novelty: NU minus FU; processing of usefulness: NU minus NS). Specifically, we predicted that the creative integration of novelty and usefulness entails not only the formation of new associations, which could be critically mediated by the hippocampus and adjacent medial temporal lobe (MTL) areas, but also the formation of new concepts or categories, which is supported by the middle temporal gyrus (MTG). We found that both the MTL and the MTG were involved in the processing of novelty and usefulness. The MTG showed distinctive patterns of information processing, reflected by strengthened functional connectivity with the hippocampus to construct new concepts and strengthened functional connectivity with the executive control system to break the boundaries of old concepts. Additionally, participants' subjective evaluations of concept distance showed that the distance between the familiar concept (FU) and the successfully constructed concept (NU) was larger than that between the FU and the unsuccessfully constructed concept (NS), and this pattern was found to correspond to the patterns of their neural representations in the MTG. These findings demonstrate the critical mechanism by which new associations and concepts are formed during novelty and usefulness processing in creative design; this mechanism may be critically mediated by the hippocampus-MTG connection.

Categorical representation from sound and sight in the ventral occipito-temporal cortex of sighted and blind

Is vision necessary for the development of the categorical organization of the Ventral Occipito-Temporal Cortex (VOTC)? We used fMRI to characterize VOTC responses to eight categories presented acoustically in sighted and early blind individuals, and visually in a separate sighted group. We observed that VOTC reliably encodes sound categories in sighted and blind people using a representational structure and connectivity partially similar to the one found in vision. Sound categories were, however, more reliably encoded in the blind than the sighted group, using a representational format closer to the one found in vision. Crucially, VOTC in blind represents the categorical membership of sounds rather than their acoustic features. Our results suggest that sounds trigger categorical responses in the VOTC of congenitally blind and sighted people that partially match the topography and functional profile of the visual response, despite qualitative nuances in the categorical organization of VOTC between modalities and groups.

The world is full of rich and dynamic visual information. To avoid information overload, the human brain groups inputs into categories such as faces, houses, or tools. A part of the brain called the ventral occipito-temporal cortex (VOTC) helps categorize visual information. Specific parts of the VOTC prefer different types of visual input; for example, one part may tend to respond more to faces, whilst another may prefer houses. However, it is not clear how the VOTC characterizes information.

One idea is that similarities between certain types of visual information may drive how information is organized in the VOTC. For example, looking at faces requires using central vision, while looking at houses requires using peripheral vision. Furthermore, all faces have a roundish shape while houses tend to have a more rectangular shape. Another possibility, however, is that the categorization of different inputs cannot be explained just by vision, and is also be driven by higher-level aspects of each category. For instance, how humans use or interact with something may also influence how an input is categorized. If categories are established depending (at least partially) on these higher-level aspects, rather than purely through visual likeness, it is likely that the VOTC would respond similarly to both sounds and images representing these categories.

Now, Mattioni et al. have tested how individuals with and without sight respond to eight different categories of information to find out whether or not categorization is driven purely by visual likeness. Each category was presented to participants using sounds while measuring their brain activity. In addition, a group of participants who could see were also presented with the categories visually. Mattioni et al. then compared what happened in the VOTC of the three groups – sighted people presented with sounds, blind people presented with sounds, and sighted people presented with images – in response to each category.

The experiment revealed that the VOTC organizes both auditory and visual information in a similar way. However, there were more similarities between the way blind people categorized auditory information and how sighted people categorized visual information than between how sighted people categorized each type of input. Mattioni et al. also found that the region of the VOTC that responds to inanimate objects massively overlapped across the three groups, whereas the part of the VOTC that responds to living things was more variable.

These findings suggest that the way that the VOTC organizes information is, at least partly, independent from vision. The experiments also provide some information about how the brain reorganizes in people who are born blind. Further studies may reveal how differences in the VOTC of people with and without sight affect regions typically associated with auditory categorization, and potentially explain how the brain reorganizes in people who become blind later in life.

Multi-time-point analysis: A time course analysis with functional near-infrared spectroscopy

In the data analysis of functional near-infrared spectroscopy (fNIRS), linear model frameworks, in particular mass univariate analysis, are often used when researchers consider examining the difference between conditions at each sampled time point. However, some statistical issues, such as assumptions of linearity, autocorrelation and multiple comparison problems, influence statistical inferences when mass univariate analysis is used on fNIRS time course data. In order to address these issues, the present study proposes a novel perspective, multi-time-point analysis (MTPA), to discriminate signal differences between conditions by combining temporal information from multiple time points in fNIRS. In addition, MTPA adopts the random forest algorithm from the statistical learning domain, followed by a series of cross-validation procedures, providing reasonable power for detecting significant time points and ensuring generalizability. Using a real fNIRS data set, the proposed MTPA outperformed mass univariate analysis in detecting more time points, showing significant differences between experimental conditions. Finally, MTPA was also able to make comparisons between different areas, leading to a novel viewpoint of fNIRS time course analysis and providing additional theoretical implications for future fNIRS studies. The data set and all source code are available for researchers to replicate the analyses and to adapt the program for their own needs in future fNIRS studies.

The empirical replicability of task-based fMRI as a function of sample size

Replicating results (i.e. obtaining consistent results using a new independent dataset) is an essential part of good science. As replicability has consequences for theories derived from empirical studies, it is of utmost importance to better understand the underlying mechanisms influencing it. A popular tool for non-invasive neuroimaging studies is functional magnetic resonance imaging (fMRI). While the effect of underpowered studies is well documented, the empirical assessment of the interplay between sample size and replicability of results for task-based fMRI studies remains limited. In this work, we extend existing work on this assessment in two ways. Firstly, we use a large database of 1400 subjects performing four types of tasks from the IMAGEN project to subsample a series of independent samples of increasing size. Secondly, replicability is evaluated using a multi-dimensional framework consisting of 3 different measures: (un)conditional test-retest reliability, coherence and stability. We demonstrate not only a positive effect of sample size, but also a trade-off between spatial resolution and replicability. When replicability is assessed voxelwise or when observing small areas of activation, a larger sample size than typically used in fMRI is required to replicate results. On the other hand, when focussing on clusters of voxels, we observe a higher replicability. In addition, we observe variability in the size of clusters of activation between experimental paradigms or contrasts of parameter estimates within these.

Decoding dynamic affective responses to naturalistic videos with shared neural patterns

This study explored the feasibility of using shared neural patterns from brief affective episodes (viewing affective pictures) to decode extended, dynamic affective sequences in a naturalistic experience (watching movie-trailers). Twenty-eight participants viewed pictures from the International Affective Picture System (IAPS) and, in a separate session, watched various movie-trailers. We first located voxels at bilateral occipital cortex (LOC) responsive to affective picture categories by GLM analysis, then performed between-subject hyperalignment on the LOC voxels based on their responses during movie-trailer watching. After hyperalignment, we trained between-subject machine learning classifiers on the affective pictures, and used the classifiers to decode affective states of an out-of-sample participant both during picture viewing and during movie-trailer watching. Within participants, neural classifiers identified valence and arousal categories of pictures, and tracked self-reported valence and arousal during video watching. In aggregate, neural classifiers produced valence and arousal time series that tracked the dynamic ratings of the movie-trailers obtained from a separate sample. Our findings provide further support for the possibility of using pre-trained neural representations to decode dynamic affective responses during a naturalistic experience.

The Rapid Emergence of Musical Pitch Structure in Human Cortex

In tonal music, continuous acoustic waveforms are mapped onto discrete, hierarchically arranged, internal representations of pitch. To examine the neural dynamics underlying this transformation, we presented male and female human listeners with tones embedded within a Western tonal context while recording their cortical activity using magnetoencephalography. Machine learning classifiers were then trained to decode different tones from their underlying neural activation patterns at each peristimulus time sample, providing a dynamic measure of their dissimilarity in cortex. Comparing the time-varying dissimilarity between tones with the predictions of acoustic and perceptual models, we observed a temporal evolution in the brain's representational structure. Whereas initial dissimilarities mirrored their fundamental-frequency separation, dissimilarities beyond 200 ms reflected the perceptual status of each tone within the tonal hierarchy of Western music. These effects occurred regardless of stimulus regularities within the context or whether listeners were engaged in a task requiring explicit pitch analysis. Lastly, patterns of cortical activity that discriminated between tones became increasingly stable in time as the information coded by those patterns transitioned from low-to-high level properties. Current results reveal the dynamics with which the complex perceptual structure of Western tonal music emerges in cortex at the timescale of an individual tone.SIGNIFICANCE STATEMENT Little is understood about how the brain transforms an acoustic waveform into the complex perceptual structure of musical pitch. Applying neural decoding techniques to the cortical activity of human subjects engaged in music listening, we measured the dynamics of information processing in the brain on a moment-to-moment basis as subjects heard each tone. In the first 200 ms after onset, transient patterns of neural activity coded the fundamental frequency of tones. Subsequently, a period emerged during which more temporally stable activation patterns coded the perceptual status of each tone within the "tonal hierarchy" of Western music. Our results provide a crucial link between the complex perceptual structure of tonal music and the underlying neural dynamics from which it emerges.

A multimodal neuroimaging classifier for alcohol dependence

With progress in magnetic resonance imaging technology and a broader dissemination of state-of-the-art imaging facilities, the acquisition of multiple neuroimaging modalities is becoming increasingly feasible. One particular hope associated with multimodal neuroimaging is the development of reliable data-driven diagnostic classifiers for psychiatric disorders, yet previous studies have often failed to find a benefit of combining multiple modalities. As a psychiatric disorder with established neurobiological effects at several levels of description, alcohol dependence is particularly well-suited for multimodal classification. To this aim, we developed a multimodal classification scheme and applied it to a rich neuroimaging battery (structural, functional task-based and functional resting-state data) collected in a matched sample of alcohol-dependent patients (N = 119) and controls (N = 97). We found that our classification scheme yielded 79.3% diagnostic accuracy, which outperformed the strongest individual modality – grey-matter density – by 2.7%. We found that this moderate benefit of multimodal classification depended on a number of critical design choices: a procedure to select optimal modality-specific classifiers, a fine-grained ensemble prediction based on cross-modal weight matrices and continuous classifier decision values. We conclude that the combination of multiple neuroimaging modalities is able to moderately improve the accuracy of machine-learning-based diagnostic classification in alcohol dependence.

Predictability Changes What We Remember in Familiar Temporal Contexts

The human brain constantly anticipates the future based on memories of the past. Encountering a familiar situation reactivates memory of previous encounters, which can trigger a prediction of what comes next to facilitate responsiveness. However, a prediction error can lead to pruning of the offending memory, a process that weakens its representation in the brain and leads to forgetting. Our goal in this study was to evaluate whether memories are spared from such pruning in situations that allow for accurate predictions at the categorical level, despite prediction errors at the item level. Participants viewed a sequence of objects, some of which reappeared multiple times ("cues"), followed always by novel items. Half of the cues were followed by new items from different (unpredictable) categories, while others were followed by new items from a single (predictable) category. Pattern classification of fMRI data was used to identify category-specific predictions after each cue. Pruning was observed only in unpredictable contexts, while encoding of new items was less robust in predictable contexts. These findings demonstrate that how associative memories are updated is influenced by the reliability of abstract-level predictions in familiar contexts.

Interpreting neural decoding models using grouped model reliance

Machine learning algorithms are becoming increasingly popular for decoding psychological constructs based on neural data. However, as a step towards bridging the gap between theory-driven cognitive neuroscience and data-driven decoding approaches, there is a need for methods that allow to interpret trained decoding models. The present study demonstrates grouped model reliance as a model-agnostic permutation-based approach to this problem. Grouped model reliance indicates the extent to which a trained model relies on conceptually related groups of variables, such as frequency bands or regions of interest in electroencephalographic (EEG) data. As a case study to demonstrate the method, random forest and support vector machine models were trained on within-participant single-trial EEG data from a Sternberg working memory task. Participants were asked to memorize a sequence of digits (0-9), varying randomly in length between one, four and seven digits, where EEG recordings for working memory load estimation were taken from a 3-second retention interval. The present results confirm previous findings insofar as both random forest and support vector machine models relied on alpha-band activity in most subjects. However, as revealed by further analyses, patterns in frequency and topography varied considerably between individuals, pointing to more pronounced inter-individual differences than previously reported.

Acute stress alters neural patterns of value representation for others

Acute stress is often evoked during social interactions, by feelings of threat or negative evaluation by other people. We also constantly interact with others while under stress - in the workplace or in private alike. However, it is not clear how stress affects social interactions. For one, individuals could become more selfish and focused on their own goals. On the other hand, individuals might also become more focused on affiliating with potential social partners, in order to secure their support. There is, indeed, accumulating behavioral evidence that prosocial behaviors increase rather than decrease under stress. Here, we tested the underlying brain processes of such findings, by assessing the effects of stress on the neural representations of (monetary) value for self and other. Participants (N ​= ​30; male, 18-40 years) played a gambling task for themselves and for another participant while undergoing functional magnetic resonance imaging (fMRI). Each participant played the gambling task twice: once immediately following acute stress induction, and once in a control session. We compared neural patterns of value representation in the dorsomedial prefrontal cortex (dmPFC), ventromedial prefrontal cortex (vmPFC) and striatum using representational similarity analysis (RSA). We found that under stress, dmPFC and striatum showed higher dissimilarity between neural patterns underlying high and low value for the other. Dissimilarity of neural patterns underlying high and low value for the self was unaffected by stress. These findings suggest that participants track the magnitude of possible rewards for others more under stress, suggesting increased prosocial orientation.

Social network proximity predicts similar trajectories of psychological states: Evidence from multi-voxel spatiotemporal dynamics

Homophily is a prevalent characteristic of human social networks: individuals tend to associate and bond with others who are similar to themselves with respect to physical traits and demographic attributes, such as age, gender, and ethnicity. Recent research using functional magnetic resonance imaging has demonstrated a positive relationship between individuals' real-world social network proximity (i.e., whether they are friends, friends-of-friends, or farther removed in social ties) and inter-subject correlation (ISC) in their time series of neural responses when viewing audiovisual movies. However, conventional ISC methods only capture information about similarity in the temporal evolution of region-averaged neural responses, and ignore information carried in fine-grained, spatially distributed response topographies. Here, we demonstrate that temporal trajectories of multi-voxel response patterns to naturalistic stimuli are exceptionally similar among friends and predictive of social network proximity, over and above the effects of response magnitude fluctuations. Furthermore, inter-subject similarity in the temporal trajectory of multi-voxel response patterns across distant points in time was particularly positively associated with individuals' proximity in their real-world social network. The fact that exceptional similarities among friends were most pronounced in long-range temporal fluctuations of response patterns located in multimodal cortical regions (e.g., regions of posterior parietal cortex) suggests that aspects of high-level processing during naturalistic stimulation may be particularly similar among friends. Given the localization of results, we speculate that socially close individuals may be particularly similar in endogenously driven shifts in how they distribute their attention (e.g., across the environment, within internal representations) over time. These results suggest that friends may experience exceptionally similar trajectories of psychological states when exposed to a common stimulus, and, more generally, that there are meaningful individual differences in the temporal evolution of multi-voxel response patterns during naturalistic stimulation.

Invariant representations of mass in the human brain

An intuitive understanding of physical objects and events is critical for successfully interacting with the world. Does the brain achieve this understanding by running simulations in a mental physics engine, which represents variables such as force and mass, or by analyzing patterns of motion without encoding underlying physical quantities? To investigate, we scanned participants with fMRI while they viewed videos of objects interacting in scenarios indicating their mass. Decoding analyses in brain regions previously implicated in intuitive physical inference revealed mass representations that generalized across variations in scenario, material, friction, and motion energy. These invariant representations were found during tasks without action planning, and tasks focusing on an orthogonal dimension (object color). Our results support an account of physical reasoning where abstract physical variables serve as inputs to a forward model of dynamics, akin to a physics engine, in parietal and frontal cortex.

A Double Dissociation in Sensitivity to Verb and Noun Semantics Across Cortical Networks

What is the neural organization of the mental lexicon? Previous research suggests that partially distinct cortical networks are active during verb and noun processing, but what information do these networks represent? We used multivoxel pattern analysis (MVPA) to investigate whether these networks are sensitive to lexicosemantic distinctions among verbs and among nouns and, if so, whether they are more sensitive to distinctions among words in their preferred grammatical class. Participants heard 4 types of verbs (light emission, sound emission, hand-related actions, mouth-related actions) and 4 types of nouns (birds, mammals, manmade places, natural places). As previously shown, the left posterior middle temporal gyrus (LMTG+), and inferior frontal gyrus (LIFG) responded more to verbs, whereas the inferior parietal lobule (LIP), precuneus (LPC), and inferior temporal (LIT) cortex responded more to nouns. MVPA revealed a double-dissociation in lexicosemantic sensitivity: classification was more accurate among verbs than nouns in the LMTG+, and among nouns than verbs in the LIP, LPC, and LIT. However, classification was similar for verbs and nouns in the LIFG, and above chance for the nonpreferred category in all regions. These results suggest that the lexicosemantic information about verbs and nouns is represented in partially nonoverlapping networks.

Mental Effort and Information-Processing Costs Are Inversely Related to Global Brain Free Energy During Visual Categorization

Mental effort is a neurocognitive process that reflects the controlled expenditure of psychological information-processing resources during perception, cognition, and action. There is a practical need to operationalize and measure mental effort in order to minimize detrimental effects of mental fatigue on real-world human performance. Previous research has identified several neurocognitive indices of mental effort, but these indices are indirect measures that are also sensitive to experimental demands or general factors such as sympathetic arousal. The present study investigated a potential direct neurocognitive index of mental effort based in theories where bounded rational decision makers (realized as embodied brains) are modeled as generalized thermodynamic systems. This index is called free energy, an information-theoretic system property of the brain that reflects the difference between the brain's current and predicted states. Theory predicts that task-related differences in a decision makers' free energy are inversely related to information-processing costs related to task decisions. The present study tested this prediction by quantifying global brain free energy from electroencephalographic (EEG) measures of human brain function. EEG signals were recorded while participants engaged in two visual categorization tasks in which categorization decisions resulted from the allocation of different levels of mental information processing resources. A novel method was developed to quantify brain free energy from machine learning classification of EEG trials. Participant information-processing resource costs were estimated via computational analysis of behavior, whereas the subjective expression of mental effort was estimated via participant ratings of mental workload. Following theoretical predictions, task-related differences in brain free energy negatively correlated with increased allocation of information-processing resource costs. These brain free energy differences were smaller for the visual categorization task that required a greater versus lesser allocation of information-processing resources. Ratings of mental workload were positively correlated with information-processing resource costs, and negatively correlated with global brain free energy differences, only for the categorization task requiring the larger amount of information-processing resource costs. These findings support theoretical thermodynamic approaches to decision making and provide the first empirical evidence of a relationship between mental effort, brain free energy, and neurocognitive information-processing.

Faster learners transfer their knowledge better: Behavioral, mnemonic, and neural mechanisms of individual differences in children's learning

Why some children learn, and transfer their knowledge to novel problems, better than others remains an important unresolved question in the science of learning. Here we developed an innovative tutoring program and data analysis approach to investigate individual differences in neurocognitive mechanisms that support math learning and "near" transfer to novel, but structurally related, problems in elementary school children. Following just five days of training, children performed recently trained math problems more efficiently, with greater use of memory-retrieval-based strategies. Crucially, children who learned faster during training performed better not only on trained problems but also on novel problems, and better discriminated trained and novel problems in a subsequent recognition memory task. Faster learners exhibited increased similarity of neural representations between trained and novel problems, and greater differentiation of functional brain circuits engaged by trained and novel problems. These results suggest that learning and near transfer are characterized by parallel learning-rate dependent local integration and large-scale segregation of functional brain circuits. Our findings demonstrate that speed of learning and near transfer are interrelated and identify the neural mechanisms by which faster learners transfer their knowledge better. Our study provides new insights into the behavioral, mnemonic, and neural mechanisms underlying children's learning.

Information Content of Prefrontal Cortex Activity Quantifies the Difficulty of Narrated Stories

The ability to realize the individuals' impressions during the verbal communication allows social robots to significantly facilitate their social interactions in such areas as child education and elderly care. However, such impressions are highly subjective and internalized and therefore cannot be easily comprehended through behavioural observations. Although brain-machine interface suggests the utility of the brain information in human-robot interaction, previous studies did not consider its potential for estimating the internal impressions during verbal communication. In this article, we introduce a novel approach to estimation of the individuals' perceived difficulty of stories using the quantified information content of their prefrontal cortex activity. We demonstrate the robustness of our approach by showing its comparable performance in face-to-face, humanoid, speaker, and video-chat settings. Our results contribute to the field of socially assistive robotics by taking a step toward enabling robots determine their human companions' perceived difficulty of conversations, thereby enabling these media to sustain their communication with humans by adapting to individuals' pace and interest in response to conversational nuances and complexity.

There is no single functional atlas even for a single individual: Functional parcel definitions change with task

The goal of human brain mapping has long been to delineate the functional subunits in the brain and elucidate the functional role of each of these brain regions. Recent work has focused on whole-brain parcellation of functional Magnetic Resonance Imaging (fMRI) data to identify these subunits and create a functional atlas. Functional connectivity approaches to understand the brain at the network level require such an atlas to assess connections between parcels and extract network properties. While no single functional atlas has emerged as the dominant atlas to date, there remains an underlying assumption that such an atlas exists. Using fMRI data from a highly sampled subject as well as two independent replication data sets, we demonstrate that functional parcellations based on fMRI connectivity data reconfigure substantially and in a meaningful manner, according to brain state.

The Effects of Audiovisual Inputs on Solving the Cocktail Party Problem in the Human Brain: An fMRI Study

At cocktail parties, our brains often simultaneously receive visual and auditory information. Although the cocktail party problem has been widely investigated under auditory-only settings, the effects of audiovisual inputs have not. This study explored the effects of audiovisual inputs in a simulated cocktail party. In our fMRI experiment, each congruent audiovisual stimulus was a synthesis of 2 facial movie clips, each of which could be classified into 1 of 2 emotion categories (crying and laughing). Visual-only (faces) and auditory-only stimuli (voices) were created by extracting the visual and auditory contents from the synthesized audiovisual stimuli. Subjects were instructed to selectively attend to 1 of the 2 objects contained in each stimulus and to judge its emotion category in the visual-only, auditory-only, and audiovisual conditions. The neural representations of the emotion features were assessed by calculating decoding accuracy and brain pattern-related reproducibility index based on the fMRI data. We compared the audiovisual condition with the visual-only and auditory-only conditions and found that audiovisual inputs enhanced the neural representations of emotion features of the attended objects instead of the unattended objects. This enhancement might partially explain the benefits of audiovisual inputs for the brain to solve the cocktail party problem.

A Guide to Representational Similarity Analysis for Social Neuroscience

Representational similarity analysis (RSA) is a computational technique that uses pairwise comparisons of stimuli to reveal their representation in higher-order space. In the context of neuroimaging, mass-univariate analyses and other multivariate analyses can provide information on what and where information is represented but have limitations in their ability to address how information is represented. Social neuroscience is a field that can particularly benefit from incorporating RSA techniques to explore hypotheses regarding the representation of multidimensional data, how representations can predict behavior, how representations differ between groups and how multimodal data can be compared to inform theories. The goal of this paper is to provide a practical as well as theoretical guide to implementing RSA in social neuroscience studies.

Representational Organization of Novel Task Sets during Proactive Encoding

Recent multivariate analyses of brain data have boosted our understanding of the organizational principles that shape neural coding. However, most of this progress has focused on perceptual visual regions (Connolly et al., 2012), whereas far less is known about the organization of more abstract, action-oriented representations. In this study, we focused on humans' remarkable ability to turn novel instructions into actions. While previous research shows that instruction encoding is tightly linked to proactive activations in frontoparietal brain regions, little is known about the structure that orchestrates such anticipatory representation. We collected fMRI data while participants (both males and females) followed novel complex verbal rules that varied across control-related variables (integrating within/across stimuli dimensions, response complexity, target category) and reward expectations. Using representational similarity analysis (Kriegeskorte et al., 2008), we explored where in the brain these variables explained the organization of novel task encoding, and whether motivation modulated these representational spaces. Instruction representations in the lateral PFC were structured by the three control-related variables, whereas intraparietal sulcus encoded response complexity and the fusiform gyrus and precuneus organized its activity according to the relevant stimulus category. Reward exerted a general effect, increasing the representational similarity among different instructions, which was robustly correlated with behavioral improvements. Overall, our results highlight the flexibility of proactive task encoding, governed by distinct representational organizations in specific brain regions. They also stress the variability of motivation-control interactions, which appear to be highly dependent on task attributes, such as complexity or novelty.SIGNIFICANCE STATEMENT In comparison with other primates, humans display a remarkable success in novel task contexts thanks to our ability to transform instructions into effective actions. This skill is associated with proactive task-set reconfigurations in frontoparietal cortices. It remains yet unknown, however, how the brain encodes in anticipation the flexible, rich repertoire of novel tasks that we can achieve. Here we explored cognitive control and motivation-related variables that might orchestrate the representational space for novel instructions. Our results showed that different dimensions become relevant for task prospective encoding, depending on the brain region, and that the lateral PFC simultaneously organized task representations following different control-related variables. Motivation exerted a general modulation upon this process, diminishing rather than increasing distances among instruction representations.

Decoding disparity categories in 3-dimensional images from fMRI data using functional connectivity patterns

Humans use binocular disparity to extract depth information from two-dimensional retinal images in a process called stereopsis. Previous studies usually introduce the standard univariate analysis to describe the correlation between disparity level and brain activity within a given brain region based on functional magnetic resonance imaging (fMRI) data. Recently, multivariate pattern analysis has been developed to extract activity patterns across multiple voxels for deciphering categories of binocular disparity. However, the functional connectivity (FC) of patterns based on regions of interest or voxels and their mapping onto disparity category perception remain unknown. The present study extracted functional connectivity patterns for three disparity conditions (crossed disparity, uncrossed disparity, and zero disparity) at distinct spatial scales to decode the binocular disparity. Results of 27 subjects' fMRI data demonstrate that FC features are more discriminatory than traditional voxel activity features in binocular disparity classification. The average binary classification of the whole brain and visual areas are respectively 87% and 79% at single subject level, and thus above the chance level (50%). Our research highlights the importance of exploring functional connectivity patterns to achieve a novel understanding of 3D image processing.

Recurrence is required to capture the representational dynamics of the human visual system

Understanding the computational principles that underlie human vision is a key challenge for neuroscience and could help improve machine vision. Feedforward neural network models process their input through a deep cascade of computations. These models can recognize objects in images and explain aspects of human rapid recognition. However, the human brain contains recurrent connections within and between stages of the cascade, which are missing from the models that dominate both engineering and neuroscience. Here, we measure and model the dynamics of human brain activity during visual perception. We compare feedforward and recurrent neural network models and find that only recurrent models can account for the dynamic transformations of representations among multiple regions of visual cortex.

The human visual system is an intricate network of brain regions that enables us to recognize the world around us. Despite its abundant lateral and feedback connections, object processing is commonly viewed and studied as a feedforward process. Here, we measure and model the rapid representational dynamics across multiple stages of the human ventral stream using time-resolved brain imaging and deep learning. We observe substantial representational transformations during the first 300 ms of processing within and across ventral-stream regions. Categorical divisions emerge in sequence, cascading forward and in reverse across regions, and Granger causality analysis suggests bidirectional information flow between regions. Finally, recurrent deep neural network models clearly outperform parameter-matched feedforward models in terms of their ability to capture the multiregion cortical dynamics. Targeted virtual cooling experiments on the recurrent deep network models further substantiate the importance of their lateral and top-down connections. These results establish that recurrent models are required to understand information processing in the human ventral stream.

Reduced Fidelity of Neural Representation Underlies Episodic Memory Decline in Normal Aging

Emerging studies have emphasized the importance of the fidelity of cortical representation in forming enduring episodic memory. No study, however, has examined whether there are age-related reductions in representation fidelity that can explain memory declines in normal aging. Using functional MRI and multivariate pattern analysis, we found that older adults showed reduced representation fidelity in the visual cortex, which accounted for their decreased memory performance even after controlling for the contribution of reduced activation level. This reduced fidelity was specifically due to older adults' poorer item-specific representation, not due to their lower activation level and variance, greater variability in neuro-vascular coupling, or decreased selectivity of categorical representation (i.e., dedifferentiation). Older adults also showed an enhanced subsequent memory effect in the prefrontal cortex based on activation level, and their prefrontal activation was associated with greater fidelity of representation in the visual cortex and better memory performance. The fidelity of cortical representation thus may serve as a promising neural index for better mechanistic understanding of the memory declines and its compensation in normal aging.

Individualized functional networks reconfigure with cognitive state

There is extensive evidence that functional organization of the human brain varies dynamically as the brain switches between task demands, or cognitive states. This functional organization also varies across subjects, even when engaged in similar tasks. To date, the functional network organization of the brain has been considered static. In this work, we use fMRI data obtained across multiple cognitive states (task-evoked and rest conditions) and across multiple subjects, to measure state- and subject-specific functional network parcellation (the assignment of nodes to networks). Our parcellation approach provides a measure of how node-to-network assignment (NNA) changes across states and across subjects. We demonstrate that the brain's functional networks are not spatially fixed, but that many nodes change their network membership as a function of cognitive state. Such reconfigurations are highly robust and reliable to the extent that they can be used to predict cognitive state with up to 97% accuracy. Our findings suggest that if functional networks are to be defined via functional clustering of nodes, then it is essential to consider that such definitions may be fluid and cognitive-state dependent.

Neural representation of current and intended task sets during sequential judgements on human faces

Engaging in a demanding activity while holding in mind another task to be performed in the near future requires the maintenance of information about both the currently-active task set and the intended one. However, little is known about how the human brain implements such action plans. While some previous studies have examined the neural representation of current task sets and others have investigated delayed intentions, to date none has examined the representation of current and intended task sets within a single experimental paradigm. In this fMRI study, we examined the neural representation of current and intended task sets, employing sequential classification tasks on human faces. Multivariate decoding analyses showed that current task sets were represented in the orbitofrontal cortex (OFC) and fusiform gyrus (FG), while intended tasks could be decoded from lateral prefrontal cortex (lPFC). Importantly, a ventromedial region in PFC/OFC contained information about both current and delayed tasks, although cross-classification between the two types of information was not possible. These results help delineate the neural representations of current and intended task sets, and highlight the importance of ventromedial PFC/OFC for maintaining task-relevant information regardless of when it is needed.

Working memory prioritization impacts neural recovery from distraction

The ability to protect goal-relevant information from disruption over short intervals is a hallmark of working memory. Recent behavioral data suggest that high-priority items in working memory are more vulnerable to disruption. We used functional magnetic resonance imaging to evaluate the hypothesis that prioritization of working memories might impact the recovery of their neural representation(s) after distraction. A delay-period retrospective cue informed participants which of two memory items (a face or a scene) to prioritize during a first delay period. Consistent with prior work, and confirming successful prioritization, multivoxel pattern classifier evidence in perceptual brain regions was higher for cued versus uncued memory items. A distraction task was then imposed before a second retrospective cue informed participants to either "stay" remembering the previously cued item or "switch" to the previously uncued item. This allowed for the evaluation of recovery for high-priority items (on stay trials) and also low-priority items (on switch trials). Classifiers showed successful reinstatement of both high- and low-priority items after distraction, but only low-priority items recovered to their pre-distraction representational levels. Moreover, the degree of prioritization before distraction predicted the amount of disruption for high-priority items after distraction, suggesting that the more a participant prioritized the cued item, the greater the impact of distraction. Our data provide neural evidence that prioritizing working memory information in perceptual regions makes that information more vulnerable to disruption.

Decision-making in sensorimotor control

Skilled sensorimotor interactions with the world result from a series of decision-making processes that determine, on the basis of information extracted during the unfolding sequence of events, which movements to make and when and how to make them. Despite this inherent link between decision-making and sensorimotor control, research into each of these two areas has largely evolved in isolation, and it is only fairly recently that researchers have begun investigating how they interact and, together, influence behaviour. Here, we review recent behavioural, neurophysiological and computational research that highlights the role of decision-making processes in the selection, planning and control of goal-directed movements in humans and nonhuman primates.

Multi-Objective Cognitive Model: a Supervised Approach for Multi-subject fMRI Analysis

In order to decode human brain, Multivariate Pattern (MVP) classification generates cognitive models by using functional Magnetic Resonance Imaging (fMRI) datasets. As a standard pipeline in the MVP analysis, brain patterns in multi-subject fMRI dataset must be mapped to a shared space and then a classification model is generated by employing the mapped patterns. However, the MVP models may not provide stable performance on a new fMRI dataset because the standard pipeline uses disjoint steps for generating these models. Indeed, each step in the pipeline includes an objective function with independent optimization approach, where the best solution of each step may not be optimum for the next steps. For tackling the mentioned issue, this paper introduces Multi-Objective Cognitive Model (MOCM) that utilizes an integrated objective function for MVP analysis rather than just using those disjoint steps. For solving the integrated problem, we proposed a customized multi-objective optimization approach, where all possible solutions are firstly generated, and then our method ranks and selects the robust solutions as the final results. Empirical studies confirm that the proposed method can generate superior performance in comparison with other techniques.

Multivariate approaches improve the reliability and validity of functional connectivity and prediction of individual behaviors

Brain functional connectivity features can predict cognition and behavior at the level of the individual. Most studies measure univariate signals, correlating timecourses from the average of constituent voxels in each node. While straightforward, this approach overlooks the spatial patterns of voxel-wise signals within individual nodes. Given that multivariate spatial activity patterns across voxels can improve fMRI measures of mental representations, here we asked whether using voxel-wise timecourses can better characterize region-by-region interactions relative to univariate approaches. Using two fMRI datasets, the Human Connectome Project sample and a local test-retest sample, we measured multivariate functional connectivity with multivariate distance correlation and univariate connectivity with Pearson's correlation. We compared multivariate and univariate connectivity estimates, demonstrating that relative to univariate estimates, multivariate estimates exhibited higher reliability at both the edge-level and connectome-level, stronger prediction of individual differences, and greater sensitivity to brain states within individuals. Our findings suggest that multivariate estimates reliably provide more powerful information about an individual's functional brain organization and its relation to cognitive skills.

Atlas-Based Classification Algorithms for Identification of Informative Brain Regions in fMRI Data

Multi-voxel pattern analysis (MVPA) has been successfully applied to neuroimaging data due to its larger sensitivity compared to univariate traditional techniques. Searchlight is the most widely employed approach to assign functional value to different regions of the brain. However, its performance depends on the size of the sphere, which can overestimate the region of activation when a large sphere size is employed. In the current study, we examined the validity of two different alternatives to Searchlight: an atlas-based local averaging method (ABLA, Schrouff et al. Neuroinformatics 16, 117-143, 2013a) and a Multi-Kernel Learning (MKL, Rakotomamonjy et al. Journal of Machine Learning 9, 2491-2521, 2008) approach, in a scenario where the goal is to find the informative brain regions that support certain mental operations. These methods employ weights to measure the informativeness of a brain region and highly reduce the large computational cost that Searchlight entails. We evaluated their performance in two different scenarios where the differential BOLD activation between experimental conditions was large vs. small, and employed nine different atlases to assess the influence of diverse brain parcellations. Results show that both methods were able to localize informative regions when differences between conditions were large, demonstrating a large sensitivity and stability in the identification of regions across atlases. Moreover, the sign of the weights reported by these methods provided the directionality of univariate approaches. However, when differences were small, only ABLA localized informative regions. Thus, our results show that atlas-based methods are useful alternatives to Searchlight, but that the nature of the classification to perform should be taken into account when choosing the specific method to implement.

Measuring shared responses across subjects using intersubject correlation

Our capacity to jointly represent information about the world underpins our social experience. By leveraging one individual's brain activity to model another's, we can measure shared information across brains-even in dynamic, naturalistic scenarios where an explicit response model may be unobtainable. Introducing experimental manipulations allows us to measure, for example, shared responses between speakers and listeners or between perception and recall. In this tutorial, we develop the logic of intersubject correlation (ISC) analysis and discuss the family of neuroscientific questions that stem from this approach. We also extend this logic to spatially distributed response patterns and functional network estimation. We provide a thorough and accessible treatment of methodological considerations specific to ISC analysis and outline best practices.

Representation of shape, space, and attention in monkey cortex

Attentional deficits are core to numerous developmental, neurological, and psychiatric disorders. At the single-cell level, much knowledge has been garnered from studies of shape and spatial properties, as well as from numerous demonstrations of attentional modulation of those properties. Despite this wealth of knowledge of single-cell responses across many brain regions, little is known about how these cellular characteristics relate to population level representations and how such representations relate to behavior; in particular, how these cellular responses relate to the representation of shape, space, and attention, and how these representations differ across cortical areas and streams. Here we will emphasize the role of population coding as a missing link for connecting single-cell properties with behavior. Using a data-driven intrinsic approach to population decoding, we show that both 'what' and 'where' cortical visual streams encode shape, space, and attention, yet demonstrate striking differences in these representations. We suggest that both pathways fully process shape and space, but that differences in representation may arise due to their differing functions and input and output constraints. Moreover, differences in the effects of attention on shape and spatial population representations in the two visual streams suggest two distinct strategies: in a ventral area, attention or task demands modulate the population representations themselves (perhaps to expand or enhance one part at the expense of other parts) while in a dorsal area, at a population representation level, attention effects are weak and nearly non-existent, perhaps in order to maintain veridical representations needed for visuomotor control. We show that an intrinsic approach, as opposed to theory-driven and labeled approaches, is useful for understanding how representations develop and differ across brain regions. Most importantly, these approaches help link cellular properties more tightly with behavior, a much-needed step to better understand and interpret cellular findings and key to providing insights to improve interventions in human disorders.

Parsing rooms: the role of the PPA and RSC in perceiving object relations and spatial layout

The perception of a scene involves grasping the global space of the scene, usually called the spatial layout, as well as the objects in the scene and the relations between them. The main brain areas involved in scene perception, the parahippocampal place area (PPA) and retrosplenial cortex (RSC), are supposed to mostly support the processing of spatial layout. Here we manipulated the objects and their relations either by arranging objects within rooms in a common way or by scattering them randomly. The rooms were then varied for spatial layout by keeping or removing the walls of the room, a typical layout manipulation. We then combined a visual search paradigm, where participants actively search for an object within the room, with multivariate pattern analysis (MVPA). Both left and right PPA were sensitive to the layout properties, but the right PPA was also sensitive to the object relations even when the information about objects and their relations is used in the cross-categorization procedure on novel stimuli. The left and right RSC were sensitive to both spatial layout and object relations, but could only use the information about object relations for cross-categorization to novel stimuli. These effects were restricted to the PPA and RSC, as other control brain areas did not display the same pattern of results. Our results underline the importance of employing paradigms that require participants to explicitly retrieve domain-specific processes and indicate that objects and their relations are processed in the scene areas to a larger extent than previously assumed.