Reliability of dissimilarity measures for multi-voxel pattern analysis

Manipulating and measuring variation in deep neural network (DNN) representations of objects

We explore how DNNs can be used to develop a computational understanding of individual differences in high-level visual cognition given their ability to generate rich meaningful object representations informed by their architecture, experience, and training protocols. As a first step to quantifying individual differences in DNN representations, we systematically explored the robustness of a variety of representational similarity measures: Representational Similarity Analysis (RSA), Centered Kernel Alignment (CKA), and Projection-Weighted Canonical Correlation Analysis (PWCCA), with an eye to how these measures are used in cognitive science, cognitive neuroscience, and vision science. To manipulate object representations, we next created a large set of models varying in random initial weights and random training image order, training image frequencies, training category frequencies, and model size and architecture and measured the representational variation caused by each manipulation. We examined both small (All-CNN-C) and commonly-used large (VGG and ResNet) DNN architectures. To provide a comparison for the magnitude of representational differences, we established a baseline based on the representational variation caused by image-augmentation techniques used to train those DNNs. We found that variation in model randomization and model size never exceeded baseline. By contrast, differences in training image frequency and training category frequencies caused representational variation that exceeded baseline, with training category frequency manipulations exceeding baseline earlier in the networks. These findings provide insights into the magnitude of representational variations that can be expected with a range of manipulations and provide a springboard for further exploration of systematic model variations aimed at modeling individual differences in high-level visual cognition.

Semi-orthogonal subspaces for value mediate a binding and generalization trade-off

When choosing between options, we must associate their values with the actions needed to select them. We hypothesize that the brain solves this binding problem through neural population subspaces. Here, in macaques performing a choice task, we show that neural populations in five reward-sensitive regions encode the values of offers presented on the left and right in distinct subspaces. This encoding is sufficient to bind offer values to their locations while preserving abstract value information. After offer presentation, all areas encode the value of the first and second offers in orthogonal subspaces; this orthogonalization also affords binding. Our binding-by-subspace hypothesis makes two new predictions confirmed by the data. First, behavioral errors should correlate with spatial, but not temporal, neural misbinding. Second, behavioral errors should increase when offers have low or high values, compared to medium values, even when controlling for value difference. Together, these results support the idea that the brain uses semi-orthogonal subspaces to bind features.

A cortical surface template for human neuroscience

Neuroimaging data analysis relies on normalization to standard anatomical templates to resolve macroanatomical differences across brains. Existing human cortical surface templates sample locations unevenly because of distortions introduced by inflation of the folded cortex into a standard shape. Here we present the onavg template, which affords uniform sampling of the cortex. We created the onavg template based on openly available high-quality structural scans of 1,031 brains-25 times more than existing cortical templates. We optimized the vertex locations based on cortical anatomy, achieving an even distribution. We observed consistently higher multivariate pattern classification accuracies and representational geometry inter-participant correlations based on onavg than on other templates, and onavg only needs three-quarters as much data to achieve the same performance compared with other templates. The optimized sampling also reduces CPU time across algorithms by 1.3-22.4% due to less variation in the number of vertices in each searchlight.

Predictions enable top-down pattern separation in the macaque face-processing hierarchy

Distinguishing faces requires well distinguishable neural activity patterns. Contextual information may separate neural representations, leading to enhanced identity recognition. Here, we use functional magnetic resonance imaging to investigate how predictions derived from contextual information affect the separability of neural activity patterns in the macaque face-processing system, a 3-level processing hierarchy in ventral visual cortex. We find that in the presence of predictions, early stages of this hierarchy exhibit well separable and high-dimensional neural geometries resembling those at the top of the hierarchy. This is accompanied by a systematic shift of tuning properties from higher to lower areas, endowing lower areas with higher-order, invariant representations instead of their feedforward tuning properties. Thus, top-down signals dynamically transform neural representations of faces into separable and high-dimensional neural geometries. Our results provide evidence how predictive context transforms flexible representational spaces to optimally use the computational resources provided by cortical processing hierarchies for better and faster distinction of facial identities.

Electroencephalogram Decoding Reveals Distinct Processes for Directing Spatial Attention and Encoding Into Working Memory

Past work reveals a tight relationship between spatial attention and storage in visual working memory. But is spatially attending an item tantamount to working memory encoding? Here, we tracked electroencephalography (EEG) signatures of spatial attention and working memory encoding while independently manipulating the number of memory items and the spatial extent of attention in two studies of adults (N = 39; N = 33). Neural measures of spatial attention tracked the position and size of the attended area independent of the number of individuated items encoded into working memory. At the same time, multivariate decoding of the number of items stored in working memory was insensitive to variations in the breadth and position of spatial attention. Finally, representational similarity analyses provided converging evidence for a pure load signal that is insensitive to the spatial extent of the stored items. Thus, although spatial attention is a persistent partner of visual working memory, it is functionally dissociable from the selection and maintenance of individuated representations in working memory.

Phase-Dependent Visual and Sensorimotor Integration of Features for Grasp Computations before and after Effector Specification

The simple act of viewing and grasping an object involves complex sensorimotor control mechanisms that have been shown to vary as a function of multiple object and other task features such as object size, shape, weight, and wrist orientation. However, these features have been mostly studied in isolation. In contrast, given the nonlinearity of motor control, its computations require multiple features to be incorporated concurrently. Therefore, the present study tested the hypothesis that grasp computations integrate multiple task features superadditively in particular when these features are relevant for the same action phase. We asked male and female human participants to reach-to-grasp objects of different shapes and sizes with different wrist orientations. Also, we delayed the movement onset using auditory signals to specify which effector to use. Using electroencephalography and representative dissimilarity analysis to map the time course of cortical activity, we found that grasp computations formed superadditive integrated representations of grasp features during different planning phases of grasping. Shape-by-size representations and size-by-orientation representations occurred before and after effector specification, respectively, and could not be explained by single-feature models. These observations are consistent with the brain performing different preparatory, phase-specific computations; visual object analysis to identify grasp points at abstract visual levels; and downstream sensorimotor preparatory computations for reach-to-grasp trajectories. Our results suggest the brain adheres to the needs of nonlinear motor control for integration. Furthermore, they show that examining the superadditive influence of integrated representations can serve as a novel lens to map the computations underlying sensorimotor control.

The neural basis of swap errors in working memory

When making decisions in a cluttered world, humans and other animals often have to hold multiple items in memory at once-such as the different items on a shopping list. Psychophysical experiments in humans and other animals have shown remembered stimuli can sometimes become confused, with participants reporting chimeric stimuli composed of features from different stimuli. In particular, subjects will often make "swap errors" where they misattribute a feature from one object as belonging to another object. While swap errors have been described behaviorally and theoretical explanations have been proposed, their neural mechanisms are unknown. Here, we elucidate these neural mechanisms by analyzing neural population recordings from monkeys performing two multistimulus working memory tasks. In these tasks, monkeys were cued to report the color of an item that either was previously shown at a corresponding location or will be shown at the corresponding location. Animals made swap errors in both tasks. In the neural data, we find evidence that the neural correlates of swap errors emerged when correctly remembered information is selected from working memory. This led to a representation of the distractor color as if it were the target color, underlying the eventual swap error. We did not find consistent evidence that swap errors arose from misinterpretation of the cue or errors during encoding or storage in working memory. These results provide evidence that swap errors emerge during selection of correctly remembered information from working memory, and highlight this selection as a crucial-yet surprisingly brittle-neural process.

Combined representation of visual features in the scene-selective cortex

Visual features of separable dimensions conjoin to represent an integrated entity. We investigated how visual features bind to form a complex visual scene. Specifically, we focused on features important for visually guided navigation: direction and distance. Previously, separate works have shown that directions and distances of navigable paths are coded in the occipital place area (OPA). Using functional magnetic resonance imaging (fMRI), we tested how separate features are concurrently represented in the OPA. Participants saw eight types of scenes, in which four of them had one path and the other four had two paths. In single-path scenes, path direction was either to the left or to the right. In double-path scenes, both directions were present. A glass wall was placed in some paths to restrict navigational distance. To test how the OPA represents path directions and distances, we took three approaches. First, the independent-features approach examined whether the OPA codes each direction and distance. Second, the integrated-features approach explored how directions and distances are integrated into path units, as compared to pooled features, using double-path scenes. Finally, the integrated-paths approach asked how separate paths are combined into a scene. Using multi-voxel pattern similarity analysis, we found that the OPA's representations of single-path scenes were similar to other single-path scenes of either the same direction or the same distance. Representations of double-path scenes were similar to the combination of two constituent single-paths, as a combined unit of direction and distance rather than as a pooled representation of all features. These results show that the OPA combines the two features to form path units, which are then used to build multiple-path scenes. Altogether, these results suggest that visually guided navigation may be supported by the OPA that automatically and efficiently combines multiple features relevant for navigation and represent a navigation file.

Concurrent response and action effect representations across the somatomotor cortices during novel task preparation

Instructions allow us to fulfill novel and complex tasks on the first try. This skill has been linked to preparatory brain signals that encode upcoming demands in advance, facilitating novel performance. To deepen insight into these processes, we explored whether instructions pre-activated task-relevant motoric and perceptual neural states. Critically, we addressed whether these representations anticipated activity patterns guiding overt sensorimotor processing, which could reflect that internally simulating novel tasks facilitates the preparation. To do so, we collected functional magnetic resonance imaging data while female and male participants encoded and implemented novel stimulus-response associations. Participants also completed localizer tasks designed to isolate the neural representations of the mappings-relevant motor responses, perceptual consequences, and stimulus categories. Using canonical template tracking, we identified whether and where these sensorimotor representations were pre-activated. We found that response-related templates were encoded in advance in regions linked with action control, entailing not only the instructed responses but also their somatosensory consequences. This result was particularly robust in primary motor and somatosensory cortices. While, following our predictions, we found a systematic decrease in the irrelevant stimulus templates' representational strength compared to the relevant ones, this difference was due to below-zero estimates linked to the irrelevant category activity patterns. Overall, our findings reflect that instruction processing relies on the sensorimotor cortices to anticipate motoric and kinesthetic representations of prospective action plans, suggesting the engagement of motor imagery during novel task preparation. More generally, they stress that the somatomotor system could participate with higher-level frontoparietal regions during anticipatory task control.

Statistical Learning of Incidental Perceptual Regularities Induces Sensory Conditioned Cortical Responses

Statistical learning of sensory patterns can lead to predictive neural processes enhancing stimulus perception and enabling fast deviancy detection. Predictive processes have been extensively demonstrated when environmental statistical regularities are relevant to task execution. Preliminary evidence indicates that statistical learning can even occur independently of task relevance and top-down attention, although the temporal profile and neural mechanisms underlying sensory predictions and error signals induced by statistical learning of incidental sensory regularities remain unclear. In our study, we adopted an implicit sensory conditioning paradigm that elicited the generation of specific perceptual priors in relation to task-irrelevant audio-visual associations, while recording Electroencephalography (EEG). Our results showed that learning task-irrelevant associations between audio-visual stimuli resulted in anticipatory neural responses to predictive auditory stimuli conveying anticipatory signals of expected visual stimulus presence or absence. Moreover, we observed specific modulation of cortical responses to probabilistic visual stimulus presentation or omission. Pattern similarity analysis indicated that predictive auditory stimuli tended to resemble the response to expected visual stimulus presence or absence. Remarkably, Hierarchical Gaussian filter modeling estimating dynamic changes of prediction error signals in relation to differential probabilistic occurrences of audio-visual stimuli further demonstrated instantiation of predictive neural signals by showing distinct neural processing of prediction error in relation to violation of expected visual stimulus presence or absence. Overall, our findings indicated that statistical learning of non-salient and task-irrelevant perceptual regularities could induce the generation of neural priors at the time of predictive stimulus presentation, possibly conveying sensory-specific information about the predicted consecutive stimulus.

Cost-benefit Tradeoff Mediates the Rule- to Memory-based Processing Transition during Practice

Practice not only improves task performance, but also changes task execution from rule- to memory-based processing by incorporating experiences from practice. However, how and when this change occurs is unclear. We tested the hypothesis that strategy transition in task learning results from cost-benefit analysis. Participants learned two task sequences and were then queried about the task type at a cued sequence and position. Behavioral improvement with practice can be accounted for by a computational model implementing cost-benefit analysis. Model-predicted strategy transition points are related to behavioral slowing and changes in fMRI activation patterns in the dorsolateral prefrontal cortex. Strategy transition is also related to increased pattern separation in the ventromedial prefrontal cortex. The cost-benefit analysis model outperforms alternative models (e.g., both strategies racing for being expressed in behavior) in accounting for empirical data. These findings support cost-benefit analysis as a mechanism of practice-induced strategy shift.

Using multivariate pattern analysis to increase effect sizes for event-related potential analyses

Multivariate pattern analysis (MVPA) approaches can be applied to the topographic distribution of event-related potential (ERP) signals to "decode" subtly different stimulus classes, such as different faces or different orientations. These approaches are extremely sensitive, and it seems possible that they could also be used to increase effect sizes and statistical power in traditional paradigms that ask whether an ERP component differs in amplitude across conditions. To assess this possibility, we leveraged the open-source ERP CORE data set and compared the effect sizes resulting from conventional univariate analyses of mean amplitude with two MVPA approaches (support vector machine decoding and the cross-validated Mahalanobis distance, both of which are easy to compute using open-source software). We assessed these approaches across seven widely studied ERP components (N170, N400, N2pc, P3b, lateral readiness potential, error related negativity, and mismatch negativity). Across all components, we found that multivariate approaches yielded effect sizes that were as large or larger than the effect sizes produced by univariate approaches. These results indicate that researchers could obtain larger effect sizes, and therefore greater statistical power, by using multivariate analysis of topographic voltage patterns instead of traditional univariate analyses in many ERP studies.

Neural Correlates of Online Action Preparation

When performing movements in rapid succession, the brain needs to coordinate ongoing execution with the preparation of an upcoming action. Here we identify the processes and brain areas involved in this ability of online preparation. Human participants (both male and female) performed pairs of single-finger presses or three-finger chords in rapid succession, while 7T fMRI was recorded. In the overlap condition, they could prepare the second movement during the first response and in the nonoverlap condition only after the first response was completed. Despite matched perceptual and movement requirements, fMRI revealed increased brain activity in the overlap condition in regions along the intraparietal sulcus and ventral visual stream. Multivariate analyses suggested that these areas are involved in stimulus identification and action selection. In contrast, the dorsal premotor cortex, known to be involved in planning upcoming movements, showed no discernible signs of heightened activity. This observation suggests that the bottleneck during simultaneous action execution and preparation arises at the level of stimulus identification and action selection, whereas movement planning in the premotor cortex can unfold concurrently with the execution of a current action without requiring additional neural activity.

Information-based TMS to mid-lateral prefrontal cortex disrupts action goals during emotional processing

The ability to respond to emotional events in a context-sensitive and goal-oriented manner is essential for adaptive functioning. In models of behavioral and emotion regulation, the lateral prefrontal cortex (LPFC) is postulated to maintain goal-relevant representations that promote cognitive control, an idea rarely tested with causal inference. Here, we altered mid-LPFC function in healthy individuals using a putatively inhibitory brain stimulation protocol (continuous theta burst; cTBS), followed by fMRI scanning. Participants performed the Affective Go/No-Go task, which requires goal-oriented action during affective processing. We targeted mid-LPFC (vs. a Control site) based on the individualized location of action-goal representations observed during the task. cTBS to mid-LPFC reduced action-goal representations in mid-LPFC and impaired goal-oriented action, particularly during processing of negative emotional cues. During negative-cue processing, cTBS to mid-LPFC reduced functional coupling between mid-LPFC and nodes of the default mode network, including frontopolar cortex-a region thought to modulate LPFC control signals according to internal states. Collectively, these results indicate that mid-LPFC goal-relevant representations play a causal role in governing context-sensitive cognitive control during emotional processing.

Visual context affects the perceived timing of tactile sensations elicited through intra-cortical microstimulation

Intra-cortical microstimulation (ICMS) is a technique to provide tactile sensations for a somatosensory brain-machine interface (BMI). A viable BMI must function within the rich, multisensory environment of the real world, but how ICMS is integrated with other sensory modalities is poorly understood. To investigate how ICMS percepts are integrated with visual information, ICMS and visual stimuli were delivered at varying times relative to one another. Both visual context and ICMS current amplitude were found to bias the qualitative experience of ICMS. In two tetraplegic participants, ICMS and visual stimuli were more likely to be experienced as occurring simultaneously when visual stimuli were more realistic, demonstrating an effect of visual context on the temporal binding window. The peak of the temporal binding window varied but was consistently offset from zero, suggesting that multisensory integration with ICMS can suffer from temporal misalignment. Recordings from primary somatosensory cortex (S1) during catch trials where visual stimuli were delivered without ICMS demonstrated that S1 represents visual information related to ICMS across visual contexts.

Orthogonal neural encoding of targets and distractors supports multivariate cognitive control

The complex challenges of our mental life require us to coordinate multiple forms of neural information processing. Recent behavioural studies have found that people can coordinate multiple forms of attention, but the underlying neural control process remains obscure. We hypothesized that the brain implements multivariate control by independently monitoring feature-specific difficulty and independently prioritizing feature-specific processing. During functional MRI, participants performed a parametric conflict task that separately tags target and distractor processing. Consistent with feature-specific monitoring, univariate analyses revealed spatially segregated encoding of target and distractor difficulty in the dorsal anterior cingulate cortex. Consistent with feature-specific attentional priority, our encoding geometry analysis revealed overlapping but orthogonal representations of target and distractor coherence in the intraparietal sulcus. Coherence representations were mediated by control demands and aligned with both performance and frontoparietal activity, consistent with top-down attention. Together, these findings provide evidence for the neural geometry necessary to coordinate multivariate cognitive control.

Repetition Suppression Reveals Cue-Specific Spatial Representations for Landmarks and Self-Motion Cues in the Human Retrosplenial Cortex

The efficient use of various spatial cues within a setting is crucial for successful navigation. Two fundamental forms of spatial navigation, landmark-based and self-motion-based, engage distinct cognitive mechanisms. The question of whether these modes invoke shared or separate spatial representations in the brain remains unresolved. While nonhuman animal studies have yielded inconsistent results, human investigation is limited. In our previous work (Chen et al., 2019), we introduced a novel spatial navigation paradigm utilizing ultra-high field fMRI to explore neural coding of positional information. We found that different entorhinal subregions in the right hemisphere encode positional information for landmarks and self-motion cues. The present study tested the generalizability of our previous finding with a modified navigation paradigm. Although we did not replicate our previous finding in the entorhinal cortex, we identified adaptation-based allocentric positional codes for both cue types in the retrosplenial cortex (RSC), which were not confounded by the path to the spatial location. Crucially, the multi-voxel patterns of these spatial codes differed between the cue types, suggesting cue-specific positional coding. The parahippocampal cortex exhibited positional coding for self-motion cues, which was not dissociable from path length. Finally, the brain regions involved in successful navigation differed from our previous study, indicating overall distinct neural mechanisms recruited in our two studies. Taken together, the current findings demonstrate cue-specific allocentric positional coding in the human RSC in the same navigation task for the first time and that spatial representations in the brain are contingent on specific experimental conditions.

Representational maps in the brain: concepts, approaches, and applications

Neural systems have evolved to process sensory stimuli in a way that allows for efficient and adaptive behavior in a complex environment. Recent technological advances enable us to investigate sensory processing in animal models by simultaneously recording the activity of large populations of neurons with single-cell resolution, yielding high-dimensional datasets. In this review, we discuss concepts and approaches for assessing the population-level representation of sensory stimuli in the form of a representational map. In such a map, not only are the identities of stimuli distinctly represented, but their relational similarity is also mapped onto the space of neuronal activity. We highlight example studies in which the structure of representational maps in the brain are estimated from recordings in humans as well as animals and compare their methodological approaches. Finally, we integrate these aspects and provide an outlook for how the concept of representational maps could be applied to various fields in basic and clinical neuroscience.

Using Multivariate Pattern Analysis to Increase Effect Sizes for Event-Related Potential Analyses

Multivariate pattern analysis approaches can be applied to the topographic distribution of event-related potential (ERP) signals to 'decode' subtly different stimulus classes, such as different faces or different orientations. These approaches are extremely sensitive, and it seems possible that they could also be used to increase effect sizes and statistical power in traditional paradigms that ask whether an ERP component differs in amplitude across conditions. To assess this possibility, we leveraged the open-source ERP CORE dataset and compared the effect sizes resulting from conventional univariate analyses of mean amplitude with two multivariate pattern analysis approaches (support vector machine decoding and the cross-validated Mahalanobis distance, both of which are easy to compute using open-source software). We assessed these approaches across seven widely studied ERP components (N170, N400, N2pc, P3b, lateral readiness potential, error related negativity, and mismatch negativity). Across all components, we found that multivariate approaches yielded effect sizes that were as large or larger than the effect sizes produced by univariate approaches. These results indicate that researchers could obtain larger effect sizes, and therefore greater statistical power, by using multivariate analysis of topographic voltage patterns instead of traditional univariate analyses in many ERP studies.

Event Integration and Temporal Differentiation: How Hierarchical Knowledge Emerges in Hippocampal Subfields through Learning

Everyday life is composed of events organized by changes in contexts, with each event containing an unfolding sequence of occurrences. A major challenge facing our memory systems is how to integrate sequential occurrences within events while also maintaining their details and avoiding over-integration across different contexts. We asked if and how distinct hippocampal subfields come to hierarchically and, in parallel, represent both event context and subevent occurrences with learning. Female and male human participants viewed sequential events defined as sequences of objects superimposed on shared color frames while undergoing high-resolution fMRI. Importantly, these events were repeated to induce learning. Event segmentation, as indexed by increased reaction times at event boundaries, was observed in all repetitions. Temporal memory decisions were quicker for items from the same event compared to across different events, indicating that events shaped memory. With learning, hippocampal CA3 multivoxel activation patterns clustered to reflect the event context, with more clustering correlated with behavioral facilitation during event transitions. In contrast, in the dentate gyrus (DG), temporally proximal items that belonged to the same event became associated with more differentiated neural patterns. A computational model explained these results by dynamic inhibition in the DG. Additional similarity measures support the notion that CA3 clustered representations reflect shared voxel populations, while DG's distinct item representations reflect different voxel populations. These findings suggest an interplay between temporal differentiation in the DG and attractor dynamics in CA3. They advance our understanding of how knowledge is structured through integration and separation across time and context.

Dorsolateral prefrontal activity supports a cognitive space organization of cognitive control

Cognitive control resolves conflicts between task-relevant and -irrelevant information to enable goal-directed behavior. As conflicts can arise from different sources (e.g., sensory input, internal representations), how a limited set of cognitive control processes can effectively address diverse conflicts remains a major challenge. Based on the cognitive space theory, different conflicts can be parameterized and represented as distinct points in a (low-dimensional) cognitive space, which can then be resolved by a limited set of cognitive control processes working along the dimensions. It leads to a hypothesis that conflicts similar in their sources are also represented similarly in the cognitive space. We designed a task with five types of conflicts that could be conceptually parameterized. Both human performance and fMRI activity patterns in the right dorsolateral prefrontal cortex support that different types of conflicts are organized based on their similarity, thus suggesting cognitive space as a principle for representing conflicts.

Does Ipsilateral Remapping Following Hand Loss Impact Motor Control of the Intact Hand?

What happens once a cortical territory becomes functionally redundant? We studied changes in brain function and behavior for the remaining hand in humans (male and female) with either a missing hand from birth (one-handers) or due to amputation. Previous studies reported that amputees, but not one-handers, show increased ipsilateral activity in the somatosensory territory of the missing hand (i.e., remapping). We used a complex finger task to explore whether this observed remapping in amputees involves recruiting more neural resources to support the intact hand to meet greater motor control demands. Using basic fMRI analysis, we found that only amputees had more ipsilateral activity when motor demand increased; however, this did not match any noticeable improvement in their behavioral task performance. More advanced multivariate fMRI analyses showed that amputees had stronger and more typical representation-relative to controls' contralateral hand representation-compared with one-handers. This suggests that in amputees, both hand areas work together more collaboratively, potentially reflecting the intact hand's efference copy. One-handers struggled to learn difficult finger configurations, but this did not translate to differences in univariate or multivariate activity relative to controls. Additional white matter analysis provided conclusive evidence that the structural connectivity between the two hand areas did not vary across groups. Together, our results suggest that enhanced activity in the missing hand territory may not reflect intact hand function. Instead, we suggest that plasticity is more restricted than generally assumed and may depend on the availability of homologous pathways acquired early in life.

Neural sensitivity to translational self- and object-motion velocities

The ability to detect and assess world-relative object-motion is a critical computation performed by the visual system. This computation, however, is greatly complicated by the observer's movements, which generate a global pattern of motion on the observer's retina. How the visual system implements this computation is poorly understood. Since we are potentially able to detect a moving object if its motion differs in velocity (or direction) from the expected optic flow generated by our own motion, here we manipulated the relative motion velocity between the observer and the object within a stationary scene as a strategy to test how the brain accomplishes object-motion detection. Specifically, we tested the neural sensitivity of brain regions that are known to respond to egomotion-compatible visual motion (i.e., egomotion areas: cingulate sulcus visual area, posterior cingulate sulcus area, posterior insular cortex [PIC], V6+, V3A, IPSmot/VIP, and MT+) to a combination of different velocities of visually induced translational self- and object-motion within a virtual scene while participants were instructed to detect object-motion. To this aim, we combined individual surface-based brain mapping, task-evoked activity by functional magnetic resonance imaging, and parametric and representational similarity analyses. We found that all the egomotion regions (except area PIC) responded to all the possible combinations of self- and object-motion and were modulated by the self-motion velocity. Interestingly, we found that, among all the egomotion areas, only MT+, V6+, and V3A were further modulated by object-motion velocities, hence reflecting their possible role in discriminating between distinct velocities of self- and object-motion. We suggest that these egomotion regions may be involved in the complex computation required for detecting scene-relative object-motion during self-motion.

Seeking the neural representation of statistical properties in print during implicit processing of visual words

Statistical learning (SL) plays a key role in literacy acquisition. Studies have increasingly revealed the influence of distributional statistical properties of words on visual word processing, including the effects of word frequency (lexical level) and mappings between orthography, phonology, and semantics (sub-lexical level). However, there has been scant evidence to directly confirm that the statistical properties contained in print can be directly characterized by neural activities. Using time-resolved representational similarity analysis (RSA), the present study examined neural representations of different types of statistical properties in visual word processing. From the perspective of predictive coding, an equal probability sequence with low built-in prediction precision and three oddball sequences with high built-in prediction precision were designed with consistent and three types of inconsistent (orthographically inconsistent, orthography-to-phonology inconsistent, and orthography-to-semantics inconsistent) Chinese characters as visual stimuli. In the three oddball sequences, consistent characters were set as the standard stimuli (probability of occurrence p = 0.75) and three types of inconsistent characters were set as deviant stimuli (p = 0.25), respectively. In the equal probability sequence, the same consistent and inconsistent characters were presented randomly with identical occurrence probability (p = 0.25). Significant neural representation activities of word frequency were observed in the equal probability sequence. By contrast, neural representations of sub-lexical statistics only emerged in oddball sequences where short-term predictions were shaped. These findings reveal that the statistical properties learned from long-term print environment continues to play a role in current word processing mechanisms and these mechanisms can be modulated by short-term predictions.

Stable Cortical Body Maps Before and After Arm Amputation

Neuroscientists have long debated the adult brain's capacity to reorganize itself in response to injury. A driving model for studying plasticity has been limb amputation. For decades, it was believed that amputation triggers large-scale reorganization of cortical body resources. However, these studies have relied on cross-sectional observations post-amputation, without directly tracking neural changes. Here, we longitudinally followed adult patients with planned arm amputations and measured hand and face representations, before and after amputation. By interrogating the representational structure elicited from movements of the hand (pre-amputation) and phantom hand (post-amputation), we demonstrate that hand representation is unaltered. Further, we observed no evidence for lower face (lip) reorganization into the deprived hand region. Collectively, our findings provide direct and decisive evidence that amputation does not trigger large-scale cortical reorganization.

Hippocampal contributions to novel spatial learning are both age-related and age-invariant

Older adults show declines in spatial memory, although the extent of these alterations is not uniform across the healthy older population. Here, we investigate the stability of neural representations for the same and different spatial environments in a sample of younger and older adults using high-resolution functional MRI of the medial temporal lobes. Older adults showed, on average, lower neural pattern similarity for retrieving the same environment and more variable neural patterns compared to young adults. We also found a positive association between spatial distance discrimination and the distinctiveness of neural patterns between environments. Our analyses suggested that one source for this association was the extent of informational connectivity to CA1 from other subfields, which was dependent on age, while another source was the fidelity of signals within CA1 itself, which was independent of age. Together, our findings suggest both age-dependent and independent neural contributions to spatial memory performance.

BackWards - Unveiling the brain's topographic organization of paraspinal sensory input

Cortical reorganization and its potential pathological significance are being increasingly studied in musculoskeletal disorders such as chronic low back pain (CLBP) patients. However, detailed sensory-topographic maps of the human back are lacking, and a baseline characterization of such representations, reflecting the somatosensory organization of the healthy back, is needed before exploring potential sensory map reorganization. To this end, a novel pneumatic vibrotactile stimulation method was used to stimulate paraspinal sensory afferents, while studying their cortical representations in unprecedented detail. In 41 young healthy participants, vibrotactile stimulations at 20 Hz and 80 Hz were applied bilaterally at nine locations along the thoracolumbar axis while functional magnetic resonance imaging (fMRI) was performed. Model-based whole-brain searchlight representational similarity analysis (RSA) was used to investigate the organizational structure of brain activity patterns evoked by thoracolumbar sensory inputs. A model based on segmental distances best explained the similarity structure of brain activity patterns that were located in different areas of sensorimotor cortices, including the primary somatosensory and motor cortices and parts of the superior parietal cortex, suggesting that these brain areas process sensory input from the back in a "dermatomal" manner. The current findings provide a sound basis for testing the "cortical map reorganization theory" and its pathological relevance in CLBP.

Moral foundations elicit shared and dissociable cortical activation modulated by political ideology

Moral foundations theory (MFT) holds that moral judgements are driven by modular and ideologically variable moral foundations but where and how these foundations are represented in the brain and shaped by political beliefs remains an open question. Using a moral vignette judgement task (n = 64), we probed the neural (dis)unity of moral foundations. Univariate analyses revealed that moral judgement of moral foundations, versus conventional norms, reliably recruits core areas implicated in theory of mind. Yet, multivariate pattern analysis demonstrated that each moral foundation elicits dissociable neural representations distributed throughout the cortex. As predicted by MFT, individuals' liberal or conservative orientation modulated neural responses to moral foundations. Our results confirm that each moral foundation recruits domain-general mechanisms of social cognition but also has a dissociable neural signature malleable by sociomoral experience. We discuss these findings in view of unified versus dissociable accounts of morality and their neurological support for MFT.

Hippocampal contributions to novel spatial learning are both age-related and age-invariant

Older adults show declines in spatial memory, although the extent of these alterations is not uniform across the healthy older population. Here, we investigate the stability of neural representations for the same and different spatial environments in a sample of younger and older adults using high-resolution functional magnetic resonance imaging (fMRI) of the medial temporal lobe. Older adults showed, on average, lower neural pattern similarity for retrieving the same environment and more variable neural patterns compared to young adults. We also found a positive association between spatial distance discrimination and the distinctiveness of neural patterns between environments. Our analyses suggested that one source for this association was the extent of informational connectivity to CA1 from other subfields, which was dependent on age, while another source was the fidelity of signals within CA1 itself, which was independent of age. Together, our findings suggest both age-dependent and independent neural contributions to spatial memory performance.

Performing a vibrotactile discrimination task modulates finger representations in primary somatosensory cortex

It is well established that vibrotactile stimuli are represented in somatotopic maps. However, less is known about whether these somatotopic representations are modulated by task demands and maybe even in the absence of tactile input. Here, we used a vibrotactile discrimination task as a tool to investigate these questions in further detail. Participants were required to actively perceive and process tactile stimuli in comparison to a no-task control condition where identical stimuli were passively perceived (no-memory condition). Importantly, both vibrotactile stimuli were either applied to the right index or little finger, allowing us to investigate whether cognitive task demands shape finger representations in primary somatosensory cortex (S1). Using multivoxel pattern analysis and representational similarity analysis, we found that S1 finger representations were more distinct during the memory than the no-memory condition. Interestingly, this effect was not only observed while tactile stimuli were presented but also during the delay period (i.e., in the absence of tactile stimulation). Our findings imply that when individuals are required to focus on tactile stimuli, retain them in their memory, and engage in active processing of distinctive stimulus features, this exerts a modulatory effect on the finger representations present in S1.NEW & NOTEWORTHY Using multivoxel pattern analysis, we found that discrimination task demands shape finger representations in the contralateral primary somatosensory cortex (S1), and that somatotopic representations are modulated by task demands not only during tactile stimulation but also to a certain extent in the absence of tactile input.

Reassessing referral of touch following peripheral deafferentation: The role of contextual bias

Some amputees have been famously reported to perceive facial touch as arising from their phantom hand. These referred sensations have since been replicated across multiple neurological disorders and were classically interpreted as a perceptual correlate of cortical plasticity. Common to all these and related studies is that participants might have been influenced in their self-reports by the experimental design or related contextual biases. Here, we investigated whether referred sensations reports might be confounded by demand characteristics (e.g., compliance, expectation, suggestion). Unilateral upper-limb amputees (N = 18), congenital one-handers (N = 19), and two-handers (N = 22) were repeatedly stimulated with computer-controlled vibrations on 10 body-parts and asked to report the occurrence of any concurrent sensations on their hand(s). To further manipulate expectations, we gave participants the suggestion that some of these vibrations had a higher probability to evoke referred sensations. We also assessed similarity between (phantom) hand and face representation in primary somatosensory cortex (S1), using functional Magnetic Resonance Imaging (fMRI) multivariate representational similarity analysis. We replicated robust reports of referred sensations in amputees towards their phantom hand; however, the frequency and distribution of reported referred sensations were similar across groups. Moreover, referred sensations were evoked by stimulation of multiple body-parts and similarly reported on both the intact and phantom hand in amputees. Face-to-phantom-hand representational similarity was not different in amputees' missing hand region, compared with controls. These findings weaken the interpretation of referred sensations as a perceptual correlate of S1 plasticity and reveal the need to account for contextual biases when evaluating anomalous perceptual phenomena.

Time-dependent memory transformation in hippocampus and neocortex is semantic in nature

Memories undergo a time-dependent neural reorganization, which is assumed to be accompanied by a transformation from detailed to more gist-like memory. However, the nature of this transformation and its underlying neural mechanisms are largely unknown. Here, we report that the time-dependent transformation of memory is semantic in nature, while we find no credible evidence for a perceptual transformation. Model-based MRI analyses reveal time-dependent increases in semantically transformed representations of events in prefrontal and parietal cortices, while specific pattern representations in the anterior hippocampus decline over time. Posterior hippocampal memory reinstatement, in turn, increases over time and is linked to the semantic gist of the original memory, without a statistically significant link to perceptual details. These findings indicate that qualitative changes in memory over time, associated with distinct representational changes in the neocortex and within the hippocampus, reflect a semantic transformation, which may promote the integration of memories into abstract knowledge structures.

The representation of occluded image regions in area V1 of monkeys and humans

Neuronal activity in the primary visual cortex (V1) is driven by feedforward input from within the neurons' receptive fields (RFs) and modulated by contextual information in regions surrounding the RF. The effect of contextual information on spiking activity occurs rapidly and is therefore challenging to dissociate from feedforward input. To address this challenge, we recorded the spiking activity of V1 neurons in monkeys viewing either natural scenes or scenes where the information in the RF was occluded, effectively removing the feedforward input. We found that V1 neurons responded rapidly and selectively to occluded scenes. V1 responses elicited by occluded stimuli could be used to decode individual scenes and could be predicted from those elicited by non-occluded images, indicating that there is an overlap between visually driven and contextual responses. We used representational similarity analysis to show that the structure of V1 representations of occluded scenes measured with electrophysiology in monkeys correlates strongly with the representations of the same scenes in humans measured with functional magnetic resonance imaging (fMRI). Our results reveal that contextual influences rapidly alter V1 spiking activity in monkeys over distances of several degrees in the visual field, carry information about individual scenes, and resemble those in human V1. VIDEO ABSTRACT.

ConnSearch: A framework for functional connectivity analysis designed for interpretability and effectiveness at limited sample sizes

Functional connectivity studies increasingly turn to machine learning methods, which typically involve fitting a connectome-wide classifier, then conducting post hoc interpretation analyses to identify the neural correlates that best predict a dependent variable. However, this traditional analytic paradigm suffers from two main limitations. First, even if classifiers are perfectly accurate, interpretation analyses may not identify all the patterns expressed by a dependent variable. Second, even if classifiers are generalizable, the patterns implicated via interpretation analyses may not replicate. In other words, this traditional approach can yield effective classifiers while falling short of most neuroscientists' goals: pinpointing the neural correlates of dependent variables. We propose a new framework for multivariate analysis, ConnSearch, which involves dividing the connectome into components (e.g., groups of highly connected regions) and fitting an independent model for each component (e.g., a support vector machine or a correlation-based model). Conclusions about the link between a dependent variable and the brain are based on which components yield predictive models rather than on interpretation analysis. We used working memory data from the Human Connectome Project (N = 50-250) to compare ConnSearch with four existing connectome-wide classification/interpretation methods. For each approach, the models attempted to classify examples as being from the high-load or low-load conditions (binary labels). Relative to traditional methods, ConnSearch identified neural correlates that were more comprehensive, had greater consistency with the WM literature, and better replicated across datasets. Hence, ConnSearch is well-positioned to be an effective tool for functional connectivity research.

Testing the generalization of neural representations

Multivariate analysis methods are widely used in neuroscience to investigate the presence and structure of neural representations. Representational similarities across time or contexts are often investigated using pattern generalization, e.g. by training and testing multivariate decoders in different contexts, or by comparable pattern-based encoding methods. It is however unclear what conclusions can be validly drawn on the underlying neural representations when significant pattern generalization is found in mass signals such as LFP, EEG, MEG, or fMRI. Using simulations, we show how signal mixing and dependencies between measurements can drive significant pattern generalization even though the true underlying representations are orthogonal. We suggest that, using an accurate estimate of the expected pattern generalization given identical representations, it is nonetheless possible to test meaningful hypotheses about the generalization of neural representations. We offer such an estimate of the expected magnitude of pattern generalization and demonstrate how this measure can be used to assess the similarity and differences of neural representations across time and contexts.

Testing cognitive theories with multivariate pattern analysis of neuroimaging data

Multivariate pattern analysis (MVPA) has emerged as a powerful method for the analysis of functional magnetic resonance imaging, electroencephalography and magnetoencephalography data. The new approaches to experimental design and hypothesis testing afforded by MVPA have made it possible to address theories that describe cognition at the functional level. Here we review a selection of studies that have used MVPA to test cognitive theories from a range of domains, including perception, attention, memory, navigation, emotion, social cognition and motor control. This broad view reveals properties of MVPA that make it suitable for understanding the 'how' of human cognition, such as the ability to test predictions expressed at the item or event level. It also reveals limitations and points to future directions.

Loss of finger control complexity and intrusion of flexor biases are dissociable in finger individuation impairment after stroke

The ability to control each finger independently is an essential component of human hand dexterity. A common observation of hand function impairment after stroke is the loss of this finger individuation ability, often referred to as enslavement, i.e., the unwanted coactivation of non-intended fingers in individuated finger movements. In the previous literature, this impairment has been attributed to several factors, such as the loss of corticospinal drive, an intrusion of flexor synergy due to upregulations of the subcortical pathways, and/or biomechanical constraints. These factors may or may not be mutually exclusive and are often difficult to tease apart. It has also been suggested, based on a prevailing impression, that the intrusion of flexor synergy appears to be an exaggerated pattern of the involuntary coactivations of task-irrelevant fingers seen in a healthy hand, often referred to as a flexor bias. Most previous studies, however, were based on assessments of enslavement in a single dimension (i.e., finger flexion/extension) that coincide with the flexor bias, making it difficult to tease apart the other aforementioned factors. Here, we set out to closely examine the nature of individuated finger control and finger coactivation patterns in all dimensions. Using a novel measurement device and a 3D finger-individuation paradigm, we aim to tease apart the contributions of lower biomechanical, subcortical constraints, and top-down cortical control to these patterns in both healthy and stroke hands. For the first time, we assessed all five fingers' full capacity for individuation. Our results show that these patterns in the healthy and paretic hands present distinctly different shapes and magnitudes that are not influenced by biomechanical constraints. Those in the healthy hand presented larger angular distances that were dependent on top-down task goals, whereas those in the paretic hand presented larger Euclidean distances that arise from two dissociable factors: a loss of complexity in finger control and the dominance of an intrusion of flexor bias. These results suggest that finger individuation impairment after stroke is due to two dissociable factors: the loss of finger control complexity present in the healthy hand reflecting a top-down neural control strategy and an intrusion of flexor bias likely due to an upregulation of subcortical pathways. Our device and paradigm are demonstrated to be a promising tool to assess all aspects of the dexterous capacity of the hand.

Statistical inference on representational geometries

Neuroscience has recently made much progress, expanding the complexity of both neural activity measurements and brain-computational models. However, we lack robust methods for connecting theory and experiment by evaluating our new big models with our new big data. Here, we introduce new inference methods enabling researchers to evaluate and compare models based on the accuracy of their predictions of representational geometries: A good model should accurately predict the distances among the neural population representations (e.g. of a set of stimuli). Our inference methods combine novel 2-factor extensions of crossvalidation (to prevent overfitting to either subjects or conditions from inflating our estimates of model accuracy) and bootstrapping (to enable inferential model comparison with simultaneous generalization to both new subjects and new conditions). We validate the inference methods on data where the ground-truth model is known, by simulating data with deep neural networks and by resampling of calcium-imaging and functional MRI data. Results demonstrate that the methods are valid and conclusions generalize correctly. These data analysis methods are available in an open-source Python toolbox (rsatoolbox.readthedocs.io).

Mapping multidimensional content representations to neural and behavioral expressions of episodic memory

Human neuroimaging studies have shown that the contents of episodic memories are represented in distributed patterns of neural activity. However, these studies have mostly been limited to decoding simple, unidimensional properties of stimuli. Semantic encoding models, in contrast, offer a means for characterizing the rich, multidimensional information that comprises episodic memories. Here, we extensively sampled four human fMRI subjects to build semantic encoding models and then applied these models to reconstruct content from natural scene images as they were viewed and recalled from memory. First, we found that multidimensional semantic information was successfully reconstructed from activity patterns across visual and lateral parietal cortices, both when viewing scenes and when recalling them from memory. Second, whereas visual cortical reconstructions were much more accurate when images were viewed versus recalled from memory, lateral parietal reconstructions were comparably accurate across visual perception and memory. Third, by applying natural language processing methods to verbal recall data, we showed that fMRI-based reconstructions reliably matched subjects' verbal descriptions of their memories. In fact, reconstructions from ventral temporal cortex more closely matched subjects' own verbal recall than other subjects' verbal recall of the same images. Fourth, encoding models reliably transferred across subjects: memories were successfully reconstructed using encoding models trained on data from entirely independent subjects. Together, these findings provide evidence for successful reconstructions of multidimensional and idiosyncratic memory representations and highlight the differential sensitivity of visual cortical and lateral parietal regions to information derived from the external visual environment versus internally-generated memories.

Effects of Experiencing CS-US Pairings on Instructed Fear Reversal

Fear learning allows us to identify and anticipate aversive events and adapt our behavior accordingly. This is often thought to rely on associative learning mechanisms where an initially neutral conditioned stimulus (CS) is repeatedly paired with an aversive unconditioned stimulus (US), eventually leading to the CS also being perceived as aversive and threatening. Importantly, however, humans also show verbal fear learning. Namely, they have the ability to change their responses to stimuli rapidly through verbal instructions about CS-US pairings. Past research on the link between experience-based and verbal fear learning indicated that verbal instructions about a reversal of CS-US pairings can fully override the effects of previously experienced CS-US pairings, as measured through fear ratings, skin conductance, and fear-potentiated startle. However, it remains an open question whether such instructions can also annul learned CS representations in the brain. Here, we used a fear reversal paradigm (female and male participants) in conjunction with representational similarity analysis of fMRI data to test whether verbal instructions fully override the effects of experienced CS-US pairings in fear-related brain regions or not. Previous research suggests that only the right amygdala should show lingering representations of previously experienced threat ("pavlovian trace"). Unexpectedly, we found evidence for the residual effect of prior CS-US experience to be much more widespread than anticipated, in the amygdala but also cortical regions like the dorsal anterior cingulate or dorsolateral prefrontal cortex. This finding shines a new light on the interaction of different fear learning mechanisms, at times with unexpected consequences.SIGNIFICANCE STATEMENT Humans are able to learn about aversive stimuli both from experience (i.e., repeated pairings of conditioned stimulus (CS) and unconditioned stimulus (US; pavlovian conditioning), and from verbal instructions about stimulus pairings. Understanding how experience-based and verbal learning processes interact is key for understanding the cognitive and neural underpinnings of fear learning. We tested whether prior aversive experiences (CS-US pairings) affected subsequent verbal learning, searching for lingering threat signals after verbal instructions reversed a CS from being threatening to being safe. While past research suggested such threat signals can only be found in the amygdala, we found evidence to be much more widespread, including the medial and lateral PFC. This highlights how experience-based and verbal learning processes interact to support adaptive behavior.

Distinct ventral stream and prefrontal cortex representational dynamics during sustained conscious visual perception

Instances of sustained stationary sensory input are ubiquitous. However, previous work focused almost exclusively on transient onset responses. This presents a critical challenge for neural theories of consciousness, which should account for the full temporal extent of experience. To address this question, we use intracranial recordings from ten human patients with epilepsy to view diverse images of multiple durations. We reveal that, in sensory regions, despite dramatic changes in activation magnitude, the distributed representation of categories and exemplars remains sustained and stable. In contrast, in frontoparietal regions, we find transient content representation at stimulus onset. Our results highlight the connection between the anatomical and temporal correlates of experience. To the extent perception is sustained, it may rely on sensory representations and to the extent perception is discrete, centered on perceptual updating, it may rely on frontoparietal representations.

Metamodal Coupling of Vibrotactile and Auditory Speech Processing Systems through Matched Stimulus Representations

It has been postulated that the brain is organized by "metamodal," sensory-independent cortical modules capable of performing tasks (e.g., word recognition) in both "standard" and novel sensory modalities. Still, this theory has primarily been tested in sensory-deprived individuals, with mixed evidence in neurotypical subjects, thereby limiting its support as a general principle of brain organization. Critically, current theories of metamodal processing do not specify requirements for successful metamodal processing at the level of neural representations. Specification at this level may be particularly important in neurotypical individuals, where novel sensory modalities must interface with existing representations for the standard sense. Here we hypothesized that effective metamodal engagement of a cortical area requires congruence between stimulus representations in the standard and novel sensory modalities in that region. To test this, we first used fMRI to identify bilateral auditory speech representations. We then trained 20 human participants (12 female) to recognize vibrotactile versions of auditory words using one of two auditory-to-vibrotactile algorithms. The vocoded algorithm attempted to match the encoding scheme of auditory speech while the token-based algorithm did not. Crucially, using fMRI, we found that only in the vocoded group did trained-vibrotactile stimuli recruit speech representations in the superior temporal gyrus and lead to increased coupling between them and somatosensory areas. Our results advance our understanding of brain organization by providing new insight into unlocking the metamodal potential of the brain, thereby benefitting the design of novel sensory substitution devices that aim to tap into existing processing streams in the brain.SIGNIFICANCE STATEMENT It has been proposed that the brain is organized by "metamodal," sensory-independent modules specialized for performing certain tasks. This idea has inspired therapeutic applications, such as sensory substitution devices, for example, enabling blind individuals "to see" by transforming visual input into soundscapes. Yet, other studies have failed to demonstrate metamodal engagement. Here, we tested the hypothesis that metamodal engagement in neurotypical individuals requires matching the encoding schemes between stimuli from the novel and standard sensory modalities. We trained two groups of subjects to recognize words generated by one of two auditory-to-vibrotactile transformations. Critically, only vibrotactile stimuli that were matched to the neural encoding of auditory speech engaged auditory speech areas after training. This suggests that matching encoding schemes is critical to unlocking the brain's metamodal potential.

Prefrontal cortex state representations shape human credit assignment

People learn adaptively from feedback, but the rate of such learning differs drastically across individuals and contexts. Here, we examine whether this variability reflects differences in what is learned. Leveraging a neurocomputational approach that merges fMRI and an iterative reward learning task, we link the specificity of credit assignment-how well people are able to appropriately attribute outcomes to their causes-to the precision of neural codes in the prefrontal cortex (PFC). Participants credit task-relevant cues more precisely in social compared vto nonsocial contexts, a process that is mediated by high-fidelity (i.e., distinct and consistent) state representations in the PFC. Specifically, the medial PFC and orbitofrontal cortex work in concert to match the neural codes from feedback to those at choice, and the strength of these common neural codes predicts credit assignment precision. Together this work provides a window into how neural representations drive adaptive learning.

Adaptive coding of stimulus information in human frontoparietal cortex during visual classification

The neural mechanisms of how frontal and parietal brain regions support flexible adaptation of behavior remain poorly understood. Here, we used functional magnetic resonance imaging (fMRI) and representational similarity analysis (RSA) to investigate frontoparietal representations of stimulus information during visual classification under varying task demands. Based on prior research, we predicted that increasing perceptual task difficulty should lead to adaptive changes in stimulus coding: task-relevant category information should be stronger, while task-irrelevant exemplar-level stimulus information should become weaker, reflecting a focus on the behaviorally relevant category information. Counter to our expectations, however, we found no evidence for adaptive changes in category coding. We did find weakened coding at the exemplar-level within categories however, demonstrating that task-irrelevant information is de-emphasized in frontoparietal cortex. These findings reveal adaptive coding of stimulus information at the exemplar-level, highlighting how frontoparietal regions might support behavior even under challenging conditions.

Task state representations in vmPFC mediate relevant and irrelevant value signals and their behavioral influence

The ventromedial prefrontal-cortex (vmPFC) is known to contain expected value signals that inform our choices. But expected values even for the same stimulus can differ by task. In this study, we asked how the brain flexibly switches between such value representations in a task-dependent manner. Thirty-five participants alternated between tasks in which either stimulus color or motion predicted rewards. We show that multivariate vmPFC signals contain a rich representation that includes the current task state or context (motion/color), the associated expected value, and crucially, the irrelevant value of the alternative context. We also find that irrelevant value representations in vmPFC compete with relevant value signals, interact with task-state representations and relate to behavioral signs of value competition. Our results shed light on vmPFC's role in decision making, bridging between its role in mapping observations onto the task states of a mental map, and computing expected values for multiple states.

Temporal lobe perceptual predictions for speech are instantiated in motor cortex and reconciled by inferior frontal cortex

Humans use predictions to improve speech perception, especially in noisy environments. Here we use 7-T functional MRI (fMRI) to decode brain representations of written phonological predictions and degraded speech signals in healthy humans and people with selective frontal neurodegeneration (non-fluent variant primary progressive aphasia [nfvPPA]). Multivariate analyses of item-specific patterns of neural activation indicate dissimilar representations of verified and violated predictions in left inferior frontal gyrus, suggestive of processing by distinct neural populations. In contrast, precentral gyrus represents a combination of phonological information and weighted prediction error. In the presence of intact temporal cortex, frontal neurodegeneration results in inflexible predictions. This manifests neurally as a failure to suppress incorrect predictions in anterior superior temporal gyrus and reduced stability of phonological representations in precentral gyrus. We propose a tripartite speech perception network in which inferior frontal gyrus supports prediction reconciliation in echoic memory, and precentral gyrus invokes a motor model to instantiate and refine perceptual predictions for speech.

Decoding and geometry of ten finger movements in human posterior parietal cortex and motor cortex

Objective. Enable neural control of individual prosthetic fingers for participants with upper-limb paralysis.Approach. Two tetraplegic participants were each implanted with a 96-channel array in the left posterior parietal cortex (PPC). One of the participants was additionally implanted with a 96-channel array near the hand knob of the left motor cortex (MC). Across tens of sessions, we recorded neural activity while the participants attempted to move individual fingers of the right hand. Offline, we classified attempted finger movements from neural firing rates using linear discriminant analysis with cross-validation. The participants then used the neural classifier online to control individual fingers of a brain-machine interface (BMI). Finally, we characterized the neural representational geometry during individual finger movements of both hands.Main Results. The two participants achieved 86% and 92% online accuracy during BMI control of the contralateral fingers (chance = 17%). Offline, a linear decoder achieved ten-finger decoding accuracies of 70% and 66% using respective PPC recordings and 75% using MC recordings (chance = 10%). In MC and in one PPC array, a factorized code linked corresponding finger movements of the contralateral and ipsilateral hands.Significance. This is the first study to decode both contralateral and ipsilateral finger movements from PPC. Online BMI control of contralateral fingers exceeded that of previous finger BMIs. PPC and MC signals can be used to control individual prosthetic fingers, which may contribute to a hand restoration strategy for people with tetraplegia.

A gradual transition from veridical to categorical representations along the visual hierarchy during working memory, but not perception

The ability to stably maintain visual information over brief delays is central to cognitive functioning. One possible way to achieve robust working memory maintenance is by having multiple concurrent mnemonic representations across multiple cortical loci. For example, early visual cortex might contribute to storage by representing information in a "sensory-like" format, while intraparietal sulcus uses a format transformed away from sensory driven responses. As an explicit test of mnemonic code transformations along the visual hierarchy, we quantitatively modeled the progression of veridical-to-categorical orientation representations in human participants. Participants directly viewed, or held in mind, an oriented grating pattern, and the similarity between fMRI activation patterns for different orientations was calculated throughout retinotopic cortex. During direct perception, similarity was clustered around cardinal orientations, while during working memory the obliques were represented more similarly. We modeled these similarity patterns based on the known distribution of orientation information in the natural world: The "veridical" model uses an efficient coding framework to capture hypothesized representations during visual perception. The "categorical" model assumes that different "psychological distances" between orientations result in orientation categorization relative to cardinal axes. During direct perception, the veridical model explained the data well in early visual areas, while the categorical model did worse. During working memory, the veridical model only explained some of the data, while the categorical model gradually gained explanatory power for increasingly anterior retinotopic regions. These findings suggest that directly viewed images are represented veridically, but once visual information is no longer tethered to the sensory world, there is a gradual progression to more categorical mnemonic formats along the visual hierarchy.

Classification of age groups and task conditions provides additional evidence for differences in electrophysiological correlates of inhibitory control across the lifespan

The aim of this study was to extend previous findings on selective attention over a lifetime using machine learning procedures. By decoding group membership and stimulus type, we aimed to study differences in the neural representation of inhibitory control across age groups at a single-trial level. We re-analyzed data from 211 subjects from six age groups between 8 and 83 years of age. Based on single-trial EEG recordings during a flanker task, we used support vector machines to predict the age group as well as to determine the presented stimulus type (i.e., congruent, or incongruent stimulus). The classification of group membership was highly above chance level (accuracy: 55%, chance level: 17%). Early EEG responses were found to play an important role, and a grouped pattern of classification performance emerged corresponding to age structure. There was a clear cluster of individuals after retirement, i.e., misclassifications mostly occurred within this cluster. The stimulus type could be classified above chance level in ~ 95% of subjects. We identified time windows relevant for classification performance that are discussed in the context of early visual attention and conflict processing. In children and older adults, a high variability and latency of these time windows were found. We were able to demonstrate differences in neuronal dynamics at the level of individual trials. Our analysis was sensitive to mapping gross changes, e.g., at retirement age, and to differentiating components of visual attention across age groups, adding value for the diagnosis of cognitive status across the lifespan. Overall, the results highlight the use of machine learning in the study of brain activity over a lifetime.

Uncertainty aversion predicts the neural expansion of semantic representations

Correctly identifying the meaning of a stimulus requires activating the appropriate semantic representation among many alternatives. One way to reduce this uncertainty is to differentiate semantic representations from each other, thereby expanding the semantic space. Here, in four experiments, we test this semantic-expansion hypothesis, finding that uncertainty-averse individuals exhibit increasingly differentiated and separated semantic representations. This effect is mirrored at the neural level, where uncertainty aversion predicts greater distances between activity patterns in the left inferior frontal gyrus when reading words, and enhanced sensitivity to the semantic ambiguity of these words in the ventromedial prefrontal cortex. Two direct tests of the behavioural consequences of semantic expansion further reveal that uncertainty-averse individuals exhibit reduced semantic interference and poorer generalization. Together, these findings show that the internal structure of our semantic representations acts as an organizing principle to make the world more identifiable.