Why neurons mix: high dimensionality for higher cognition

Modular representations emerge in neural networks trained to perform context-dependent tasks

The brain has large-scale modular structure in the form of brain regions, which are thought to arise from constraints on connectivity and the physical geometry of the cortical sheet. In contrast, experimental and theoretical work has argued both for and against the existence of specialized sub-populations of neurons (modules) within single brain regions. By studying artificial neural networks, we show that this local modularity emerges to support context-dependent behavior, but only when the input is low-dimensional. No anatomical constraints are required. We also show when modular specialization emerges at the population level (different modules correspond to orthogonal subspaces). Modularity yields abstract representations, allows for rapid learning and generalization on novel tasks, and facilitates the rapid learning of related contexts. Non-modular representations facilitate the rapid learning of unrelated contexts. Our findings reconcile conflicting experimental results and make predictions for future experiments.

Language proficiency is associated with neural representational dimensionality of semantic concepts

Previous studies suggest that semantic concepts are characterized by high-dimensional neural representations and that language proficiency affects semantic processing. However, it is not clear whether language proficiency modulates the dimensional representations of semantic concepts at the neural level. To address this question, the present study adopted principal component analysis (PCA) and representational similarity analysis (RSA) to examine the differences in representational dimensionalities (RDs) and in semantic representations between words in highly proficient (Chinese) and less proficient (English) language. PCA results revealed that language proficiency increased the dimensions of lexical representations in the left inferior frontal gyrus, temporal pole, inferior temporal gyrus, supramarginal gyrus, angular gyrus, and fusiform gyrus. RSA results further showed that these regions represented semantic information and that higher semantic representations were observed in highly proficient language relative to less proficient language. These results suggest that language proficiency is associated with the neural representational dimensionality of semantic concepts.

A transient high-dimensional geometry affords stable conjunctive subspaces for efficient action selection

Flexible action selection requires cognitive control mechanisms capable of mapping the same inputs to different output actions depending on the context. From a neural state-space perspective, this requires a control representation that separates similar input neural states by context. Additionally, for action selection to be robust and time-invariant, information must be stable in time, enabling efficient readout. Here, using EEG decoding methods, we investigate how the geometry and dynamics of control representations constrain flexible action selection in the human brain. Participants performed a context-dependent action selection task. A forced response procedure probed action selection different states in neural trajectories. The result shows that before successful responses, there is a transient expansion of representational dimensionality that separated conjunctive subspaces. Further, the dynamics stabilizes in the same time window, with entry into this stable, high-dimensional state predictive of individual trial performance. These results establish the neural geometry and dynamics the human brain needs for flexible control over behavior.

The big mixup: Neural representation during natural modes of primate visual behavior

The primate brain has evolved specialized visual capacities to navigate complex physical and social environments. Researchers studying cortical circuits underlying these capacities have traditionally favored the use of simplified tasks and brief stimulus presentations in order to isolate cognitive variables with tight experimental control. As a result, operational theories about visual brain function have come to emphasize feature detection, hierarchical stimulus encoding, top-down task modulation, and functional segregation in distinct cortical areas. Recently, however, experimental paradigms combining natural behavior with electrophysiological recordings have begun to offer a distinctly different portrait of how the brain takes in and analyzes its visual surroundings. The present article reviews recent work in this area, highlighting some of the more surprising findings in domains of social vision and spatial navigation along with shifts in thinking that have begun to emanate from this approach.

Locus coeruleus modulation of prefrontal dynamics and encoding of flexible rule switching

Behavioral flexibility, the ability to adjust behavioral strategies in response to changing environmental contingencies and internal demands, is fundamental to cognitive functions. Despite a large body of pharmacology and lesion studies, the underlying neurophysiological correlates and mechanisms that support flexible rule switching are under active investigation. To address this question, we trained mice to distinguish complex sensory cues comprising different perceptual dimensions (set shifting). Endoscopic calcium imaging revealed that medial prefrontal cortex (mPFC) neurons exhibited pronounced dynamic changes during rule switching. Notably, prominent encoding capacity in the mPFC was associated with switching across, but not within perceptual dimensions. We then showed the functional importance of the ascending input from the locus coeruleus (LC), as LC inhibition impaired rule switching behavior and impeded mPFC dynamic processes and encoding. Our results highlight the pivotal role of the mPFC in set shifting processes and demonstrate the profound impact of ascending neuromodulation on shaping prefrontal neural dynamics and behavioral flexibility.

High-dimensional encoding of movement by single neurons in basal ganglia output

The substantia nigra pars reticulata (SNpr), an output structure of the basal ganglia, is hypothesized to gate movement execution. Previous studies in the eye movement system focusing mostly on saccades have reported that SNpr neurons are tonically active and either pause or increase their firing during movements, consistent with the gating role. We recorded activity in the SNpr of two monkeys during smooth pursuit and saccadic eye movements. SNpr neurons exhibited highly diverse reaction patterns during pursuit, including frequent increases and decreases in firing rate, uncorrelated responses in different movement directions and in reward conditions that resulted in the high dimensional activity of single neurons. These diverse temporal patterns surpassed those in other oculomotor areas in the medial-temporal cortex, frontal cortex, basal ganglia, and cerebellum. These findings suggest that temporal properties of the responses enrich the coding capacity of the basal ganglia output beyond gating or permitting movement.

A geometrical solution underlies general neural principle for serial ordering

A general mathematical description of how the brain sequentially encodes knowledge remains elusive. We propose a linear solution for serial learning tasks, based on the concept of mixed selectivity in high-dimensional neural state spaces. In our framework, neural representations of items in a sequence are projected along a "geometric" mental line learned through classical conditioning. The model successfully solves serial position tasks and explains behaviors observed in humans and animals during transitive inference tasks amidst noisy sensory input and stochastic neural activity. This approach extends to recurrent neural networks performing motor decision tasks, where the same geometric mental line correlates with motor plans and modulates network activity according to the symbolic distance between items. Serial ordering is thus predicted to emerge as a monotonic mapping between sensory input and behavioral output, highlighting a possible pivotal role for motor-related associative cortices in transitive inference tasks.

Semi-orthogonal subspaces for value mediate a binding and generalization trade-off

When choosing between options, we must associate their values with the actions needed to select them. We hypothesize that the brain solves this binding problem through neural population subspaces. Here, in macaques performing a choice task, we show that neural populations in five reward-sensitive regions encode the values of offers presented on the left and right in distinct subspaces. This encoding is sufficient to bind offer values to their locations while preserving abstract value information. After offer presentation, all areas encode the value of the first and second offers in orthogonal subspaces; this orthogonalization also affords binding. Our binding-by-subspace hypothesis makes two new predictions confirmed by the data. First, behavioral errors should correlate with spatial, but not temporal, neural misbinding. Second, behavioral errors should increase when offers have low or high values, compared to medium values, even when controlling for value difference. Together, these results support the idea that the brain uses semi-orthogonal subspaces to bind features.

Practice Reshapes the Geometry and Dynamics of Task-tailored Representations

Extensive practice makes task performance more efficient and precise, leading to automaticity. However, theories of automaticity differ on which levels of task representations (e.g., low-level features, stimulus-response mappings, or high-level conjunctive memories of individual events) change with practice, despite predicting the same pattern of improvement (e.g., power law of practice). To resolve this controversy, we built on recent theoretical advances in understanding computations through neural population dynamics. Specifically, we hypothesized that practice optimizes the neural representational geometry of task representations to minimally separate the highest-level task contingencies needed for successful performance. This involves efficiently reaching conjunctive neural states that integrate task-critical features nonlinearly while abstracting over non-critical dimensions. To test this hypothesis, human participants (n = 40) engaged in extensive practice of a simple, context-dependent action selection task over 3 days while recording EEG. During initial rapid improvement in task performance, representations of the highest-level, context-specific conjunctions of task-features were enhanced as a function of the number of successful episodes. Crucially, only enhancement of these conjunctive representations, and not lower-order representations, predicted the power-law improvement in performance. Simultaneously, over sessions, these conjunctive neural states became more stable earlier in time and more aligned, abstracting over redundant task features, which correlated with offline performance gain in reducing switch costs. Thus, practice optimizes the dynamic representational geometry as task-tailored neural states that minimally tesselate the task space, taming their high-dimensionality.

Brain structure and activity predicting cognitive maturation in adolescence

Cognitive abilities of primates, including humans, continue to improve through adolescence 1,2. While a range of changes in brain structure and connectivity have been documented 3,4, how they affect neuronal activity that ultimately determines performance of cognitive functions remains unknown. Here, we conducted a multilevel longitudinal study of monkey adolescent neurocognitive development. The developmental trajectory of neural activity in the prefrontal cortex accounted remarkably well for working memory improvements. While complex aspects of activity changed progressively during adolescence, such as the rotation of stimulus representation in multidimensional neuronal space, which has been implicated in cognitive flexibility, even simpler attributes, such as the baseline firing rate in the period preceding a stimulus appearance had predictive power over behavior. Unexpectedly, decreases in brain volume and thickness, which are widely thought to underlie cognitive changes in humans 5 did not predict well the trajectory of neural activity or cognitive performance changes. Whole brain cortical volume in particular, exhibited an increase and reached a local maximum in late adolescence, at a time of rapid behavioral improvement. Maturation of long-distance white matter tracts linking the frontal lobe with areas of the association cortex and subcortical regions best predicted changes in neuronal activity and behavior. Our results provide evidence that optimization of neural activity depending on widely distributed circuitry effects cognitive development in adolescence.

Integration and competition between space and time in the hippocampus

Episodic memory is organized in both spatial and temporal contexts. The hippocampus is crucial for episodic memory and has been demonstrated to encode spatial and temporal information. However, how the representations of space and time interact in the hippocampal memory system is still unclear. Here, we recorded the activity of hippocampal CA1 neurons in mice in a variety of one-dimensional navigation tasks while systematically varying the speed of the animals. For all tasks, we found neurons simultaneously represented space and elapsed time. There was a negative correlation between the preferred space and lap duration, e.g., the preferred spatial position shifted more toward the origin when the lap duration became longer. A similar relationship between the preferred time and traveled distance was also observed. The results strongly suggest a competitive and integrated representation of space-time by single hippocampal neurons, which may provide the neural basis for spatiotemporal contexts.

Future movement plans interact in sequential arm movements

Real-world actions often comprise a series of movements that cannot be entirely planned before initiation. When these actions are executed rapidly, the planning of multiple future movements needs to occur simultaneously with the ongoing action. How the brain solves this task remains unknown. Here, we address this question with a new sequential arm reaching paradigm that manipulates how many future reaches are available for planning while controlling execution of the ongoing reach. We show that participants plan at least two future reaches simultaneously with an ongoing reach. Further, the planning processes of the two future reaches are not independent of one another. Evidence that the planning processes interact is twofold. First, correcting for a visual perturbation of the ongoing reach target is slower when more future reaches are planned. Second, the curvature of the current reach is modified based on the next reach only when their planning processes temporally overlap. These interactions between future planning processes may enable smooth production of sequential actions by linking individual segments of a long sequence at the level of motor planning.

Population-level coding of avoidance learning in medial prefrontal cortex

The medial prefrontal cortex (mPFC) has been proposed to link sensory inputs and behavioral outputs to mediate the execution of learned behaviors. However, how such a link is implemented has remained unclear. To measure prefrontal neural correlates of sensory stimuli and learned behaviors, we performed population calcium imaging during a new tone-signaled active avoidance paradigm in mice. We developed an analysis approach based on dimensionality reduction and decoding that allowed us to identify interpretable task-related population activity patterns. While a large fraction of tone-evoked activity was not informative about behavior execution, we identified an activity pattern that was predictive of tone-induced avoidance actions and did not occur for spontaneous actions with similar motion kinematics. Moreover, this avoidance-specific activity differed between distinct avoidance actions learned in two consecutive tasks. Overall, our results are consistent with a model in which mPFC contributes to the selection of goal-directed actions by transforming sensory inputs into specific behavioral outputs through distributed population-level computations.

Timescales of learning in prefrontal cortex

The lateral prefrontal cortex (PFC) in humans and other primates is critical for immediate, goal-directed behaviour and working memory, which are classically considered distinct from the cognitive and neural circuits that support long-term learning and memory. Over the past few years, a reconsideration of this textbook perspective has emerged, in that different timescales of memory-guided behaviour are in constant interaction during the pursuit of immediate goals. Here, we will first detail how neural activity related to the shortest timescales of goal-directed behaviour (which requires maintenance of current states and goals in working memory) is sculpted by long-term knowledge and learning - that is, how the past informs present behaviour. Then, we will outline how learning across different timescales (from seconds to years) drives plasticity in the primate lateral PFC, from single neuron firing rates to mesoscale neuroimaging activity patterns. Finally, we will review how, over days and months of learning, dense local and long-range connectivity patterns in PFC facilitate longer-lasting changes in population activity by changing synaptic weights and recruiting additional neural resources to inform future behaviour. Our Review sheds light on how the machinery of plasticity in PFC circuits facilitates the integration of learned experiences across time to best guide adaptive behaviour.

Centering cognitive neuroscience on task demands and generalization

Cognitive neuroscience seeks generalizable theories explaining the relationship between behavioral, physiological and mental states. In pursuit of such theories, we propose a theoretical and empirical framework that centers on understanding task demands and the mutual constraints they impose on behavior and neural activity. Task demands emerge from the interaction between an agent's sensory impressions, goals and behavior, which jointly shape the activity and structure of the nervous system on multiple spatiotemporal scales. Understanding this interaction requires multitask studies that vary more than one experimental component (for example, stimuli and instructions) combined with dense behavioral and neural sampling and explicit testing for generalization across tasks and data modalities. By centering task demands rather than mental processes that tasks are assumed to engage, this framework paves the way for the discovery of new generalizable concepts unconstrained by existing taxonomies, and moves cognitive neuroscience toward an action-oriented, dynamic and integrated view of the brain.

A Transient High-dimensional Geometry Affords Stable Conjunctive Subspaces for Efficient Action Selection

Flexible action selection requires cognitive control mechanisms capable of mapping the same inputs to different output actions depending on the context. From a neural state-space perspective, this requires a control representation that separates similar input neural states by context. Additionally, for action selection to be robust and time-invariant, information must be stable in time, enabling efficient readout. Here, using EEG decoding methods, we investigate how the geometry and dynamics of control representations constrain flexible action selection in the human brain. Participants performed a context-dependent action selection task. A forced response procedure probed action selection different states in neural trajectories. The result shows that before successful responses, there is a transient expansion of representational dimensionality that separated conjunctive subspaces. Further, the dynamics stabilizes in the same time window, with entry into this stable, high-dimensional state predictive of individual trial performance. These results establish the neural geometry and dynamics the human brain needs for flexible control over behavior.

Dissociable Representations of Decision Variables within Subdivisions of the Macaque Orbital and Ventrolateral Frontal Cortex

The ventral frontal cortex (VFC) in macaques is involved in many affective and cognitive processes and has a key role in flexibly guiding reward-based decision-making. VFC is composed of a set of anatomically distinct subdivisions that are within the orbitofrontal cortex, ventrolateral prefrontal cortex, and anterior insula. In part, because prior studies have lacked the resolution to test for differences, it is unclear if neural representations related to decision-making are dissociable across these subdivisions. Here we recorded the activity of thousands of neurons within eight anatomically defined subdivisions of VFC in male macaque monkeys performing a two-choice probabilistic task for different fruit juice outcomes. We found substantial variation in the encoding of decision variables across these eight subdivisions. Notably, ventrolateral Area 12l was unique relative to the other areas that we recorded from as the activity of single neurons integrated multiple attributes when monkeys evaluated the different choice options. Activity within Area 12o, in contrast, more closely represented reward probability and whether reward was received on a given trial. Orbitofrontal Area 11m/l contained more specific representations of the quality of the outcome that could be earned later on. We also found that reward delivery encoding was highly distributed across all VFC subdivisions, while the properties of the reward, such as its flavor, were more strongly represented in Areas 11m/l and 13m. Taken together, our work reveals the diversity of encoding within the various anatomically distinct subdivisions of VFC in primates.

Distinct roles of prefrontal cortex neurons in set shifting

Cognitive flexibility, the ability to adjust behavioral strategies in response to changing environmental contingencies, requires adaptive processing of internal states and contextual cues to guide goal-oriented behavior, and is dependent on prefrontal cortex (PFC) functions. However, the neurophysiological underpinning of how the PFC supports cognitive flexibility is not well understood and has been under active investigation. We recorded spiking activity from single PFC neurons in mice performing the attentional set-shifting task, where mice learned to associate different contextually relevant sensory stimuli to reward. We identified subgroups of PFC neurons encoding task context, choice and trial outcome. Putative fast-spiking neurons were more involved in representing outcome and choice than putative regular-spiking neurons. Regression model further revealed that task context and trial outcome modulated the activity of choice-encoding neurons in rule-dependent and cell type-dependent manners. Together, our data provide new evidence to elucidate PFC's role in cognitive flexibility, suggesting differential cell type-specific engagement during set shifting, and that both contextual rule representation and trial outcome monitoring underlie PFC's unique capacity to support flexible behavioral switching.

The Dimensions of dimensionality

Cognitive scientists often infer multidimensional representations from data. Whether the data involve text, neuroimaging, neural networks, or human judgments, researchers frequently infer and analyze latent representational spaces (i.e., embeddings). However, the properties of a latent representation (e.g., prediction performance, interpretability, compactness) depend on the inference procedure, which can vary widely across endeavors. For example, dimensions are not always globally interpretable and the dimensionality of different embeddings may not be readily comparable. Moreover, the dichotomy between multidimensional spaces and purportedly richer representational formats, such as graph representations, is misleading. We review what the different notions of dimension in cognitive science imply for how these latent representations should be used and interpreted.

Stability of cross-sensory input to primary somatosensory cortex across experience

Merging information from across sensory modalities is key to forming robust, disambiguated percepts of the world, yet how the brain achieves this feat remains unclear. Recent observations of cross-modal influences in primary sensory cortical areas have suggested that multisensory integration may occur in the earliest stages of cortical processing, but the role of these responses is still poorly understood. We address these questions by testing several hypotheses about the possible functions served by auditory influences on the barrel field of mouse primary somatosensory cortex (S1) using in vivo 2-photon calcium imaging. We observed sound-evoked spiking activity in a small fraction of cells overall, and moreover that this sparse activity was insufficient to encode auditory stimulus identity; few cells responded preferentially to one sound or another, and a linear classifier trained to decode auditory stimuli from population activity performed barely above chance. Moreover S1 did not encode information about specific audio-tactile feature conjunctions that we tested. Our ability to decode auditory audio-tactile stimuli from neural activity remained unchanged after both passive experience and reinforcement. Collectively, these results suggest that while a primary sensory cortex is highly plastic with regard to its own modality, the influence of other modalities are remarkably stable and play a largely stimulus-non-specific role.

Revealing unexpected complex encoding but simple decoding mechanisms in motor cortex via separating behaviorally relevant neural signals

In motor cortex, behaviorally relevant neural responses are entangled with irrelevant signals, which complicates the study of encoding and decoding mechanisms. It remains unclear whether behaviorally irrelevant signals could conceal some critical truth. One solution is to accurately separate behaviorally relevant and irrelevant signals at both single-neuron and single-trial levels, but this approach remains elusive due to the unknown ground truth of behaviorally relevant signals. Therefore, we propose a framework to define, extract, and validate behaviorally relevant signals. Analyzing separated signals in three monkeys performing different reaching tasks, we found neural responses previously considered to contain little information actually encode rich behavioral information in complex nonlinear ways. These responses are critical for neuronal redundancy and reveal movement behaviors occupy a higher-dimensional neural space than previously expected. Surprisingly, when incorporating often-ignored neural dimensions, behaviorally relevant signals can be decoded linearly with comparable performance to nonlinear decoding, suggesting linear readout may be performed in motor cortex. Our findings prompt that separating behaviorally relevant signals may help uncover more hidden cortical mechanisms.

Desegregation of neuronal predictive processing

Neural circuits construct internal 'world-models' to guide behavior. The predictive processing framework posits that neural activity signaling sensory predictions and concurrently computing prediction-errors is a signature of those internal models. Here, to understand how the brain generates predictions for complex sensorimotor signals, we investigate the emergence of high-dimensional, multi-modal predictive representations in recurrent networks. We find that robust predictive processing arises in a network with loose excitatory/inhibitory balance. Contrary to previous proposals of functionally specialized cell-types, the network exhibits desegregation of stimulus and prediction-error representations. We confirmed these model predictions by experimentally probing predictive-coding circuits using a rich stimulus-set to violate learned expectations. When constrained by data, our model further reveals and makes concrete testable experimental predictions for the distinct functional roles of excitatory and inhibitory neurons, and of neurons in different layers along a laminar hierarchy, in computing multi-modal predictions. These results together imply that in natural conditions, neural representations of internal models are highly distributed, yet structured to allow flexible readout of behaviorally-relevant information. The generality of our model advances the understanding of computation of internal models across species, by incorporating different types of predictive computations into a unified framework.

Mixing and mingling in visual working memory: Inter-item competition is feature-specific during encoding and feature-general during maintenance

Visual working memory (WM) is a central cognitive ability but is capacity-limited due to competition between remembered items. Understanding whether inter-item competition depends on the similarity of the features being remembered has important implications for determining if competition occurs in sensory or post-sensory stages of processing. Experiment 1 compared the precision of WM across homogeneous displays, where items belonged to the same feature type (e.g., colorful circles), and heterogeneous displays (e.g., colorful circles and oriented bars). Performance was better for heterogeneous displays, suggesting a feature-specific component of interference. However, Experiment 2 used a retro-cueing task to isolate encoding from online maintenance and revealed that inter-item competition during storage was not feature-specific. The data support recent models of WM in which inter-item interference - and hence capacity limits in WM - occurs in higher-order structures that receive convergent input from a diverse array of feature-specific representations.

Neural representational geometries reflect behavioral differences in monkeys and recurrent neural networks

Animals likely use a variety of strategies to solve laboratory tasks. Traditionally, combined analysis of behavioral and neural recording data across subjects employing different strategies may obscure important signals and give confusing results. Hence, it is essential to develop techniques that can infer strategy at the single-subject level. We analyzed an experiment in which two male monkeys performed a visually cued rule-based task. The analysis of their performance shows no indication that they used a different strategy. However, when we examined the geometry of stimulus representations in the state space of the neural activities recorded in dorsolateral prefrontal cortex, we found striking differences between the two monkeys. Our purely neural results induced us to reanalyze the behavior. The new analysis showed that the differences in representational geometry are associated with differences in the reaction times, revealing behavioral differences we were unaware of. All these analyses suggest that the monkeys are using different strategies. Finally, using recurrent neural network models trained to perform the same task, we show that these strategies correlate with the amount of training, suggesting a possible explanation for the observed neural and behavioral differences.

Thalamocortical architectures for flexible cognition and efficient learning

The brain exhibits a remarkable ability to learn and execute context-appropriate behaviors. How it achieves such flexibility, without sacrificing learning efficiency, is an important open question. Neuroscience, psychology, and engineering suggest that reusing and repurposing computations are part of the answer. Here, we review evidence that thalamocortical architectures may have evolved to facilitate these objectives of flexibility and efficiency by coordinating distributed computations. Recent work suggests that distributed prefrontal cortical networks compute with flexible codes, and that the mediodorsal thalamus provides regularization to promote efficient reuse. Thalamocortical interactions resemble hierarchical Bayesian computations, and their network implementation can be related to existing gating, synchronization, and hub theories of thalamic function. By reviewing recent findings and providing a novel synthesis, we highlight key research horizons integrating computation, cognition, and systems neuroscience.

Irrational choices via a curvilinear representational geometry for value

We make decisions by comparing values, but it is not yet clear how value is represented in the brain. Many models assume, if only implicitly, that the representational geometry of value is linear. However, in part due to a historical focus on noisy single neurons, rather than neuronal populations, this hypothesis has not been rigorously tested. Here, we examine the representational geometry of value in the ventromedial prefrontal cortex (vmPFC), a part of the brain linked to economic decision-making, in two male rhesus macaques. We find that values are encoded along a curved manifold in vmPFC. This curvilinear geometry predicts a specific pattern of irrational decision-making: that decision-makers will make worse choices when an irrelevant, decoy option is worse in value, compared to when it is better. We observe this type of irrational choices in behavior. Together, these results not only suggest that the representational geometry of value is nonlinear, but that this nonlinearity could impose bounds on rational decision-making.

Neural dynamics underlying minute-timescale persistent behavior in the human brain

The ability to pursue long-term goals relies on a representations of task context that can both be maintained over long periods of time and switched flexibly when goals change. Little is known about the neural substrate for such minute-scale maintenance of task sets. Utilizing recordings in neurosurgical patients, we examined how groups of neurons in the human medial frontal cortex and hippocampus represent task contexts. When cued explicitly, task context was encoded in both brain areas and changed rapidly at task boundaries. Hippocampus exhibited a temporally dynamic code with fast decorrelation over time, preventing cross-temporal generalization. Medial frontal cortex exhibited a static code that decorrelated slowly, allowing generalization across minutes of time. When task context needed to be inferred as a latent variable, hippocampus encoded task context with a static code. These findings reveal two possible regimes for encoding minute-scale task-context representations that were engaged differently based on task demands.

Mixed selectivity: Cellular computations for complexity

The property of mixed selectivity has been discussed at a computational level and offers a strategy to maximize computational power by adding versatility to the functional role of each neuron. Here, we offer a biologically grounded implementational-level mechanistic explanation for mixed selectivity in neural circuits. We define pure, linear, and nonlinear mixed selectivity and discuss how these response properties can be obtained in simple neural circuits. Neurons that respond to multiple, statistically independent variables display mixed selectivity. If their activity can be expressed as a weighted sum, then they exhibit linear mixed selectivity; otherwise, they exhibit nonlinear mixed selectivity. Neural representations based on diverse nonlinear mixed selectivity are high dimensional; hence, they confer enormous flexibility to a simple downstream readout neural circuit. However, a simple neural circuit cannot possibly encode all possible mixtures of variables simultaneously, as this would require a combinatorially large number of mixed selectivity neurons. Gating mechanisms like oscillations and neuromodulation can solve this problem by dynamically selecting which variables are mixed and transmitted to the readout.

Orthogonality of sensory and contextual categorical dynamics embedded in a continuum of responses from the second somatosensory cortex

How does the brain simultaneously process signals that bring complementary information, like raw sensory signals and their transformed counterparts, without any disruptive interference? Contemporary research underscores the brain's adeptness in using decorrelated responses to reduce such interference. Both neurophysiological findings and artificial neural networks support the notion of orthogonal representation for signal differentiation and parallel processing. Yet, where, and how raw sensory signals are transformed into more abstract representations remains unclear. Using a temporal pattern discrimination task in trained monkeys, we revealed that the second somatosensory cortex (S2) efficiently segregates faithful and transformed neural responses into orthogonal subspaces. Importantly, S2 population encoding for transformed signals, but not for faithful ones, disappeared during a nondemanding version of this task, which suggests that signal transformation and their decoding from downstream areas are only active on-demand. A mechanistic computation model points to gain modulation as a possible biological mechanism for the observed context-dependent computation. Furthermore, individual neural activities that underlie the orthogonal population representations exhibited a continuum of responses, with no well-determined clusters. These findings advocate that the brain, while employing a continuum of heterogeneous neural responses, splits population signals into orthogonal subspaces in a context-dependent fashion to enhance robustness, performance, and improve coding efficiency.

Coordinated Response Modulations Enable Flexible Use of Visual Information

We use sensory information in remarkably flexible ways. We can generalize by ignoring task-irrelevant features, report different features of a stimulus, and use different actions to report a perceptual judgment. These forms of flexible behavior are associated with small modulations of the responses of sensory neurons. While the existence of these response modulations is indisputable, efforts to understand their function have been largely relegated to theory, where they have been posited to change information coding or enable downstream neurons to read out different visual and cognitive information using flexible weights. Here, we tested these ideas using a rich, flexible behavioral paradigm, multi-neuron, multi-area recordings in primary visual cortex (V1) and mid-level visual area V4. We discovered that those response modulations in V4 (but not V1) contain the ingredients necessary to enable flexible behavior, but not via those previously hypothesized mechanisms. Instead, we demonstrated that these response modulations are precisely coordinated across the population such that downstream neurons have ready access to the correct information to flexibly guide behavior without making changes to information coding or synapses. Our results suggest a novel computational role for task-dependent response modulations: they enable flexible behavior by changing the information that gets out of a sensory area, not by changing information coding within it.

Context-invariant beliefs are supported by dynamic reconfiguration of single unit functional connectivity in prefrontal cortex of male macaques

Natural behaviors occur in closed action-perception loops and are supported by dynamic and flexible beliefs abstracted away from our immediate sensory milieu. How this real-world flexibility is instantiated in neural circuits remains unknown. Here, we have male macaques navigate in a virtual environment by primarily leveraging sensory (optic flow) signals, or by more heavily relying on acquired internal models. We record single-unit spiking activity simultaneously from the dorsomedial superior temporal area (MSTd), parietal area 7a, and the dorso-lateral prefrontal cortex (dlPFC). Results show that while animals were able to maintain adaptive task-relevant beliefs regardless of sensory context, the fine-grain statistical dependencies between neurons, particularly in 7a and dlPFC, dynamically remapped with the changing computational demands. In dlPFC, but not 7a, destroying these statistical dependencies abolished the area's ability for cross-context decoding. Lastly, correlational analyses suggested that the more unit-to-unit couplings remapped in dlPFC, and the less they did so in MSTd, the less were population codes and behavior impacted by the loss of sensory evidence. We conclude that dynamic functional connectivity between neurons in prefrontal cortex maintain a stable population code and context-invariant beliefs during naturalistic behavior.

Acquisition of non-olfactory encoding improves odour discrimination in olfactory cortex

Olfaction is influenced by contextual factors, past experiences, and the animal's internal state. Whether this information is integrated at the initial stages of cortical odour processing is not known, nor how these signals may influence odour encoding. Here we revealed multiple and diverse non-olfactory responses in the primary olfactory (piriform) cortex (PCx), which dynamically enhance PCx odour discrimination according to behavioural demands. We performed recordings of PCx neurons from mice trained in a virtual reality task to associate odours with visual contexts to obtain a reward. We found that learning shifts PCx activity from encoding solely odours to a regime in which positional, contextual, and associative responses emerge on odour-responsive neurons that become mixed-selective. The modulation of PCx activity by these non-olfactory signals was dynamic, improving odour decoding during task engagement and in rewarded contexts. This improvement relied on the acquired mixed-selectivity, demonstrating how integrating extra-sensory inputs in sensory cortices can enhance sensory processing while encoding the behavioural relevance of stimuli.

A comparison of hippocampal and retrosplenial cortical spatial and contextual firing patterns

The hippocampus (HPC) and retrosplenial cortex (RSC) are key components of the brain's memory and navigation systems. Lesions of either region produce profound deficits in spatial cognition and HPC neurons exhibit well-known spatial firing patterns (place fields). Recent studies have also identified an array of navigation-related firing patterns in the RSC. However, there has been little work comparing the response properties and information coding mechanisms of these two brain regions. In the present study, we examined the firing patterns of HPC and RSC neurons in two tasks which are commonly used to study spatial cognition in rodents, open field foraging with an environmental context manipulation and continuous T-maze alternation. We found striking similarities in the kinds of spatial and contextual information encoded by these two brain regions. Neurons in both regions carried information about the rat's current spatial location, trajectories and goal locations, and both regions reliably differentiated the contexts. However, we also found several key differences. For example, information about head direction was a prominent component of RSC representations but was only weakly encoded in the HPC. The two regions also used different coding schemes, even when they encoded the same kind of information. As expected, the HPC employed a sparse coding scheme characterized by compact, high contrast place fields, and information about spatial location was the dominant component of HPC representations. RSC firing patterns were more consistent with a distributed coding scheme. Instead of compact place fields, RSC neurons exhibited broad, but reliable, spatial and directional tuning, and they typically carried information about multiple navigational variables. The observed similarities highlight the closely related functions of the HPC and RSC, whereas the differences in information types and coding schemes suggest that these two regions likely make somewhat different contributions to spatial cognition.

Advancing mechanistic explanations through natural and artificial embodied cognitive systems

Mougenot and Matheson propose that mechanistic models can explain behavior by describing the complex interactions among components of the brain, body, and environment as an integrated system, which aligns with embodied cognition. However, we suggest incorporating cognitive ontology theory and addressing degeneracy and neuronal reuse. We also recommend studying natural embodied cognition through artificial systems to develop a comprehensive mechanistic framework.

Computational role of structure in neural activity and connectivity

One major challenge of neuroscience is identifying structure in seemingly disorganized neural activity. Different types of structure have different computational implications that can help neuroscientists understand the functional role of a particular brain area. Here, we outline a unified approach to characterize structure by inspecting the representational geometry and the modularity properties of the recorded activity and show that a similar approach can also reveal structure in connectivity. We start by setting up a general framework for determining geometry and modularity in activity and connectivity and relating these properties with computations performed by the network. We then use this framework to review the types of structure found in recent studies of model networks performing three classes of computations.

Co-existence of synaptic plasticity and metastable dynamics in a spiking model of cortical circuits

Evidence for metastable dynamics and its role in brain function is emerging at a fast pace and is changing our understanding of neural coding by putting an emphasis on hidden states of transient activity. Clustered networks of spiking neurons have enhanced synaptic connections among groups of neurons forming structures called cell assemblies; such networks are capable of producing metastable dynamics that is in agreement with many experimental results. However, it is unclear how a clustered network structure producing metastable dynamics may emerge from a fully local plasticity rule, i.e., a plasticity rule where each synapse has only access to the activity of the neurons it connects (as opposed to the activity of other neurons or other synapses). Here, we propose a local plasticity rule producing ongoing metastable dynamics in a deterministic, recurrent network of spiking neurons. The metastable dynamics co-exists with ongoing plasticity and is the consequence of a self-tuning mechanism that keeps the synaptic weights close to the instability line where memories are spontaneously reactivated. In turn, the synaptic structure is stable to ongoing dynamics and random perturbations, yet it remains sufficiently plastic to remap sensory representations to encode new sets of stimuli. Both the plasticity rule and the metastable dynamics scale well with network size, with synaptic stability increasing with the number of neurons. Overall, our results show that it is possible to generate metastable dynamics over meaningful hidden states using a simple but biologically plausible plasticity rule which co-exists with ongoing neural dynamics.

Neural interactions in the human frontal cortex dissociate reward and punishment learning

How human prefrontal and insular regions interact while maximizing rewards and minimizing punishments is unknown. Capitalizing on human intracranial recordings, we demonstrate that the functional specificity toward reward or punishment learning is better disentangled by interactions compared to local representations. Prefrontal and insular cortices display non-selective neural populations to rewards and punishments. Non-selective responses, however, give rise to context-specific interareal interactions. We identify a reward subsystem with redundant interactions between the orbitofrontal and ventromedial prefrontal cortices, with a driving role of the latter. In addition, we find a punishment subsystem with redundant interactions between the insular and dorsolateral cortices, with a driving role of the insula. Finally, switching between reward and punishment learning is mediated by synergistic interactions between the two subsystems. These results provide a unifying explanation of distributed cortical representations and interactions supporting reward and punishment learning.

Evoking artificial speech perception through invasive brain stimulation for brain-computer interfaces: current challenges and future perspectives

Encoding artificial perceptions through brain stimulation, especially that of higher cognitive functions such as speech perception, is one of the most formidable challenges in brain-computer interfaces (BCI). Brain stimulation has been used for functional mapping in clinical practices for the last 70 years to treat various disorders affecting the nervous system, including epilepsy, Parkinson's disease, essential tremors, and dystonia. Recently, direct electrical stimulation has been used to evoke various forms of perception in humans, ranging from sensorimotor, auditory, and visual to speech cognition. Successfully evoking and fine-tuning artificial perceptions could revolutionize communication for individuals with speech disorders and significantly enhance the capabilities of brain-computer interface technologies. However, despite the extensive literature on encoding various perceptions and the rising popularity of speech BCIs, inducing artificial speech perception is still largely unexplored, and its potential has yet to be determined. In this paper, we examine the various stimulation techniques used to evoke complex percepts and the target brain areas for the input of speech-like information. Finally, we discuss strategies to address the challenges of speech encoding and discuss the prospects of these approaches.

Co-existence of synaptic plasticity and metastable dynamics in a spiking model of cortical circuits

Evidence for metastable dynamics and its role in brain function is emerging at a fast pace and is changing our understanding of neural coding by putting an emphasis on hidden states of transient activity. Clustered networks of spiking neurons have enhanced synaptic connections among groups of neurons forming structures called cell assemblies; such networks are capable of producing metastable dynamics that is in agreement with many experimental results. However, it is unclear how a clustered network structure producing metastable dynamics may emerge from a fully local plasticity rule, i.e., a plasticity rule where each synapse has only access to the activity of the neurons it connects (as opposed to the activity of other neurons or other synapses). Here, we propose a local plasticity rule producing ongoing metastable dynamics in a deterministic, recurrent network of spiking neurons. The metastable dynamics co-exists with ongoing plasticity and is the consequence of a self-tuning mechanism that keeps the synaptic weights close to the instability line where memories are spontaneously reactivated. In turn, the synaptic structure is stable to ongoing dynamics and random perturbations, yet it remains sufficiently plastic to remap sensory representations to encode new sets of stimuli. Both the plasticity rule and the metastable dynamics scale well with network size, with synaptic stability increasing with the number of neurons. Overall, our results show that it is possible to generate metastable dynamics over meaningful hidden states using a simple but biologically plausible plasticity rule which co-exists with ongoing neural dynamics.

Spiking networks that efficiently process dynamic sensory features explain receptor information mixing in somatosensory cortex

How do biological neural systems efficiently encode, transform and propagate information between the sensory periphery and the sensory cortex about sensory features evolving at different time scales? Are these computations efficient in normative information processing terms? While previous work has suggested that biologically plausible models of of such neural information processing may be implemented efficiently within a single processing layer, how such computations extend across several processing layers is less clear. Here, we model propagation of multiple time-varying sensory features across a sensory pathway, by extending the theory of efficient coding with spikes to efficient encoding, transformation and transmission of sensory signals. These computations are optimally realized by a multilayer spiking network with feedforward networks of spiking neurons (receptor layer) and recurrent excitatory-inhibitory networks of generalized leaky integrate-and-fire neurons (recurrent layers). Our model efficiently realizes a broad class of feature transformations, including positive and negative interaction across features, through specific and biologically plausible structures of feedforward connectivity. We find that mixing of sensory features in the activity of single neurons is beneficial because it lowers the metabolic cost at the network level. We apply the model to the somatosensory pathway by constraining it with parameters measured empirically and include in its last node, analogous to the primary somatosensory cortex (S1), two types of inhibitory neurons: parvalbumin-positive neurons realizing lateral inhibition, and somatostatin-positive neurons realizing winner-take-all inhibition. By implementing a negative interaction across stimulus features, this model captures several intriguing empirical observations from the somatosensory system of the mouse, including a decrease of sustained responses from subcortical networks to S1, a non-linear effect of the knock-out of receptor neuron types on the activity in S1, and amplification of weak signals from sensory neurons across the pathway.

Abstract deliberation by visuomotor neurons in prefrontal cortex

During visually guided behavior, the prefrontal cortex plays a pivotal role in mapping sensory inputs onto appropriate motor plans. When the sensory input is ambiguous, this involves deliberation. It is not known whether the deliberation is implemented as a competition between possible stimulus interpretations or between possible motor plans. Here we study neural population activity in the prefrontal cortex of macaque monkeys trained to flexibly report perceptual judgments of ambiguous visual stimuli. We find that the population activity initially represents the formation of a perceptual choice before transitioning into the representation of the motor plan. Stimulus strength and prior expectations both bear on the formation of the perceptual choice, but not on the formation of the action plan. These results suggest that prefrontal circuits involved in action selection are also used for the deliberation of abstract propositions divorced from a specific motor plan, thus providing a crucial mechanism for abstract reasoning.

Remapping revisited: how the hippocampus represents different spaces

The representation of distinct spaces by hippocampal place cells has been linked to changes in their place fields (the locations in the environment where the place cells discharge strongly), a phenomenon that has been termed 'remapping'. Remapping has been assumed to be accompanied by the reorganization of subsecond cofiring relationships among the place cells, potentially maximizing hippocampal information coding capacity. However, several observations challenge this standard view. For example, place cells exhibit mixed selectivity, encode non-positional variables, can have multiple place fields and exhibit unreliable discharge in fixed environments. Furthermore, recent evidence suggests that, when measured at subsecond timescales, the moment-to-moment cofiring of a pair of cells in one environment is remarkably similar in another environment, despite remapping. Here, I propose that remapping is a misnomer for the changes in place fields across environments and suggest instead that internally organized manifold representations of hippocampal activity are actively registered to different environments to enable navigation, promote memory and organize knowledge.

Simultaneous, cortex-wide dynamics of up to 1 million neurons reveal unbounded scaling of dimensionality with neuron number

The brain's remarkable properties arise from the collective activity of millions of neurons. Widespread application of dimensionality reduction to multi-neuron recordings implies that neural dynamics can be approximated by low-dimensional "latent" signals reflecting neural computations. However, can such low-dimensional representations truly explain the vast range of brain activity, and if not, what is the appropriate resolution and scale of recording to capture them? Imaging neural activity at cellular resolution and near-simultaneously across the mouse cortex, we demonstrate an unbounded scaling of dimensionality with neuron number in populations up to 1 million neurons. Although half of the neural variance is contained within sixteen dimensions correlated with behavior, our discovered scaling of dimensionality corresponds to an ever-increasing number of neuronal ensembles without immediate behavioral or sensory correlates. The activity patterns underlying these higher dimensions are fine grained and cortex wide, highlighting that large-scale, cellular-resolution recording is required to uncover the full substrates of neuronal computations.

Primacy of vision shapes behavioral strategies and neural substrates of spatial navigation in marmoset hippocampus

The role of the hippocampus in spatial navigation has been primarily studied in nocturnal mammals, such as rats, that lack many adaptations for daylight vision. Here we demonstrate that during 3D navigation, the common marmoset, a new world primate adapted to daylight, predominantly uses rapid head-gaze shifts for visual exploration while remaining stationary. During active locomotion marmosets stabilize the head, in contrast to rats that use low-velocity head movements to scan the environment as they locomote. Pyramidal neurons in the marmoset hippocampus CA3/CA1 regions predominantly show mixed selectivity for 3D spatial view, head direction, and place. Exclusive place selectivity is scarce. Inhibitory interneurons are predominantly mixed selective for angular head velocity and translation speed. Finally, we found theta phase resetting of local field potential oscillations triggered by head-gaze shifts. Our findings indicate that marmosets adapted to their daylight ecological niche by modifying exploration/navigation strategies and their corresponding hippocampal specializations.

Approximating Nonlinear Functions With Latent Boundaries in Low-Rank Excitatory-Inhibitory Spiking Networks

Deep feedforward and recurrent neural networks have become successful functional models of the brain, but they neglect obvious biological details such as spikes and Dale's law. Here we argue that these details are crucial in order to understand how real neural circuits operate. Towards this aim, we put forth a new framework for spike-based computation in low-rank excitatory-inhibitory spiking networks. By considering populations with rank-1 connectivity, we cast each neuron's spiking threshold as a boundary in a low-dimensional input-output space. We then show how the combined thresholds of a population of inhibitory neurons form a stable boundary in this space, and those of a population of excitatory neurons form an unstable boundary. Combining the two boundaries results in a rank-2 excitatory-inhibitory (EI) network with inhibition-stabilized dynamics at the intersection of the two boundaries. The computation of the resulting networks can be understood as the difference of two convex functions and is thereby capable of approximating arbitrary non-linear input-output mappings. We demonstrate several properties of these networks, including noise suppression and amplification, irregular activity and synaptic balance, as well as how they relate to rate network dynamics in the limit that the boundary becomes soft. Finally, while our work focuses on small networks (5-50 neurons), we discuss potential avenues for scaling up to much larger networks. Overall, our work proposes a new perspective on spiking networks that may serve as a starting point for a mechanistic understanding of biological spike-based computation.

The representational geometry of emotional states in basolateral amygdala

Sensory stimuli associated with aversive outcomes cause multiple behavioral responses related to an animal's evolving emotional state, but neural mechanisms underlying these processes remain unclear. Here aversive stimuli were presented to mice, eliciting two responses reflecting fear and flight to safety: tremble and ingress into a virtual burrow. Inactivation of basolateral amygdala (BLA) eliminated differential responses to aversive and neutral stimuli without eliminating responses themselves, suggesting BLA signals valence, not motor commands. However, two-photon imaging revealed that neurons typically exhibited mixed selectivity for stimulus identity, valence, tremble and/or ingress. Despite heterogeneous selectivity, BLA representational geometry was lower-dimensional when encoding valence, tremble and safety, enabling generalization of emotions across conditions. Further, tremble and valence coding directions were orthogonal, allowing linear readouts to specialize. Thus BLA representational geometry confers two computational properties that identify specialized neural circuits encoding variables describing emotional states: generalization across conditions, and readouts lacking interference from other readouts.

Tuned geometries of hippocampal representations meet the computational demands of social memory

Social memory consists of two processes: the detection of familiar compared with novel conspecifics and the detailed recollection of past social episodes. We investigated the neural bases for these processes using calcium imaging of dorsal CA2 hippocampal pyramidal neurons, known to be important for social memory, during social/spatial encounters with novel conspecifics and familiar littermates. Whereas novel individuals were represented in a low-dimensional geometry that allows for generalization of social identity across different spatial locations and of location across different identities, littermates were represented in a higher-dimensional geometry that supports high-capacity memory storage. Moreover, familiarity was represented in an abstract format, independent of individual identity. The degree to which familiarity increased the dimensionality of CA2 representations for individual mice predicted their performance in a social novelty recognition memory test. Thus, by tuning the geometry of structured neural activity, CA2 is able to meet the demands of distinct social memory processes.

A hippocampus-accumbens code guides goal-directed appetitive behavior

The dorsal hippocampus (dHPC) is a key brain region for the expression of spatial memories, such as navigating towards a learned reward location. The nucleus accumbens (NAc) is a prominent projection target of dHPC and implicated in value-based action selection. Yet, the contents of the dHPC→NAc information stream and their acute role in behavior remain largely unknown. Here, we found that optogenetic stimulation of the dHPC→NAc pathway while mice navigated towards a learned reward location was both necessary and sufficient for spatial memory-related appetitive behaviors. To understand the task-relevant coding properties of individual NAc-projecting hippocampal neurons (dHPC→NAc), we used in vivo dual-color two-photon imaging. In contrast to other dHPC neurons, the dHPC→NAc subpopulation contained more place cells, with enriched spatial tuning properties. This subpopulation also showed enhanced coding of non-spatial task-relevant behaviors such as deceleration and appetitive licking. A generalized linear model revealed enhanced conjunctive coding in dHPC→NAc neurons which improved the identification of the reward zone. We propose that dHPC routes specific reward-related spatial and behavioral state information to guide NAc action selection.

Multiplicative joint coding in preparatory activity for reaching sequence in macaque motor cortex

Although the motor cortex has been found to be modulated by sensory or cognitive sequences, the linkage between multiple movement elements and sequence-related responses is not yet understood. Here, we recorded neuronal activity from the motor cortex with implanted micro-electrode arrays and single electrodes while monkeys performed a double-reach task that was instructed by simultaneously presented memorized cues. We found that there existed a substantial multiplicative component jointly tuned to impending and subsequent reaches during preparation, then the coding mechanism transferred to an additive manner during execution. This multiplicative joint coding, which also spontaneously emerged in recurrent neural networks trained for double reach, enriches neural patterns for sequential movement, and might explain the linear readout of elemental movements.

Differential clustering of visual and choice- and saccade-related activity in macaque V3A and CIP

Neurons in sensory and motor cortices tend to aggregate in clusters with similar functional properties. Within the primate dorsal ("where") pathway, an important interface between three-dimensional (3-D) visual processing and motor-related functions consists of two hierarchically organized areas: V3A and the caudal intraparietal (CIP) area. In these areas, 3-D visual information, choice-related activity, and saccade-related activity converge, often at the single-neuron level. Characterizing the clustering of functional properties in areas with mixed selectivity, such as these, may help reveal organizational principles that support sensorimotor transformations. Here we quantified the clustering of visual feature selectivity, choice-related activity, and saccade-related activity by performing correlational and parametric comparisons of the responses of well-isolated, simultaneously recorded neurons in macaque monkeys. Each functional domain showed statistically significant clustering in both areas. However, there were also domain-specific differences in the strength of clustering across the areas. Visual feature selectivity and saccade-related activity were more strongly clustered in V3A than in CIP. In contrast, choice-related activity was more strongly clustered in CIP than in V3A. These differences in clustering may reflect the areas' roles in sensorimotor processing. Stronger clustering of visual and saccade-related activity in V3A may reflect a greater role in within-domain processing, as opposed to cross-domain synthesis. In contrast, stronger clustering of choice-related activity in CIP may reflect a greater role in synthesizing information across functional domains to bridge perception and action.NEW & NOTEWORTHY The occipital and parietal cortices of macaque monkeys are bridged by hierarchically organized areas V3A and CIP. These areas support 3-D visual transformations, carry choice-related activity during 3-D perceptual tasks, and possess saccade-related activity. This study quantifies the functional clustering of neuronal response properties within V3A and CIP for each of these domains. The findings reveal domain-specific cross-area differences in clustering that may reflect the areas' roles in sensorimotor processing.