Learning task-state representations

A boundedly rational model for category learning

The computational modeling of category learning is typically evaluated in terms of the model's accuracy. For a model to accurately infer category membership of stimuli, it has to have sufficient representational precision. Thus, many category learning models infer category representations that guide decision-making and the model's fitness is evaluated by its ability to accurately choose. Substantial decision-making research, however, indicates that noise plays an important role. Specifically, noisy representations are assumed to introduce an element of stochasticity to decision-making. Noise can be minimized at the cost of cognitive resource expenditure. Thus, a more biologically plausible model of category learning should balance representational precision with costs. Here, we tested an autoencoder model that learns categories (the six category structures introduced by Roger Shepard and colleagues) by balancing the minimization of error with minimization of resource usage. By incorporating the goal of reducing category complexity, the currently proposed model biases category decisions toward previously learned central tendencies. We show that this model is still able to account for category learning performance in a traditional category learning benchmark. The currently proposed model additionally makes some novel predictions about category learning that future studies can test empirically. The goal of this paper is to make progress toward development of an ecologically and neurobiologically plausible model of category learning that can guide future studies and theoretical frameworks.

Representational spaces in orbitofrontal and ventromedial prefrontal cortex: task states, values, and beyond

The orbitofrontal cortex (OFC) and ventromedial-prefrontal cortex (vmPFC) play a key role in decision-making and encode task states in addition to expected value. We review evidence suggesting a connection between value and state representations and argue that OFC / vmPFC integrate stimulus, context, and outcome information. Comparable encoding principles emerge in late layers of deep reinforcement learning (RL) models, where single nodes exhibit similar forms of mixed-selectivity, which enables flexible readout of relevant variables by downstream neurons. Based on these lines of evidence, we suggest that outcome-maximization leads to complex representational spaces that are insufficiently characterized by linear value signals that have been the focus of most prior research on the topic. Major outstanding questions concern the role of OFC/ vmPFC in learning across tasks, in encoding of task-irrelevant aspects, and the role of hippocampus-PFC interactions.

Adaptive empathic response selection is sensitive to multiple dimensions of social interaction

During empathic response selection, individuals draw from both past experiences and social cues, including the distressed person's identity, their emotional state, and the cause of distress. To study how these social dimensions influence empathic-response learning we integrated a multidimensional learning paradigm, computational modelling, and adaptive empathy framework. Participants identified effective empathic responses across two blocks of distress scenarios, with one social dimension altered between blocks. We anticipated two learning patterns: dimension-sensitive, treating each change as a new learning experience, and dimension-insensitive, relying on previous experience as a baseline. We found that participants were sensitive to changes in person, emotional state, and distress cause, but to different degree. The person dimension was the most salient, suggesting that the distressed person's identity is the primary reference point when interacting with others. Our findings provide a quantitative evaluation of the weight given to different dimensions of social interactions, which may help understand how people perceive and react in such scenarios.

Human hippocampus and dorsomedial prefrontal cortex infer and update latent causes during social interaction

Latent-cause inference is the process of identifying features of the environment that have caused an outcome. This problem is especially important in social settings where individuals may not make equal contributions to the outcomes they achieve together. Here, we designed a novel task in which participants inferred which of two characters was more likely to have been responsible for outcomes achieved by working together. Using computational modeling, univariate and multivariate analysis of human fMRI, and continuous theta-burst stimulation, we identified two brain regions that solved the task. Notably, as each outcome occurred, it was possible to decode the inference of its cause (the responsible character) from hippocampal activity. Activity in dorsomedial prefrontal cortex (dmPFC) updated estimates of association between cause-responsible character-and the outcome. Disruption of dmPFC activity impaired participants' ability to update their estimate as a function of inferred responsibility but spared their ability to infer responsibility.

A hierarchical active inference model of spatial alternation tasks and the hippocampal-prefrontal circuit

Cognitive problem-solving benefits from cognitive maps aiding navigation and planning. Physical space navigation involves hippocampal (HC) allocentric codes, while abstract task space engages medial prefrontal cortex (mPFC) task-specific codes. Previous studies show that challenging tasks, like spatial alternation, require integrating these two types of maps. The disruption of the HC-mPFC circuit impairs performance. We propose a hierarchical active inference model clarifying how this circuit solves spatial interaction tasks by bridging physical and task-space maps. Simulations demonstrate that the model's dual layers develop effective cognitive maps for physical and task space. The model solves spatial alternation tasks through reciprocal interactions between the two layers. Disrupting its communication impairs decision-making, which is consistent with empirical evidence. Additionally, the model adapts to switching between multiple alternation rules, providing a mechanistic explanation of how the HC-mPFC circuit supports spatial alternation tasks and the effects of disruption.

Working memory as a representational template for reinforcement learning

Working memory (WM) and reinforcement learning (RL) both influence decision-making, but how they interact to affect behaviour remains unclear. We assessed whether RL is influenced by the format of visual stimuli held in WM, either feature-based or unified, object-based representations. In a pre-registered paradigm, participants learned stimulus-action combinations that provided reward through 80% probabilistic feedback. In parallel, participants retained the RL stimulus in WM and were asked to recall this stimulus after each RL choice. Crucially, the format of representation probed in WM was manipulated, with blocks encouraging either separate features or bound objects to be remembered. Incentivising a feature-based WM representation facilitated feature-based learning, shown by an improved choice strategy. This reveals a role of WM in providing sustained internal representations that are harnessed by RL, providing a framework by which these two cognitive processes cooperate.

An inductive bias for slowly changing features in human reinforcement learning

Identifying goal-relevant features in novel environments is a central challenge for efficient behaviour. We asked whether humans address this challenge by relying on prior knowledge about common properties of reward-predicting features. One such property is the rate of change of features, given that behaviourally relevant processes tend to change on a slower timescale than noise. Hence, we asked whether humans are biased to learn more when task-relevant features are slow rather than fast. To test this idea, 295 human participants were asked to learn the rewards of two-dimensional bandits when either a slowly or quickly changing feature of the bandit predicted reward. Across two experiments and one preregistered replication, participants accrued more reward when a bandit's relevant feature changed slowly, and its irrelevant feature quickly, as compared to the opposite. We did not find a difference in the ability to generalise to unseen feature values between conditions. Testing how feature speed could affect learning with a set of four function approximation Kalman filter models revealed that participants had a higher learning rate for the slow feature, and adjusted their learning to both the relevance and the speed of feature changes. The larger the improvement in participants' performance for slow compared to fast bandits, the more strongly they adjusted their learning rates. These results provide evidence that human reinforcement learning favours slower features, suggesting a bias in how humans approach reward learning.

Computational processes of simultaneous learning of stochasticity and volatility in humans

Making adaptive decisions requires predicting outcomes, and this in turn requires adapting to uncertain environments. This study explores computational challenges in distinguishing two types of noise influencing predictions: volatility and stochasticity. Volatility refers to diffusion noise in latent causes, requiring a higher learning rate, while stochasticity introduces moment-to-moment observation noise and reduces learning rate. Dissociating these effects is challenging as both increase the variance of observations. Previous research examined these factors mostly separately, but it remains unclear whether and how humans dissociate them when they are played off against one another. In two large-scale experiments, through a behavioral prediction task and computational modeling, we report evidence of humans dissociating volatility and stochasticity solely based on their observations. We observed contrasting effects of volatility and stochasticity on learning rates, consistent with statistical principles. These results are consistent with a computational model that estimates volatility and stochasticity by balancing their dueling effects.

Toward a computational role for locus coeruleus/norepinephrine arousal systems

Brain and behavior undergo measurable changes in their underlying state and neuromodulators are thought to contribute to these fluctuations. Why do we undergo such changes, and what function could the underlying neuromodulatory systems perform? Here we examine theoretical answers to these questions with respect to the locus coeruleus/norepinephrine system focusing on peripheral markers for arousal, such as pupil diameter, that are thought to provide a window into brain wide noradrenergic signaling. We explore a computational role for arousal systems in facilitating internal state transitions that facilitate credit assignment and promote accurate perceptions in non-stationary environments. We summarize recent work that supports this idea and highlight open questions as well as alternative views of how arousal affects cognition.

Cracking and Packing Information about the Features of Expected Rewards in the Orbitofrontal Cortex

The orbitofrontal cortex (OFC) is crucial for tracking various aspects of expected outcomes, thereby helping to guide choices and support learning. Our previous study showed that the effects of reward timing and size on the activity of single units in OFC were dissociable when these attributes were manipulated independently ( Roesch et al., 2006). However, in real-life decision-making scenarios, outcome features often change simultaneously, so here we investigated how OFC neurons in male rats integrate information about the timing and identity (flavor) of reward and respond to changes in these features, according to whether they were changed simultaneously or separately. We found that a substantial number of OFC neurons fired differentially to immediate versus delayed reward and to the different reward flavors. However, contrary to the previous study, selectivity for timing was strongly correlated with selectivity for identity. Taken together with the previous research, these results suggest that when reward features are correlated, OFC tends to "pack" them into unitary constructs, whereas when they are independent, OFC tends to "crack" them into separate constructs. Furthermore, we found that when both reward timing and flavor were changed, reward-responsive OFC neurons showed unique activity patterns preceding and during the omission of an expected reward. Interestingly, this OFC activity is similar and slightly preceded the ventral tegmental area dopamine (VTA DA) activity observed in a previous study ( Takahashi et al., 2023), consistent with the role of OFC in providing predictive information to VTA DA neurons.

Convolutional Neural Network-Based Lane-Change Strategy via Motion Image Representation for Automated and Connected Vehicles

The lane-change decision-making module of automated and connected vehicles (ACVs) is one of the most crucial and challenging issues to be addressed. Motivated by human beings' underlying driving paradigm and the convolutional neural network's (CNN) dramatic capability of extracting features and learning strategies, this article proposes a CNN-based lane-change decision-making method via the dynamic motion image representation. Human drivers take proper driving maneuvers after they subconsciously construct the dynamic traffic scene representation in their brains, so this study first proposes the dynamic motion image representation method to reveal informative traffic situations in the motion-sensitive area (MSA), which provides a full view of surrounding cars. Then, this article develops a CNN model to extract the underlying features and learn driving policies from labeled datasets of MSA motion images. Besides, a safety-constrained layer is added to avoid vehicle collisions. We build a simulation platform based on the simulation of urban mobility (SUMO) to collect traffic datasets and test our proposed method. In addition, real-world traffic datasets are also involved to further investigate the proposed method's performance. The rule-based strategy and reinforcement learning (RL)-based method are used to compare with our approach. All results demonstrate that the proposed method performs lane-change decision-making much better than prevailing methods, which suggests our scheme has huge potential to accelerate the deployment of ACVs and is worth further study.

Space is a latent sequence: A theory of the hippocampus

Fascinating phenomena such as landmark vector cells and splitter cells are frequently discovered in the hippocampus. Without a unifying principle, each experiment seemingly uncovers new anomalies or coding types. Here, we provide a unifying principle that the mental representation of space is an emergent property of latent higher-order sequence learning. Treating space as a sequence resolves numerous phenomena and suggests that the place field mapping methodology that interprets sequential neuronal responses in Euclidean terms might itself be a source of anomalies. Our model, clone-structured causal graph (CSCG), employs higher-order graph scaffolding to learn latent representations by mapping aliased egocentric sensory inputs to unique contexts. Learning to compress sequential and episodic experiences using CSCGs yields allocentric cognitive maps that are suitable for planning, introspection, consolidation, and abstraction. By explicating the role of Euclidean place field mapping and demonstrating how latent sequential representations unify myriad observed phenomena, our work positions the hippocampus in a sequence-centric paradigm, challenging the prevailing space-centric view.

A feature-specific prediction error model explains dopaminergic heterogeneity

The hypothesis that midbrain dopamine (DA) neurons broadcast a reward prediction error (RPE) is among the great successes of computational neuroscience. However, recent results contradict a core aspect of this theory: specifically that the neurons convey a scalar, homogeneous signal. While the predominant family of extensions to the RPE model replicates the classic model in multiple parallel circuits, we argue that these models are ill suited to explain reports of heterogeneity in task variable encoding across DA neurons. Instead, we introduce a complementary 'feature-specific RPE' model, positing that individual ventral tegmental area DA neurons report RPEs for different aspects of an animal's moment-to-moment situation. Further, we show how our framework can be extended to explain patterns of heterogeneity in action responses reported among substantia nigra pars compacta DA neurons. This theory reconciles new observations of DA heterogeneity with classic ideas about RPE coding while also providing a new perspective of how the brain performs reinforcement learning in high-dimensional environments.

Reward processes in extinction learning and applications to exposure therapy

Anxiety disorders are common and highly distressing mental health conditions. Exposure therapy is a gold-standard treatment for anxiety disorders. Mechanisms of Pavlovian fear learning, and particularly fear extinction, are central to exposure therapy. A growing body of evidence suggests an important role of reward processes during Pavlovian fear extinction. Nonetheless, predominant models of exposure therapy do not currently incorporate reward processes. Herein, we present a theoretical model of reward processes in relation to Pavlovian mechanisms of exposure therapy, including a focus on dopaminergic prediction error signaling, coinciding positive emotional experiences (i.e., relief), and unexpected positive outcomes. We then highlight avenues for further research and discuss potential strategies to leverage reward processes to maximize exposure therapy response, such as pre-exposure interventions to increase reward sensitivity or post-exposure rehearsal (e.g., savoring, imaginal recounting strategies) to enhance retrieval and retention of learned associations.

Reconciling shared versus context-specific information in a neural network model of latent causes

It has been proposed that, when processing a stream of events, humans divide their experiences in terms of inferred latent causes (LCs) to support context-dependent learning. However, when shared structure is present across contexts, it is still unclear how the "splitting" of LCs and learning of shared structure can be simultaneously achieved. Here, we present the Latent Cause Network (LCNet), a neural network model of LC inference. Through learning, it naturally stores structure that is shared across tasks in the network weights. Additionally, it represents context-specific structure using a context module, controlled by a Bayesian nonparametric inference algorithm, which assigns a unique context vector for each inferred LC. Across three simulations, we found that LCNet could (1) extract shared structure across LCs in a function learning task while avoiding catastrophic interference, (2) capture human data on curriculum effects in schema learning, and (3) infer the underlying event structure when processing naturalistic videos of daily events. Overall, these results demonstrate a computationally feasible approach to reconciling shared structure and context-specific structure in a model of LCs that is scalable from laboratory experiment settings to naturalistic settings.

Neural dynamics underlying minute-timescale persistent behavior in the human brain

The ability to pursue long-term goals relies on a representations of task context that can both be maintained over long periods of time and switched flexibly when goals change. Little is known about the neural substrate for such minute-scale maintenance of task sets. Utilizing recordings in neurosurgical patients, we examined how groups of neurons in the human medial frontal cortex and hippocampus represent task contexts. When cued explicitly, task context was encoded in both brain areas and changed rapidly at task boundaries. Hippocampus exhibited a temporally dynamic code with fast decorrelation over time, preventing cross-temporal generalization. Medial frontal cortex exhibited a static code that decorrelated slowly, allowing generalization across minutes of time. When task context needed to be inferred as a latent variable, hippocampus encoded task context with a static code. These findings reveal two possible regimes for encoding minute-scale task-context representations that were engaged differently based on task demands.

Placebo treatment affects brain systems related to affective and cognitive processes, but not nociceptive pain

Drug treatments for pain often do not outperform placebo, and a better understanding of placebo mechanisms is needed to improve treatment development and clinical practice. In a large-scale fMRI study (N = 392) with pre-registered analyses, we tested whether placebo analgesic treatment modulates nociceptive processes, and whether its effects generalize from conditioned to unconditioned pain modalities. Placebo treatment caused robust analgesia in conditioned thermal pain that generalized to unconditioned mechanical pain. However, placebo did not decrease pain-related fMRI activity in brain measures linked to nociceptive pain, including the Neurologic Pain Signature (NPS) and spinothalamic pathway regions, with strong support for null effects in Bayes Factor analyses. In addition, surprisingly, placebo increased activity in some spinothalamic regions for unconditioned mechanical pain. In contrast, placebo reduced activity in a neuromarker associated with higher-level contributions to pain, the Stimulus Intensity Independent Pain Signature (SIIPS), and affected activity in brain regions related to motivation and value, in both pain modalities. Individual differences in behavioral analgesia were correlated with neural changes in both modalities. Our results indicate that cognitive and affective processes primarily drive placebo analgesia, and show the potential of neuromarkers for separating treatment influences on nociception from influences on evaluative processes.

Analysis of Continual Learning Techniques for Image Generative Models with Learned Class Information Management

The advancements in deep learning have significantly enhanced the capability of image generation models to produce images aligned with human intentions. However, training and adapting these models to new data and tasks remain challenging because of their complexity and the risk of catastrophic forgetting. This study proposes a method for addressing these challenges involving the application of class-replacement techniques within a continual learning framework. This method utilizes selective amnesia (SA) to efficiently replace existing classes with new ones while retaining crucial information. This approach improves the model's adaptability to evolving data environments while preventing the loss of past information. We conducted a detailed evaluation of class-replacement techniques, examining their impact on the "class incremental learning" performance of models and exploring their applicability in various scenarios. The experimental results demonstrated that our proposed method could enhance the learning efficiency and long-term performance of image generation models. This study broadens the application scope of image generation technology and supports the continual improvement and adaptability of corresponding models.

Forming cognitive maps for abstract spaces: the roles of the human hippocampus and orbitofrontal cortex

How does the human brain construct cognitive maps for decision-making and inference? Here, we conduct an fMRI study on a navigation task in multidimensional abstract spaces. Using a deep neural network model, we assess learning levels and categorized paths into exploration and exploitation stages. Univariate analyses show higher activation in the bilateral hippocampus and lateral prefrontal cortex during exploration, positively associated with learning level and response accuracy. Conversely, the bilateral orbitofrontal cortex (OFC) and retrosplenial cortex show higher activation during exploitation, negatively associated with learning level and response accuracy. Representational similarity analysis show that the hippocampus, entorhinal cortex, and OFC more accurately represent destinations in exploitation than exploration stages. These findings highlight the collaboration between the medial temporal lobe and prefrontal cortex in learning abstract space structures. The hippocampus may be involved in spatial memory formation and representation, while the OFC integrates sensory information for decision-making in multidimensional abstract spaces.

Surprise-minimization as a solution to the structural credit assignment problem

The structural credit assignment problem arises when the causal structure between actions and subsequent outcomes is hidden from direct observation. To solve this problem and enable goal-directed behavior, an agent has to infer structure and form a representation thereof. In the scope of this study, we investigate a possible solution in the human brain. We recorded behavioral and electrophysiological data from human participants in a novel variant of the bandit task, where multiple actions lead to multiple outcomes. Crucially, the mapping between actions and outcomes was hidden and not instructed to the participants. Human choice behavior revealed clear hallmarks of credit assignment and learning. Moreover, a computational model which formalizes action selection as the competition between multiple representations of the hidden structure was fit to account for participants data. Starting in a state of uncertainty about the correct representation, the central mechanism of this model is the arbitration of action control towards the representation which minimizes surprise about outcomes. Crucially, single-trial latent-variable analysis reveals that the neural patterns clearly support central quantitative predictions of this surprise minimization model. The results suggest that neural activity is not only related to reinforcement learning under correct as well as incorrect task representations but also reflects central mechanisms of credit assignment and behavioral arbitration.

Neurocomputational mechanisms involved in adaptation to fluctuating intentions of others

Humans frequently interact with agents whose intentions can fluctuate between competition and cooperation over time. It is unclear how the brain adapts to fluctuating intentions of others when the nature of the interactions (to cooperate or compete) is not explicitly and truthfully signaled. Here, we use model-based fMRI and a task in which participants thought they were playing with another player. In fact, they played with an algorithm that alternated without signaling between cooperative and competitive strategies. We show that a neurocomputational mechanism with arbitration between competitive and cooperative experts outperforms other learning models in predicting choice behavior. At the brain level, the fMRI results show that the ventral striatum and ventromedial prefrontal cortex track the difference of reliability between these experts. When attributing competitive intentions, we find increased coupling between these regions and a network that distinguishes prediction errors related to competition and cooperation. These findings provide a neurocomputational account of how the brain arbitrates dynamically between cooperative and competitive intentions when making adaptive social decisions.

Neuroscience: Memory modification without catastrophe

Adaptive behaviour is supported by changes in neuronal networks. Insight into maintaining these memories - preventing their catastrophic loss - despite further network changes occurring due to novel learning is provided in a new study.

The ventromedial prefrontal cortex in response to threat omission is associated with subsequent explicit safety memory

In order to memorize and discriminate threatening and safe stimuli, the processing of the actual absence of threat seems crucial. Here, we measured brain activity with fMRI in response to both threat conditioned stimuli and their outcomes by combining threat learning with a subsequent memory paradigm. Participants (N = 38) repeatedly saw a variety of faces, half of which (CS+) were associated with an aversive unconditioned stimulus (US) and half of which were not (CS-). When an association was later remembered, the hippocampus had been more active (than when forgotten). However, the ventromedial prefrontal cortex predicted subsequent memory specifically during safe associations (CS- and US omission responses) and the left dorsolateral prefrontal cortex during outcomes in general (US and US omissions). In exploratory analyses of the theoretically important US omission, we found extended involvement of the medial prefrontal cortex and an enhanced functional connectivity to visual and somatosensory cortices, suggesting a possible function in sustaining sensory information for an integration with semantic memory. Activity in visual and somatosensory cortices together with the inferior frontal gyrus also predicted memory performance one week after learning. The findings imply the importance of a close interplay between prefrontal and sensory areas during the processing of safe outcomes-or 'nothing'-to establish declarative safety memory.

Event-specific and persistent representations for contextual states in orbitofrontal neurons

Flexible and context-dependent behaviors require animals, including humans, to identify their current contextual state for proper rules to apply, especially when information that defines these states is partially observable. Depending on behavioral needs, contextual states usually persist for prolonged periods and across other events, including sensory stimuli, actions, and rewards, highlighting prominent challenges of holding a reliable state representation. The orbitofrontal cortex (OFC) is crucial in behaviors requiring the identification of the current context (e.g., reversal learning); however, how single units in the OFC accomplish this function has not been assessed. Do they maintain such information persistently, in separate populations from those responding phasically to events within a task, or is contextual information dynamic and embedded in these phasic responses? Here, we investigated this question by recording single units from OFC in rats performing a task that required them to identify the current contextual state related to estimated proximity to future reward with distracting olfactory cues. We found that while some OFC neurons encode contextual states, most change their selectivity upon the transition of task events. Nevertheless, despite dynamic activities in single neurons, the neural populations maintain persistent representations regarding current contextual states within particular neural subspaces.

Transcranial direct current stimulation suggests a causal role of the medial prefrontal cortex in learning social hierarchy

Social hierarchies can be inferred through observational learning of social relationships between individuals. Yet, little is known about the causal role of specific brain regions in learning hierarchies. Here, using transcranial direct current stimulation, we show a causal role of the medial prefrontal cortex (mPFC) in learning social versus non-social hierarchies. In a Training phase, participants acquired knowledge about social and non-social hierarchies by trial and error. During a Test phase, they were presented with two items from hierarchies that were never encountered together, requiring them to make transitive inferences. Anodal stimulation over mPFC impaired social compared with non-social hierarchy learning, and this modulation was influenced by the relative social rank of the members (higher or lower status). Anodal stimulation also impaired transitive inference making, but only during early blocks before learning was established. Together, these findings demonstrate a causal role of the mPFC in learning social ranks by observation.

Dorsolateral prefrontal activity supports a cognitive space organization of cognitive control

Cognitive control resolves conflicts between task-relevant and -irrelevant information to enable goal-directed behavior. As conflicts can arise from different sources (e.g., sensory input, internal representations), how a limited set of cognitive control processes can effectively address diverse conflicts remains a major challenge. Based on the cognitive space theory, different conflicts can be parameterized and represented as distinct points in a (low-dimensional) cognitive space, which can then be resolved by a limited set of cognitive control processes working along the dimensions. It leads to a hypothesis that conflicts similar in their sources are also represented similarly in the cognitive space. We designed a task with five types of conflicts that could be conceptually parameterized. Both human performance and fMRI activity patterns in the right dorsolateral prefrontal cortex support that different types of conflicts are organized based on their similarity, thus suggesting cognitive space as a principle for representing conflicts.

The successor representation subserves hierarchical abstraction for goal-directed behavior

Humans have the ability to craft abstract, temporally extended and hierarchically organized plans. For instance, when considering how to make spaghetti for dinner, we typically concern ourselves with useful "subgoals" in the task, such as cutting onions, boiling pasta, and cooking a sauce, rather than particulars such as how many cuts to make to the onion, or exactly which muscles to contract. A core question is how such decomposition of a more abstract task into logical subtasks happens in the first place. Previous research has shown that humans are sensitive to a form of higher-order statistical learning named "community structure". Community structure is a common feature of abstract tasks characterized by a logical ordering of subtasks. This structure can be captured by a model where humans learn predictions of upcoming events multiple steps into the future, discounting predictions of events further away in time. One such model is the "successor representation", which has been argued to be useful for hierarchical abstraction. As of yet, no study has convincingly shown that this hierarchical abstraction can be put to use for goal-directed behavior. Here, we investigate whether participants utilize learned community structure to craft hierarchically informed action plans for goal-directed behavior. Participants were asked to search for paintings in a virtual museum, where the paintings were grouped together in "wings" representing community structure in the museum. We find that participants' choices accord with the hierarchical structure of the museum and that their response times are best predicted by a successor representation. The degree to which the response times reflect the community structure of the museum correlates with several measures of performance, including the ability to craft temporally abstract action plans. These results suggest that successor representation learning subserves hierarchical abstractions relevant for goal-directed behavior.

Naturalistic reinforcement learning

Humans possess a remarkable ability to make decisions within real-world environments that are expansive, complex, and multidimensional. Human cognitive computational neuroscience has sought to exploit reinforcement learning (RL) as a framework within which to explain human decision-making, often focusing on constrained, artificial experimental tasks. In this article, we review recent efforts that use naturalistic approaches to determine how humans make decisions in complex environments that better approximate the real world, providing a clearer picture of how humans navigate the challenges posed by real-world decisions. These studies purposely embed elements of naturalistic complexity within experimental paradigms, rather than focusing on simplification, generating insights into the processes that likely underpin humans' ability to navigate complex, multidimensional real-world environments so successfully.

Nicotine-related beliefs induce dose-dependent responses in the human brain

Beliefs have a powerful influence on our behavior, yet their neural mechanisms remain elusive. Here we investigate whether beliefs could impact brain activities in a way akin to pharmacological dose-dependent effects. Nicotine-dependent humans were told that nicotine strength in an electronic cigarette was either 'low', 'medium' or 'high', while nicotine content was held constant. After vaping, participants underwent functional neuroimaging and performed a decision-making task known to engage neural circuits affected by nicotine. Beliefs about nicotine strength induced dose-dependent responses in the thalamus, a key binding site for nicotine, but not in other brain regions such as the striatum. Nicotine-related beliefs also parametrically modulated the connectivity between the thalamus and ventromedial prefrontal cortex, a region important for decision-making. These findings reveal a high level of precision in the way beliefs influence the brain, offering mechanistic insights into humans' heterogeneous responses to drugs and a pivotal role of beliefs in addiction.

Grid codes underlie multiple cognitive maps in the human brain

Grid cells fire at multiple positions that organize the vertices of equilateral triangles tiling a 2D space and are well studied in rodents. The last decade witnessed rapid progress in two other research lines on grid codes-empirical studies on distributed human grid-like representations in physical and multiple non-physical spaces, and cognitive computational models addressing the function of grid cells based on principles of efficient and predictive coding. Here, we review the progress in these fields and integrate these lines into a systematic organization. We also discuss the coordinate mechanisms of grid codes in the human entorhinal cortex and medial prefrontal cortex and their role in neurological and psychiatric diseases.

Latent-state and model-based learning in PTSD

Post-traumatic stress disorder (PTSD) is characterized by altered emotional and behavioral responding following a traumatic event. In this article, we review the concepts of latent-state and model-based learning (i.e., learning and inferring abstract task representations) and discuss their relevance for clinical and neuroscience models of PTSD. Recent data demonstrate evidence for brain and behavioral biases in these learning processes in PTSD. These new data potentially recast excessive fear towards trauma cues as a problem in learning and updating abstract task representations, as opposed to traditional conceptualizations focused on stimulus-specific learning. Biases in latent-state and model-based learning may also be a common mechanism targeted in common therapies for PTSD. We highlight key knowledge gaps that need to be addressed to further elaborate how latent-state learning and its associated neurocircuitry mechanisms function in PTSD and how to optimize treatments to target these processes.

Latent representations in hippocampal network model co-evolve with behavioral exploration of task structure

To successfully learn real-life behavioral tasks, animals must pair actions or decisions to the task's complex structure, which can depend on abstract combinations of sensory stimuli and internal logic. The hippocampus is known to develop representations of this complex structure, forming a so-called "cognitive map". However, the precise biophysical mechanisms driving the emergence of task-relevant maps at the population level remain unclear. We propose a model in which plateau-based learning at the single cell level, combined with reinforcement learning in an agent, leads to latent representational structures codependently evolving with behavior in a task-specific manner. In agreement with recent experimental data, we show that the model successfully develops latent structures essential for task-solving (cue-dependent "splitters") while excluding irrelevant ones. Finally, our model makes testable predictions concerning the co-dependent interactions between split representations and split behavioral policy during their evolution.

The emergence of task-relevant representations in a nonlinear decision-making task

This paper describes the relationship between performance in a decision-making task and the emergence of task-relevant representations. Participants learnt two tasks in which the appropriate response depended on multiple relevant stimuli and the underlying stimulus-outcome associations were governed by a latent feature that participants could discover. We divided participants into good and bad performers based on their overall classification rate and computed behavioural accuracy for each feature value. We found that participants with better performance had a better representation of the latent feature space. We then used representation similarity analysis on Electroencephalographic (EEG) data to identify when these representations emerge. We were able to decode task-relevant representations in a time window emerging 700 ms after stimulus presentation, but only for participants with good task performance. Our findings suggest that, in order to make good decisions, it is necessary to create and extract a low-dimensional representation of the task at hand.

Abstract representations emerge in human hippocampal neurons during inference behavior

Humans have the remarkable cognitive capacity to rapidly adapt to changing environments. Central to this capacity is the ability to form high-level, abstract representations that take advantage of regularities in the world to support generalization 1 . However, little is known about how these representations are encoded in populations of neurons, how they emerge through learning, and how they relate to behavior 2,3 . Here we characterized the representational geometry of populations of neurons (single-units) recorded in the hippocampus, amygdala, medial frontal cortex, and ventral temporal cortex of neurosurgical patients who are performing an inferential reasoning task. We find that only the neural representations formed in the hippocampus simultaneously encode multiple task variables in an abstract, or disentangled, format. This representational geometry is uniquely observed after patients learn to perform inference, and consisted of disentangled directly observable and discovered latent task variables. Interestingly, learning to perform inference by trial and error or through verbal instructions led to the formation of hippocampal representations with similar geometric properties. The observed relation between representational format and inference behavior suggests that abstract/disentangled representational geometries are important for complex cognition.

Common Mechanisms of Learning in Motor and Cognitive Systems

Rapid progress in our understanding of the brain's learning mechanisms has been accomplished over the past decade, particularly with conceptual advances, including representing behavior as a dynamical system, large-scale neural population recordings, and new methods of analysis of neuronal populations. However, motor and cognitive systems have been traditionally studied with different methods and paradigms. Recently, some common principles, evident in both behavior and neural activity, that underlie these different types of learning have become to emerge. Here we review results from motor and cognitive learning, relying on different techniques and studying different systems to understand the mechanisms of learning. Movement is intertwined with cognitive operations, and its dynamics reflect cognitive variables. Training, in either motor or cognitive tasks, involves recruitment of previously unresponsive neurons and reorganization of neural activity in a low dimensional manifold. Mapping of new variables in neural activity can be very rapid, instantiating flexible learning of new tasks. Communication between areas is just as critical a part of learning as are patterns of activity within an area emerging with learning. Common principles across systems provide a map for future research.

Latent profiles of substance use, early life stress, and attention/externalizing problems and their association with neural correlates of reinforcement learning in adolescents

BACKGROUND

Adolescent substance use, externalizing and attention problems, and early life stress (ELS) commonly co-occur. These psychopathologies show overlapping neural dysfunction in the form of reduced recruitment of reward processing neuro-circuitries. However, it is unclear to what extent these psychopathologies show common v. different neural dysfunctions as a function of symptom profiles, as no studies have directly compared neural dysfunctions associated with each of these psychopathologies to each other.

METHODS

In study 1, a latent profile analysis (LPA) was conducted in a sample of 266 adolescents (aged 13-18, 41.7% female, 58.3% male) from a residential youth care facility and the surrounding community to investigate substance use, externalizing and attention problems, and ELS psychopathologies and their co-presentation. In study 2, we examined a subsample of 174 participants who completed the Passive Avoidance learning task during functional magnetic resonance imaging to examine differential and/or common reward processing neuro-circuitry dysfunctions associated with symptom profiles based on these co-presentations.

RESULTS

In study 1, LPA identified profiles of substance use plus rule-breaking behaviors, attention-deficit hyperactivity disorder, and ELS. In study 2, the substance use/rule-breaking profile was associated with reduced recruitment of reward processing and attentional neuro-circuitries during the Passive Avoidance task (p < 0.05, corrected for multiple comparisons).

CONCLUSIONS

Findings indicate that there is reduced responsivity of striato-cortical regions when receiving outcomes on an instrumental learning task within a profile of adolescents with substance use and rule-breaking behaviors. Mitigating reward processing dysfunction specifically may represent a potential intervention target for substance-use psychopathologies accompanied by rule-breaking behaviors.

Continuous multiplexed population representations of task context in the mouse primary visual cortex

Effective task execution requires the representation of multiple task-related variables that determine how stimuli lead to correct responses. Even the primary visual cortex (V1) represents other task-related variables such as expectations, choice, and context. However, it is unclear how V1 can flexibly accommodate these variables without interfering with visual representations. We trained mice on a context-switching cross-modal decision task, where performance depends on inferring task context. We found that the context signal that emerged in V1 was behaviorally relevant as it strongly covaried with performance, independent from movement. Importantly, this signal was integrated into V1 representation by multiplexing visual and context signals into orthogonal subspaces. In addition, auditory and choice signals were also multiplexed as these signals were orthogonal to the context representation. Thus, multiplexing allows V1 to integrate visual inputs with other sensory modalities and cognitive variables to avoid interference with the visual representation while ensuring the maintenance of task-relevant variables.

Economic and social values in the brain: evidence from lesions to the human ventromedial prefrontal cortex

Making good economic and social decisions is essential for individual and social welfare. Decades of research have provided compelling evidence that damage to the ventromedial prefrontal cortex (vmPFC) is associated with dramatic personality changes and impairments in economic and social decision-making. However, whether the vmPFC subserves a unified mechanism in the social and non-social domains remains unclear. When choosing between economic options, the vmPFC is thought to guide decision by encoding value signals that reflect the motivational relevance of the options on a common scale. A recent framework, the "extended common neural currency" hypothesis, suggests that the vmPFC may also assign values to social factors and principles, thereby guiding social decision-making. Although neural value signals have been observed in the vmPFC in both social and non-social studies, it is yet to be determined whether they have a causal influence on behavior or merely correlate with decision-making. In this review, we assess whether lesion studies of patients with vmPFC damage offer evidence for such a causal role of the vmPFC in shaping economic and social behavior.

Predictions transform memories: How expected versus unexpected events are integrated or separated in memory

Our brains constantly generate predictions about the environment based on prior knowledge. Many of the events we experience are consistent with these predictions, while others might be inconsistent with prior knowledge and thus violate our predictions. To guide future behavior, the memory system must be able to strengthen, transform, or add to existing knowledge based on the accuracy of our predictions. We synthesize recent evidence suggesting that when an event is consistent with our predictions, it leads to neural integration between related memories, which is associated with enhanced associative memory, as well as memory biases. Prediction errors, in turn, can promote both neural integration and separation, and lead to multiple mnemonic outcomes. We review these findings and how they interact with factors such as memory reactivation, prediction error strength, and task goals, to offer insight into what determines memory for events that violate our predictions. In doing so, this review brings together recent neural and behavioral research to advance our understanding of how predictions shape memory, and why.

Placebo treatment affects brain systems related to affective and cognitive processes, but not nociceptive pain

Placebo analgesia is a replicable and well-studied phenomenon, yet it remains unclear to what degree it includes modulation of nociceptive processes. Some studies find effects consistent with nociceptive effects, but meta-analyses show that these effects are often small. We analyzed placebo analgesia in a large fMRI study (N = 392), including placebo effects on brain responses to noxious stimuli. Placebo treatment caused robust analgesia in both conditioned thermal and unconditioned mechanical pain. Placebo did not decrease fMRI activity in nociceptive pain regions, including the Neurologic Pain Signature (NPS) and pre-registered spinothalamic pathway regions, with strong support from Bayes Factor analyses. However, placebo treatment affected activity in pre-registered analyses of a second neuromarker, the Stimulus Intensity Independent Pain Signature (SIIPS), and several associated a priori brain regions related to motivation and value, in both thermal and mechanical pain. Individual differences in behavioral analgesia were correlated with neural changes in both thermal and mechanical pain. Our results indicate that processes related to affective and cognitive aspects of pain primarily drive placebo analgesia.

Distinct roles of the orbitofrontal cortex, ventral striatum, and dopamine neurons in counterfactual thinking of decision outcomes

Individuals often assess past decisions by comparing what was gained with what would have been gained had they acted differently. Thoughts of past alternatives that counter what actually happened are called "counterfactuals." Recent theories emphasize the role of the prefrontal cortex in processing counterfactual outcomes in decision-making, although how subcortical regions contribute to this process remains to be elucidated. Here we report a clear distinction among the roles of the orbitofrontal cortex, ventral striatum and midbrain dopamine neurons in processing counterfactual outcomes in monkeys. Our findings suggest that actually gained and counterfactual outcome signals are both processed in the cortico-subcortical network constituted by these regions but in distinct manners and integrated only in the orbitofrontal cortex in a way to compare these outcomes. This study extends the prefrontal theory of counterfactual thinking and provides key insights regarding how the prefrontal cortex cooperates with subcortical regions to make decisions using counterfactual information.

Enhancing reinforcement learning models by including direct and indirect pathways improves performance on striatal dependent tasks

A major advance in understanding learning behavior stems from experiments showing that reward learning requires dopamine inputs to striatal neurons and arises from synaptic plasticity of cortico-striatal synapses. Numerous reinforcement learning models mimic this dopamine-dependent synaptic plasticity by using the reward prediction error, which resembles dopamine neuron firing, to learn the best action in response to a set of cues. Though these models can explain many facets of behavior, reproducing some types of goal-directed behavior, such as renewal and reversal, require additional model components. Here we present a reinforcement learning model, TD2Q, which better corresponds to the basal ganglia with two Q matrices, one representing direct pathway neurons (G) and another representing indirect pathway neurons (N). Unlike previous two-Q architectures, a novel and critical aspect of TD2Q is to update the G and N matrices utilizing the temporal difference reward prediction error. A best action is selected for N and G using a softmax with a reward-dependent adaptive exploration parameter, and then differences are resolved using a second selection step applied to the two action probabilities. The model is tested on a range of multi-step tasks including extinction, renewal, discrimination; switching reward probability learning; and sequence learning. Simulations show that TD2Q produces behaviors similar to rodents in choice and sequence learning tasks, and that use of the temporal difference reward prediction error is required to learn multi-step tasks. Blocking the update rule on the N matrix blocks discrimination learning, as observed experimentally. Performance in the sequence learning task is dramatically improved with two matrices. These results suggest that including additional aspects of basal ganglia physiology can improve the performance of reinforcement learning models, better reproduce animal behaviors, and provide insight as to the role of direct- and indirect-pathway striatal neurons.

Decision-making for concurrent reward and threat is differentially modulated by trauma exposure and PTSD symptom severity

Trauma exposure, particularly interpersonal violence (IPV) traumas, are significant risk factors for development of mental health disorders, particularly posttraumatic stress disorder (PTSD). Studies attempting to disentangle mechanisms by which trauma confers risk and maintenance of PTSD have often investigated threat or reward learning in isolation. However, real-world decision-making often involves navigating concurrent and conflicting probabilities for threat and reward. We sought to understand how threat and reward learning interact to impact decision-making, and how these processes are modulated by trauma exposure and PTSD symptom severity. 429 adult participants with a range of trauma exposure and symptom severities completed an online version of the two stage Markov task, where participants make a series of decisions towards the goal of obtaining a reward, that embedded an intermediate threat or neutral image along the sequence of decisions to be made. This task design afforded the possibility to differentiate between threat avoidance vs diminished reward learning in the presence of threat, and whether these two processes reflect model-based vs model-free decision-making. Results demonstrated that trauma exposure severity, particularly IPV exposure, was associated with impairment in model-based learning for reward independent of threat, as well as with model-based threat avoidance. PTSD symptom severity was associated with diminished model-based learning for reward in the presence of threat, consistent with a threat-induced impairment in cognitively-demanding strategies for reward learning, but no evidence of heightened threat avoidance. These results highlight the complex interactions between threat and reward learning as a function of trauma exposure and PTSD symptom severity. Findings have potential implications for treatment augmentation and suggest a need for continued research.

The influence of aerobic exercise on model-based decision making in women with posttraumatic stress disorder

Individuals with PTSD often exhibit deficits in executive functioning. An unexplored aspect of neurocognitive functions associated with PTSD is the type of learning system engaged in during decision-making. A model-free (MF) system is habitual in nature and involves trial-and-error learning that is often updated based on the most recent experience (e.g., repeat action if rewarded). A model-based (MB) system is goal-directed in nature and involves the development of an abstract representation of the environment to facilitate decisions (e.g., choose sequence of actions according to current contextual state and predicted outcomes). The existing neurocognitive literature on PTSD suggests the hypothesis of greater reliance on MF vs MB learning strategies when navigating their environment. While MF systems may be more cognitively efficient, they do not afford flexibility when making prospective predictions about likely outcomes of different decision-tree branches. Emerging research suggests that an acute bout of aerobic exercise improves certain aspects of neurocognition, and thereby could promote the utilization of MB over MF systems during decision making, although prior research has not yet tested this hypothesis. Accordingly, the current study administered a lab-based two-stage Markov decision-making task capable of discriminating MF vs MB decision making, in order to determine if moderate-intensity aerobic exercise (either shortly after or 30-minutes after the exercise bout has ended) promotes greater engagement in MB behavioral strategies compared to light-intensity aerobic exercise in adult women with and without PTSD (N=61). Results revealed that control women generally displayed higher levels of MB behavior that was further increased following immediate exercise, particularly moderate-intensity exercise. By contrast, the PTSD group generally displayed lower levels of MB behavior, and exhibited greater MB behavior when completing the task following moderate-intensity aerobic exercise compared to light-intensity aerobic exercise regardless of whether there was a short or long delay between exercise and the task. Additionally, women with PTSD demonstrated less impairment in MB decision-making compared to controls following moderate-intensity aerobic exercise. These results suggest that an acute bout of moderate-intensity aerobic exercise boosts MB behavior in women with PTSD, and suggests that aerobic exercise may play an important role in enhancing cognitive outcomes for PTSD.

Prior experience modifies acquisition trajectories via response-strategy sampling

Few studies have considered how signal detection parameters evolve during acquisition periods. We addressed this gap by training mice with differential prior experience in a conditional discrimination, auditory signal detection task. Naïve mice, mice given separate experience with each of the later correct choice options (Correct Choice Response Transfer, CCRT), and mice experienced in conditional discriminations (Conditional Discrimination Transfer, CDT) were trained to detect the presence or absence of a tone in white noise. We analyzed data assuming a two-period model of acquisition: a pre-solution and solution period (Heinemann EG (1983) in The Presolution period and the detection of statistical associations. In: Quantitative analyses of behavior: discrimination processes, vol. 4, pp. 21-36). Ballinger. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.536.1978andrep=rep1andtype=pdf ). The pre-solution period was characterized by a selective sampling of biased response strategies until adoption of a conditional responding strategy in the solution period. Correspondingly, discriminability remained low until the solution period; criterion took excursions reflecting response-strategy sampling. Prior experience affected the length and composition of the pre-solution period. Whereas CCRT and CDT mice had shorter pre-solution periods than naïve mice, CDT and Naïve mice developed substantial criterion biases and acquired asymptotic discriminability faster than CCRT mice. To explain these data, we propose a learning model in which mice selectively sample and test different response-strategies and corresponding task structures until they exit the pre-solution period. Upon exit, mice adopt the conditional responding strategy and task structure, with action values updated via inference and generalization from the other task structures. Simulations of representative mouse data illustrate the viability of this model.

Behavioral and neural generalization: Hitting the right notes

Successful generalization to novel stimuli is a core goal of learning and memory systems, but how do we do it? In this issue of Neuron, Miller et al.1 use a novel auditory structure learning task to reveal neural and behavioral signatures of generalization.

Emergence of a predictive model in the hippocampus

The brain organizes experiences into memories that guide future behavior. Hippocampal CA1 population activity is hypothesized to reflect predictive models that contain information about future events, but little is known about how they develop. We trained mice on a series of problems with or without a common statistical structure to observe how memories are formed and updated. Mice that learned structured problems integrated their experiences into a predictive model that contained the solutions to upcoming novel problems. Retrieving the model during learning improved discrimination accuracy and facilitated learning. Using calcium imaging to track CA1 activity during learning, we found that hippocampal ensemble activity became more stable as mice formed a predictive model. The hippocampal ensemble was reactivated during training and incorporated new activity patterns from each training problem. These results show how hippocampal activity supports building predictive models by organizing new information with respect to existing memories.

GluN2B inhibition confers resilience against long-term cocaine-induced neurocognitive sequelae

Cocaine self-administration can disrupt the capacity of humans and rodents to flexibly modify familiar behavioral routines, even when they become maladaptive or unbeneficial. However, mechanistic factors, particularly those driving long-term behavioral changes, are still being determined. Here, we capitalized on individual differences in oral cocaine self-administration patterns in adolescent mice and revealed that the post-synaptic protein PSD-95 was reduced in the orbitofrontal cortex (OFC) of escalating, but not stable, responders, which corresponded with later deficits in flexible decision-making behavior. Meanwhile, NMDA receptor GluN2B subunit content was lower in the OFC of mice that were resilient to escalatory oral cocaine seeking. This discovery led us to next co-administer the GluN2B-selective antagonist ifenprodil with cocaine, blocking the later emergence of cocaine-induced decision-making abnormalities. GluN2B inhibition also prevented cocaine-induced dysregulation of neuronal structure and function in the OFC, preserving mature, mushroom-shaped dendritic spine densities on deep-layer pyramidal neurons, which were otherwise lower with cocaine, and safeguarding functional BLA→OFC connections necessary for action flexibility. We posit that cocaine potentiates GluN2B-dependent signaling, which triggers a series of durable adaptations that result in the dysregulation of post-synaptic neuronal structure in the OFC and disruption of BLA→OFC connections, ultimately weakening the capacity for flexible choice. And thus, inhibiting GluN2B-NMDARs promotes resilience to long-term cocaine-related sequelae.