Action selection and action value in frontal-striatal circuits

Dynamics of striatal action selection and reinforcement learning

Spiny projection neurons (SPNs) in dorsal striatum are often proposed as a locus of reinforcement learning in the basal ganglia. Here, we identify and resolve a fundamental inconsistency between striatal reinforcement learning models and known SPN synaptic plasticity rules. Direct-pathway (dSPN) and indirect-pathway (iSPN) neurons, which promote and suppress actions, respectively, exhibit synaptic plasticity that reinforces activity associated with elevated or suppressed dopamine release. We show that iSPN plasticity prevents successful learning, as it reinforces activity patterns associated with negative outcomes. However, this pathological behavior is reversed if functionally opponent dSPNs and iSPNs, which promote and suppress the current behavior, are simultaneously activated by efferent input following action selection. This prediction is supported by striatal recordings and contrasts with prior models of SPN representations. In our model, learning and action selection signals can be multiplexed without interference, enabling learning algorithms beyond those of standard temporal difference models.

Distinct dopaminergic spike-timing-dependent plasticity rules are suited to different functional roles

Various mathematical models have been formulated to describe the changes in synaptic strengths resulting from spike-timing-dependent plasticity (STDP). A subset of these models include a third factor, dopamine, which interacts with the timing of pre- and postsynaptic spiking to contribute to plasticity at specific synapses, notably those from cortex to striatum at the input layer of the basal ganglia. Theoretical work to analyze these plasticity models has largely focused on abstract issues, such as the conditions under which they may promote synchronization and the weight distributions induced by inputs with simple correlation structures, rather than on scenarios associated with specific tasks, and has generally not considered dopamine-dependent forms of STDP. In this paper, we analyze, mathematically and with simulations, three forms of dopamine-modulated STDP in three scenarios that are relevant to corticostriatal synapses. Two of the models considered comprise previously proposed STDP rules with modifications to incorporate dopamine, while the third is a corticostriatal dopamine-dependent STDP rule adapted from a similar one already in the literature. We test the ability of each of the three models to maintain its weights in the face of noise and to complete simple reward prediction and action selection tasks, studying the learned weight distributions and corresponding task performance in each setting. Interestingly, we find that each of the three plasticity rules is well suited to a subset of the scenarios studied but falls short in others. These results show that different tasks may require different forms of synaptic plasticity, yielding the prediction that the precise form of the STDP mechanism may vary across regions of the striatum, and other brain areas impacted by dopamine, that are involved in distinct computational functions.

Complementary opposing D2-MSNs and D1-MSNs dynamics during interval timing

The role of striatal pathways in cognitive processing is unclear. We studied dorsomedial striatal cognitive processing during interval timing, an elementary cognitive task that requires mice to estimate intervals of several seconds and involves working memory for temporal rules as well as attention to the passage of time. We harnessed optogenetic tagging to record from striatal D2-dopamine receptor-expressing medium spiny neurons (D2-MSNs) in the indirect pathway and from D1-dopamine receptor-expressing MSNs (D1-MSNs) in the direct pathway. We found that D2-MSNs and D1-MSNs exhibited distinct dynamics over temporal intervals as quantified by principal component analyses and trial-by-trial generalized linear models. MSN recordings helped construct and constrain a four-parameter drift-diffusion computational model. This model predicted that disrupting either D2-MSNs or D1-MSNs would increase interval timing response times and alter MSN firing. In line with this prediction, we found that optogenetic inhibition or pharmacological disruption of either D2-MSNs or D1-MSNs increased interval timing response times. Pharmacologically disrupting D2-MSNs or D1-MSNs also changed MSN dynamics and degraded trial-by-trial temporal decoding. Together, our findings demonstrate that D2-MSNs and D1-MSNs make complementary contributions to interval timing despite opposing dynamics, implying that striatal direct and indirect pathways work together to shape temporal control of action. These data provide novel insight into basal ganglia cognitive operations beyond movement and have implications for human striatal diseases and therapies targeting striatal pathways.

Reconfiguration of Behavioral Signals in the Anterior Cingulate Cortex Based on Emotional State

Behaviors and their execution depend on the context and emotional state in which they are performed. The contextual modulation of behavior likely relies on regions such as the anterior cingulate cortex (ACC) that multiplex information about emotional/autonomic states and behaviors. The objective of the present study was to understand how the representations of behaviors by ACC neurons become modified when performed in different emotional states. A pipeline of machine learning techniques was developed to categorize and classify complex, spontaneous behaviors in male rats from the video. This pipeline, termed Hierarchical Unsupervised Behavioural Discovery Tool (HUB-DT), discovered a range of statistically separable behaviors during a task in which motivationally significant outcomes were delivered in blocks of trials that created three unique "emotional contexts." HUB-DT was capable of detecting behaviors specific to each emotional context and was able to identify and segregate the portions of a neural signal related to a behavior and to emotional context. Overall, ∼10× as many neurons responded to behaviors in a contextually dependent versus a fixed manner, highlighting the extreme impact of emotional state on representations of behaviors that were precisely defined based on detailed analyses of limb kinematics. This type of modulation may be a key mechanism that allows the ACC to modify the behavioral output based on emotional states and contextual demands.

Assessment on interoceptive awareness on alcohol use and gambling disorders reveals dissociable interoceptive abilities linked to external and internal dependencies: Practical use of Body Perception Questionnaire Very Short Form (BPQ-VSF) in clinical settings

Interoception is one of the pivotal cognitive functions for mechanisms of our body awareness, and malfunction of the interoceptive network is thought to be associated with mental illness, including addiction. Within addictive disorders, substance-based and non-substance-based addictions are known to hold dissociable reward systems. However, little is known about how interoceptive awareness between these addiction sub-types would differ. Subjective interoceptive awareness was assessed among patients with alcohol use disorder (n = 50) who were subsequently hospitalized or remained out-patient and gambling addiction (n = 41) by the Body Awareness component of the Japanese version of the Body Perception Questionnaire (BPQ-VSFBA-J) and compared them against healthy control (n = 809). Both addiction groups showed significantly lower BPQ than the control, with no substantial differences between inpatients and outpatients for alcohol samples. Notably, BPQ scores for gambling patients were significantly lower than those for the alcohol group. This evidence may suggest a putative role of interoceptive ability on the severity of behavioral addiction over substance-based addiction.

Motor System-Dependent Effects of Amygdala and Ventral Striatum Lesions on Explore-Exploit Behaviors

Deciding whether to forego immediate rewards or explore new opportunities is a key component of flexible behavior and is critical for the survival of the species. Although previous studies have shown that different cortical and subcortical areas, including the amygdala and ventral striatum (VS), are implicated in representing the immediate (exploitative) and future (explorative) value of choices, the effect of the motor system used to make choices has not been examined. Here, we tested male rhesus macaques with amygdala or VS lesions on two versions of a three-arm bandit task where choices were registered with either a saccade or an arm movement. In both tasks we presented the monkeys with explore-exploit tradeoffs by periodically replacing familiar options with novel options that had unknown reward probabilities. We found that monkeys explored more with saccades but showed better learning with arm movements. VS lesions caused the monkeys to be more explorative with arm movements and less explorative with saccades, although this may have been due to an overall decrease in performance. VS lesions affected the monkeys' ability to learn novel stimulus-reward associations in both tasks, while after amygdala lesions this effect was stronger when choices were made with saccades. Further, on average, VS and amygdala lesions reduced the monkeys' ability to choose better options only when choices were made with a saccade. These results show that learning reward value associations to manage explore-exploit behaviors is motor system dependent and they further define the contributions of amygdala and VS to reinforcement learning.

Neural Representations of Post-Decision Accuracy and Reward Expectation in the Caudate Nucleus and Frontal Eye Field

Performance monitoring that supports ongoing behavioral adjustments is often examined in the context of either choice confidence for perceptual decisions (i.e., "did I get it right?") or reward expectation for reward-based decisions (i.e., "what reward will I receive?"). However, our understanding of how the brain encodes these distinct evaluative signals remains limited because they are easily conflated, particularly in commonly used two-alternative tasks with symmetric rewards for correct choices. Previously we used a motion-discrimination task with asymmetric rewards to identify neural substrates of forming reward-biased perceptual decisions in the caudate nucleus (part of the striatum in the basal ganglia) and the frontal eye field (FEF, in prefrontal cortex). Here we leveraged this task design to partially decouple estimates of accuracy and reward expectation and examine their impacts on subsequent decisions and their representations in those two brain areas. We identified distinguishable representations of these two evaluative signals in individual caudate and FEF neurons, with regional differences in their distribution patterns and time courses. We observed that well-trained monkeys (both sexes) used both evaluative signals, infrequently but consistently, to adjust their subsequent decisions. We found further that these behavioral adjustments had reliable relationships with the neural representations of both evaluative signals in caudate, but not FEF. These results suggest that the cortico-striatal decision network may use diverse evaluative signals to monitor and adjust decision-making behaviors, adding to our understanding of the different roles that the FEF and caudate nucleus play in a diversity of decision-related computations.

Prefrontal signals precede striatal signals for biased credit assignment in motivational learning biases

Actions are biased by the outcomes they can produce: Humans are more likely to show action under reward prospect, but hold back under punishment prospect. Such motivational biases derive not only from biased response selection, but also from biased learning: humans tend to attribute rewards to their own actions, but are reluctant to attribute punishments to having held back. The neural origin of these biases is unclear. Specifically, it remains open whether motivational biases arise primarily from the architecture of subcortical regions or also reflect cortical influences, the latter being typically associated with increased behavioral flexibility and control beyond stereotyped behaviors. Simultaneous EEG-fMRI allowed us to track which regions encoded biased prediction errors in which order. Biased prediction errors occurred in cortical regions (dorsal anterior and posterior cingulate cortices) before subcortical regions (striatum). These results highlight that biased learning is not a mere feature of the basal ganglia, but arises through prefrontal cortical contributions, revealing motivational biases to be a potentially flexible, sophisticated mechanism.

The neurocognitive role of working memory load when Pavlovian motivational control affects instrumental learning

Research suggests that a fast, capacity-limited working memory (WM) system and a slow, incremental reinforcement learning (RL) system jointly contribute to instrumental learning. Thus, situations that strain WM resources alter instrumental learning: under WM loads, learning becomes slow and incremental, the reliance on computationally efficient learning increases, and action selection becomes more random. It is also suggested that Pavlovian learning influences people's behavior during instrumental learning by providing hard-wired instinctive responses including approach to reward predictors and avoidance of punishment predictors. However, it remains unknown how constraints on WM resources affect instrumental learning under Pavlovian influence. Thus, we conducted a functional magnetic resonance imaging (fMRI) study (N = 49) in which participants completed an instrumental learning task with Pavlovian-instrumental conflict (the orthogonalized go/no-go task) both with and without extra WM load. Behavioral and computational modeling analyses revealed that WM load reduced the learning rate and increased random choice, without affecting Pavlovian bias. Model-based fMRI analysis revealed that WM load strengthened RPE signaling in the striatum. Moreover, under WM load, the striatum showed weakened connectivity with the ventromedial and dorsolateral prefrontal cortex when computing reward expectations. These results suggest that the limitation of cognitive resources by WM load promotes slow and incremental learning through the weakened cooperation between WM and RL; such limitation also makes action selection more random, but it does not directly affect the balance between instrumental and Pavlovian systems.

Attractor dynamics reflect decision confidence in macaque prefrontal cortex

Decisions are made with different degrees of consistency, and this consistency can be linked to the confidence that the best choice has been made. Theoretical work suggests that attractor dynamics in networks can account for choice consistency, but how this is implemented in the brain remains unclear. Here we provide evidence that the energy landscape around attractor basins in population neural activity in the prefrontal cortex reflects choice consistency. We trained two rhesus monkeys to make accept/reject decisions based on pretrained visual cues that signaled reward offers with different magnitudes and delays to reward. Monkeys made consistent decisions for very good and very bad offers, but decisions were less consistent for intermediate offers. Analysis of neural data showed that the attractor basins around patterns of activity reflecting decisions had steeper landscapes for offers that led to consistent decisions. Therefore, we provide neural evidence that energy landscapes predict decision consistency, which reflects decision confidence.

Attractor dynamics reflect decision confidence in macaque prefrontal cortex

Decisions are made with different degrees of consistency, and this consistency can be linked to the confidence that the best choice has been made. Theoretical work suggests that attractor dynamics in networks can account for choice consistency, but how this is implemented in the brain remains unclear. Here, we provide evidence that the energy landscape around attractor basins in population neural activity in prefrontal cortex reflects choice consistency. We trained two rhesus monkeys to make accept/reject decisions based on pretrained visual cues that signaled reward offers with different magnitudes and delays-to-reward. Monkeys made consistent decisions for very good and very bad offers, but decisions were less consistent for intermediate offers. Analysis of neural data showed that the attractor basins around patterns of activity reflecting decisions had steeper landscapes for offers that led to consistent decisions. Therefore, we provide neural evidence that energy landscapes predict decision consistency, which reflects decision confidence.

Contributions of the Basal Ganglia to Visual Perceptual Decisions

The basal ganglia (BG) make up a prominent nexus between visual and motor-related brain regions. In contrast to the BG's well-established roles in movement control and value-based decision making, their contributions to the transformation of visual input into an action remain unclear, especially in the context of perceptual decisions based on uncertain visual evidence. This article reviews recent progress in our understanding of the BG's contributions to the formation, evaluation, and adjustment of such decisions. From theoretical and experimental perspectives, the review focuses on four key stations in the BG network, namely, the striatum, pallidum, subthalamic nucleus, and midbrain dopamine neurons, which can have different roles and together support the decision process.

Dead rats, dopamine, performance metrics, and peacock tails: Proxy failure is an inherent risk in goal-oriented systems

When a measure becomes a target, it ceases to be a good measure. For example, when standardized test scores in education become targets, teachers may start "teaching to the test," leading to breakdown of the relationship between the measure - test performance - and the underlying goal - quality education. Similar phenomena have been named and described across a broad range of contexts, such as economics, academia, machine learning, and ecology. Yet it remains unclear whether these phenomena bear only superficial similarities, or if they derive from some fundamental unifying mechanism. Here, we propose such a unifying mechanism, which we label proxy failure. We first review illustrative examples and their labels, such as the "cobra effect," "Goodhart's law," and "Campbell's law." Second, we identify central prerequisites and constraints of proxy failure, noting that it is often only a partial failure or divergence. We argue that whenever incentivization or selection is based on an imperfect proxy measure of the underlying goal, a pressure arises that tends to make the proxy a worse approximation of the goal. Third, we develop this perspective for three concrete contexts, namely neuroscience, economics, and ecology, highlighting similarities and differences. Fourth, we outline consequences of proxy failure, suggesting it is key to understanding the structure and evolution of goal-oriented systems. Our account draws on a broad range of disciplines, but we can only scratch the surface within each. We thus hope the present account elicits a collaborative enterprise, entailing both critical discussion as well as extensions in contexts we have missed.

Nutrient-Sensitive Reinforcement Learning in Monkeys

In reinforcement learning (RL), animals choose by assigning values to options and learn by updating these values from reward outcomes. This framework has been instrumental in identifying fundamental learning variables and their neuronal implementations. However, canonical RL models do not explain how reward values are constructed from biologically critical intrinsic reward components, such as nutrients. From an ecological perspective, animals should adapt their foraging choices in dynamic environments to acquire nutrients that are essential for survival. Here, to advance the biological and ecological validity of RL models, we investigated how (male) monkeys adapt their choices to obtain preferred nutrient rewards under varying reward probabilities. We found that the nutrient composition of rewards strongly influenced learning and choices. Preferences of the animals for specific nutrients (sugar, fat) affected how they adapted to changing reward probabilities; the history of recent rewards influenced choices of the monkeys more strongly if these rewards contained the their preferred nutrients (nutrient-specific reward history). The monkeys also chose preferred nutrients even when they were associated with lower reward probability. A nutrient-sensitive RL model captured these processes; it updated the values of individual sugar and fat components of expected rewards based on experience and integrated them into subjective values that explained the choices of the monkeys. Nutrient-specific reward prediction errors guided this value-updating process. Our results identify nutrients as important reward components that guide learning and choice by influencing the subjective value of choice options. Extending RL models with nutrient-value functions may enhance their biological validity and uncover nutrient-specific learning and decision variables.SIGNIFICANCE STATEMENT RL is an influential framework that formalizes how animals learn from experienced rewards. Although reward is a foundational concept in RL theory, canonical RL models cannot explain how learning depends on specific reward properties, such as nutrients. Intuitively, learning should be sensitive to the nutrient components of the reward to benefit health and survival. Here, we show that the nutrient (fat, sugar) composition of rewards affects how the monkeys choose and learn in an RL paradigm and that key learning variables including reward history and reward prediction error should be modified with nutrient-specific components to account for the choice behavior observed in the monkeys. By incorporating biologically critical nutrient rewards into the RL framework, our findings help advance the ecological validity of RL models.

Deficient dynamics of prefrontal-striatal and striatal-default mode network (DMN) neural circuits in internet gaming disorder

BACKGROUND

Studies have proven that individuals with internet gaming disorder (IGD) show impaired cognitive control over game craving; however, the neural mechanism underlying this process remains unclear. Accordingly, the present study aimed to investigate the dynamic features of brain functional networks of individuals with IGD during rest, which have barely been understood until now.

METHODS

Resting-state fMRI data were collected from 333 subjects (123 subjects with IGD (males/females: 73/50) and 210 healthy controls (males/females: 135/75)). First, the data-driven methodology, named co-activation pattern analysis, was applied to investigate the dynamic features of nucleus accumbens (the core region involved in craving/reward processing and addiction)-centered brain networks in IGD. Further, machine learning analysis was conducted to investigate the prediction effect of the dynamic features on participants' addiction severity.

RESULTS

Compared to controls, subjects in the IGD group showed decreased resilience, betweenness centrality and occurrence in the prefrontal-striatal neural circuit, and decreased in-degree in the striatal-default mode network (DMN) circuit. Moreover, these decreased dynamic features could significantly predict participants' addiction severity.

LIMITATIONS

The causal relationship between IGD and the abnormal dynamic features cannot be identified in this study. All the subjects were university students.

CONCLUSIONS

The present results revealed the underlying brain networks of uncontrollable craving and game-seeking behaviors in individuals with IGD during rest. The decreased dynamics of the prefrontal-striatal and striatal-DMN neural circuits might be potential biomarkers for predicting the addiction severity of IGD and potential targets for effective interventions to reduce game craving of this disorder.

Thalamocortical contribution to flexible learning in neural systems

Animal brains evolved to optimize behavior in dynamic environments, flexibly selecting actions that maximize future rewards in different contexts. A large body of experimental work indicates that such optimization changes the wiring of neural circuits, appropriately mapping environmental input onto behavioral outputs. A major unsolved scientific question is how optimal wiring adjustments, which must target the connections responsible for rewards, can be accomplished when the relation between sensory inputs, action taken, and environmental context with rewards is ambiguous. The credit assignment problem can be categorized into context-independent structural credit assignment and context-dependent continual learning. In this perspective, we survey prior approaches to these two problems and advance the notion that the brain's specialized neural architectures provide efficient solutions. Within this framework, the thalamus with its cortical and basal ganglia interactions serves as a systems-level solution to credit assignment. Specifically, we propose that thalamocortical interaction is the locus of meta-learning where the thalamus provides cortical control functions that parametrize the cortical activity association space. By selecting among these control functions, the basal ganglia hierarchically guide thalamocortical plasticity across two timescales to enable meta-learning. The faster timescale establishes contextual associations to enable behavioral flexibility, while the slower one enables generalization to new contexts.

Hierarchical Reinforcement Learning, Sequential Behavior, and the Dorsal Frontostriatal System

To effectively behave within ever-changing environments, biological agents must learn and act at varying hierarchical levels such that a complex task may be broken down into more tractable subtasks. Hierarchical reinforcement learning (HRL) is a computational framework that provides an understanding of this process by combining sequential actions into one temporally extended unit called an option. However, there are still open questions within the HRL framework, including how options are formed and how HRL mechanisms might be realized within the brain. In this review, we propose that the existing human motor sequence literature can aid in understanding both of these questions. We give specific emphasis to visuomotor sequence learning tasks such as the discrete sequence production task and the M × N (M steps × N sets) task to understand how hierarchical learning and behavior manifest across sequential action tasks as well as how the dorsal cortical-subcortical circuitry could support this kind of behavior. This review highlights how motor chunks within a motor sequence can function as HRL options. Furthermore, we aim to merge findings from motor sequence literature with reinforcement learning perspectives to inform experimental design in each respective subfield.

Brain Functional Connectivity Mapping of Behavioral Flexibility in Rhesus Monkeys

The predisposition to engage in autonomous habitual behaviors has been associated with behavioral disorders, such as obsessive-compulsive disorder and addiction. Attentional set-shifting tasks (ASSTs), which incorporate changes governing the association of discriminative stimuli with contingent reinforcement, are commonly used to measure underlying processes of cognitive/behavioral flexibility. The purpose of this study was to identify primate brain networks that mediate trait-like deficits in ASST performance using resting-state fMRI. A self-pacing ASST was administered to three cohorts of rhesus monkeys (total n = 35, 18 female). Increased performance over 30 consecutive sessions segregated the monkeys into two populations, termed High Performers (HP, n = 17) and Low Performers (LP, n = 17), with one anomaly. Compared with LPs, HPs had higher rates of improving performance over sessions and completed the 8 sets/sessions with fewer errors. LP monkeys, on the other hand, spent most of each session in the first set and often did not acquire the first reversal. A whole-brain independent components analysis of resting-state fMRI under isoflurane identified four strong networks. Of these, a dual regression analysis revealed that a designated "executive control network," differed between HPs and LPs. Specific areas of connectivity in the rhesus executive control network, including frontal cortices (ventrolateral, ventromedial, and orbital) and the dorsal striatum (caudate, putamen) correlated with perseverative errors and response latency. Overall, the results identify trait-like characteristics of behavioral flexibility that are associated with correlated brain activity involving specific nuclei of frontostriatal networks.SIGNIFICANCE STATEMENT Resting state functional connectivity MRI in rhesus monkeys identified specific nuclei in frontostriatal circuitry that were associated with population differences in perseverative and impulsive aspects of cognitive flexibility.

Longer screen time utilization is associated with the polygenic risk for Attention-deficit/hyperactivity disorder with mediation by brain white matter microstructure

BACKGROUND

Attention-deficit/hyperactivity disorder (ADHD) has been reported to be associated with longer screen time utilization (STU) at the behavioral level. However, whether there are shared neural links between ADHD symptoms and prolonged STU is not clear and has not been explored in a single large-scale dataset.

METHODS

Leveraging the genetics, neuroimaging and behavioral data of 11,000+ children aged 9-11 from the Adolescent Brain Cognitive Development cohort, this study investigates the associations between the polygenic risk and trait for ADHD, STU, and white matter microstructure through cross-sectionally and longitudinal analyses.

FINDINGS

Children with higher polygenic risk scores for ADHD tend to have longer STU and more severe ADHD symptoms. Fractional anisotropy (FA) values in several white matter tracts are negatively correlated with both the ADHD polygenic risk score and STU, including the inferior frontal-striatal tract, inferior frontal-occipital fasciculus, superior longitudinal fasciculus and corpus callosum. Most of these tracts are linked to visual-related functions. Longitudinal analyses indicate a directional effect of white matter microstructure on the ADHD scale, and a bi-directional effect between the ADHD scale and STU. Furthermore, reduction of FA in several white matter tracts mediates the association between the ADHD polygenic risk score and STU.

INTERPRETATION

These findings shed new light on the shared neural overlaps between ADHD symptoms and prolonged STU, and provide evidence that the polygenic risk for ADHD is related, via white matter microstructure and the ADHD trait, to STU.

FUNDING

This study was mainly supported by NSFC and National Key R&D Program of China.

Choice-selective sequences dominate in cortical relative to thalamic inputs to NAc to support reinforcement learning

How are actions linked with subsequent outcomes to guide choices? The nucleus accumbens, which is implicated in this process, receives glutamatergic inputs from the prelimbic cortex and midline regions of the thalamus. However, little is known about whether and how representations differ across these input pathways. By comparing these inputs during a reinforcement learning task in mice, we discovered that prelimbic cortical inputs preferentially represent actions and choices, whereas midline thalamic inputs preferentially represent cues. Choice-selective activity in the prelimbic cortical inputs is organized in sequences that persist beyond the outcome. Through computational modeling, we demonstrate that these sequences can support the neural implementation of reinforcement-learning algorithms, in both a circuit model based on synaptic plasticity and one based on neural dynamics. Finally, we test and confirm a prediction of our circuit models by direct manipulation of nucleus accumbens input neurons.

Differential coding of goals and actions in ventral and dorsal corticostriatal circuits during goal-directed behavior

Goal-directed behavior requires identifying objects in the environment that can satisfy internal needs and executing actions to obtain those objects. The current study examines ventral and dorsal corticostriatal circuits that support complementary aspects of goal-directed behavior. We analyze activity from the amygdala, ventral striatum, orbitofrontal cortex, and lateral prefrontal cortex (LPFC) while monkeys perform a three-armed bandit task. Information about chosen stimuli and their value is primarily encoded in the amygdala, ventral striatum, and orbitofrontal cortex, while the spatial information is primarily encoded in the LPFC. Before the options are presented, information about the to-be-chosen stimulus is represented in the amygdala, ventral striatum, and orbitofrontal cortex; at the time of choice, the information is passed to the LPFC to direct a saccade. Thus, learned value information specifying behavioral goals is maintained throughout the ventral corticostriatal circuit, and it is routed through the dorsal circuit at the time actions are selected.

Reinforcement-learning in fronto-striatal circuits

We review the current state of knowledge on the computational and neural mechanisms of reinforcement-learning with a particular focus on fronto-striatal circuits. We divide the literature in this area into five broad research themes: the target of the learning-whether it be learning about the value of stimuli or about the value of actions; the nature and complexity of the algorithm used to drive the learning and inference process; how learned values get converted into choices and associated actions; the nature of state representations, and of other cognitive machinery that support the implementation of various reinforcement-learning operations. An emerging fifth area focuses on how the brain allocates or arbitrates control over different reinforcement-learning sub-systems or "experts". We will outline what is known about the role of the prefrontal cortex and striatum in implementing each of these functions. We then conclude by arguing that it will be necessary to build bridges from algorithmic level descriptions of computational reinforcement-learning to implementational level models to better understand how reinforcement-learning emerges from multiple distributed neural networks in the brain.

Striatal BOLD and Midfrontal Theta Power Express Motivation for Action

Action selection is biased by the valence of anticipated outcomes. To assess mechanisms by which these motivational biases are expressed and controlled, we measured simultaneous EEG-fMRI during a motivational Go/NoGo learning task (N = 36), leveraging the temporal resolution of EEG and subcortical access of fMRI. VmPFC BOLD encoded cue valence, importantly predicting trial-by-trial valence-driven response speed differences and EEG theta power around cue onset. In contrast, striatal BOLD encoded selection of active Go responses and correlated with theta power around response time. Within trials, theta power ramped in the fashion of an evidence accumulation signal for the value of making a "Go" response, capturing the faster responding to reward cues. Our findings reveal a dual nature of midfrontal theta power, with early components reflecting the vmPFC contribution to motivational biases, and late components reflecting their striatal translation into behavior, in line with influential recent "value of work" theories of striatal processing.

The neural underpinnings of allocentric thinking in a novel signaling task

The ability to adopt the perspectives of others is fundamental to effective communication in social interactions. However, the neural correlates of allocentric thinking in communicative signaling remain unclear. We adapted a novel signaling task in which the signaler was given the target word and must choose a one-word signal to help the receiver guess the target. Behavioral results suggest that speakers can use allocentric thinking to choose signals that are salient from the perspective of the receiver rather than their own point of view. At the neural level, functional magnetic resonance imaging (fMRI) data reveal that the medial prefrontal cortex (mPFC), ventral striatum, and temporal-parietal junction are more activated when signalers engage in allocentric than egocentric thinking. Moreover, functional connectivity between the mPFC and ventral striatum predicted individuals' perspective-taking ability during successful communication. These findings reveal that neural representations in the mPFC-striatum network support perspective-taking in complex social decision making, providing a new perspective on how the brain arbitrates between allocentric thinking and egocentric thinking in communication and social coordination.

Dynamic decision making and value computations in medial frontal cortex

Dynamic decision making requires an intact medial frontal cortex. Recent work has combined theory and single-neuron measurements in frontal cortex to advance models of decision making. We review behavioral tasks that have been used to study dynamic decision making and algorithmic models of these tasks using reinforcement learning theory. We discuss studies linking neurophysiology and quantitative decision variables. We conclude with hypotheses about the role of other cortical and subcortical structures in dynamic decision making, including ascending neuromodulatory systems.

Activity of fast-spiking interneurons in the monkey striatum during reaching movements guided by external cues or by a free choice

Parvalbumin-containing GABAergic interneurons in the striatum, electrophysiologically identified as fast-spiking interneurons (FSIs), exert inhibitory control over striatal output to drive appropriate behavior. While a number of studies have emphasized their importance in motor control, it is unknown how these putative interneurons adapt their functional properties to different modes of movement selection. Here, we tested whether FSIs are sensitive to externally versus internally selected movements by recording their activity while two male rhesus monkeys performed reaching movements to visual targets. Two variants were used: an external condition, in which movements were instructed via external cues, and an internal condition, in which movements were guided by an internal representation of the target location. These conditions allowed to contrast the FSI activity associated with either externally cued or internally driven movement selection. After extensive training, reaching performance was only marginally affected by the type of movement, albeit with some differences between the monkeys. Over two-thirds of the FSIs were modulated around movement onset, regardless of the condition, and consisting mostly of increased activity. We found that a subset of FSIs showed stronger activation related to the initiation of movements in the external condition than in the internal condition, suggesting a dependence on movement selection mode. Moreover, this difference in the strength of FSI activation was predominant in the motor striatum. These data indicate that changes in FSI activity carry information that is scaled by constraints on action selection reflecting the involvement of local striatal inhibitory circuits in adaptation of behavior according to task demands.

Learning to select actions shapes recurrent dynamics in the corticostriatal system

Learning to select appropriate actions based on their values is fundamental to adaptive behavior. This form of learning is supported by fronto-striatal systems. The dorsal-lateral prefrontal cortex (dlPFC) and the dorsal striatum (dSTR), which are strongly interconnected, are key nodes in this circuitry. Substantial experimental evidence, including neurophysiological recordings, have shown that neurons in these structures represent key aspects of learning. The computational mechanisms that shape the neurophysiological responses, however, are not clear. To examine this, we developed a recurrent neural network (RNN) model of the dlPFC-dSTR circuit and trained it on an oculomotor sequence learning task. We compared the activity generated by the model to activity recorded from monkey dlPFC and dSTR in the same task. This network consisted of a striatal component which encoded action values, and a prefrontal component which selected appropriate actions. After training, this system was able to autonomously represent and update action values and select actions, thus being able to closely approximate the representational structure in corticostriatal recordings. We found that learning to select the correct actions drove action-sequence representations further apart in activity space, both in the model and in the neural data. The model revealed that learning proceeds by increasing the distance between sequence-specific representations. This makes it more likely that the model will select the appropriate action sequence as learning develops. Our model thus supports the hypothesis that learning in networks drives the neural representations of actions further apart, increasing the probability that the network generates correct actions as learning proceeds. Altogether, this study advances our understanding of how neural circuit dynamics are involved in neural computation, revealing how dynamics in the corticostriatal system support task learning.

Frontal eye field and caudate neurons make different contributions to reward-biased perceptual decisions

Many decisions require trade-offs between sensory evidence and internal preferences. Potential neural substrates include the frontal eye field (FEF) and caudate nucleus, but their distinct roles are not understood. Previously we showed that monkeys’ decisions on a direction-discrimination task with asymmetric rewards reflected a biased accumulate-to-bound decision process (Fan et al., 2018) that was affected by caudate microstimulation (Doi et al., 2020). Here we compared single-neuron activity in FEF and caudate to each other and to accumulate-to-bound model predictions derived from behavior. Task-dependent neural modulations were similar in both regions. However, choice-selective neurons in FEF, but not caudate, encoded behaviorally derived biases in the accumulation process. Baseline activity in both regions was sensitive to reward context, but this sensitivity was not reliably associated with behavioral biases. These results imply distinct contributions of FEF and caudate neurons to reward-biased decision-making and put experimental constraints on the neural implementation of accumulation-to-bound-like computations.

Increased Dopamine Type 2 Gene Expression in the Dorsal Striatum in Individuals With Autism Spectrum Disorder Suggests Alterations in Indirect Pathway Signaling and Circuitry

Autism spectrum disorder (ASD) is behaviorally defined and diagnosed by delayed and/or impeded language, stereotyped repetitive behaviors, and difficulties with social interactions. Additionally, there are disruptions in motor processing, which includes the intent to execute movements, interrupted/inhibited action chain sequences, impaired execution of speech, and repetitive motor behaviors. Cortical loops through basal ganglia (BG) structures are known to play critical roles in the typical functioning of these actions. Specifically, corticostriate projections to the dorsal striatum (caudate and putamen) convey abundant input from motor, cognitive and limbic cortices and subsequently project to other BG structures. Excitatory dopamine (DA) type 1 receptors are predominantly expressed on GABAergic medium spiny neurons (MSNs) in the dorsal striatum as part of the "direct pathway" to GPi and SNpr whereas inhibitory DA type 2 receptors are predominantly expressed on MSNs that primarily project to GPe. This study aimed to better understand how this circuitry may be altered in ASD, especially concerning the neurochemical modulation of GABAergic MSNs within the two major BG pathways. We utilized two classical methods to analyze the postmortem BG in ASD in comparison to neurotypical cases: ligand binding autoradiography to quantify densities of GABA-A, GABA-B, 5-HT₂, and DA type 1 and 2 receptors and in situ hybridization histochemistry (ISHH) to quantify mRNA for D1, D2 receptors and three key GABAergic subunits (α1, β2, and γ2), as well as the GABA synthesizing enzymes (GAD65/67). Results demonstrated significant increases in D2 mRNA within MSNs in both the caudate and putamen, which was further verified by proenkephalin mRNA that is co-expressed with the D2 receptor in the indirect pathway MSNs. In contrast, all other GABAergic, serotonergic and dopaminergic markers in the dorsal striatum had comparable labeling densities. These results indicate alterations in the indirect pathway of the BG, with possible implications for the execution of competing motor programs and E/I imbalance in the direct/indirect motor feedback pathways through thalamic and motor cortical areas. Results also provide insights regarding the efficacy of FDA-approved drugs used to treat individuals with ASD acting on specific DA and 5-HT receptor subtypes.

Time will tell: Decision making in premanifest and manifest Huntington's disease

OBJECTIVE

To investigate cognitive flexibility in premanifest and manifest Huntington's disease (HD).

BACKGROUND

HD is an autosomal dominant neurodegenerative disease characterized by motor, cognitive, and behavioral abnormalities with typical motor symptoms. In this study, we wanted to assess decision making in premanifest (pre-HD) and manifest HD patients.

METHODS

A total of 77 non-demented subjects including 29 pre-HD, 22 manifest HD patients, and 26 healthy controls (HC) were included. We stratified the pre-HD group based on their estimated years to disease onset into a far (FAR, n = 13) and a near (NEAR, n = 16) group. Furthermore, participants performed the Montreal cognitive assessment battery (MoCA), the trail making task part A and B (TMT A, TMT B), the Symbol digit modalities test (SDMT), and the beads task.

RESULTS

In the beads task, HD patients gathered less information than all other groups (all p-values < .001). Furthermore, the NEAR group gathered less information than the FAR group (p < .001) and HC (p = .001). There was no difference between the HC and the FAR group (p = 1.0). In the TMT and the SDMT, HD patients were slower than all other groups (all p-values < .01) but there were no other significant differences.

CONCLUSIONS

Decision making with a higher degree of uncertainty may be an early neuropsychological sign to indicate the disease process prior to reaching criteria for motor diagnosis of HD.

Development and evaluation of an artificial intelligence system for COVID-19 diagnosis

Early detection of COVID-19 based on chest CT enables timely treatment of patients and helps control the spread of the disease. We proposed an artificial intelligence (AI) system for rapid COVID-19 detection and performed extensive statistical analysis of CTs of COVID-19 based on the AI system. We developed and evaluated our system on a large dataset with more than 10 thousand CT volumes from COVID-19, influenza-A/B, non-viral community acquired pneumonia (CAP) and non-pneumonia subjects. In such a difficult multi-class diagnosis task, our deep convolutional neural network-based system is able to achieve an area under the receiver operating characteristic curve (AUC) of 97.81% for multi-way classification on test cohort of 3,199 scans, AUC of 92.99% and 93.25% on two publicly available datasets, CC-CCII and MosMedData respectively. In a reader study involving five radiologists, the AI system outperforms all of radiologists in more challenging tasks at a speed of two orders of magnitude above them. Diagnosis performance of chest x-ray (CXR) is compared to that of CT. Detailed interpretation of deep network is also performed to relate system outputs with CT presentations. The code is available at https://github.com/ChenWWWeixiang/diagnosis_covid19 .

Effects of Amygdala Lesions on Object-Based Versus Action-Based Learning in Macaques

The neural systems that underlie reinforcement learning (RL) allow animals to adapt to changes in their environment. In the present study, we examined the hypothesis that the amygdala would have a preferential role in learning the values of visual objects. We compared a group of monkeys (Macaca mulatta) with amygdala lesions to a group of unoperated controls on a two-armed bandit reversal learning task. The task had two conditions. In the What condition, the animals had to learn to select a visual object, independent of its location. And in the Where condition, the animals had to learn to saccade to a location, independent of the object at the location. In both conditions choice-outcome mappings reversed in the middle of the block. We found that monkeys with amygdala lesions had learning deficits in both conditions. Monkeys with amygdala lesions did not have deficits in learning to reverse choice-outcome mappings. Rather, amygdala lesions caused the monkeys to become overly sensitive to negative feedback which impaired their ability to consistently select the more highly valued action or object. These results imply that the amygdala is generally necessary for RL.

Attention-related modulation of caudate neurons depends on superior colliculus activity

Recent work has implicated the primate basal ganglia in visual perception and attention, in addition to their traditional role in motor control. The basal ganglia, especially the caudate nucleus 'head' (CDh) of the striatum, receive indirect anatomical connections from the superior colliculus (SC), a midbrain structure that is known to play a crucial role in the control of visual attention. To test the possible functional relationship between these subcortical structures, we recorded CDh neuronal activity of macaque monkeys before and during unilateral SC inactivation in a spatial attention task. SC inactivation significantly altered the attention-related modulation of CDh neurons and strongly impaired the classification of task-epochs based on CDh activity. Only inactivation of SC on the same side of the brain as recorded CDh neurons, not the opposite side, had these effects. These results demonstrate a novel interaction between SC activity and attention-related visual processing in the basal ganglia.

Hypothalamic Interactions with Large-Scale Neural Circuits Underlying Reinforcement Learning and Motivated Behavior

Biological agents adapt behavior to support the survival needs of the individual and the species. In this review we outline the anatomical, physiological, and computational processes that support reinforcement learning (RL). We describe two circuits in the primate brain that are linked to specific aspects of learning and goal-directed behavior. The ventral circuit, that includes the amygdala, ventral medial prefrontal cortex, and ventral striatum, has substantial connectivity with the hypothalamus. The dorsal circuit, that includes inferior parietal cortex, dorsal lateral prefrontal cortex, and the dorsal striatum, has minimal connectivity with the hypothalamus. The hypothalamic connectivity suggests distinct roles for these circuits. We propose that the ventral circuit defines behavioral goals, and the dorsal circuit orchestrates behavior to achieve those goals.

Differential encoding of action selection by orbitofrontal and striatal population dynamics

Survival relies on the ability to flexibly choose between different actions according to varying environmental circumstances. Many lines of evidence indicate that action selection involves signaling in corticostriatal circuits, including the orbitofrontal cortex (OFC) and dorsomedial striatum (DMS). While choice-specific responses have been found in individual neurons from both areas, it is unclear whether populations of OFC or DMS neurons are better at encoding an animal's choice. To address this, we trained head-fixed mice to perform an auditory guided two-alternative choice task, which required moving a joystick forward or backward. We then used silicon microprobes to simultaneously measure the spiking activity of OFC and DMS ensembles, allowing us to directly compare population dynamics between these areas within the same animals. Consistent with previous literature, both areas contained neurons that were selective for specific stimulus-action associations. However, analysis of concurrently recorded ensemble activity revealed that the animal's trial-by-trial behavior could be decoded more accurately from DMS dynamics. These results reveal substantial regional differences in encoding action selection, suggesting that DMS neural dynamics are more specialized than OFC at representing an animal's choice of action.NEW & NOTEWORTHY While previous literature shows that both orbitofrontal cortex (OFC) and dorsomedial striatum (DMS) represent information relevant to selecting specific actions, few studies have directly compared neural signals between these areas. Here we compared OFC and DMS dynamics in mice performing a two-alternative choice task. We found that the animal's choice could be decoded more accurately from DMS population activity. This work provides among the first evidence that OFC and DMS differentially represent information about an animal's selected action.

The caudate nucleus contributes causally to decisions that balance reward and uncertain visual information

Our decisions often balance what we observe and what we desire. A prime candidate for implementing this complex balancing act is the basal ganglia pathway, but its roles have not yet been examined experimentally in detail. Here, we show that a major input station of the basal ganglia, the caudate nucleus, plays a causal role in integrating uncertain visual evidence and reward context to guide adaptive decision-making. In monkeys making saccadic decisions based on motion cues and asymmetric reward-choice associations, single caudate neurons encoded both sources of information. Electrical microstimulation at caudate sites during motion viewing affected the monkeys’ decisions. These microstimulation effects included coordinated changes in multiple computational components of the decision process that mimicked the monkeys’ similarly coordinated voluntary strategies for balancing visual and reward information. These results imply that the caudate nucleus plays causal roles in coordinating decision processes that balance external evidence and internal preferences.

Caudate stimulation enhances learning

Neuromodulation is a promising treatment modality for disorders of learning and memory, offering the possibility of precise alteration of disordered neural circuits. Studies to date have failed to identify an optimal target and stimulation paradigm. Six epilepsy patients with depth electrodes implanted for seizure localization participated in our study. We recorded local field potentials from implanted electrodes while subjects participated in an associative learning task requiring them to learn an association between presented images and a button press. Three subjects participated in stimulation sessions during which caudate or putamen stimulation was delivered for some images during feedback after correct responses. Caudate stimulation enhanced learning. Both caudate and dorsolateral prefrontal cortex demonstrated a beta power increase during the feedback period of the learning task that was greater following correct than incorrect trials. In dorsolateral prefrontal cortex, this difference increased with learning and persisted beyond the end of the feedback period. Caudate stimulation was associated with increased dorsolateral prefrontal cortex beta power following feedback. These findings suggest that temporally specific caudate stimulation is a promising neuromodulation strategy to improve learning in disorders of learning and memory.

Prefrontal Regulation of Punished Ethanol Self-administration

BACKGROUND

A clinical hallmark of alcohol use disorder is persistent drinking despite potential adverse consequences. The ventromedial prefrontal cortex (vmPFC) and dorsomedial prefrontal cortex (dmPFC) are positioned to exert top-down control over subcortical regions, such as the nucleus accumbens shell (NAcS) and basolateral amygdala, which encode positive and negative valence of ethanol (EtOH)-related stimuli. Prior rodent studies have implicated these regions in regulation of punished EtOH self-administration (EtOH-SA).

METHODS

We conducted in vivo electrophysiological recordings in mouse vmPFC and dmPFC to obtain neuronal correlates of footshock-punished EtOH-SA. Ex vivo recordings were performed in NAcS D₁ receptor-expressing medium spiny neurons receiving vmPFC input to examine punishment-related plasticity in this pathway. Optogenetic photosilencing was employed to assess the functional contribution of the vmPFC, dmPFC, vmPFC projections to NAcS, or vmPFC projections to basolateral amygdala, to punished EtOH-SA.

RESULTS

Punishment reduced EtOH lever pressing and elicited aborted presses (lever approach followed by rapid retraction). Neurons in the vmPFC and dmPFC exhibited phasic firing to EtOH lever presses and aborts, but only in the vmPFC was there a population-level shift in coding from lever presses to aborts with punishment. Closed-loop vmPFC, but not dmPFC, photosilencing on a postpunishment probe test negated the reduction in EtOH lever presses but not in aborts. Punishment was associated with altered plasticity at vmPFC inputs to D₁ receptor-expressing medium spiny neurons in the NAcS. Photosilencing vmPFC projections to the NAcS, but not to the basolateral amygdala, partially reversed suppression of EtOH lever presses on probe testing.

CONCLUSIONS

These findings demonstrate a key role for the vmPFC in regulating EtOH-SA after punishment, with implications for understanding the neural basis of compulsive drinking in alcohol use disorder.

Dimensionality, information and learning in prefrontal cortex

Learning leads to changes in population patterns of neural activity. In this study we wanted to examine how these changes in patterns of activity affect the dimensionality of neural responses and information about choices. We addressed these questions by carrying out high channel count recordings in dorsal-lateral prefrontal cortex (dlPFC; 768 electrodes) while monkeys performed a two-armed bandit reinforcement learning task. The high channel count recordings allowed us to study population coding while monkeys learned choices between actions or objects. We found that the dimensionality of neural population activity was higher across blocks in which animals learned the values of novel pairs of objects, than across blocks in which they learned the values of actions. The increase in dimensionality with learning in object blocks was related to less shared information across blocks, and therefore patterns of neural activity that were less similar, when compared to learning in action blocks. Furthermore, these differences emerged with learning, and were not a simple function of the choice of a visual image or action. Therefore, learning the values of novel objects increases the dimensionality of neural representations in dlPFC.

In this study we found that learning to choose rewarding objects increased the diversity of patterns of activity, measured as the dimensionality of the response, observed in dorsal-lateral prefrontal cortex. The dimensionality increase for learning to choose rewarding objects was larger than the dimensionality increase for learning to choose rewarding actions. The dimensionality increase was not a simple function of the diverse set of images used, as the patterns of activity only appeared after learning.

Prefrontal Cortex Predicts State Switches during Reversal Learning

Reinforcement learning allows organisms to predict future outcomes and to update their beliefs about value in the world. The dorsal-lateral prefrontal cortex (dlPFC) integrates information carried by reward circuits, which can be used to infer the current state of the world under uncertainty. Here, we explored the dlPFC computations related to updating current beliefs during stochastic reversal learning. We recorded the activity of populations up to 1,000 neurons, simultaneously, in two male macaques while they executed a two-armed bandit reversal learning task. Behavioral analyses using a Bayesian framework showed that animals inferred reversals and switched their choice preference rapidly, rather than slowly updating choice values, consistent with state inference. Furthermore, dlPFC neural populations accurately encoded choice preference switches. These results suggest that prefrontal neurons dynamically encode decisions associated with Bayesian subjective values, highlighting the role of the PFC in representing a belief about the current state of the world.

Basal Ganglia Circuits for Action Specification

Behavior is readily classified into patterns of movements with inferred common goals-actions. Goals may be discrete; movements are continuous. Through the careful study of isolated movements in laboratory settings, or via introspection, it has become clear that animals can exhibit exquisite graded specification to their movements. Moreover, graded control can be as fundamental to success as the selection of which action to perform under many naturalistic scenarios: a predator adjusting its speed to intercept moving prey, or a tool-user exerting the perfect amount of force to complete a delicate task. The basal ganglia are a collection of nuclei in vertebrates that extend from the forebrain (telencephalon) to the midbrain (mesencephalon), constituting a major descending extrapyramidal pathway for control over midbrain and brainstem premotor structures. Here we discuss how this pathway contributes to the continuous specification of movements that endows our voluntary actions with vigor and grace.

The Neurobiology of Impulsive Decision-Making and Reinforcement Learning in Nonhuman Animals

Impulsive decisions are those that favor immediate over delayed rewards, involve the acceptance of undue risk or uncertainty, or fail to adapt to environmental changes. Pathological levels of impulsive decision-making have been observed in individuals with mental illness, but there may be substantial heterogeneity in the processes that drive impulsive choices. Understanding this behavioral heterogeneity may be critical for understanding associated diverseness in the neural mechanisms that give rise to impulsivity. The application of reinforcement learning algorithms in the deconstruction of impulsive decision-making phenotypes can help bridge the gap between biology and behavior and provide insights into the biobehavioral heterogeneity of impulsive choice. This chapter will review the literature on the neurobiological mechanisms of impulsive decision-making in nonhuman animals; specifically, the role of the amine neuromodulatory systems (dopamine, serotonin, norepinephrine, and acetylcholine) in impulsive decision-making and reinforcement learning processes is discussed. Ultimately, the integration of reinforcement learning algorithms with sophisticated behavioral and neuroscience techniques may be critical for advancing the understanding of the neurochemical basis of impulsive decision-making.

Movement-related coupling of human subthalamic nucleus spikes to cortical gamma

Cortico-basal ganglia interactions continuously shape the way we move. Ideas about how this circuit works are based largely on models those consider only firing rate as the mechanism of information transfer. A distinct feature of neural activity accompanying movement, however, is increased motor cortical and basal ganglia gamma synchrony. To investigate the relationship between neuronal firing in the basal ganglia and cortical gamma activity during movement, we analysed human ECoG and subthalamic nucleus (STN) unit activity during hand gripping. We found that fast reaction times were preceded by enhanced STN spike-to-cortical gamma phase coupling, indicating a role in motor preparation. Importantly, increased gamma phase coupling occurred independent of changes in mean STN firing rates, and the relative timing of STN spikes was offset by half a gamma cycle for ipsilateral vs. contralateral movements, indicating that relative spike timing is as relevant as firing rate for understanding cortico-basal ganglia information transfer.

Primate Orbitofrontal Cortex Codes Information Relevant for Managing Explore-Exploit Tradeoffs

Reinforcement learning (RL) refers to the behavioral process of learning to obtain reward and avoid punishment. An important component of RL is managing explore-exploit tradeoffs, which refers to the problem of choosing between exploiting options with known values and exploring unfamiliar options. We examined correlates of this tradeoff, as well as other RL related variables, in orbitofrontal cortex (OFC) while three male monkeys performed a three-armed bandit learning task. During the task, novel choice options periodically replaced familiar options. The values of the novel options were unknown, and the monkeys had to explore them to see if they were better than other currently available options. The identity of the chosen stimulus and the reward outcome were strongly encoded in the responses of single OFC neurons. These two variables define the states and state transitions in our model that are relevant to decision-making. The chosen value of the option and the relative value of exploring that option were encoded at intermediate levels. We also found that OFC value coding was stimulus specific, as opposed to coding value independent of the identity of the option. The location of the option and the value of the current environment were encoded at low levels. Therefore, we found encoding of the variables relevant to learning and managing explore-exploit tradeoffs in OFC. These results are consistent with findings in the ventral striatum and amygdala and show that this monosynaptically connected network plays an important role in learning based on the immediate and future consequences of choices.SIGNIFICANCE STATEMENT Orbitofrontal cortex (OFC) has been implicated in representing the expected values of choices. Here we extend these results and show that OFC also encodes information relevant to managing explore-exploit tradeoffs. Specifically, OFC encodes an exploration bonus, which characterizes the relative value of exploring novel choice options. OFC also strongly encodes the identity of the chosen stimulus, and reward outcomes, which are necessary for computing the value of novel and familiar options.

Information-Limiting Correlations in Large Neural Populations

Understanding the neural code requires understanding how populations of neurons code information. Theoretical models predict that information may be limited by correlated noise in large neural populations. Nevertheless, analyses based on tens of neurons have failed to find evidence of saturation. Moreover, some studies have shown that noise correlations can be very small, and therefore may not affect information coding. To determine whether information-limiting correlations exist, we implanted eight Utah arrays in prefrontal cortex (PFC; area 46) of two male macaque monkeys, recording >500 neurons simultaneously. We estimated information in PFC about saccades as a function of ensemble size. Noise correlations were, on average, small (∼10^-3). However, information scaled strongly sublinearly with ensemble size. After shuffling trials, destroying noise correlations, information was a linear function of ensemble size. Thus, we provide evidence for the existence of information-limiting noise correlations in large populations of PFC neurons.SIGNIFICANCE STATEMENT Recent theoretical work has shown that even small correlations can limit information if they are "differential correlations," which are difficult to measure directly. However, they can be detected through decoding analyses on recordings from a large number of neurons over a large number of trials. We have achieved both by collecting neural activity in dorsal-lateral prefrontal cortex of macaques using eight microelectrode arrays (768 electrodes), from which we were able to compute accurate information estimates. We show, for the first time, strong evidence for information-limiting correlations. Despite pairwise correlations being small (on the order of 10^-3), they affect information coding in populations on the order of 100 s of neurons.

The Role of the Striatum in Learning to Orthogonalize Action and Valence: A Combined PET and 7 T MRI Aging Study

Pavlovian biases influence instrumental learning by coupling reward seeking with action invigoration and punishment avoidance with action suppression. Using a probabilistic go/no-go task designed to orthogonalize action (go/no-go) and valence (reward/punishment), recent studies have shown that the interaction between the two is dependent on the striatum and its key neuromodulator dopamine. Using this task, we sought to identify how structural and neuromodulatory age-related differences in the striatum may influence Pavlovian biases and instrumental learning in 25 young and 31 older adults. Computational modeling revealed a significant age-related reduction in reward and punishment sensitivity and marked (albeit not significant) reduction in learning rate and lapse rate (irreducible noise). Voxel-based morphometry analysis using 7 Tesla MRI images showed that individual differences in learning rate in older adults were related to the volume of the caudate nucleus. In contrast, dopamine synthesis capacity in the dorsal striatum, assessed using [18F]-DOPA positron emission tomography in 22 of these older adults, was not associated with learning performance and did not moderate the relationship between caudate volume and learning rate. This multiparametric approach suggests that age-related differences in striatal volume may influence learning proficiency in old age.

A neural network for information seeking

Humans and other animals often show a strong desire to know the uncertain rewards their future has in store, even when they cannot use this information to influence the outcome. However, it is unknown how the brain predicts opportunities to gain information and motivates this information-seeking behavior. Here we show that neurons in a network of interconnected subregions of primate anterior cingulate cortex and basal ganglia predict the moment of gaining information about uncertain rewards. Spontaneous increases in their information prediction signals are followed by gaze shifts toward objects associated with resolving uncertainty, and pharmacologically disrupting this network reduces the motivation to seek information. These findings demonstrate a cortico-basal ganglia mechanism responsible for motivating actions to resolve uncertainty by seeking knowledge about the future.

Animals resolve uncertainty by seeking knowledge about the future. How the brain controls this is unclear. The authors show that a network including primate anterior cingulate cortex and basal ganglia encodes opportunities to gain information about uncertain rewards and mediates information seeking.