Action-outcome relationships are represented differently by medial prefrontal and orbitofrontal cortex neurons during action execution

Orbitofrontal and Prelimbic Cortices Serve Complementary Roles in Adapting Reward Seeking to Learned Anxiety

BACKGROUND

Anxiety is a common symptom of several mental health disorders and adversely affects motivated behaviors. Anxiety can emerge from associating risk of future harm while engaged in goal-guided actions. Using a recently developed behavioral paradigm to model this aspect of anxiety, we investigated the role of 2 cortical subregions, the prelimbic medial frontal cortex (PL) and lateral orbitofrontal cortex (lOFC), which have been implicated in anxiety and outcome expectation, in flexible representation of actions associated with harm risk.

METHODS

A seek-take reward-guided instrumental task design was used to train animals (N = 8) to associate the seek action with a variable risk of punishment. After learning, animals underwent extinction training for this association. Fiber photometry was used to measure and compare neuronal activity in the PL and lOFC during learning and extinction.

RESULTS

Animals increased action suppression in response to punishment contingencies. This increase dissipated after extinction training. These behavioral changes were associated with region-specific changes in neuronal activity. PL neuronal activity preferentially adapted to the threat of punishment, whereas lOFC activity adapted to safe aspects of the task. Moreover, correlated activity between these regions was suppressed during actions associated with harm risk, suggesting that these regions may guide behavior independently under anxiety.

CONCLUSIONS

These findings suggest that the PL and lOFC serve distinct but complementary roles in the representation of learned anxiety. This dissociation may provide a mechanism to explain how overlapping cortical systems are implicated in reward-guided action execution during anxiety.

Cortico-striatal beta oscillations as a reward-related signal

The value associated with reward is sensitive to external factors, such as the time between the choice and reward delivery as classically manipulated in temporal discounting tasks. Subjective preference for two reward options is dependent on objective variables of reward magnitude and reward delay. Single neuron correlates of reward value have been observed in regions, including ventral striatum, orbital, and medial prefrontal cortex. Brain imaging studies show cortico-striatal-limbic network activity related to subjective preferences. To explore how oscillatory dynamics represent reward processing across brain regions, we measured local field potentials of rats performing a temporal discounting task. Our goal was to use a data-driven approach to identify an electrophysiological marker that correlates with reward preference. We found that reward-locked oscillations at beta frequencies signaled the magnitude of reward and decayed with longer temporal delays. Electrodes in orbitofrontal/medial prefrontal cortex, anterior insula, ventral striatum, and amygdala individually increased power and were functionally connected at beta frequencies during reward outcome. Beta power during reward outcome correlated with subjective value as defined by a computational model fit to the discounting behavior. These data suggest that cortico-striatal beta oscillations are a reward signal correlated, which may represent subjective value and hold potential to serve as a biomarker and potential therapeutic target.

Lateral Orbitofrontal Cortex Encodes Presence of Risk and Subjective Risk Preference During Decision-Making

Adaptive decision-making requires consideration of objective risks and rewards associated with each option, as well as subjective preference for risky/safe alternatives. Inaccurate risk/reward estimations can engender excessive risk-taking, a central trait in many psychiatric disorders. The lateral orbitofrontal cortex (lOFC) has been linked to many disorders associated with excessively risky behavior and is ideally situated to mediate risky decision-making. Here, we used single-unit electrophysiology to measure neuronal activity from lOFC of freely moving rats performing in a punishment-based risky decision-making task. Subjects chose between a small, safe reward and a large reward associated with either 0% or 50% risk of concurrent punishment. lOFC activity repeatedly encoded current risk in the environment throughout the decision-making sequence, signaling risk before, during, and after a choice. In addition, lOFC encoded reward magnitude, although this information was only evident during action selection. A Random Forest classifier successfully used neural data accurately to predict the risk of punishment in any given trial, and the ability to predict choice via lOFC activity differentiated between and risk-preferring and risk-averse rats. Finally, risk preferring subjects demonstrated reduced lOFC encoding of risk and increased encoding of reward magnitude. These findings suggest lOFC may serve as a central decision-making hub in which external, environmental information converges with internal, subjective information to guide decision-making in the face of punishment risk.

Dynamic Foraging Behavior Performance Is Not Affected by Scn2a Haploinsufficiency

Dysfunction in the gene SCN2A, which encodes the voltage-gated sodium channel Nav1.2, is strongly associated with neurodevelopmental disorders including autism spectrum disorder and intellectual disability (ASD/ID). This dysfunction typically manifests in these disorders as a haploinsufficiency, where loss of one copy of a gene cannot be compensated for by the other allele. Scn2a haploinsufficiency affects a range of cells and circuits across the brain, including associative neocortical circuits that are important for cognitive flexibility and decision-making behaviors. Here, we tested whether Scn2a haploinsufficiency has any effect on a dynamic foraging task that engages such circuits. Scn2a +/- mice and wild-type (WT) littermates were trained on a choice behavior where the probability of reward between two options varied dynamically across trials and where the location of the high reward underwent uncued reversals. Despite impairments in Scn2a-related neuronal excitability, we found that both male and female Scn2a +/- mice performed these tasks as well as wild-type littermates, with no behavioral difference across genotypes in learning or performance parameters. Varying the number of trials between reversals or probabilities of receiving reward did not result in an observable behavioral difference, either. These data suggest that, despite heterozygous loss of Scn2a, mice can perform relatively complex foraging tasks that make use of higher-order neuronal circuits.

Using lickometry to infer differential contributions of salience network regions during compulsion-like alcohol drinking

Alcohol use disorder extracts substantial personal, social and clinical costs, and continued intake despite negative consequences (compulsion-like consumption) can contribute strongly. Here we discuss lickometry, a simple method where lick times are determined across a session, while analysis across many aspects of licking can offer important insights into underlying psychological and action strategies, including their brain mechanisms. We first describe studies implicating anterior insula (AIC) and dorsal medial prefrontal cortex (dMPF) in compulsion-like responding for alcohol, then review work suggesting that AIC/ventral frontal cortex versus dMPF regulate different aspects of behavior (oral control and overall response strategy, versus moment-to-moment action organization). We then detail our lickometer work comparing alcohol-only drinking (AOD) and compulsion-like drinking under moderate- or higher-challenge (ModChD or HiChD, using quinine-alcohol). Many studies have suggested utilization of one of two main strategies, with higher motivation indicated by more bouts, and greater palatability suggested by longer, faster bouts. Instead, ModChD shows decreased variability in many lick measures, which is unexpected but consistent with the suggested importance of automaticity for addiction. Also surprising is that HiChD retains several behavior changes seen with ModChD, reduced tongue variability and earlier bout start, even though intake is otherwise disrupted. Since AIC-related measures are retained under both moderate- and higher-challenge, we propose a novel hypothesis that AIC sustains overall commitment regardless of challenge level, while disordered licking during HiChD mirrors the effects of dMPF inhibition. Thus, while AIC provides overall drive despite challenge, the ability to act is ultimately determined within the dMPF.

Abstraction of Reward Context Facilitates Relative Reward Coding in Neural Populations of the Macaque Anterior Cingulate Cortex

The anterior cingulate cortex (ACC) is believed to be involved in many cognitive processes, including linking goals to actions and tracking decision-relevant contextual information. ACC neurons robustly encode expected outcomes, but how this relates to putative functions of ACC remains unknown. Here, we approach this question from the perspective of population codes by analyzing neural spiking data in the ventral and dorsal banks of the ACC in two male monkeys trained to perform a stimulus-motor mapping task to earn rewards or avoid losses. We found that neural populations favor a low dimensional representational geometry that emphasizes the valence of potential outcomes while also facilitating the independent, abstract representation of multiple task-relevant variables. Valence encoding persisted throughout the trial, and realized outcomes were primarily encoded in a relative sense, such that cue valence acted as a context for outcome encoding. This suggests that the population coding we observe could be a mechanism that allows feedback to be interpreted in a context-dependent manner. Together, our results point to a prominent role for ACC in context setting and relative interpretation of outcomes, facilitated by abstract, or untangled, representations of task variables.SIGNIFICANCE STATEMENT The ability to interpret events in light of the current context is a critical facet of higher-order cognition. The ACC is suggested to be important for tracking contextual information, whereas alternate views hold that its function is more related to the motor system and linking goals to appropriate actions. We evaluated these possibilities by analyzing geometric properties of neural population activity in monkey ACC when contexts were determined by the valence of potential outcomes and found that this information was represented as a dominant, abstract concept. Ensuing outcomes were then coded relative to these contexts, suggesting an important role for these representations in context-dependent evaluation. Such mechanisms may be critical for the abstract reasoning and generalization characteristic of biological intelligence.

Sex-specific alterations in corticotropin-releasing factor regulation of coerulear-cortical network activity

The locus coeruleus (LC)-norepinephrine system is a stress responsive system that regulates arousal and cognitive functions through extensive projections, including to the prefrontal cortex. LC-cortical circuits are activated by stressors, and this activation is thought to contribute to stress-induced impairments in executive function. Because corticotropin-releasing factor (CRF) is a mediator of stress-induced LC activation, we examined the effects of CRF administered into the LC of male and female rats on network activity of two functionally distinct regions of the PFC, the medial PFC (mPFC) and the orbitofrontal cortex (OFC). Network activity, measured as local field potentials, was recorded in awake animals before and after intra-LC infusion of aCSF or CRF (2 or 20 ng). CRF had qualitatively distinct effects on network activity in males and females with respect to dose, region and timecourse. CRF (20 ng) produced a prominent theta oscillation (7-9 Hz) selectively in female rats shortly after LC infusion and 20 min later. In contrast, in male rats, CRF (2 and 20 ng) decreased the amplitude of power in the 4-6 Hz range in the mPFC 10 min after injection. Lastly, CRF (20 ng) increased mPFC-OFC coherence in females and decreased mPFC-OFC coherence in males. In sum, these results show sex differences in CRF modulation of the LC-norepinephrine system that regulates prefrontal cortical networks, which may underlie sex differences in cognitive and behavioral responses to stress.

Orbitofrontal cortex populations are differentially recruited to support actions

The ability to use information from one's prior actions is necessary for decision-making. While orbitofrontal cortex (OFC) has been hypothesized as key for inferences made using cue and value-related information, whether OFC populations contribute to the use of information from volitional actions to guide behavior is not clear. Here, we used a self-paced lever-press hold-down task in which mice infer prior lever-press durations to guide subsequent action performance. We show that the activity of genetically identified lateral OFC (lOFC) subpopulations differentially instantiate current and prior action information during ongoing action execution. Transient state-dependent lOFC circuit disruptions of specified subpopulations reduced the encoding of ongoing press durations but did not disrupt the use of prior action information to guide future action performance. In contrast, a chronic functional loss of lOFC circuit activity resulted in increased reliance on recently executed lever-press durations and impaired contingency reversal, suggesting the recruitment of compensatory mechanisms that resulted in repetitive action control. Our results identify a novel role for lOFC in the integration of action information to guide adaptive behavior.

Learning of probabilistic punishment as a model of anxiety produces changes in action but not punisher encoding in the dmPFC and VTA

Previously, we developed a novel model for anxiety during motivated behavior by training rats to perform a task where actions executed to obtain a reward were probabilistically punished and observed that after learning, neuronal activity in the ventral tegmental area (VTA) and dorsomedial prefrontal cortex (dmPFC) represent the relationship between action and punishment risk (Park and Moghaddam, 2017). Here, we used male and female rats to expand on the previous work by focusing on neural changes in the dmPFC and VTA that were associated with the learning of probabilistic punishment, and anxiolytic treatment with diazepam after learning. We find that adaptive neural responses of dmPFC and VTA during the learning of anxiogenic contingencies are independent from the punisher experience and occur primarily during the peri-action and reward period. Our results also identify peri-action ramping of VTA neural calcium activity, and VTA-dmPFC correlated activity, as potential markers for the anxiolytic properties of diazepam.

Lateral Orbitofrontal Cortex and Basolateral Amygdala Regulate Sensitivity to Delayed Punishment during Decision-making

In real-world decision-making scenarios, negative consequences do not always occur immediately after a choice. This delay between action and outcome drives the underestimation, or "delay discounting", of punishment. While the neural substrates underlying sensitivity to immediate punishment have been well-studied, there has been minimal investigation of delayed consequences. Here, we assessed the role of lateral orbitofrontal cortex (LOFC) and basolateral amygdala (BLA), two regions implicated in cost/benefit decision-making, in sensitivity to delayed vs immediate punishment. The delayed punishment decision-making task (DPDT) was used to measure delay discounting of punishment in rodents. During DPDT, rats choose between a small, single pellet reward and a large, three pellet reward accompanied by a mild foot shock. As the task progresses, the shock is preceded by a delay that systematically increases or decreases throughout the session. We observed that rats avoid choices associated with immediate punishment, then shift preference toward these options when punishment is delayed. LOFC inactivation did not influence choice of rewards with immediate punishment, but decreased choice of delayed punishment. We also observed that BLA inactivation reduced choice of delayed punishment for ascending but not descending delays. Inactivation of either brain region produced comparable effects on decision-making in males and females, but there were sex differences observed in omissions and latency to make a choice. In summary, both LOFC and BLA contribute to the delay discounting of punishment and may serve as promising therapeutic targets to improve sensitivity to delayed punishment during decision-making.Significance StatementNegative consequences occurring after a delay are often underestimated, which can lead to maladaptive decision-making. While sensitivity to immediate punishment during reward-seeking has been well-studied, the neural substrates underlying sensitivity to delayed punishment remain unclear. Here, we used the Delayed Punishment Decision-making Task to determine that lateral orbitofrontal cortex and basolateral amygdala both regulate the discounting of delayed punishment, suggesting that these regions may be potential targets to improve decision-making in psychopathology.

Measuring maladaptive avoidance: from animal models to clinical anxiety

Avoiding stimuli that predict danger is required for survival. However, avoidance can become maladaptive in individuals who overestimate threat and thus avoid safe situations as well as dangerous ones. Excessive avoidance is a core feature of anxiety disorders, post-traumatic stress disorder (PTSD), and obsessive-compulsive disorder (OCD). This avoidance prevents patients from confronting maladaptive threat beliefs, thereby maintaining disordered anxiety. Avoidance is associated with high levels of psychosocial impairment yet is poorly understood at a mechanistic level. Many objective laboratory assessments of avoidance measure adaptive avoidance, in which an individual learns to successfully avoid a truly noxious stimulus. However, anxiety disorders are characterized by maladaptive avoidance, for which there are fewer objective laboratory measures. We posit that maladaptive avoidance behavior depends on a combination of three altered neurobehavioral processes: (1) threat appraisal, (2) habitual avoidance, and (3) trait avoidance tendency. This heterogeneity in underlying processes presents challenges to the objective measurement of maladaptive avoidance behavior. Here we first review existing paradigms for measuring avoidance behavior and its underlying neural mechanisms in both human and animal models, and identify how existing paradigms relate to these neurobehavioral processes. We then propose a new framework to improve the translational understanding of maladaptive avoidance behavior by adapting paradigms to better differentiate underlying processes and mechanisms and applying these paradigms in clinical populations across diagnoses with the goal of developing novel interventions to engage specific identified neurobehavioral targets.

Multiplexed action-outcome representation by striatal striosome-matrix compartments detected with a mouse cost-benefit foraging task

Learning about positive and negative outcomes of actions is crucial for survival and underpinned by conserved circuits including the striatum. How associations between actions and outcomes are formed is not fully understood, particularly when the outcomes have mixed positive and negative features. We developed a novel foraging (‘bandit’) task requiring mice to maximize rewards while minimizing punishments. By 2-photon Ca⁺⁺ imaging, we monitored activity of visually identified anterodorsal striatal striosomal and matrix neurons. We found that action-outcome associations for reward and punishment were encoded in parallel in partially overlapping populations. Single neurons could, for one action, encode outcomes of opposing valence. Striosome compartments consistently exhibited stronger representations of reinforcement outcomes than matrix, especially for high reward or punishment prediction errors. These findings demonstrate multiplexing of action-outcome contingencies by single identified striatal neurons and suggest that striosomal neurons are particularly important in action-outcome learning.

The role that the striatum plays in tracking the association between actions and combinations of rewarding and aversive outcomes remains unclear. Here, by using both calcium imaging in mice and reinforcement learning models, the authors find that individual striatal neurons can encode associations between actions and multiple, sometimes conflicting, outcomes.

Coherent theta activity in the medial and orbital frontal cortices encodes reward value

This study examined how the medial frontal (MFC) and orbital frontal (OFC) cortices process reward information. We simultaneously recorded local field potentials in the two areas as rats consumed liquid sucrose rewards. Both areas exhibited a 4-8 Hz 'theta' rhythm that was phase-locked to the lick cycle. The rhythm tracked shifts in sucrose concentrations and fluid volumes, demonstrating that it is sensitive to differences in reward magnitude. The coupling between the rhythm and licking was stronger in MFC than OFC and varied with response vigor and absolute reward value in the MFC. Spectral analysis revealed zero-lag coherence between the cortical areas, and found evidence for a directionality of the rhythm, with MFC leading OFC. Our findings suggest that consummatory behavior generates simultaneous theta range activity in the MFC and OFC that encodes the value of consumed fluids, with the MFC having a top-down role in the control of consumption.

The orbitofrontal cortex in temporal cognition

One of the most important factors in decision-making is estimating the value of available options. Subregions of the prefrontal cortex, including the orbitofrontal cortex (OFC), have been deemed essential for this process. Value computations require a complex integration across numerous dimensions, including, reward magnitude, effort, internal state, and time. The importance of the temporal dimension is well illustrated by temporal discounting tasks, in which subjects select between smaller-sooner versus larger-later rewards. The specific role of OFC in telling time and integrating temporal information into decision-making remains unclear. Based on the current literature, in this review we reevaluate current theories of OFC function, accounting for the influence of time. Incorporating temporal information into value estimation and decision-making requires distinct, yet interrelated, forms of temporal information including the ability to tell time, represent time, create temporal expectations, and the ability to use this information for optimal decision-making in a wide range of tasks, including temporal discounting and wagering. We use the term "temporal cognition" to refer to the integrated use of these different aspects of temporal information. We suggest that the OFC may be a critical site for the integration of reward magnitude and delay, and thus important for temporal cognition. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Enhanced synaptic properties of the prefrontal cortex and hippocampus after learning a spatial working memory task in adult male mice

Working memory (WM) is the ability to hold on-line and manipulate information. The prefrontal cortex (PFC) is a key brain region involved in WM, while the hippocampus is also involved, particularly, in spatial WM. Although several studies have investigated the neuronal substrates of WM in trained animals, the effects and the mechanisms underlying learning WM tasks have not been explored. In our study, we investigated the effects of learning WM tasks in mice on the function of PFC and hippocampus, by training mice in the delayed alternation task for 9 days (adaptive group). This group was compared to naïve mice (which stayed in their homecage) and mice trained in the alternation procedure only (non-adaptive). Following training, a cohort of mice (Experiment A) was tested in the left-right discrimination task and the reversal learning task, while another cohort (Experiment B) was tested in the attention set-shifting task (AST). The adaptive group performed significantly better in the reversal learning task (Experiment A) and AST (Experiment B), compared to non-adaptive and naïve groups. At the end of the behavioral experiments in Experiment A, field excitatory post-synaptic potential (fEPSP) recordings were performed in PFC and hippocampal brain slices. The adaptive group had enhanced the long-term potentiation (LTP) in the PFC, compared to the other groups. In the hippocampus, both the adaptive and the non-adaptive groups exhibited increased fEPSP compared to the naïve group, but no differences in LTP. In Experiment B, the dendritic spine density was measured, which, in the PFC, was found increased in the adaptive group, compared to the non-adaptive and naïve groups. In the hippocampus, there was an increase in mature dendritic spine density in the adaptive group, compared to the other two groups. Our results indicate a role for LTP and dendritic spine density in learning WM tasks.

Consolidating the Circuit Model for Addiction

Addiction is a disease characterized by compulsive drug seeking and consumption observed in 20-30% of users. An addicted individual will favor drug reward over natural rewards, despite major negative consequences. Mechanistic research on rodents modeling core components of the disease has identified altered synaptic transmission as the functional substrate of pathological behavior. While the initial version of a circuit model for addiction focused on early drug adaptive behaviors observed in all individuals, it fell short of accounting for the stochastic nature of the transition to compulsion. The model builds on the initial pharmacological effect common to all addictive drugs-an increase in dopamine levels in the mesolimbic system. Here, we consolidate this early model by integrating circuits underlying compulsion and negative reinforcement. We discuss the genetic and epigenetic correlates of individual vulnerability. Many recent data converge on a gain-of-function explanation for circuit remodeling, revealing blueprints for novel addiction therapies.

Dynamic decision making and value computations in medial frontal cortex

Dynamic decision making requires an intact medial frontal cortex. Recent work has combined theory and single-neuron measurements in frontal cortex to advance models of decision making. We review behavioral tasks that have been used to study dynamic decision making and algorithmic models of these tasks using reinforcement learning theory. We discuss studies linking neurophysiology and quantitative decision variables. We conclude with hypotheses about the role of other cortical and subcortical structures in dynamic decision making, including ascending neuromodulatory systems.

Integration of value and action in medial prefrontal neural systems

The rodent medial prefrontal cortex (mPFC) plays a key role in regulating cognition, emotion, and behavior. mPFC neurons are activated in diverse experimental paradigms, raising the questions of whether there are specific task elements or dimensions encoded by mPFC neurons, and whether these encoded parameters are selective to neurons in particular mPFC subregions or networks. Here, we consider the role of mPFC neurons in processing appetitive and aversive cues, outcomes, and related behaviors. mPFC neurons are strongly activated in tasks probing value and outcome-associated actions, but these responses vary across experimental paradigms. Can we identify specific categories of responses (e.g., positive or negative value), or do mPFC neurons exhibit response properties that are too heterogeneous/complex to cluster into distinct conceptual groups? Based on a review of relevant studies, we consider what has been done and what needs to be further explored in order to address these questions.

Differential encoding of action selection by orbitofrontal and striatal population dynamics

Survival relies on the ability to flexibly choose between different actions according to varying environmental circumstances. Many lines of evidence indicate that action selection involves signaling in corticostriatal circuits, including the orbitofrontal cortex (OFC) and dorsomedial striatum (DMS). While choice-specific responses have been found in individual neurons from both areas, it is unclear whether populations of OFC or DMS neurons are better at encoding an animal's choice. To address this, we trained head-fixed mice to perform an auditory guided two-alternative choice task, which required moving a joystick forward or backward. We then used silicon microprobes to simultaneously measure the spiking activity of OFC and DMS ensembles, allowing us to directly compare population dynamics between these areas within the same animals. Consistent with previous literature, both areas contained neurons that were selective for specific stimulus-action associations. However, analysis of concurrently recorded ensemble activity revealed that the animal's trial-by-trial behavior could be decoded more accurately from DMS dynamics. These results reveal substantial regional differences in encoding action selection, suggesting that DMS neural dynamics are more specialized than OFC at representing an animal's choice of action.NEW & NOTEWORTHY While previous literature shows that both orbitofrontal cortex (OFC) and dorsomedial striatum (DMS) represent information relevant to selecting specific actions, few studies have directly compared neural signals between these areas. Here we compared OFC and DMS dynamics in mice performing a two-alternative choice task. We found that the animal's choice could be decoded more accurately from DMS population activity. This work provides among the first evidence that OFC and DMS differentially represent information about an animal's selected action.

Prefrontal Cortex Representation of Learning of Punishment Probability During Reward-Motivated Actions

Actions executed toward obtaining a reward are frequently associated with the probability of harm occurring during action execution. Learning this probability allows for appropriate computation of future harm to guide action selection. Impaired learning of this probability may be critical for the pathogenesis of anxiety or reckless and impulsive behavior. Here we designed a task for punishment probability learning during reward-guided actions to begin to understand the neuronal basis of this form of learning, and the biological or environmental variables that influence action selection after learning. Male and female Long-Evans rats were trained in a seek-take behavioral paradigm where the seek action was associated with varying probability of punishment. The take action remained safe and was followed by reward delivery. Learning was evident as subjects selectively adapted seek action behavior as a function of punishment probability. Recording of neural activity in the mPFC during learning revealed changes in phasic mPFC neuronal activity during risky-seek actions but not during the safe take actions or reward delivery, revealing that this region is involved in learning of probabilistic punishment. After learning, the variables that influenced behavior included reinforcer and punisher value, pretreatment with the anxiolytic diazepam, and biological sex. In particular, females were more sensitive to probabilistic punishment than males. These data demonstrate that flexible encoding of risky actions by mPFC is involved in probabilistic punishment learning and provide a novel behavioral approach for studying the pathogenesis of anxiety and impulsivity with inclusion of sex as a biological variable.SIGNIFICANCE STATEMENT Actions we choose to execute toward obtaining a reward are often associated with the probability of harm occurring. Impaired learning of this probability may be critical for the pathogenesis of anxiety or reckless behavior and impulsivity. We developed a behavioral model to assess this mode of learning. This procedure allowed us to determine biological and environmental factors that influence the resistance of reward seeking to probabilistic punishment and to identify the mPFC as a region that flexibly adapts its response to risky actions as contingencies are learned.

Differential Rapid Plasticity in Auditory and Visual Responses in the Primarily Multisensory Orbitofrontal Cortex

Given the connectivity of orbitofrontal cortex (OFC) with the sensory areas and areas involved in goal execution, it is likely that OFC, along with its function in reward processing, also has a role to play in perception-based multisensory decision-making. To understand mechanisms involved in multisensory decision-making, it is important to first know the encoding of different sensory stimuli in single neurons of the mouse OFC. Ruling out effects of behavioral state, memory, and others, we studied the anesthetized mouse OFC responses to auditory, visual, and audiovisual/multisensory stimuli, multisensory associations and sensory-driven input organization to the OFC. Almost all, OFC single neurons were found to be multisensory in nature, with sublinear to supralinear integration of the component unisensory stimuli. With a novel multisensory oddball stimulus set, we show that the OFC receives both unisensory as well as multisensory inputs, further corroborated by retrograde tracers showing labeling in secondary auditory and visual cortices, which we find to also have similar multisensory integration and responses. With long audiovisual pairing/association, we show rapid plasticity in OFC single neurons, with a strong visual bias, leading to a strong depression of auditory responses and effective enhancement of visual responses. Such rapid multisensory association driven plasticity is absent in the auditory and visual cortices, suggesting its emergence in the OFC. Based on the above results, we propose a hypothetical local circuit model in the OFC that integrates auditory and visual information which participates in computing stimulus value in dynamic multisensory environments.

Learning from Action: Reconsidering Movement Signaling in Midbrain Dopamine Neuron Activity

Animals infer when and where a reward is available from experience with informative sensory stimuli and their own actions. In vertebrates, this is thought to depend upon the release of dopamine from midbrain dopaminergic neurons. Studies of the role of dopamine have focused almost exclusively on their encoding of informative sensory stimuli; however, many dopaminergic neurons are active just prior to movement initiation, even in the absence of sensory stimuli. How should current frameworks for understanding the role of dopamine incorporate these observations? To address this question, we review recent anatomical and functional evidence for action-related dopamine signaling. We conclude by proposing a framework in which dopaminergic neurons encode subjective signals of action initiation to solve an internal credit assignment problem.

Credit assignment to state-independent task representations and its relationship with model-based decision making

It is widely accepted that agents learn action values based on experience in a “model-free” manner (i.e., without holding a model of the environment). Environments usually embody many features, where a subset is considered relevant for model-free outcome learning. In this study, we show that a putative model-free system assigns credit to outcome-irrelevant task representations, regardless of stimulus features. The degree of impact of these associations is strongly linked to deployment of model-based strategies. Our findings motivate a reconsideration of how model-free representations are formed and regulated according to the structure of the environment.

Model-free learning enables an agent to make better decisions based on prior experience while representing only minimal knowledge about an environment’s structure. It is generally assumed that model-free state representations are based on outcome-relevant features of the environment. Here, we challenge this assumption by providing evidence that a putative model-free system assigns credit to task representations that are irrelevant to an outcome. We examined data from 769 individuals performing a well-described 2-step reward decision task where stimulus identity but not spatial-motor aspects of the task predicted reward. We show that participants assigned value to spatial-motor representations despite it being outcome irrelevant. Strikingly, spatial-motor value associations affected behavior across all outcome-relevant features and stages of the task, consistent with credit assignment to low-level state-independent task representations. Individual difference analyses suggested that the impact of spatial-motor value formation was attenuated for individuals who showed greater deployment of goal-directed (model-based) strategies. Our findings highlight a need for a reconsideration of how model-free representations are formed and regulated according to the structure of the environment.

Stable Representations of Decision Variables for Flexible Behavior

Decisions occur in dynamic environments. In the framework of reinforcement learning, the probability of performing an action is influenced by decision variables. Discrepancies between predicted and obtained rewards (reward prediction errors) update these variables, but they are otherwise stable between decisions. Although reward prediction errors have been mapped to midbrain dopamine neurons, it is unclear how the brain represents decision variables themselves. We trained mice on a dynamic foraging task in which they chose between alternatives that delivered reward with changing probabilities. Neurons in the medial prefrontal cortex, including projections to the dorsomedial striatum, maintained persistent firing rate changes over long timescales. These changes stably represented relative action values (to bias choices) and total action values (to bias response times) with slow decay. In contrast, decision variables were weakly represented in the anterolateral motor cortex, a region necessary for generating choices. Thus, we define a stable neural mechanism to drive flexible behavior.

The predictive value of neural reward processing on exposure therapy outcome: Results from a randomized controlled trial

BACKGROUND

Exposure is the gold standard treatment for phobic anxiety and is thought to represent the clinical application of extinction learning. Reward sensitivity might however also represent a predictive factor for exposure therapy outcome, as this therapy promotes positive experiences and involves positive comments by the therapist. We hypothesized that high reward sensitivity, as expressed by elevated reward expectancy and reward value, can be associated with better outcome to exposure therapy specifically.

METHODS

Forty-four participants with a specific phobia for spiders were included in the current study. Participants were randomly assigned to exposure therapy (n = 25) or progressive muscle relaxation (PMR) (n = 19). Treatment outcome was defined as pre- versus post-therapy phobia symptoms. Before treatment, functional brain responses and behavioral responses (i.e. reaction time and accuracy) during reward anticipation and consumption were assessed with the Monetary Incentive Delay task (MID). Behavioral and neural responses in regions of interest (i.e. nucleus accumbens, ventromedial prefrontal cortex and the ventral tegmental area) as well as across the whole-brain were subsequently regressed on treatment outcomes.

RESULTS

Exposure therapy was more effective in reducing phobia symptoms than PMR. Longer reaction times to reward cues and lower activation in the left posterior cingulate cortex during reward consumption were selectively associated with symptoms reductions following exposure therapy but not following PMR. Only within the exposure therapy group, greater symptom reduction was related to increased activation in the ventrolateral prefrontal cortex during reward anticipation, and decreased activation in the medial prefrontal cortex during reward consumption.

CONCLUSION

Results indicate that individual differences in reward sensitivity can specifically predict exposure therapy outcome. Although activation in regions of interest were not related to therapy outcome, regions involved in attentional processing of reward cues were predictive of phobic symptom change following exposure therapy but not PMR.

Small Networks Encode Decision-Making in Primary Auditory Cortex

Sensory detection tasks enhance representations of behaviorally meaningful stimuli in primary auditory cortex (A1). However, it remains unclear how A1 encodes decision-making. Neurons in A1 layer 2/3 (L2/3) show heterogeneous stimulus selectivity and complex anatomical connectivity, and receive input from prefrontal cortex. Thus, task-related modulation of activity in A1 L2/3 might differ across subpopulations. To study the neural coding of decision-making, we used two-photon imaging in A1 L2/3 of mice performing a tone-detection task. Neural responses to targets showed attentional gain and encoded behavioral choice. To characterize network representation of behavioral choice, we analyzed functional connectivity using Granger causality, pairwise noise correlations, and neural decoding. During task performance, small groups of four to five neurons became sparsely linked, locally clustered, and rostro-caudally oriented, while noise correlations both increased and decreased. Our results suggest that sensory-based decision-making involves small neural networks driven by the sum of sensory input, attentional gain, and behavioral choice.

Differential Encoding of Time by Prefrontal and Striatal Network Dynamics

Telling time is fundamental to many forms of learning and behavior, including the anticipation of rewarding events. Although the neural mechanisms underlying timing remain unknown, computational models have proposed that the brain represents time in the dynamics of neural networks. Consistent with this hypothesis, changing patterns of neural activity dynamically in a number of brain areas-including the striatum and cortex-has been shown to encode elapsed time. To date, however, no studies have explicitly quantified and contrasted how well different areas encode time by recording large numbers of units simultaneously from more than one area. Here, we performed large-scale extracellular recordings in the striatum and orbitofrontal cortex of mice that learned the temporal relationship between a stimulus and a reward and reported their response with anticipatory licking. We used a machine-learning algorithm to quantify how well populations of neurons encoded elapsed time from stimulus onset. Both the striatal and cortical networks encoded time, but the striatal network outperformed the orbitofrontal cortex, a finding replicated both in simultaneously and nonsimultaneously recorded corticostriatal datasets. The striatal network was also more reliable in predicting when the animals would lick up to ∼1 s before the actual lick occurred. Our results are consistent with the hypothesis that temporal information is encoded in a widely distributed manner throughout multiple brain areas, but that the striatum may have a privileged role in timing because it has a more accurate "clock" as it integrates information across multiple cortical areas.

SIGNIFICANCE STATEMENT

The neural representation of time is thought to be distributed across multiple functionally specialized brain structures, including the striatum and cortex. However, until now, the neural code for time has not been compared quantitatively between these areas. Here, we performed large-scale recordings in the striatum and orbitofrontal cortex of mice trained on a stimulus-reward association task involving a delay period and used a machine-learning algorithm to quantify how well populations of simultaneously recorded neurons encoded elapsed time from stimulus onset. We found that, although both areas encoded time, the striatum consistently outperformed the orbitofrontal cortex. These results suggest that the striatum may refine the code for time by integrating information from multiple inputs.

Ethanol Mediated Inhibition of Synaptic Vesicle Recycling at Amygdala Glutamate Synapses Is Dependent upon Munc13-2

Chronic exposure to alcohol produces adaptations within the basolateral amygdala (BLA) that are associated with the development of anxiety-like behaviors during withdrawal. In part, these adaptations are mediated by plasticity in glutamatergic synapses occurring through an AMPA receptor mediated form of post-synaptic facilitation in addition to a unique form of presynaptic facilitation. In comparison to the post-synaptic compartment, relatively less is understood about the mechanisms involved in the acute and chronic effects of ethanol in the presynaptic terminal. Previous research has demonstrated that glutamatergic terminals in the mouse BLA are sensitive to ethanol mediated inhibition of synaptic vesicle recycling in a strain-dependent fashion. Importantly, the strain-dependent differences in presynaptic ethanol sensitivity are in accordance with known strain-dependent differences in ethanol/anxiety interactions. In the present study, we have used a short-hairpin RNA to knockdown the expression of the presynaptic Munc13-2 protein in C57BL/6J mice, whose BLA glutamate terminals are normally ethanol-insensitive. We injected this shRNA, or a scrambled control virus, into the medial prefrontal cortex (mPFC) which sends dense projections to the BLA. Accordingly, this knockdown strategy reduces the expression of the Munc13-2 isoform in mPFC terminals within the BLA and alters presynaptic terminal function in C57BL/6J mice in a manner that phenocopies DBA/2J glutamate terminals which are normally ethanol-sensitive. Here, we provide evidence that manipulation of this single protein, Munc13-2, renders C57BL/6J terminals sensitive to ethanol mediated inhibition of synaptic vesicle recycling and post-tetanic potentiation. Furthermore, we found that this ethanol inhibition was dose dependent. Considering the important role of Munc13 proteins in synaptic plasticity, this study potentially identifies a molecular mechanism regulating the acute presynaptic effects of ethanol to the long lasting adaptations in the BLA that occur during chronic ethanol exposure.

Representational changes of latent strategies in rat medial prefrontal cortex precede changes in behaviour

The ability to change behavioural strategies in the face of a changing world has been linked to the integrity of medial prefrontal cortex (mPFC) function in several species. While recording studies have found that mPFC representations reflect the strategy being used, lesion studies suggest that mPFC is necessary for changing strategy. Here we examine the relationship between representational changes in mPFC and behavioural strategy changes in the rat. We found that on tasks with a forced change in reward criterion, strategy-related representational transitions in mPFC occurred after animals learned that the reward contingency had changed, but before their behaviour changed. On tasks in which animals made their own strategic decisions, representational transitions in mPFC preceded changes in behaviour. These results suggest that mPFC does not merely reflect the action–selection policy of the animal, but rather that mPFC processes information related to a need for a change in strategy.

The medial prefrontal cortex (mPFC) is involved in changing behavioural strategies. Recording neural ensembles in rats, Powell and Redish find that the requirement for those changes is represented in mPFC before they manifest behaviourally, both in tasks that externally force a change and in tasks with self-determined change.

Impact of anxiety on prefrontal cortex encoding of cognitive flexibility

Anxiety often is studied as a stand-alone construct in laboratory models. But in the context of coping with real-life anxiety, its negative impacts extend beyond aversive feelings and involve disruptions in ongoing goal-directed behaviors and cognitive functioning. Critical examples of cognitive constructs affected by anxiety are cognitive flexibility and decision making. In particular, anxiety impedes the ability to shift flexibly between strategies in response to changes in task demands, as well as the ability to maintain a strategy in the presence of distractors. The brain region most critically involved in behavioral flexibility is the prefrontal cortex (PFC), but little is known about how anxiety impacts PFC encoding of internal and external events that are critical for flexible behavior. Here we review animal and human neurophysiological and neuroimaging studies implicating PFC neural processing in anxiety-induced deficits in cognitive flexibility. We then suggest experimental and analytical approaches for future studies to gain a better mechanistic understanding of impaired cognitive inflexibility in anxiety and related disorders.