Rat nucleus accumbens neurons predominantly respond to the outcome-related properties of conditioned stimuli rather than their behavioral-switching properties

Lesions of the nucleus accumbens core modulate development of matching behavior

Background

The development of choice is a crucial determinant in the performance of appetitive responses. Given two options with different reinforcement rates, animals match their relative rate of responding to the relative rates of reinforcement (i.e., matching behavior). A previous study has shown that the nucleus accumbens core (AcbC) is involved in the performance of matching behavior in trained animals. However, the role of the AcbC in the acquisition of matching behavior has not been addressed.

Results

We conducted a series of experimental sessions to examine the role of the AcbC on the development of matching behavior. Instrumental responding was measured in rats with excitotoxic lesions of the AcbC. Rats were given two options that differed in the relative rate of reinforcement under concurrent variable-interval schedules. The locations of the more frequently reinforced option and the alternative option were randomly switched between sessions. Lesions of the AcbC accelerated the development of matching behavior compared to the sham-operated group. The AcbC-lesioned rats exhibited closer conformity to the matching law than shams when the options were in the same positions as in the previous session (the same condition), but not when the option locations had been switched (the different condition). The AcbC rats showed smaller probabilities of switching behavior between alternatives than shams. Post-reinforcement pausing was not affected by the AcbC lesion. Neither numbers of rewards obtained nor number of lever presses were different between the AcbC-lesioned rats and shams over session blocks.

Conclusions

Our results suggest that the AcbC plays a regulatory role in the development of matching behavior through switching probabilities rather than perception of reward magnitude. The differential effect of AcbC lesions on the matching behavior between the same and different conditions suggests influence of the spontaneous recovery, that is, reversion to a previously reinforced choice at the beginning of the next session, on the development of matching behavior in the AcbC-lesioned rats.

Stress intensifies demands on response selection during action cascading processes

Stress has been shown to modulate a number of cognitive processes including action control. These functions are important in daily life and are mediated by various cognitive subprocesses. However, it is unknown if stress affects the whole processing cascade, or exerts specific effects on a restricted subset of processes involved in the chaining of actions. We examine the effects of stress on action selection processes in a stop-change paradigm and apply event-related potentials (ERPs) combined with source localization analysis to examine potentially restricted effects of stress on subprocesses mediating action cascading. The results show that attentional selection processes, as well as processes related to allocation of processing resources were not affected by stress. Stress only seems to affect response selection functions during action cascading and leads to slowing of responses when two actions are executed in succession. These changes are related to the anterior cingulate cortex (ACC). Changes in response selection were predictable on the basis of individual salivary cortisol levels. The results show that stress does not affect the whole processing cascade involved in the cascading of different actions, but seems to exert circumscribed effects on response selection processes which have previously been shown to depend on dopaminergic neural transmission.

Enhanced striatal sensitivity to aversive reinforcement in adolescents versus adults

Neurodevelopmental changes in mesolimbic regions are associated with adolescent risk-taking behavior. Numerous studies have shown exaggerated activation in the striatum in adolescents compared with children and adults during reward processing. However, striatal sensitivity to aversion remains elusive. Given the important role of the striatum in tracking both appetitive and aversive events, addressing this question is critical to understanding adolescent decision-making, as both positive and negative factors contribute to this behavior. In this study, human adult and adolescent participants performed a task in which they received squirts of appetitive or aversive liquid while undergoing fMRI, a novel approach in human adolescents. Compared with adults, adolescents showed greater behavioral and striatal sensitivity to both appetitive and aversive stimuli, an effect that was exaggerated in response to delivery of the aversive stimulus. Collectively, these findings contribute to understanding how neural responses to positive and negative outcomes differ between adolescents and adults and how they may influence adolescent behavior.

Neural systems analysis of decision making during goal-directed navigation

The ability to make adaptive decisions during goal-directed navigation is a fundamental and highly evolved behavior that requires continual coordination of perceptions, learning and memory processes, and the planning of behaviors. Here, a neurobiological account for such coordination is provided by integrating current literatures on spatial context analysis and decision-making. This integration includes discussions of our current understanding of the role of the hippocampal system in experience-dependent navigation, how hippocampal information comes to impact midbrain and striatal decision making systems, and finally the role of the striatum in the implementation of behaviors based on recent decisions. These discussions extend across cellular to neural systems levels of analysis. Not only are key findings described, but also fundamental organizing principles within and across neural systems, as well as between neural systems functions and behavior, are emphasized. It is suggested that studying decision making during goal-directed navigation is a powerful model for studying interactive brain systems and their mediation of complex behaviors.

Nucleus accumbens neurons encode predicted and ongoing reward costs in rats

Efficient decision-making requires that animals consider both the benefits and the costs of potential actions, such as the amount of effort or temporal delay involved in reward seeking. The nucleus accumbens (NAc) has been implicated in the ability to choose between options with different costs and overcome high costs when necessary, but it is not clear how NAc processing contributes to this role. Here, neuronal activity in the rat NAc was monitored using multi-neuron electrophysiology during two cost-based decision tasks in which either reward effort or reward delay was manipulated. In each task, distinct visual cues predicted high-value (low effort/immediate) and low-value (high effort/delayed) rewards. After training, animals exhibited a behavioral preference for high-value rewards, yet overcame high costs when necessary to obtain rewards. Electrophysiological analysis indicated that a subgroup of NAc neurons exhibited phasic increases in firing rate during cue presentations. In the effort-based decision task (but not the delay-based task), this population reflected the cost-discounted value of the future response. In contrast, other subgroups of cells were activated during response initiation or reward delivery, but activity did not differ on the basis of reward cost. Finally, another population of cells exhibited sustained changes in firing rate while animals completed high-effort requirements or waited for delayed rewards. These findings are consistent with previous reports that implicate NAc function in reward prediction and behavioral allocation during reward-seeking behavior, and suggest a mechanism by which NAc activity contributes to both cost-based decisions and actual cost expenditure.

The dorsomedial striatum reflects response bias during learning

Previous studies have established that neurons in the dorsomedial striatum track the behavioral significance of external stimuli, are sensitive to contingencies between actions and outcomes, and show rapid flexibility in representing task-related information. Here, we describe how neural activity in the dorsomedial striatum changes during the initial acquisition of a Go/NoGo task and during an initial reversal of stimulus-response contingencies. Rats made nosepoke responses over delay periods and then received one of two acoustic stimuli. Liquid rewards were delivered after one stimulus (S+) if the rats made a Go response (entering a reward port on the opposite wall of the chamber). If a Go response was made to other stimulus (S-), rats experienced a timeout. On 10% of trials, no stimulus was presented. These trials were used to assess response bias, the animals' tendency to collect reward independent of the stimulus. Response bias increased during the reversal, corresponding to the animals' uncertainty about the stimulus-response contingencies. Most task-modulated neurons fired during the response at the end of the delay period. The fraction of response-modulated neurons was correlated with response bias and neural activity was sensitive to the behavioral response made on the previous trial. During initial task acquisition and initial reversal learning, there was a remarkable change in the percentages of neurons that fired in relation to the task events, especially during withdrawal from the nosepoke aperture. These results suggest that changes in task-related activity in the dorsomedial striatum during learning are driven by the animal's bias to collect rewards.

Beyond the Reward Pathway: Coding Reward Magnitude and Error in the Rat Subthalamic Nucleus

It was recently shown that subthalamic nucleus (STN) lesions affect motivation for food, cocaine, and alcohol, differentially, according to either the nature of the reward or the preference for it. The STN may thus code a reward according to its value. Here, we investigated how the firing of subthalamic neurons is modulated during expectation of a predicted reward between two possibilities (4 or 32% sucrose solution). The firing pattern of neurons responding to predictive cues and to reward delivery indicates that STN neurons can be divided into subpopulations responding specifically to one reward and less or giving no response to the other. In addition, some neurons (“oops” neurons) specifically encode errors as they respond only during error trials. These results reveal that the STN plays a critical role in ascertaining the value of the reward and seems to encode that value differently depending on the magnitude of the reward. These data highlight the importance of the STN in the reward circuitry of the brain.

Validation of decision-making models and analysis of decision variables in the rat basal ganglia

Reinforcement learning theory plays a key role in understanding the behavioral and neural mechanisms of choice behavior in animals and humans. Especially, intermediate variables of learning models estimated from behavioral data, such as the expectation of reward for each candidate choice (action value), have been used in searches for the neural correlates of computational elements in learning and decision making. The aims of the present study are as follows: (1) to test which computational model best captures the choice learning process in animals and (2) to elucidate how action values are represented in different parts of the corticobasal ganglia circuit. We compared different behavioral learning algorithms to predict the choice sequences generated by rats during a free-choice task and analyzed associated neural activity in the nucleus accumbens (NAc) and ventral pallidum (VP). The major findings of this study were as follows: (1) modified versions of an action-value learning model captured a variety of choice strategies of rats, including win-stay-lose-switch and persevering behavior, and predicted rats' choice sequences better than the best multistep Markov model; and (2) information about action values and future actions was coded in both the NAc and VP, but was less dominant than information about trial types, selected actions, and reward outcome. The results of our model-based analysis suggest that the primary role of the NAc and VP is to monitor information important for updating choice behaviors. Information represented in the NAc and VP might contribute to a choice mechanism that is situated elsewhere.

Organization of interneuronal connections in the nucleus accumbens in "impulsive" and "self-controlled" behavior in cats

In behavioral experiments, cats placed in a situation of choosing between a high-value time-delayed and a low-value rapid food reinforcement elected to wait for the preferred reward (they demonstrated "self-control") or to obtain the worse reward quickly (they demonstrated impulsive behavior). On the basis of the selected behavioral strategy, the cats were divided into three groups - "impulsive," "ambivalent," and "self-controlled." Cross-correlation analysis was used to assess the linked activity of cells in the nucleus accumbens, which reflects the nature of interactions between close-lying neurons. In cats with self-control, interneuronal interactions appeared in a significantly larger proportion of cases than in impulsive cats. In combinations resulting in long-latency reactions, cats with self-controlled and impulsive behavior showed no significant difference in the occurrence frequency of interneuronal interactions. The numbers of interneuronal interactions were greater during erroneous responses as compared with correctly performed reactions in animals of the different groups. These data indicate a key role for the interrelated activity of nucleus accumbens neurons in organizing the pattern of long-latency responses typical of self-controlled behavior.

Anticipatory reward signals in ventral striatal neurons of behaving rats

It has been proposed that the striatum plays a crucial role in learning to select appropriate actions, optimizing rewards according to the principles of 'Actor-Critic' models of trial-and-error learning. The ventral striatum (VS), as Critic, would employ a temporal difference (TD) learning algorithm to predict rewards and drive dopaminergic neurons. This study examined this model's adequacy for VS responses to multiple rewards in rats. The respective arms of a plus-maze provided rewards of varying magnitudes; multiple rewards were provided at 1-s intervals while the rat stood still. Neurons discharged phasically prior to each reward, during both initial approach and immobile waiting, demonstrating that this signal is predictive and not simply motor-related. In different neurons, responses could be greater for early, middle or late droplets in the sequence. Strikingly, this activity often reappeared after the final reward, as if in anticipation of yet another. In contrast, previous TD learning models show decremental reward-prediction profiles during reward consumption due to a temporal-order signal introduced to reproduce accurate timing in dopaminergic reward-prediction error signals. To resolve this inconsistency in a biologically plausible manner, we adapted the TD learning model such that input information is nonhomogeneously distributed among different neurons. By suppressing reward temporal-order signals and varying richness of spatial and visual input information, the model reproduced the experimental data. This validates the feasibility of a TD-learning architecture where different groups of neurons participate in solving the task based on varied input information.

The bivalent side of the nucleus accumbens

An increasing body of evidence suggests that the nucleus accumbens (NAcc) is engaged in both incentive reward processes and in adaptive responses to conditioned and unconditioned aversive stimuli. Yet, it has been argued that NAcc activation to aversive stimuli may be a consequence of the rewarding effects of their termination, i.e., relief. To address this question we used fMRI to delineate brain response to the onset and offset of unpleasant and pleasant auditory stimuli in the absence of learning or motor response. Increased NAcc activity was seen for the onset of both pleasant and unpleasant stimuli. Our results support the expanded bivalent view of NAcc function and call for expansion of current models of NAcc function that are solely focused on reward.

A unified framework for addiction: vulnerabilities in the decision process

The understanding of decision-making systems has come together in recent years to form a unified theory of decision-making in the mammalian brain as arising from multiple, interacting systems (a planning system, a habit system, and a situation-recognition system). This unified decision-making system has multiple potential access points through which it can be driven to make maladaptive choices, particularly choices that entail seeking of certain drugs or behaviors. We identify 10 key vulnerabilities in the system: (1) moving away from homeostasis, (2) changing allostatic set points, (3) euphorigenic "reward-like" signals, (4) overvaluation in the planning system, (5) incorrect search of situation-action-outcome relationships, (6) misclassification of situations, (7) overvaluation in the habit system, (8) a mismatch in the balance of the two decision systems, (9) over-fast discounting processes, and (10) changed learning rates. These vulnerabilities provide a taxonomy of potential problems with decision-making systems. Although each vulnerability can drive an agent to return to the addictive choice, each vulnerability also implies a characteristic symptomology. Different drugs, different behaviors, and different individuals are likely to access different vulnerabilities. This has implications for an individual's susceptibility to addiction and the transition to addiction, for the potential for relapse, and for the potential for treatment.

Cue-evoked encoding of movement planning and execution in the rat nucleus accumbens

The nucleus accumbens is involved in the modulation of motivated behaviour by reward-associated sensory information. However, little is known about the specific nature of the nucleus accumbens' contribution to generating movement. We investigated motor encoding by nucleus accumbens neurons in rats performing a delayed response task that allowed us to dissociate the effects of sensory and motor events on firing. In a subset of neurons, firing in the delay period preceding movement was highly selective; this selectivity was tightly correlated with the direction of the subsequent movement, but not with the sensory properties of the instructive cue. Direction selectivity in this population of neurons developed over the course of the delay period, with the strongest selectivity apparent just prior to movement onset. Selectivity was also apparent in nucleus accumbens neurons during movement, such that firing showed a tight correlation with movement direction, but not the instructive cue presented nor the spatial destination of the movement. These results are consistent with the hypothesis that a subpopulation of nucleus accumbens neurons contributes to the selection and execution of specific motivated behaviours.

Differential encoding of losses and gains in the human striatum

Studies on human monetary prediction and decision making emphasize the role of the striatum in encoding prediction errors for financial reward. However, less is known about how the brain encodes financial loss. Using Pavlovian conditioning of visual cues to outcomes that simultaneously incorporate the chance of financial reward and loss, we show that striatal activation reflects positively signed prediction errors for both. Furthermore, we show functional segregation within the striatum, with more anterior regions showing relative selectivity for rewards and more posterior regions for losses. These findings mirror the anteroposterior valence-specific gradient reported in rodents and endorse the role of the striatum in aversive motivational learning about financial losses, illustrating functional and anatomical consistencies with primary aversive outcomes such as pain.

The nucleus accumbens as part of a basal ganglia action selection circuit

BACKGROUND

The nucleus accumbens is the ventral extent of the striatum, the main input nucleus of the basal ganglia. Recent hypotheses propose that the accumbens and its dopamine projection from the midbrain contribute to appetitive behaviors required to obtain reward. However, the specific nature of this contribution is unclear. In contrast, significant advances have been made in understanding the role of the dorsal striatum in action selection and decision making.

OBJECTIVE

In order to develop a hypothesis of the role of nucleus accumbens dopamine in action selection, the physiology and behavioral pharmacology of the nucleus accumbens are compared to those of the dorsal striatum.

HYPOTHESES

Three hypotheses concerning the role of dopamine in these structures are proposed: (1) that dopamine release in the dorsal striatum serves to facilitate the ability to respond appropriately to temporally predictable stimuli (that is, stimuli that are so predictable that animals engage in anticipatory behavior just prior to the stimulus); (2) that dopamine in the nucleus accumbens facilitates the ability to respond to temporally unpredictable stimuli (which require interruption of ongoing behavior); and (3) that accumbens neurons participate in action selection in response to such stimuli by virtue of their direct (monosynaptic inhibitory) and indirect (polysynaptic excitatory) projections to basal ganglia output nuclei.

Rat Nucleus Accumbens Neurons Persistently Encode Locations Associated With Morphine Reward

When rats and mice are free to explore a familiar environment they spend more time in a previously rewarded location. This conditioned place preference (CPP) results from an increased probability of initiating transitions from an unrewarded location to one previously paired with reward. We recorded nucleus accumbens (NAc) neurons while rats explored a three-room in-line apparatus. Before place conditioning, approximately equal proportions of NAc neurons show excitations or inhibitions when the rat is in each of the rooms (morphine paired, center or saline paired). Conditioning increased the proportion of neurons inhibited while the rat was in the morphine room and neurons excited in the saline room. Many of the neurons in these two groups responded during room transitions. Furthermore, the postconditioning increase in the population of neurons with room-selective responding persisted for several weeks after the last morphine treatment. This long-lasting change in population responses of NAc neurons to initially neutral locations is a neural correlate of the change in location preference manifest as CPP.

L-DOPA disrupts activity in the nucleus accumbens during reversal learning in Parkinson's disease

Evidence indicates that dopaminergic medication in Parkinson's disease may impair certain aspects of cognitive function, such as reversal learning. We used functional magnetic resonance imaging in patients with mild Parkinson's disease to investigate the neural site at which L-DOPA acts during reversal learning. Patients were scanned both ON and OFF their normal dopamine-enhancing L-DOPA medication during the performance of a probabilistic reversal learning task. We demonstrate that L-DOPA modulated reversal-related activity in the nucleus accumbens, but not in the dorsal striatum or the prefrontal cortex. These data concur with evidence from studies with experimental animals and indicate an important role for the human nucleus accumbens in the dopaminergic modulation of reversal learning.

Firing Patterns of Accumbal Neurons During a Pavlovian-Conditioned Approach Task

The nucleus accumbens (NAc) is necessary for the expression of Pavlovian-conditioned approach behavior but not for the expression of instrumental behavior conditioned in sessions that set a low response requirement. Although numerous studies have characterized firing patterns of NAc neurons in relation to instrumental behavior, very little is known about how NAc neurons encode information in Pavlovian tasks. In the present study, recordings of accumbal firing patterns were made during sessions in which rats performed a Pavlovian-conditioned approach task. Most of the recorded neurons (74/83, 89%) exhibited significant responses during the conditioned stimulus (CS) presentation and/or the reward exposure. The reward responses were prevalent, predominantly inhibitory, and comparable to reward responses observed in various types of behavioral paradigms, including instrumental tasks. The CS responses could be segregated into multiple subtypes on the basis of directionality, onset latency, and duration. Several characteristics of the CS firing patterns were unique relative to cue responses observed previously during alternative types of conditioning sessions. It is possible that the novel firing patterns correspond to the differential contributions of the accumbens to Pavlovian-conditioned approach behavior and instrumentally conditioned behavior. Regardless, the novel patterns of firing add to existing evidence that characterization of accumbal firing patterns in Pavlovian tasks may provide additional information about the neurophysiological mechanisms that mediate accumbal contributions to behavior.

Neurons in dopamine-rich areas of the rat medial midbrain predominantly encode the outcome-related rather than behavioural switching properties of conditioned stimuli

Midbrain dopamine neurons are phasically activated by a variety of sensory stimuli. It has been hypothesized that these activations contribute to reward prediction or behavioural switching. To test the latter hypothesis we recorded from 131 single neurons in the ventral tegmental area and retrorubral field of thirsty rats responding during a modified go/no-go task. One-quarter (n = 33) of these neurons responded to conditioned stimuli in the task, which varied according to the outcome with which they were associated (saccharin or quinine solution) and according to whether they triggered a switch in the ongoing sequence of the animal's behaviour ('behavioural switching'). Almost half the neurons (45%) responded differentially to saccharin- vs. quinine-conditioned stimuli; the activity of a minority (15%) correlated with an aspect of behavioural switching (mostly exhibiting changes from baseline activity in the absence of a behavioural switch) and one-third (33%) encoded various outcome-switch combinations. The strongest response was excitation to the saccharin-conditioned stimulus. Additionally, a proportion (38%) of neurons responded during outcome delivery, typically exhibiting inhibition during saccharin consumption. The neurons sampled did not fall into distinct clusters on the basis of their electrophysiological characteristics. However, most neurons that responded to the outcome-related properties of conditioned stimuli had long action potentials (> 1.2 ms), a reported characteristic of dopamine neurons. Moreover, responses to saccharin-conditioned stimuli were functionally akin to dopamine responses found in the macaque and rat nucleus accumbens responses observed within the same task. In conclusion, our data are more consistent with the reward-prediction than the behavioural switching hypothesis.