Differential role of the hippocampus in response-outcome and context-outcome learning: Evidence from selective satiation procedures

A hippocampo-cortical pathway detects changes in the validity of an action as a predictor of reward

Much research has been dedicated to understanding the psychological and neural bases of goal-directed action, yet the relationship between context and goal-directed action is not well understood. Here, we used excitotoxic lesions, chemogenetics, and circuit-specific manipulations to demonstrate the role of the ventral hippocampus (vHPC) in contextual learning that supports sensitivity to action-outcome contingencies, a hallmark of goal-directed action. We found that chemogenetic inhibition of the ventral, but not dorsal, hippocampus attenuated sensitivity to instrumental contingency degradation. We then tested the hypothesis that this deficit was due to an inability to discern the relative validity of the action compared with the context as a predictor of reward. Using latent inhibition and Pavlovian context conditioning, we confirm that degradation of action-outcome contingencies relies on intact context-outcome learning and show that this learning is dependent on vHPC. Finally, we show that chemogenetic inhibition of vHPC terminals in the medial prefrontal cortex also impairs both instrumental contingency degradation and context-outcome learning. These results implicate a hippocampo-cortical pathway in adapting to changes in instrumental contingencies and indicate that the psychological basis of this deficit is an inability to learn the predictive value of the context. Our findings contribute to a broader understanding of the neural bases of goal-directed action and its contextual regulation.

Enhancing reinforcement learning models by including direct and indirect pathways improves performance on striatal dependent tasks

A major advance in understanding learning behavior stems from experiments showing that reward learning requires dopamine inputs to striatal neurons and arises from synaptic plasticity of cortico-striatal synapses. Numerous reinforcement learning models mimic this dopamine-dependent synaptic plasticity by using the reward prediction error, which resembles dopamine neuron firing, to learn the best action in response to a set of cues. Though these models can explain many facets of behavior, reproducing some types of goal-directed behavior, such as renewal and reversal, require additional model components. Here we present a reinforcement learning model, TD2Q, which better corresponds to the basal ganglia with two Q matrices, one representing direct pathway neurons (G) and another representing indirect pathway neurons (N). Unlike previous two-Q architectures, a novel and critical aspect of TD2Q is to update the G and N matrices utilizing the temporal difference reward prediction error. A best action is selected for N and G using a softmax with a reward-dependent adaptive exploration parameter, and then differences are resolved using a second selection step applied to the two action probabilities. The model is tested on a range of multi-step tasks including extinction, renewal, discrimination; switching reward probability learning; and sequence learning. Simulations show that TD2Q produces behaviors similar to rodents in choice and sequence learning tasks, and that use of the temporal difference reward prediction error is required to learn multi-step tasks. Blocking the update rule on the N matrix blocks discrimination learning, as observed experimentally. Performance in the sequence learning task is dramatically improved with two matrices. These results suggest that including additional aspects of basal ganglia physiology can improve the performance of reinforcement learning models, better reproduce animal behaviors, and provide insight as to the role of direct- and indirect-pathway striatal neurons.

Over the river, through the woods: cognitive maps in the hippocampus and orbitofrontal cortex

The hippocampus and the orbitofrontal cortex (OFC) both have important roles in cognitive processes such as learning, memory and decision making. Nevertheless, research on the OFC and hippocampus has proceeded largely independently, and little consideration has been given to the importance of interactions between these structures. Here, evidence is reviewed that the hippocampus and OFC encode parallel, but interactive, cognitive 'maps' that capture complex relationships between cues, actions, outcomes and other features of the environment. A better understanding of the interactions between the OFC and hippocampus is important for understanding the neural bases of flexible, goal-directed decision making.

Selective lesions of the cholinergic neurons within the posterior pedunculopontine do not alter operant learning or nicotine sensitization

Cholinergic neurons within the pedunculopontine tegmental nucleus have been implicated in a range of functions, including behavioral state control, attention, and modulation of midbrain and basal ganglia systems. Previous experiments with excitotoxic lesions have found persistent learning impairment and altered response to nicotine following lesion of the posterior component of the PPTg (pPPTg). These effects have been attributed to disrupted input to midbrain dopamine systems, particularly the ventral tegmental area. The pPPTg contains a dense collection of cholinergic neurons and also large numbers of glutamatergic and GABAergic neurons. Because these interdigitated populations of neurons are all susceptible to excitotoxins, the effects of such lesions cannot be attributed to one neuronal population. We wished to assess whether the learning impairments and altered responses to nicotine in excitotoxic PPTg-lesioned rats were due to loss of cholinergic neurons within the pPPTg. Selective depletion of cholinergic pPPTg neurons is achievable with the fusion toxin Dtx-UII, which targets UII receptors expressed only by cholinergic neurons in this region. Rats bearing bilateral lesions of cholinergic pPPTg neurons (>90% ChAT+ neuronal loss) displayed no deficits in the learning or performance of fixed and variable ratio schedules of reinforcement for pellet reward. Separate rats with the same lesions had a normal locomotor response to nicotine and furthermore sensitized to repeated administration of nicotine at the same rate as sham controls. Previously seen changes in these behaviors following excitotoxic pPPTg lesions cannot be attributed solely to loss of cholinergic neurons. These findings indicate that non-cholinergic neurons within the pPPTg are responsible for the learning deficits and altered responses to nicotine seen after excitotoxic lesions. The functions of cholinergic neurons may be related to behavioral state control and attention rather than learning.

Cafeteria diet impairs expression of sensory-specific satiety and stimulus-outcome learning

A range of animal and human data demonstrates that excessive consumption of palatable food leads to neuroadaptive responses in brain circuits underlying reward. Unrestrained consumption of palatable food has been shown to increase the reinforcing value of food and weaken inhibitory control; however, whether it impacts upon the sensory representations of palatable solutions has not been formally tested. These experiments sought to determine whether exposure to a cafeteria diet consisting of palatable high fat foods impacts upon the ability of rats to learn about food-associated cues and the sensory properties of ingested foods. We found that rats fed a cafeteria diet for 2 weeks were impaired in the control of Pavlovian responding in accordance to the incentive value of palatable outcomes associated with auditory cues following devaluation by sensory-specific satiety. Sensory-specific satiety is one mechanism by which a diet containing different foods increases ingestion relative to one lacking variety. Hence, choosing to consume greater quantities of a range of foods may contribute to the current prevalence of obesity. We observed that rats fed a cafeteria diet for 2 weeks showed impaired sensory-specific satiety following consumption of a high calorie solution. The deficit in expression of sensory-specific satiety was also present 1 week following the withdrawal of cafeteria foods. Thus, exposure to obesogenic diets may impact upon neurocircuitry involved in motivated control of behavior.

Interoceptive conditioning with a nicotine stimulus is susceptible to reinforcer devaluation

Pavlovian conditioning processes contribute to the etiology of nicotine dependence. Conditioning involving interoceptive stimuli is increasingly recognized as playing a role in many diseases and psychopathologies, including drug addiction. Previous animal research on diminishing the influence of interoceptive conditioning has been limited to antagonism and nonreinforced exposures to the drug stimulus. The goal of the present research was to determine whether interoceptive conditioning with a nicotine stimulus could be diminished through an unconditioned stimulus (US) devaluation procedure. In two separate experiments, male Sprague-Dawley rats received nicotine injections (0.4 mg base/kg) followed by intermittent sucrose (26%) access in a conditioning chamber. On intermixed saline sessions, sucrose was withheld. Conditioning was demonstrated by a reliable increase in head entries in the dipper receptacle on nicotine versus saline sessions. After conditioning, rats in a devaluation condition were given access to sucrose in their home cages immediately followed by a lithium chloride (LiCl) injection on 3 consecutive days. On subsequent test days, nicotine-evoked conditioned responding was significantly attenuated. Within-subject (Experiment 1) and between-subjects (Experiment 2) controls revealed that the diminished responding was not attributable to mere exposure to the sucrose US in the devaluation phase. Experiment 2 included a LiCl-alone control group. Repeated illness induced by LiCl did not reduce later nicotine-evoked responding. These findings suggest that there is a direct association between the interoceptive stimulus effects of nicotine and the appetitive sucrose US (i.e., stimulus-stimulus) rather than a stimulus-response association.

Impaired formation of stimulus-response, but not action-outcome, associations in rats with methamphetamine-induced neurotoxicity

Methamphetamine (METH) induces neurotoxic changes, including partial striatal dopamine depletions, which are thought to contribute to cognitive dysfunction in rodents and humans. The dorsal striatum is implicated in action-outcome (A-O) and stimulus-response (S-R) associations underlying instrumental learning. Thus, the present study examined the long-term consequences of METH-induced neurotoxicity on A-O and S-R associations underlying appetitive instrumental behavior. Rats were pretreated with saline or a neurotoxic regimen of METH (4 × 7.5-10 mg/kg). Rats trained on random ratio (RR) or random interval (RI) schedules of reinforcement were then subjected to outcome devaluation or contingency degradation, followed by an extinction test. All rats then were killed, and brains removed for determination of striatal dopamine loss. The results show that: (1) METH pretreatment induced a partial 45-50% decrease in striatal dopamine tissue content in dorsomedial and dorsolateral striatum; (2) METH-induced neurotoxicity did not alter acquisition of instrumental behavior on either RR or RI schedules; (3) outcome devaluation and contingency degradation similarly decreased responding in saline- and METH-pretreated rats trained on the RR schedule, suggesting intact A-O associations guiding behavior; (4) outcome devaluation after training on the RI schedule decreased extinction responding only in METH-pretreated rats, suggesting impaired S-R associations. Overall, these data suggest that METH-induced neurotoxicity, possibly due to impairment of the function of dorsolateral striatal circuitry, may decrease cognitive flexibility by impairing the ability to automatize behavioral patterns.