The response strategy and the place strategy in a plus-maze have different sensitivities to devaluation of expected outcome

Making and breaking habits: Revisiting the definitions and behavioral factors that influence habits in animals

Habits have garnered significant interest in studies of associative learning and maladaptive behavior. However, habit research has faced scrutiny and challenges related to the definitions and methods. Differences in the conceptualizations of habits between animal and human studies create difficulties for translational research. Here, we review the definitions and commonly used methods for studying habits in animals and humans and discuss potential alternative ways to assess habits, such as automaticity. To better understand habits, we then focus on the behavioral factors that have been shown to make or break habits in animals, as well as potential mechanisms underlying the influence of these factors. We discuss the evidence that habitual and goal-directed systems learn in parallel and that they seem to interact in competitive and cooperative manners. Finally, we draw parallels between habitual responding and compulsive drug seeking in animals to delineate the similarities and differences in these behaviors.

NMDA receptor blockade in the dorsolateral striatum impairs consolidation but not retrieval of habit memory

The present study investigated whether N-methyl-d-aspartate (NMDA) receptors in the dorsolateral striatum (DLS) mediate consolidation and retrieval of habit memory. Adult male Long-Evans rats were trained in a response learning version of a water plus-maze task in which rats were reinforced to make a habitual and consistent body-turn response at the maze choice point in order to mount a hidden escape platform. Prior research indicates that acquisition, consolidation, and retrieval in this task requires DLS function. The present study consisted of two experiments. In Experiment 1, rats received intra-DLS post-training injections of the NMDA receptor antagonist 2-amino-5- phosphonopentanoic acid (AP5; 2 µg/side) to examine the role of NMDA receptors in consolidation of habit memory. In Experiment 2, different groups of rats received a single pre-retrieval injection of AP5 in the DLS (AP5; 2 µg/side) during the last day of maze training to examine the potential role of NMDA receptors in retrieval of habit memory. Results indicated that post-training intra-DLS AP5 injections impaired memory consolidation. However, administration of AP5 at the same dose that impaired consolidation had no effect on memory retrieval. The findings are consistent with previous research indicating a role for NMDA receptors in the DLS in memory consolidation, and suggest that NMDA-dependent synaptic activity in the DLS may not be a critical component of habit memory retrieval.

Navigation task and action space drive the emergence of egocentric and allocentric spatial representations

In general, strategies for spatial navigation could employ one of two spatial reference frames: egocentric or allocentric. Notwithstanding intuitive explanations, it remains unclear however under what circumstances one strategy is chosen over another, and how neural representations should be related to the chosen strategy. Here, we first use a deep reinforcement learning model to investigate whether a particular type of navigation strategy arises spontaneously during spatial learning without imposing a bias onto the model. We then examine the spatial representations that emerge in the network to support navigation. To this end, we study two tasks that are ethologically valid for mammals—guidance, where the agent has to navigate to a goal location fixed in allocentric space, and aiming, where the agent navigates to a visible cue. We find that when both navigation strategies are available to the agent, the solutions it develops for guidance and aiming are heavily biased towards the allocentric or the egocentric strategy, respectively, as one might expect. Nevertheless, the agent can learn both tasks using either type of strategy. Furthermore, we find that place-cell-like allocentric representations emerge preferentially in guidance when using an allocentric strategy, whereas egocentric vector representations emerge when using an egocentric strategy in aiming. We thus find that alongside the type of navigational strategy, the nature of the task plays a pivotal role in the type of spatial representations that emerge.

Most species rely on navigation in space to find water, food, and mates, as well as to return home. When navigating, humans and animals can use one of two reference frames: one based on stable landmarks in the external environment, such as moving due north and then east, or one centered on oneself, such as moving forward and turning left. However, it remains unclear how these reference frames are chosen and interact in navigation tasks, as well as how they are supported by representations in the brain. We therefore modeled two navigation tasks that would each benefit from using one of these reference frames, and trained an artificial agent to learn to solve them through trial and error. Our results show that when given the choice, the agent leveraged the appropriate reference frame to solve the task, but surprisingly could also use the other reference frame when constrained to do so. We also show that the representations that emerge to enable the agent to solve the tasks exist on a spectrum, and are more complex than commonly thought. These representations reflect both the task and reference frame being used, and provide useful insights for the design of experimental tasks to study the use of navigational strategies.

Hippocampus-sensitive and striatum-sensitive learning one month after morphine or cocaine exposure in male rats

These experiments examined whether morphine and cocaine alter the balance between hippocampal and striatal memory systems measured long after drug exposure. Male rats received injections of morphine (5 mg/kg), cocaine (20 mg/kg), or saline for five consecutive days. One month later, rats were trained to find food on a hippocampus-sensitive place task or a striatum-sensitive response task. Relative to saline controls, morphine-treated rats exhibited impaired place learning but enhanced response learning; prior cocaine exposure did not significantly alter learning on either task. Another set of rats was trained on a dual-solution T-maze that can be solved with either place or response strategies. While a majority (67%) of control rats used place solutions, morphine treatment one month prior resulted in the exclusive use of response solutions (100%). Prior cocaine treatment did not significantly alter strategy selection. Molecular markers related to learning and drug abuse were measured in the hippocampus and striatum one month after drug exposure in behaviorally untested rats. Protein levels of glial-fibrillary acidic protein (GFAP), an intermediate filament specific to astrocytes, increased significantly in the hippocampus after morphine exposure, but not after cocaine exposure. Exposure to morphine or cocaine did not significantly change levels of brain-derived neurotrophic factor (BDNF) or a downstream target of BDNF signaling, glycogen synthase kinase 3β (GSK3β), in the hippocampus or striatum. Thus, exposure to morphine resulted in a long-lasting shift from hippocampal toward striatal dominance during learning, an effect that may be associated with lasting alterations in hippocampal astrocytes. Cocaine produced changes in the same direction, suggesting that use of a higher dose or longer duration of exposure might produce effects comparable to those seen with morphine.

Do spatial and recognition memories have a lateralized processing by the dorsal hippocampus CA3?

The present study evaluated the function of the right and left CA3 of the dorsal hippocampus (dHPC) in the processing of (i) recognition memory, (ii) recent and remote spatial memory, (iii) working memory and (iv) navigation strategy. Wistar rats were divided into four experimental groups: vehicle group (VG), animals received a bilateral injection of phosphate-saline buffer (PBS) in both right and left dorsal CA3; dHPC-R group, animals received an injection of ibotenic acid (IBO) in the right dorsal CA3; dHPC-L group, animals received an IBO injection in left dorsal CA3; and dHPC-Bi group, animals received bilateral injections of IBO in both dorsal CA3. Rats were submitted to a sequence of behavioral tests: Morris water maze (MWM), object recognition test (ORT), forced T-maze and MWM 30 days after the first exposure. The results showed no evidence of functional lateralization and the dorsal CA3 does not seem to be essential for learning and memory (recent and remote) processing and allocentric navigation analyzed in the MWM and T-maze, respectively. However, rats with right or bilateral lesions in the dorsal CA3 failed to recognize the familiar object in the ORT, suggesting a lateralized processing of recognition memory. That result is unprecedented and contributes to the knowledge about the compartmentalization of HPC functions.

The effects of spatial stability and cue type on spatial learning: Implications for theories of parallel memory systems

Some theories of spatial learning predict that associative rules apply under only limited circumstances. For example, learning based on a boundary has been claimed to be immune to cue competition effects because boundary information is the basis for the formation of a cognitive map, whilst landmark learning does not involve cognitive mapping. This is referred to as the cue type hypothesis. However, it has also been claimed that cue stability is a prerequisite for the formation of a cognitive map, meaning that whichever cue type was perceived as stable would enter a cognitive map and thus be immune to cue competition, while unstable cues will be subject to cue competition, regardless of cue type. In experiments 1 and 2 we manipulated the stability of boundary and landmark cues when learning the location of two hidden goals. One goal location was constant with respect to the boundary, and the other constant with respect to the landmark cues. For both cue types, the presence of distal orientation cues provided directional information. For half the participants the landmark cues were unstable relative to the boundary and orientation cues, whereas for the remainder of the participants the boundary was unstable relative to landmarks and orientation cues. In a second stage of training, all cues remained stable so that both goal locations could be learned with respect to both landmark and boundary information. According to the cue type hypothesis, boundary information should block learning about landmarks regardless of cue stability. According to the cue stability hypothesis, however, landmarks should block learning about the boundary when the landmarks appear stable relative to the boundary. Regardless of cue type or stability the results showed reciprocal blocking, contrary to both formulations of incidental cognitive mapping. Experiment 3 established that the results of Experiments 1 and 2 could not be explained in terms of difficulty in learning certain locations with respect to different cue types. In a final experiment, following training in which both landmarks and boundary cues signalled two goal locations, a new goal location was established with respect to the landmark cues, before testing with the boundary, which had never been used to define the new goal location. The results of this novel test of the interaction between boundary and landmark cues indicated that new learning with respect to the landmark had a profound effect on navigation with respect to the boundary, counter to the predictions of incidental cognitive mapping of boundaries.

Two-dimensional reward evaluation in mice

When choosing among multi-attribute options, integrating the full information may be computationally costly and time-consuming. So-called non-compensatory decision rules only rely on partial information, for example when a difference on a single attribute overrides all others. Such rules may be ecologically more advantageous, despite being economically suboptimal. Here, we present a study that investigates to what extent animals rely on integrative rules (using the full information) versus non-compensatory rules when choosing where to forage. Groups of mice were trained to obtain water from dispensers varying along two reward dimensions: volume and probability. The mice’s choices over the course of the experiment suggested an initial reliance on integrative rules, later displaced by a sequential rule, in which volume was evaluated before probability. Our results also demonstrate that while the evaluation of probability differences may depend on the reward volumes, the evaluation of volume differences is seemingly unaffected by the reward probabilities.

Place vs. Response Learning: History, Controversy, and Neurobiology

The present article provides a historical review of the place and response learning plus-maze tasks with a focus on the behavioral and neurobiological findings. The article begins by reviewing the conflict between Edward C. Tolman's cognitive view and Clark L. Hull's stimulus-response (S-R) view of learning and how the place and response learning plus-maze tasks were designed to resolve this debate. Cognitive learning theorists predicted that place learning would be acquired faster than response learning, indicating the dominance of cognitive learning, whereas S-R learning theorists predicted that response learning would be acquired faster, indicating the dominance of S-R learning. Here, the evidence is reviewed demonstrating that either place or response learning may be dominant in a given learning situation and that the relative dominance of place and response learning depends on various parametric factors (i.e., amount of training, visual aspects of the learning environment, emotional arousal, et cetera). Next, the neurobiology underlying place and response learning is reviewed, providing strong evidence for the existence of multiple memory systems in the mammalian brain. Research has indicated that place learning is principally mediated by the hippocampus, whereas response learning is mediated by the dorsolateral striatum. Other brain regions implicated in place and response learning are also discussed in this section, including the dorsomedial striatum, amygdala, and medial prefrontal cortex. An exhaustive review of the neurotransmitter systems underlying place and response learning is subsequently provided, indicating important roles for glutamate, dopamine, acetylcholine, cannabinoids, and estrogen. Closing remarks are made emphasizing the historical importance of the place and response learning tasks in resolving problems in learning theory, as well as for examining the behavioral and neurobiological mechanisms of multiple memory systems. How the place and response learning tasks may be employed in the future for examining extinction, neural circuits of memory, and human psychopathology is also briefly considered.

A general model of hippocampal and dorsal striatal learning and decision making

Humans and other animals use multiple strategies for making decisions. Reinforcement-learning theory distinguishes between stimulus-response (model-free; MF) learning and deliberative (model-based; MB) planning. The spatial-navigation literature presents a parallel dichotomy between navigation strategies. In "response learning," associated with the dorsolateral striatum (DLS), decisions are anchored to an egocentric reference frame. In "place learning," associated with the hippocampus, decisions are anchored to an allocentric reference frame. Emerging evidence suggests that the contribution of hippocampus to place learning may also underlie its contribution to MB learning by representing relational structure in a cognitive map. Here, we introduce a computational model in which hippocampus subserves place and MB learning by learning a "successor representation" of relational structure between states; DLS implements model-free response learning by learning associations between actions and egocentric representations of landmarks; and action values from either system are weighted by the reliability of its predictions. We show that this model reproduces a range of seemingly disparate behavioral findings in spatial and nonspatial decision tasks and explains the effects of lesions to DLS and hippocampus on these tasks. Furthermore, modeling place cells as driven by boundaries explains the observation that, unlike navigation guided by landmarks, navigation guided by boundaries is robust to "blocking" by prior state-reward associations due to learned associations between place cells. Our model, originally shaped by detailed constraints in the spatial literature, successfully characterizes the hippocampal-striatal system as a general system for decision making via adaptive combination of stimulus-response learning and the use of a cognitive map.

Complementary Control over Habits and Behavioral Vigor by Phasic Activity in the Dorsolateral Striatum

Despite clear evidence linking the basal ganglia to the control of outcome insensitivity (i.e., habit) and behavioral vigor (i.e., its behavioral speed/fluidity), it remains unclear whether or how these functions relate to one another. Here, using male Long-Evans rats in response-based and cue-based maze-running tasks, we demonstrate that phasic dorsolateral striatum (DLS) activity occurring at the onset of a learned behavior regulates how vigorous and habitual it is. In a response-based task, brief optogenetic excitation at the onset of runs decreased run duration and the occurrence of deliberative behaviors, whereas midrun stimulation carried little effect. Outcome devaluation showed these runs to be habitual. DLS inhibition at run start did not produce robust effects on behavior until after outcome devaluation. At that time, when the DLS was plausibly most critically required for performance (i.e., habitual), inhibition reduced performance vigor measures and caused a dramatic loss of habitual responding (i.e., animals quit the task). In a second cue-based "beacon" task requiring behavior initiation at the start of the run and again in the middle of the run, DLS excitation at both time points could improve the vigor of runs. Postdevaluation testing showed behavior on the beacon task to be habitual as well. This pattern of results suggests that one role for phasic DLS activity at behavior initiation is to promote the execution of the behavior in a vigorous and habitual fashion by a diverse set of measures.SIGNIFICANCE STATEMENT Our research expands the literature twofold. First, we find that features of a habitual behavior that are typically studied separately (i.e., maze response performance, deliberation movements, running vigor, and outcome insensitivity) are quite closely linked together. Second, efforts have been made to understand "what" the dorsolateral striatum (DLS) does for habitual behavior, and our research provides a key set of results showing "when" it is important (i.e., at behavior initiation). By showing such dramatic control over habits by DLS activity in a phasic time window, plausible real-world applications could involve more informed DLS perturbations to curb intractably problematic habits.

Hippocampal Contributions to Model-Based Planning and Spatial Memory

Little is known about the neural mechanisms that allow humans and animals to plan actions using knowledge of task contingencies. Emerging theories hypothesize that it involves the same hippocampal mechanisms that support self-localization and memory for locations. Yet limited direct evidence supports the link between planning and the hippocampal place map. We addressed this by investigating model-based planning and place memory in healthy controls and epilepsy patients treated using unilateral anterior temporal lobectomy with hippocampal resection. Both functions were impaired in the patient group. Specifically, the planning impairment was related to right hippocampal lesion size, controlling for overall lesion size. Furthermore, although planning and boundary-driven place memory covaried in the control group, this relationship was attenuated in patients, consistent with both functions relying on the same structure in the healthy brain. These findings clarify both the neural mechanism of model-based planning and the scope of hippocampal contributions to behavior.

Reduced sensitivity to devaluation for instrumental but not consummatory behaviors in binge eating prone rats

Binge eating is characterized by the consumption of a large amount of palatable food in a short period of time and is a core feature of many eating disorders. Patients with eating disorders are also known to display impairments in inhibitory control, cognition and decision-making, which may promote and maintain binge eating symptomology. In the current study, we examined whether rats that were subsequently characterized as displaying a higher propensity to binge eat would show pre-existing deficits in reinforcer devaluation-a paradigm used to examine decision-making following reductions in the value of a food reinforcer. Female rats were first trained to respond on two levers for the delivery of two food reinforcers (sucrose and maltodextrin solutions). At the test stage, rats were provided 1 h access to one of the two reinforcers to allow for devaluation via sensory specific satiety, immediately followed by an extinction test with both levers. Normal rats typically show reductions in responding on the lever associated with the devalued reinforcer (i.e., intact goal-directed responding). Subsequently, we used intermittent access to palatable food to identify high (BE prone [BEP]; n = 14), intermediate (BE neutral [BEN]; n = 48), and low (BE resistant [BER]; n = 13) phenotypes of binge eating. Prior reinforcer devaluation performance showed BEN and BER rats suppressed responding on the lever associated with the devalued reinforcer while BEP rats did not. This insensitivity to instrumental reinforcer devaluation in BEP rats did not reflect impaired sensory-specific satiety as during a food choice test, BEP rats showed a more robust alteration in food preferences following devaluation. Additionally, across all rats sensory specific satiety was correlated with subsequent intake of palatable food. Collectively, these findings suggest dissociable effects of devaluation procedures on instrumental actions and consummatory behaviors in BEP rats, and may indicate that pre-existing differences in goal-directed behavior and sensory-specific satiety contribute to the propensity to overeat palatable food.

Hippocampal proBDNF facilitates place learning strategy associated with neural activity in rats

Mature brain-derived neurotrophic factor has been shown to have a promotive effect on synaptic plasticity and spatial memory. The precursor of BDNF (proBDNF) has emerged as a protein against its mature form. However, it is unknown whether and how proBDNF regulates neural excitability and spatial behavior. Through infusion of cleavage-resistant proBDNF or its antibody into HPC, we sought evidence for the influences by employing multiple behavioral tests and recording hippocampal single-unit activity. Our behavioral findings showed that proBDNF induced beneficial effects on spatial learning by facilitating the use of the place strategy and inhibiting the response strategy, including (1) using more place search strategies but less response strategies, and (2) increasing the number of rats in choosing place strategies but not response strategies. Intriguingly, infusion of an anti-proBDNF antibody did not affect rats' training process but rendered the adaption to learning reversal training more difficult, indicating deficits in choosing the proper learning strategy. The training-induced increase in proBDNF promoted the firing rate of pyramidal neurons but not fast-spiking (FS) interneurons. Importantly, endogenous proBDNF facilitated the neural correlate of spatial, but not response, learning behavior. However, the anti-proBDNF antibody effectively reversed the strategy preference and inhibited neural activity. We herein propose that proBDNF exerts pivotal effects on neural excitability and the use of cognitive strategies to facilitate the spatial learning process.

The response strategy and the place strategy in a plus-maze have different sensitivities to devaluation of expected outcome

Previous studies have suggested that spatial navigation can be achieved with at least two distinct learning processes, involving either cognitive map‐like representations of the local environment, referred to as the “place strategy”, or simple stimulus‐response (S‐R) associations, the “response strategy”. A similar distinction between cognitive/behavioral processes has been made in the context of non‐spatial, instrumental conditioning, with the definition of two processes concerning the sensitivity of a given behavior to the expected value of its outcome as well as to the response‐outcome contingency (“goal‐directed action” and “S‐R habit”). Here we investigated whether these two versions of dichotomist definitions of learned behavior, one spatial and the other non‐spatial, correspond to each other in a formal way. Specifically, we assessed the goal‐directed nature of two navigational strategies, using a combination of an outcome devaluation procedure and a spatial probe trial frequently used to dissociate the two navigational strategies. In Experiment 1, rats trained in a dual‐solution T‐maze task were subjected to an extinction probe trial from the opposite start arm, with or without prefeeding‐induced devaluation of the expected outcome. We found that a non‐significant preference for the place strategy in the non‐devalued condition was completely reversed after devaluation, such that significantly more animals displayed the use of the response strategy. The result suggests that the place strategy is sensitive to the expected value of the outcome, while the response strategy is not. In Experiment 2, rats with hippocampal lesions showed significant reliance on the response strategy, regardless of whether the expected outcome was devalued or not. The result thus offers further evidence that the response strategy conforms to the definition of an outcome‐insensitive, habitual form of instrumental behavior. These results together attest a formal correspondence between two types of dual‐process accounts of animal learning and behavior.