Using optogenetics to study habits

Revealing abrupt transitions from goal-directed to habitual behavior

A fundamental tenet of animal behavior is that decision-making involves multiple 'controllers.' Initially, behavior is goal-directed, driven by desired outcomes, shifting later to habitual control, where cues trigger actions independent of motivational state. Clark Hull's question from 1943 still resonates today: "Is this transition abrupt, or is it gradual and progressive?"1 Despite a century-long belief in gradual transitions, this question remains unanswered2,3 as current methods cannot disambiguate goal-directed versus habitual control in real-time. Here, we introduce a novel 'volitional engagement' approach, motivating animals by palatability rather than biological need. Offering less palatable water in the home cage4,5 reduced motivation to 'work' for plain water in an auditory discrimination task when compared to water-restricted animals. Using quantitative behavior and computational modeling6, we found that palatability-driven animals learned to discriminate as quickly as water-restricted animals but exhibited state-like fluctuations when responding to the reward-predicting cue-reflecting goal-directed behavior. These fluctuations spontaneously and abruptly ceased after thousands of trials, with animals now always responding to the reward-predicting cue. In line with habitual control, post-transition behavior displayed motor automaticity, decreased error sensitivity (assessed via pupillary responses), and insensitivity to outcome devaluation. Bilateral lesions of the habit-related dorsolateral striatum7 blocked transitions to habitual behavior. Thus, 'volitional engagement' reveals spontaneous and abrupt transitions from goal-directed to habitual behavior, suggesting the involvement of a higher-level process that arbitrates between the two.

Optogenetic disruption of the prelimbic cortex alters long-term decision strategy but not valuation on a spatial delay discounting task

Rats demonstrate a preference for smaller, immediate rewards over larger, delayed ones, a phenomenon known as delay-discounting (DD). Behavior arises from the interaction of multiple decision-making systems, and the medial prefrontal cortex (mPFC) has been identified as a central component in the mediation between these decision systems. To investigate the role of the prelimbic (PL) subregion of mPFC on decision strategy interaction, we compared two cohorts of rats (ChR2-opsin-expressing 'Active' and opsin-absent 'Control') on a spatial delay-discounting task while delivering in-vivo light stimulation into PL at the choice point of select trials. By analyzing the overall delay-adjustment along with deliberative and procedural behavioral strategy markers, our study revealed differences in the decision strategies used between the active and control animals despite both groups showing similar valuations. Control animals developed the expected shift from deliberative to procedural decision strategy on this task (indicated by reaching delay-stability, particularly during late-session laps); however, active-virus animals repeatedly over-adjusted around their preferred delay throughout the entire session, suggesting a significant deficit in procedural decision-making on this task. Active animals showed a significant decrease in proportion of vicarious trial and error events (VTE, a behavior correlated with deliberative processes) on delay adjustment laps relative to control animals. This points to a more nuanced role for VTE, not just in executing deliberation, but in shifting from deliberative to procedural processes. This opto-induced change in VTE was especially pronounced for late-session adjustment laps. We found no other session-by-session or lap-by-lap effects, leaving a particular role for PL in the long-term development of procedural strategies on this task.

Task history dictates how the dorsolateral striatum controls action strategy and vigor

The dorsolateral striatum (DLS) is linked to the learning and honing of action routines. However, the DLS is also important for performing behaviors that have been successful in the past. The learning function can be thought of as prospective, helping to plan ongoing actions to be efficient and often optimal. The performance function is more retrospective, helping the animal continue to behave in a way that had worked previously. How the DLS manages this all is curious. What happens when a learned behavior becomes sub-optimal due to environment changes. In this case, the prospective function of the DLS would cause animals to (adaptively) learn and plan more optimal actions. In contrast, the retrospective function would cause animals to (maladaptively) favor the old behavior. Here we find that, during a change in learned task rules, DLS inhibition causes animals to adjust less rapidly to the new task (and to behave less vigorously) in a 'maladaptive' way. Yet, when the task is changed back to the initially learned rules, DLS inhibition instead causes a rapid and vigorous adjustment of behavior in an 'adaptive' way. These results show that inhibiting the DLS biases behavior towards initially acquired strategies, implying a more retrospective outlook in action selection when the DLS is offline. Thus, an active DLS could encourage planning and learning action routines more prospectively. Moreover, the DLS control over behavior can appear to be either advantageous/flexible or disadvantageous/inflexible depending on task context, and its control over vigor can change depending on task context.

Significant Statement

Basal ganglia networks aid behavioral learning (a prospective planning function) but also favor the use of old behaviors (a retrospective performance function), making it unclear what happens when learned behaviors become suboptimal. Here we inhibit the dorsolateral striatum (DLS) as animals encounter a change in task rules, and again when they shift back to those learned task rules. DLS inhibition reduces adjustment to new task rules (and reduces behavioral vigor), but it increases adjustment back to the initially learned task rules later (and increases vigor). Thus, in both cases, DLS inhibition favored the use of the initially learned behavioral strategy, which could appear either maladaptive or adaptive. We suggest that the DLS might promote a prospective orientation of action control.

The effect of instruction on motor skill learning

Many motor skills are learned with the help of instructions. In the context of complex motor sequences, instructions often break down the movement into chunks that can then be practiced in isolation. Thus, instructions shape an initial cognitive representation of the skill, which in turn guides practice. Are there ways of breaking up a motor sequence that are better than others? If participants are instructed in a way that hinders performance, how much practice does it take to overcome the influence of the instruction? To answer these questions, we used a paradigm in which participants were asked to perform finger sequences as fast and accurately as possible on a keyboard-like device. In the initial phases of training, participants had to explicitly remember and practice two- or three-digit chunks. These chunks were then combined to form seven 11-digit sequences that participants practiced for the remainder of the study. Each sequence was broken up into chunks in a way such that the instruction was either aligned or misaligned with the basic execution-level constraints. We found that misaligned chunk instruction led to an initial performance deficit compared with the aligned chunk instruction. Overall, instructions still influenced the temporal pattern of performance after 10 days of subsequent training, with shorter interpress intervals within a chunk compared with between chunks. However, for the misaligned instructed sequences, this temporal pattern was altered more rapidly, such that participants could overcome the induced performance deficit in the last week. At the end of training, participants found idiosyncratic, but interindividually stable, ways of performing each sequence.NEW & NOTEWORTHY Instructions often break down motor sequences into smaller parts, such that they can be more easily remembered. Here, we show that different ways of breaking down a finger sequence can subsequently lead to better or worse performance. The initial instruction still influenced the temporal performance pattern after 10 days of practice. The results demonstrate that the initial cognitive representation of a motor skill strongly influences how a skill is learned and performed.

Complementary Control over Habits and Behavioral Vigor by Phasic Activity in the Dorsolateral Striatum

Despite clear evidence linking the basal ganglia to the control of outcome insensitivity (i.e., habit) and behavioral vigor (i.e., its behavioral speed/fluidity), it remains unclear whether or how these functions relate to one another. Here, using male Long-Evans rats in response-based and cue-based maze-running tasks, we demonstrate that phasic dorsolateral striatum (DLS) activity occurring at the onset of a learned behavior regulates how vigorous and habitual it is. In a response-based task, brief optogenetic excitation at the onset of runs decreased run duration and the occurrence of deliberative behaviors, whereas midrun stimulation carried little effect. Outcome devaluation showed these runs to be habitual. DLS inhibition at run start did not produce robust effects on behavior until after outcome devaluation. At that time, when the DLS was plausibly most critically required for performance (i.e., habitual), inhibition reduced performance vigor measures and caused a dramatic loss of habitual responding (i.e., animals quit the task). In a second cue-based "beacon" task requiring behavior initiation at the start of the run and again in the middle of the run, DLS excitation at both time points could improve the vigor of runs. Postdevaluation testing showed behavior on the beacon task to be habitual as well. This pattern of results suggests that one role for phasic DLS activity at behavior initiation is to promote the execution of the behavior in a vigorous and habitual fashion by a diverse set of measures.SIGNIFICANCE STATEMENT Our research expands the literature twofold. First, we find that features of a habitual behavior that are typically studied separately (i.e., maze response performance, deliberation movements, running vigor, and outcome insensitivity) are quite closely linked together. Second, efforts have been made to understand "what" the dorsolateral striatum (DLS) does for habitual behavior, and our research provides a key set of results showing "when" it is important (i.e., at behavior initiation). By showing such dramatic control over habits by DLS activity in a phasic time window, plausible real-world applications could involve more informed DLS perturbations to curb intractably problematic habits.

Probing Synaptic Transmission and Behavior in Drosophila with Optogenetics: A Laboratory Exercise

Optogenetics is possibly the most revolutionary advance in neuroscience research techniques within the last decade. Here, we describe lab modules, presented at a workshop for undergraduate neuroscience educators, using optogenetic control of neurons in the fruit fly Drosophila melanogaster. Drosophila is a genetically accessible model system that combines behavioral and neurophysiological complexity, ease of use, and high research relevance. One lab module utilized two transgenic Drosophila strains, each activating specific circuits underlying startle behavior and backwards locomotion, respectively. The red-shifted channelrhodopsin, CsChrimson, was expressed in neurons sharing a common transcriptional profile, with the expression pattern further refined by the use of a Split GAL4 intersectional activation system. Another set of strains was used to investigate synaptic transmission at the larval neuromuscular junction. These expressed Channelrhodopsin 2 (ChR2) in glutamatergic neurons, including the motor neurons. The first strain expressed ChR2 in a wild type background, while the second contained the SNAP-25ts mutant allele, which confers heightened evoked potential amplitude and greatly increased spontaneous vesicle release frequency at the larval neuromuscular junction. These modules introduced educators and students to the use of optogenetic stimulation to control behavior and evoked release at a model synapse, and establish a basis for students to explore neurophysiology using this technique, through recapitulating classic experiments and conducting independent research.

Pharmacological Blockade of Adenosine A2A but Not A1 Receptors Enhances Goal-Directed Valuation in Satiety-Based Instrumental Behavior

The balance and smooth shift between flexible, goal-directed behaviors and repetitive, habitual actions are critical to optimal performance of behavioral tasks. The striatum plays an essential role in control of goal-directed versus habitual behaviors through a rich interplay of the numerous neurotransmitters and neuromodulators to modify the input, processing and output functions of the striatum. The adenosine receptors (namely A_2AR and A₁R), with their high expression pattern in the striatum and abilities to interact and integrate dopamine, glutamate and cannabinoid signals in the striatum, may represent novel therapeutic targets for modulating instrumental behavior. In this study, we examined the effects of pharmacological blockade of the A_2ARs and A₁Rs on goal-directed versus habitual behaviors in different information processing phases of instrumental learning using a satiety-based instrumental behavior procedure. We found that A_2AR antagonist acts at the coding, consolidation and expression phases of instrumental learning to modulate animals’ sensitivity to goal-directed valuation without modifying action-outcome contingency. However, pharmacological blockade and genetic knockout of A₁Rs did not affect acquisition or sensitivity to goal-valuation of instrumental behavior. These findings provide pharmacological evidence for a potential therapeutic strategy to control abnormal instrumental behaviors associated with drug addiction and obsessive-compulsive disorder by targeting the A_2AR.

Habit formation coincides with shifts in reinforcement representations in the sensorimotor striatum

Evaluating outcomes of behavior is a central function of the striatum. In circuits engaging the dorsomedial striatum, sensitivity to goal value is accentuated during learning, whereas outcome sensitivity is thought to be minimal in the dorsolateral striatum and its habit-related corticostriatal circuits. However, a distinct population of projection neurons in the dorsolateral striatum exhibits selective sensitivity to rewards. Here, we evaluated the outcome-related signaling in such neurons as rats performed an instructional T-maze task for two rewards. As the rats formed maze-running habits and then changed behavior after reward devaluation, we detected outcome-related spike activity in 116 units out of 1,479 recorded units. During initial training, nearly equal numbers of these units fired preferentially either after rewarded runs or after unrewarded runs, and the majority were responsive at only one of two reward locations. With overtraining, as habits formed, firing in nonrewarded trials almost disappeared, and reward-specific firing declined. Thus error-related signaling was lost, and reward signaling became generalized. Following reward devaluation, in an extinction test, postgoal activity was nearly undetectable, despite accurate running. Strikingly, when rewards were then returned, postgoal activity reappeared and recapitulated the original early response pattern, with nearly equal numbers responding to rewarded and unrewarded runs and to single rewards. These findings demonstrate that outcome evaluation in the dorsolateral striatum is highly plastic and tracks stages of behavioral exploration and exploitation. These signals could be a new target for understanding compulsive behaviors that involve changes to dorsal striatum function.

A Brain Motivated to Play: Insights into the Neurobiology of Playfulness

Play is an important part of normal childhood development and is seen in varied forms among many mammals. While not indispensable to normal development, playful social experiences as juveniles may provide an opportunity to develop flexible behavioral strategies when novel and uncertain situations arise as an adult. To understand the neurobiological mechanisms responsible for play and how the functions of play may relate to these neural substrates, the rat has become the model of choice. Play in the rat is easily quantified, tightly regulated, and can be modulated by genetic factors and postnatal experiences. Brain areas most likely to be involved in the modulation of play include regions within the prefrontal cortex, dorsal and ventral striatum, some regions of the amygdala, and habenula. This paper discusses what we currently know about the neurobiological substrates of play and how this can help illuminate functional questions about the putative benefits of play.

Using Optogenetics to Dissect the Neural Circuits Underlying OCD and Related Disorders

Clinical and preclinical studies have uncovered substantial evidence that dysfunction in cortico-striatal-thalamo-cortical (CSTC) loops central to the selection of action strategies may underlie obsessive compulsive disorder (OCD) symptoms. In human OCD, data suggest that the balance between selections of habitual versus goal-directed action strategies is disrupted, with concomitant hyperactivation of CSTC regions associated with these strategies. Preclinical lesion and inactivation studies of homologous CSTC regions in rodents have shed light on how sub-regions of the frontal cortex and striatum can have dissociable effects on the exhibition of goal-directed or habitual behavior. However, these traditional methods lack the precision necessary to dissect the exact projections and cell types underlying these behaviors. It is essential to uncover this information to begin to determine how disruption in these circuits may lead to disease pathology. Here, we summarize several recent studies that utilize optogenetics, a technique that allows stimulation or inhibition of specific neural projections and cell types using light, to further understand the contribution of CSTC activity to both action selection and the OCD-relevant behavior of perseverative grooming. Based on these experiments and findings in human OCD patients, we argue that OCD symptoms may not only be associated with an enhancement of habitual behavior, but also with aberrant recruitment of goal-directed neural circuits. We also discuss the current status of translating optogenetic technology to primates, as well as how findings in rodents may help inform treatment of patients suffering from OCD and related disorders.

Habitual alcohol seeking: modeling the transition from casual drinking to addiction

The transition from goal-directed actions to habitual ethanol seeking models the development of addictive behavior that characterizes alcohol use disorders. The progression to habitual ethanol-seeking behavior occurs more rapidly than for natural rewards, suggesting that ethanol may act on habit circuit to drive the loss of behavioral flexibility. This review will highlight recent research that has focused on the formation and expression of habitual ethanol seeking, and the commonalities and distinctions between ethanol and natural reward-seeking habits, with the goal of highlighting important, understudied research areas that we believe will lead toward the development of novel treatment and prevention strategies for uncontrolled drinking.

The contribution of medial prefrontal cortical regions to conditioned inhibition

Few studies have considered the process by which individuals learn to omit a response, which is an essential aspect of adaptive behavior. Several lines of evidence indicate that two regions of the medial prefrontal cortex have disparate roles in behavioral flexibility. In particular, the prelimbic cortex (PL) is thought to facilitate the generation of a strategy to inhibit a prepotent response, whereas the infralimbic cortex (IL) appears to be more important for maintaining extensively trained inhibitory behaviors. The present experiments were designed to elucidate the contributions of PL and IL to the acquisition and maintenance of Pavlovian conditioned inhibition. In Experiment 1, damage to PL before training in a compound feature negative discrimination task impaired inhibitory learning. By comparison, lesions of IL had little effect. In Experiment 2, lesions of PL or IL occurred after overtraining, and damage to IL significantly impaired subsequent performance in the task, suggesting that this region is involved in the continued expression of Pavlovian conditioned inhibition after thorough training. PL may also be involved in maintaining inhibition, as evidenced by a marginally significant lesion-induced performance deficit. These data support the notion that PL and IL have distinguishable roles in modulating inhibition, while contributing important information about the specific role for PL in acquisition of an inhibitory response and IL in performance.

Investigating habits: strategies, technologies and models

Understanding habits at a biological level requires a combination of behavioral observations and measures of ongoing neural activity. Theoretical frameworks as well as definitions of habitual behaviors emerging from classic behavioral research have been enriched by new approaches taking account of the identification of brain regions and circuits related to habitual behavior. Together, this combination of experimental and theoretical work has provided key insights into how brain circuits underlying action-learning and action-selection are organized, and how a balance between behavioral flexibility and fixity is achieved. New methods to monitor and manipulate neural activity in real time are allowing us to have a first look “under the hood” of a habit as it is formed and expressed. Here we discuss ideas emerging from such approaches. We pay special attention to the unexpected findings that have arisen from our own experiments suggesting that habitual behaviors likely require the simultaneous activity of multiple distinct components, or operators, seen as responsible for the contrasting dynamics of neural activity in both cortico-limbic and sensorimotor circuits recorded concurrently during different stages of habit learning. The neural dynamics identified thus far do not fully meet expectations derived from traditional models of the structure of habits, and the behavioral measures of habits that we have made also are not fully aligned with these models. We explore these new clues as opportunities to refine an understanding of habits.

Goals and habits in the brain

An enduring and richly elaborated dichotomy in cognitive neuroscience is that of reflective versus reflexive decision making and choice. Other literatures refer to the two ends of what is likely to be a spectrum with terms such as goal-directed versus habitual, model-based versus model-free or prospective versus retrospective. One of the most rigorous traditions of experimental work in the field started with studies in rodents and graduated via human versions and enrichments of those experiments to a current state in which new paradigms are probing and challenging the very heart of the distinction. We review four generations of work in this tradition and provide pointers to the forefront of the field's fifth generation.

Mouse models of neurodevelopmental disease of the basal ganglia and associated circuits

This chapter focuses on neurodevelopmental diseases that are tightly linked to abnormal function of the striatum and connected structures. We begin with an overview of three representative diseases in which striatal dysfunction plays a key role--Tourette syndrome and obsessive-compulsive disorder, Rett's syndrome, and primary dystonia. These diseases highlight distinct etiologies that disrupt striatal integrity and function during development, and showcase the varied clinical manifestations of striatal dysfunction. We then review striatal organization and function, including evidence for striatal roles in online motor control/action selection, reinforcement learning, habit formation, and action sequencing. A key barrier to progress has been the relative lack of animal models of these diseases, though recently there has been considerable progress. We review these efforts, including their relative merits providing insight into disease pathogenesis, disease symptomatology, and basal ganglia function.