Goal-directed control on interval schedules does not depend on the action-outcome correlation

Random interval schedule of reinforcement influences punishment resistance for cocaine in rats

In an animal model of compulsive drug use, a subset of rats continues to self-administer cocaine despite footshock consequences and is considered punishment resistant. We recently found that punishment resistance is associated with habits that persist under conditions that typically encourage a transition to goal-directed control. Given that random ratio (RR) and random interval (RI) schedules of reinforcement influence whether responding is goal-directed or habitual, we investigated the influence of these schedules on punishment resistance for cocaine or food. Male and female Sprague Dawley rats were trained to self-administer either intravenous cocaine or food pellets on a seeking-taking chained schedule of reinforcement, with the seeking lever requiring completion of either an RR20 or RI60 schedule. Rats were then given four days of punishment testing with footshock administered at the completion of seeking on a random one-third of trials. For cocaine-trained rats, the RI60 schedule led to greater punishment resistance (i.e., more trials completed) than the RR20 schedule in males and females. For food-trained rats, the RI60 schedule led to greater punishment resistance (i.e., higher reward rates) than the RR20 schedule in female rats, although male rats showed punishment resistance on both RR20 and RI60 schedules. For both cocaine and food, we found that seeking responses were suppressed to a greater degree than reward rate with the RI60 schedule, whereas response rate and reward rate were equally suppressed with the RR20 schedule. This dissociation between punishment effects on reward rate and response rate with the RI60 schedule can be explained by the nonlinear relation between these variables on RI schedules, but it does not account for the enhanced resistance to punishment. Overall, the results show greater punishment resistance with the RI60 schedule as compared to the RR20 schedule, indicating that schedules of reinforcement are an influencing factor on resistance to negative consequences.

Manipulations of the context-response relationship reduce the expression of response habits

Habitual instrumental behaviour is believed to rely on stimulus-response (S-R) associations. However, the method most commonly used to identify habitual behaviour, outcome devaluation, provides only indirect evidence of S-R control. Therefore, it is important to have a better understanding of the S-R association believed to underlie habitual responding. Under free-operant conditions, the context itself likely serves as at least part of the relevant stimuli in the association, and so modifications to the predictive power of the context should alter the expression of habits. The following experiments investigated how changes to the relationship between the training context and performance of the response, either by changing the context during testing or by exposing animals to the context alone, without the response lever present, impacted behavioural control during a devaluation test. We found evidence that the training context is important for the expression of habits; testing animals in a different context than where they were trained resulted in increased goal-directed control (Experiment 1). Furthermore, context alone exposure also increased goal-directed control with animals that received context alone exposure showing stronger devaluation effects, whether the context alone exposure happened on the last day of training (Experiment 2) or throughout training (Experiment 3). These findings are consistent with prior reports that the training context is important for the expression of habits and extends these findings by using sensory-specific satiety as a means for devaluation and by using context alone exposure to alter behavioural control.

Bad habits-good goals? Meta-analysis and translation of the habit construct to alcoholism

Excessive alcohol consumption remains a global public health crisis, with millions suffering from alcohol use disorder (AUD, or simply "alcoholism"), leading to significantly reduced life expectancy. This review examines the interplay between habitual and goal-directed behaviors and the associated neurobiological changes induced by chronic alcohol exposure. Contrary to a strict habit-goal dichotomy, our meta-analysis of the published animal experiments combined with a review of human studies reveals a nuanced transition between these behavioral control systems, emphasizing the need for refined terminology to capture the probabilistic nature of decision biases in individuals with a history of chronic alcohol exposure. Furthermore, we distinguish habitual responding from compulsivity, viewing them as separate entities with diverse roles throughout the stages of the addiction cycle. By addressing species-specific differences and translational challenges in habit research, we provide insights to enhance future investigations and inform strategies for combatting AUD.

Characterizing Human Habits in the Lab

Habits pose a fundamental puzzle for those aiming to understand human behavior. They pervade our everyday lives and dominate some forms of psychopathology but are extremely hard to elicit in the lab. In this Registered Report, we developed novel experimental paradigms grounded in computational models, which suggest that habit strength should be proportional to the frequency of behavior and, in contrast to previous research, independent of value. Specifically, we manipulated how often participants performed responses in two tasks varying action repetition without, or separately from, variations in value. Moreover, we asked how this frequency-based habitization related to value-based operationalizations of habit and self-reported propensities for habitual behavior in real life. We find that choice frequency during training increases habit strength at test and that this form of habit shows little relation to value-based operationalizations of habit. Our findings empirically ground a novel perspective on the constituents of habits and suggest that habits may arise in the absence of external reinforcement. We further find no evidence for an overlap between different experimental approaches to measuring habits and no associations with self-reported real-life habits. Thus, our findings call for a rigorous reassessment of our understanding and measurement of human habitual behavior in the lab.

The temporal structure of goal-directed and habitual operant behavior

Operant behavior can reflect the influence of goal-directed and habitual processes. These can be distinguished by changes to response rate following devaluation of the reinforcing outcome. Whether a response is goal directed or habitual depends on whether devaluation affects response rate. Response rate can be decomposed into frequencies of bouts and pauses by analyzing the distribution of interresponse times. This study sought to characterize goal-directed and habitual behaviors in terms of bout-initiation rate, within-bout response rate, bout length, and bout duration. Data were taken from three published studies that compared sensitivity to devaluation following brief and extended training with variable-interval schedules. Analyses focused on goal-directed and habitual responding, a comparison of a habitual response to a similarly trained response that had been converted back to goal-directed status after a surprising event, and a demonstration of contextual control of habit and goal direction in the same subjects. Across experiments and despite responses being clearly distinguished as goal directed and habitual by total response rate, analyses of bout-initiation rate, within-bout rate, bout length, and bout duration did not reveal a pattern that distinguished goal-directed from habitual responding.

Making and breaking habits: Revisiting the definitions and behavioral factors that influence habits in animals

Habits have garnered significant interest in studies of associative learning and maladaptive behavior. However, habit research has faced scrutiny and challenges related to the definitions and methods. Differences in the conceptualizations of habits between animal and human studies create difficulties for translational research. Here, we review the definitions and commonly used methods for studying habits in animals and humans and discuss potential alternative ways to assess habits, such as automaticity. To better understand habits, we then focus on the behavioral factors that have been shown to make or break habits in animals, as well as potential mechanisms underlying the influence of these factors. We discuss the evidence that habitual and goal-directed systems learn in parallel and that they seem to interact in competitive and cooperative manners. Finally, we draw parallels between habitual responding and compulsive drug seeking in animals to delineate the similarities and differences in these behaviors.

Sex and individual differences in the effect of chronic low-dose ethanol on behavioral strategy selection

BACKGROUND

The development of an alcohol use disorder (AUD) involves impaired behavioral control and flexibility. Behavioral inflexibility includes an inability to shift behavior in response to changes in behavioral outcomes. Low levels of ethanol drinking may promote the formation of inflexible, habitual reward seeking, but this may depend on the timing of ethanol exposure in relation to learning. The goal of this study was to determine whether a history of low-dose ethanol exposure promoted contingency-insensitive sucrose seeking and altered behavioral strategy selection.

METHODS

Male and female C57BL/6J mice were trained to perform a response (lever press) for sucrose on two different reinforcement schedules: one that is thought to promote inflexible responding (random interval) and one that maintains flexible responding (variable ratio [VR]). Following instrumental training each day, mice were exposed to saline or low-dose ethanol (0.5 g/kg; i.p.) either proximal (1 h after) or distal (4 h after) to learning. Mice were then tested for sensitivity to changes in contingency in a contingency degradation test.

RESULTS

A history of low-dose ethanol exposure shifted behavioral strategy selection, as measured by reward tracking behavior, but this depended on sex and reinforcement schedule history. Both male and female mice used different strategies depending on the reinforcement schedule, but only males exhibited ethanol-induced shifts in strategy selection. A history of low-dose ethanol exposure did not impact contingency sensitivity in males but promoted insensitivity in females specifically on the VR lever.

CONCLUSIONS

Female mice show distinct behavioral effects of repeated, low-dose ethanol exposure as compared to males, with sex differences in the use of reward tracking strategies to guide behavior. Future studies should investigate sex differences in the neural consequences of chronic low-dose ethanol exposure that may underlie behavioral changes.

Positive correlation between measures of habitual responding and motivated responding in mice

Habit and motivation are thought to be separate processes, with motivated behavior often considered to be goal directed, whereas habits are defined by the absence of goal-directed control over behavior. However, there has been increasing interrogation of the binary nature of habitual versus goal-directed behavior. Furthermore, although drug and alcohol exposure can promote the formation of habits, drug seeking itself can also be highly flexible, pointing toward the need for complex consideration of the parallel processes that drive behavior. The goal of the current study was to determine whether there was a relation between motivation-as measured by progressive ratio-and habit-as measured by contingency degradation-and whether this relation was affected by ethanol exposure history and sex. The results showed that these measures were positively correlated such that greater contingency insensitivity was associated with achieving higher break points on the progressive-ratio task. However, this relation depended on reinforcement schedule history, ethanol exposure history, and sex. These point to potential relations between measures of habit and motivation and stress the importance of carefully parsing behavioral findings and assays. These findings are also expected to inform future substance use research, as drug history may affect these relations.

Resurgence of goal-directed actions and habits

This study investigated how goal-directed and habitual behaviors recover after extinction within the context of the resurgence effect, a form of relapse induced by the removal or worsening of alternative reinforcement. Rats were trained to press a target lever with one reinforcer (O1) for either minimal (4) or extended (16) sessions. An extinction test after the completion of O1 devaluation confirmed that minimal and extended training formed goal-directed and habitual behaviors, respectively. Then, pressing an alternative lever was reinforced with a second reinforcer (O2) while the target response was placed on extinction. When O2 was discontinued, the minimally trained target response resurged with goal-directed status as in the extinction test. However, the extinguished habitual behavior in the extensively trained rats did not recover as a habit but instead with goal-directed status, possibly due to the context specificity of habits or the introduction of a new response-reinforcer contingency. The critical finding that reinforcer devaluation consistently led to less resurgence regardless of the amount of acquisition training provides a clinical implication that coupling differential-reinforcement-of-alternative-behavior (DRA) treatments with the devaluation of the associated reinforcer of problematic behavior could effectively diminish its recurrence.

Food Restriction Level and Reinforcement Schedule Differentially Influence Behavior during Acquisition and Devaluation Procedures in Mice

Behavioral strategies are often classified based on whether reinforcer value controls reinforcement. Value-sensitive behaviors, in which animals update their actions when reinforcer value is changed, are classified as goal-directed; conversely, value-insensitive actions, where behavior remains consistent when the reinforcer is removed or devalued, are considered habitual. Basic reinforcement schedules can help to bias behavior toward either process: random ratio (RR) schedules are thought to promote the formation of goal-directed behaviors while random intervals (RIs) promote habitual control. However, how the schedule-specific features of these tasks interact with other factors that influence learning to control behavior has not been well characterized. Using male and female mice, we asked how distinct food restriction levels, a strategy often used to increase task engagement, interact with RR and RI schedules to control performance during task acquisition and devaluation procedures. We determined that food restriction level has a stronger effect on the behavior of mice following RR schedules compared with RI schedules, and that it promotes a decrease in response rate during devaluation procedures that is best explained by the effects of extinction rather than devaluation. Surprisingly, food restriction accelerated the decrease in response rates observed following devaluation across sequential extinction sessions, but not within a single session. Our results support the idea that the relationships between schedules and behavioral control strategies are not clear-cut and suggest that an animal's engagement in a task must be accounted for, together with the structure of reinforcement schedules, to appropriately interpret the cognitive underpinnings of behavior.

Habit formation viewed as structural change in the behavioral network

Habit formation is a process in which an action becomes involuntary. While goal-directed behavior is driven by its consequences, habits are elicited by a situation rather than its consequences. Existing theories have proposed that actions are controlled by corresponding two distinct systems. Although canonical theories based on such distinctions are starting to be challenged, there are a few theoretical frameworks that implement goal-directed behavior and habits within a single system. Here, we propose a novel theoretical framework by hypothesizing that behavior is a network composed of several responses. With this framework, we have shown that the transition of goal-directed actions to habits is caused by a change in a single network structure. Furthermore, we confirmed that the proposed network model behaves in a manner consistent with the existing experimental results reported in animal behavioral studies. Our results revealed that habit could be formed under the control of a single system rather than two distinct systems. By capturing the behavior as a single network change, this framework provides a new perspective on studying the structure of the behavior for experimental and theoretical research.

Lack of action monitoring as a prerequisite for habitual and chunked behavior: Behavioral and neural correlates

We previously reported the rapid development of habitual behavior in a discrete-trials instrumental task in which lever insertion and retraction act as reward-predictive cues delineating sequence execution. Here we asked whether lever cues or performance variables reflective of skill and automaticity might account for habitual behavior in male rats. Behavior in the discrete-trials habit-promoting task was compared with two task variants lacking the sequence-delineating cues of lever extension and retraction. We find that behavior is under goal-directed control in absence of sequence-delineating cues but not in their presence, and that skilled performance does not predict goal-directed vs. habitual behavior. Neural activity recordings revealed an engagement of dorsolateral striatum and a disengagement of dorsomedial striatum during the sequence execution of the habit-promoting task, specifically. Together, these results indicate that sequence delineation cues promote habit and differential engagement of striatal subregions during instrumental responding, a pattern that may reflect cue-elicited behavioral chunking.

Habitual behaviour associated with exposure to high-calorie diet is prevented by an orexin-receptor-1 antagonist

Habitual actions, which are associated with addictive behaviours, contribute to the loss of control of food seeking seen following exposure to calorie-dense foods in rats. Antagonism of orexin-receptor-1 (ORX-R1) has been shown to reduce a range of stimulus-driven feeding behaviours, but have yet to be implicated in the regulation of habitual actions. In the current study, male Long-Evans rats were given 'binge-like' access to high-calorie diet (HCD) or standard chow diet, and were subsequently trained to press a lever for food outcome. When lever responses were tested following outcome devaluation, chow-fed rats displayed goal-directed actions, whereas HCD-exposed rats displayed habitual actions. In study 1, it was shown that systemic administration of the ORX-R1 antagonist, SB-334867, prior to test restored goal-directed behaviour in HCD-exposed rats. In study 2, intra-nigral administration of SB-334867 similarly restored goal-directed behaviour, thereby implicating the substantia nigra as an important site for this effect. This study demonstrates that targeting ORX-R1 reduces habitual food seeking in male rats which may be important for understanding and treating compulsive feeding, obesity and binge eating disorder. This study also implicates the lateral hypothalamus, where ORX is produced, in mediating the expression of habits for the first time, and thus extends on the neurocircuits known to regulate habitual actions. Further investigation is required to determine whether the same effects are also seen in female rats, given that there are recognised sexual dimorphisms in feeding behaviour and a higher incidence of disordered eating in female than male populations.

Sequential delay and probability discounting tasks in mice reveal anchoring effects partially attributable to decision noise

Delay discounting and probability discounting decision making tasks in rodent models have high translational potential. However, it is unclear whether the discounted value of the large reward option is the main contributor to variability in animals' choices in either task, which may limit translation to humans. Male and female mice underwent sessions of delay and probability discounting in sequence to assess how choice behavior adapts over experience with each task. To control for "anchoring" (persistent choices based on the initial delay or probability), mice experienced "Worsening" schedules where the large reward was offered under initially favorable conditions that became less favorable during testing, followed by "Improving" schedules where the large reward was offered under initially unfavorable conditions that improved over a session. During delay discounting, both male and female mice showed elimination of anchoring effects over training. In probability discounting, both sexes of mice continued to show some anchoring even after months of training. One possibility is that "noisy", exploratory choices could contribute to these persistent anchoring effects, rather than constant fluctuations in value discounting. We fit choice behavior in individual animals using models that included both a value-based discounting parameter and a decision noise parameter that captured variability in choices deviating from value maximization. Changes in anchoring behavior over time were tracked by changes in both the value and decision noise parameters in delay discounting, but by the decision noise parameter in probability discounting. Exploratory decision making was also reflected in choice response times that tracked the degree of conflict caused by both uncertainty and temporal cost, but was not linked with differences in locomotor activity reflecting chamber exploration. Thus, variable discounting behavior in mice can result from changes in exploration of the decision options rather than changes in reward valuation.

Maintained goal-directed control with overtraining on ratio schedules

It is thought that goal-directed control of actions weakens or becomes masked by habits over time. We tested the opposing hypothesis that goal-directed control becomes stronger over time, and that this growth is modulated by the overall action-outcome contiguity. Despite group differences in action-outcome contiguity early in training, rats trained under random and fixed ratio schedules showed equivalent goal-directed control of lever pressing that appeared to grow over time. We confirmed that goal-directed control was maintained after extended training under another type of ratio schedule-continuous reinforcement-using specific satiety and taste aversion devaluation methods. These results add to the growing literature showing that extensive training does not reliably weaken goal-directed control and that it may strengthen it, or at least maintain it.

Context, attention, and the switch between habit and goal-direction in behavior

This article reviews recent findings from the author’s laboratory that may provide new insights into how habits are made and broken. Habits are extensively practiced behaviors that are automatically evoked by antecedent cues and performed without their goal (or reinforcer) “in mind.” Goal-directed actions, in contrast, are instrumental behaviors that are performed because their goal is remembered and valued. New results suggest that actions may transition to habit after extended practice when conditions encourage reduced attention to the behavior. Consistent with theories of attention and learning, a behavior may command less attention (and become habitual) as its reinforcer becomes well-predicted by cues in the environment; habit learning is prevented if presentation of the reinforcer is uncertain. Other results suggest that habits are not permanent, and that goal-direction can be restored by several environmental manipulations, including exposure to unexpected reinforcers or context change. Habits are more context-dependent than goal-directed actions are. Habit learning causes retroactive interference in a way that is reminiscent of extinction: It inhibits, but does not erase, goal-direction in a context-dependent way. The findings have implications for the understanding of habitual and goal-directed control of behavior as well as disordered behaviors like addictions.

Context and renewal of habits and goal-directed actions after extinction

Instrumental behaviors that are goal-directed actions after moderate amounts of training can become habits after more extended training. Little research has asked how actions and habits are affected by retroactive interference treatments like extinction. The present experiments begin to fill this gap in the literature. In Experiments 1a and 1b, lever pressing in rats was minimally trained (1a) or extensively trained (1b) in one context (Context A), extinguished in a second context (Context B), and then tested in the acquisition context (Context A). Exposure to both contexts was equated and controlled throughout, and the status of the behavior as action or habit was determined by reinforcer devaluation methods (taste aversion conditioning). Results confirmed that action (1a) and habit (1b) renewed with action or habit status, respectively, when they were returned to Context A. Experiments 2a and 2b then similarly tested action and habit after extinction in an ABC renewal paradigm. Here, lever pressing that was trained in Context A and extinguished in Context B renewed as action in Context C regardless of whether it had been an action or habit before extinction. The apparent conversion of habit to action during renewal testing in Context C was consistent with other results suggesting that habits converted to action when the context was changed at the start of extinction. Together, the results suggest that extinction in a second context inhibits instrumental behaviors trained as either actions or habits in a context-specific manner. They also expand on prior findings suggesting that actions transfer across contexts, and that habits do not. A change of context may be sufficient to convert a habit to goal-directed action. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Lesion of striatal patches disrupts habitual behaviors and increases behavioral variability

Habits are automated behaviors that are insensitive to changes in behavioral outcomes. Habitual responding is thought to be mediated by the striatum, with medial striatum guiding goal-directed action and lateral striatum promoting habits. However, interspersed throughout the striatum are neurochemically differing subcompartments known as patches, which are characterized by distinct molecular profiles relative to the surrounding matrix tissue. These structures have been thoroughly characterized neurochemically and anatomically, but little is known regarding their function. Patches have been shown to be selectively activated during inflexible motor stereotypies elicited by stimulants, suggesting that patches may subserve habitual behaviors. To explore this possibility, we utilized transgenic mice (Sepw1 NP67) preferentially expressing Cre recombinase in striatal patch neurons to target these neurons for ablation with a virus driving Cre-dependent expression of caspase 3. Mice were then trained to press a lever for sucrose rewards on a variable interval schedule to elicit habitual responding. Mice were not impaired on the acquisition of this task, but lesioning striatal patches disrupted behavioral stability across training, and lesioned mice utilized a more goal-directed behavioral strategy during training. Similarly, when mice were forced to omit responses to receive sucrose rewards, habitual responding was impaired in lesioned mice. To rule out effects of lesion on motor behaviors, mice were then tested for impairments in motor learning on a rotarod and locomotion in an open field. We found that patch lesions partially impaired initial performance on the rotarod without modifying locomotor behaviors in open field. This work indicates that patches promote behavioral stability and habitual responding, adding to a growing literature implicating striatal patches in stimulus-response behaviors.