Context, attention, and the switch between habit and goal-direction in behavior

How can we apply decision-making theories to wild animal behavior? Predictions arising from dual process theory and Bayesian decision theory

Our understanding of decision-making processes and cognitive biases is ever increasing, thanks to an accumulation of testable models and a large body of research over the last several decades. The vast majority of this work has been done in humans and laboratory animals because these study subjects and situations allow for tightly controlled experiments. However, it raises questions about how this knowledge can be applied to wild animals in their complex environments. Here, we review two prominent decision-making theories, dual process theory and Bayesian decision theory, to assess the similarities in these approaches and consider how they may apply to wild animals living in heterogenous environments within complicated social groupings. In particular, we wanted to assess when wild animals are likely to respond to a situation with a quick heuristic decision and when they are likely to spend more time and energy on the decision-making process. Based on the literature and evidence from our multi-destination routing experiments on primates, we find that individuals are likely to make quick, heuristic decisions when they encounter routine situations, or signals/cues that accurately predict a certain outcome, or easy problems that experience or evolutionary history has prepared them for. Conversely, effortful decision-making is likely in novel or surprising situations, when signals and cues have unpredictable or uncertain relationships to an outcome, and when problems are computationally complex. Though if problems are overly complex, satisficing via heuristics is likely, to avoid costly mental effort. We present hypotheses for how animals with different socio-ecologies may have to distribute their cognitive effort. Finally, we examine the conservation implications and potential cognitive overload for animals experiencing increasingly novel situations caused by current human-induced rapid environmental change.

Adherence Challenges and Strategies in Specific Groups With Asthma: Adolescents, Pregnancy, and Older Adults

Poor adherence to controller therapies is a universal challenge to asthma control. Several high-risk groups, including adolescents, pregnant women, and older adults, have their own unique challenges to adherence. The rates of asthma controller therapy use are low in each of these populations, but secondary to different causes. Adolescents have increased independence and a transition to new self-management responsibilities; pregnant women may be concerned about adverse effects of medications to the fetus; and older adults may have age-related physical and cognitive challenges to effectively taking medication. Only by understanding the nuances of care in these populations can health care professionals develop strategies to address barriers to adherence. Tailored education focused on empowering patients and dispelling misconceptions can serve as tools to improve adherence and ultimately asthma control.

Response-specificity or response-generality of inhibition in an operant feature-negative discrimination: Influence of the amount of inhibition training and attention to the response

The suppression of behavior that occurs in instrumental extinction is strikingly specific to the response. In contrast, Steinfeld and Bouton (2022) recently reported that inhibition developing in an operant feature-negative (FN) discrimination is not specific to the response. In two experiments, we tested two potential explanations of why inhibition in FN learning is relatively response-general. In each, we used Steinfeld and Bouton's method and concurrently trained two FN discriminations with different operant responses (AR1+/ABR1- and CR2+/CDR2-). We then assessed the extent to which the inhibitory cues (B and D) suppressed the response they were trained with (same-response inhibition) and the alternative response (cross-response inhibition). Experiment 1 tested the idea that FN inhibition might be response-general because it can create strong inhibition. Rats received either 3, 6, or 12 sessions of FN discrimination training (Steinfeld and Bouton's rats had received 12). Inhibition was response-general at every level of training. In Experiment 2, the inhibitors (B and D) were first trained as cues that set the occasion for R1 and R2 (respectively) before they were turned into inhibitors in the FN discriminations. In the end, there was less cross-response inhibition, and thus more response-specificity. We suggest that inhibition in FN learning may be response-general because the unambiguous inhibitory cue (B or D) can draw attention away from the response.

Outcome devaluation as a method for identifying goal-directed behaviors in rats

Goal-directed behaviors allow animals to act to satisfy needs and desires. The outcome devaluation task is an effective method for identifying goal-directed behaviors and distinguishing these from other types of behavior. Rats can be trained to lever-press for one or multiple distinct food rewards. During testing, the previously earned food-or a control food for comparison-is devalued by allowing the animal to freely feed on it until they are sated before testing lever-press performance under extinction conditions (no rewards are delivered). Behavior that adapts to reflect the new value of the outcome is considered goal-directed, whereas behavior that continues as in previous training despite the change in outcome value, is not. As more research groups have used this task, variability in the procedures used has increased. Here, we provide a reliable procedure for conducting the outcome devaluation task with appropriate controls. We describe the most common variants of the task and control conditions and discuss troubleshooting measures such as outcome pre-exposure, habituation to pre-feeding chambers and attention to animals' hunger levels. The method outlined can be executed in ~2 weeks including training (~8 d) and testing (1-4 d) by researchers who are familiar with performing behavioral tasks in laboratory rodents, although longer training may be considered for those who are interested in observing habitual control of behavior. This protocol should facilitate the comparison of results from different studies and laboratories, while allowing flexibility in the application of the outcome devaluation task to different research questions.

Role of dopamine in reward expectation and predictability during execution of action sequences

Reward-associated cues serve different functions depending on whether they precede or terminate action sequences. Cues that precede action sequences and signal opportunity for reward could serve as GO signals to initiate the sequence, whereas sequence termination cues could serve as response feedback by signaling reward delivery. Reward expectation during sequence execution depends on these cues and might condition whether behavior is habitual or goal-directed. However, it remains unknown how sequence initiation and termination cues differentially affect reward expectation and contribute to habit learning. Further, while mesolimbic dopamine plays a key role in cue-induced reward expectation and sequence learning, how dynamic changes in dopamine signals differ depending on the response strategy is unclear. Here, we determined how mesolimbic DA signals change over training during cue-mediated sequence learning, depending on the type of cue and the nature of behavioral control. We found sequence initiation and termination cues differentially affect reward expectation during action sequences, with the termination cue contributing to habit and automaticity. Distinct response strategies induced by sequence initiation and termination cues induced differential changes in mesolimbic DA signals that captured variations in reward expectation along sequence execution. Notably, habit-like behavior induced by the sequence termination cue was associated with a rapid shift in DA signals from reward retrieval to the cue. This habit-like behavior was reflected in behavioral inflexibility and attenuated DA reward prediction error signals. Finally, using optogenetics, we provide evidence that phasic DA activity elicited by the sequence termination cue is critical for the development of habit-like behavior.

Manipulations of the context-response relationship reduce the expression of response habits

Habitual instrumental behaviour is believed to rely on stimulus-response (S-R) associations. However, the method most commonly used to identify habitual behaviour, outcome devaluation, provides only indirect evidence of S-R control. Therefore, it is important to have a better understanding of the S-R association believed to underlie habitual responding. Under free-operant conditions, the context itself likely serves as at least part of the relevant stimuli in the association, and so modifications to the predictive power of the context should alter the expression of habits. The following experiments investigated how changes to the relationship between the training context and performance of the response, either by changing the context during testing or by exposing animals to the context alone, without the response lever present, impacted behavioural control during a devaluation test. We found evidence that the training context is important for the expression of habits; testing animals in a different context than where they were trained resulted in increased goal-directed control (Experiment 1). Furthermore, context alone exposure also increased goal-directed control with animals that received context alone exposure showing stronger devaluation effects, whether the context alone exposure happened on the last day of training (Experiment 2) or throughout training (Experiment 3). These findings are consistent with prior reports that the training context is important for the expression of habits and extends these findings by using sensory-specific satiety as a means for devaluation and by using context alone exposure to alter behavioural control.

Random interval schedule of reinforcement influences punishment resistance for cocaine in rats

In an animal model of compulsive drug use, a subset of rats continues to self-administer cocaine despite footshock consequences and is considered punishment resistant. We recently found that punishment resistance is associated with habits that persist under conditions that typically encourage a transition to goal-directed control. Given that random ratio (RR) and random interval (RI) schedules of reinforcement influence whether responding is goal-directed or habitual, we investigated the influence of these schedules on punishment resistance for cocaine or food. Male and female Sprague Dawley rats were trained to self-administer either intravenous cocaine or food pellets on a seeking-taking chained schedule of reinforcement, with the seeking lever requiring completion of either an RR20 or RI60 schedule. Rats were then given four days of punishment testing with footshock administered at the completion of seeking on a random one-third of trials. For cocaine-trained rats, the RI60 schedule led to greater punishment resistance (i.e., more trials completed) than the RR20 schedule in males and females. For food-trained rats, the RI60 schedule led to greater punishment resistance (i.e., higher reward rates) than the RR20 schedule in female rats, although male rats showed punishment resistance on both RR20 and RI60 schedules. For both cocaine and food, we found that seeking responses were suppressed to a greater degree than reward rate with the RI60 schedule, whereas response rate and reward rate were equally suppressed with the RR20 schedule. This dissociation between punishment effects on reward rate and response rate with the RI60 schedule can be explained by the nonlinear relation between these variables on RI schedules, but it does not account for the enhanced resistance to punishment. Overall, the results show greater punishment resistance with the RI60 schedule as compared to the RR20 schedule, indicating that schedules of reinforcement are an influencing factor on resistance to negative consequences.

The nature of training in flavor preference learning determines the underlying associative structure

Pairing a palatable flavor (US) with an initially neutral flavor cue (CS) results in an acquired conditioned preference for the latter. Two main associations have been proposed to explain the acquisition of flavor preferences: Flavor-Flavor and Flavor-Nutrient learning. Although the hedonic reaction triggered by US consumption has also been suggested as a possible additional component underlying acquired flavor preference, this issue has received little attention. Here we explored whether the amount of training to the CS-US compound can favor the formation of a Flavor-Hedonic reaction association using rats as subjects and sucrose as the US. We expected that the more exposure to the CS-US compound, the stronger the S-R type association. Since S-R associations are not sensitive to devaluation procedures, we used a Sensory-Specific Satiety procedure to devalue the US after conditioning and then measured preferences for the CS. On Experiment 1 with a short restrictive training (classic procedure), preference for the CS was decreased after devaluation of the US compared to the control condition. On Experiment 2, with short unrestrictive training, preference for the CS was again weakened. Experiment 3 with a long unrestrictive training, rats expressed preference for the CS regardless of the devaluation procedure. These results suggest that, as with an instrumental paradigm, extensive training in flavor preference learning undermines the US devaluation effect.

Punishment resistance for cocaine is associated with inflexible habits in rats

Addiction is characterized by continued drug use despite negative consequences. In an animal model, a subset of rats continues to self-administer cocaine despite footshock consequences, showing punishment resistance. We sought to test the hypothesis that punishment resistance arises from failure to exert goal-directed control over habitual cocaine seeking. While habits are not inherently permanent or maladaptive, continued use of habits under conditions that should encourage goal-directed control makes them maladaptive and inflexible. We trained male and female Sprague Dawley rats on a seeking-taking chained schedule of cocaine self-administration. We then exposed them to four days of punishment testing in which footshock was delivered randomly on one-third of trials. Before and after punishment testing (four days pre-punishment and ≥ four days post-punishment), we assessed whether cocaine seeking was goal-directed or habitual using outcome devaluation via cocaine satiety. We found that punishment resistance was associated with continued use of habits, whereas punishment sensitivity was associated with increased goal-directed control. Although punishment resistance for cocaine was not predicted by habitual responding pre-punishment, it was associated with habitual responding post-punishment. In parallel studies of food self-administration, we similarly observed that punishment resistance was associated with habitual responding post-punishment but not pre-punishment in males, although it was associated with habitual responding both pre- and post-punishment in females, indicating that punishment resistance was predicted by habitual responding in food-seeking females. These findings indicate that punishment resistance is related to habits that have become inflexible and persist under conditions that should encourage a transition to goal-directed behavior.

Action sequence learning, habits, and automaticity in obsessive-compulsive disorder

This study investigates the goal/habit imbalance theory of compulsion in obsessive-compulsive disorder (OCD), which postulates enhanced habit formation, increased automaticity, and impaired goal/habit arbitration. It directly tests these hypotheses using newly developed behavioral tasks. First, OCD patients and healthy participants were trained daily for a month using a smartphone app to perform chunked action sequences. Despite similar procedural learning and attainment of habitual performance (measured by an objective automaticity criterion) by both groups, OCD patients self-reported higher subjective habitual tendencies via a recently developed questionnaire. Subsequently, in a re-evaluation task assessing choices between established automatic and novel goal-directed actions, both groups were sensitive to re-evaluation based on monetary feedback. However, OCD patients, especially those with higher compulsive symptoms and habitual tendencies, showed a clear preference for trained/habitual sequences when choices were based on physical effort, possibly due to their higher attributed intrinsic value. These patients also used the habit-training app more extensively and reported symptom relief post-study. The tendency to attribute higher intrinsic value to familiar actions may be a potential mechanism leading to compulsions and an important addition to the goal/habit imbalance hypothesis in OCD. We also highlight the potential of smartphone app training as a habit reversal therapeutic tool.

A novel free-operant framework enables experimental habit induction in humans

Habits are a prominent feature of both adaptive and maladaptive behavior. Yet, despite substantial research efforts, there are currently no well-established experimental procedures for habit induction in humans. It is likely that laboratory experimental settings, as well as the session-based structure typically used in controlled experiments (also outside the lab), impose serious constraints on studying habits and other effects that are sensitive to context, motivation, and training duration and frequency. To overcome these challenges, we devised a unique real-world free-operant task structure, implemented through a novel smartphone application, whereby participants could freely enter the app (24 hours a day, 7 days a week) to win rewards. This procedure is free of typical laboratory constraints, yet well controlled. Using the canonical sensitivity to outcome devaluation criterion, we successfully demonstrated habit formation as a function of training duration, a long-standing challenge in the field. Additionally, we show a positive relationship between multiple facets of engagement/motivation and goal-directedness. We suggest that our novel paradigm can be used to study the neurobehavioral and psychological mechanism underlying habits in humans. Moreover, the real-world free-operant framework can potentially be used to examine other instrumental behavior-related questions, with greater face validity in naturalistic conditions.

Cold executive function processes and their hot analogs in schizotypy

OBJECTIVE

To examine cold (based on logical reasoning) versus hot (having emotional components) executive function processes in groups with high individual schizotypal traits.

METHOD

Two-hundred and forty-seven participants were administered the Schizotypal Personality Questionnaire and were allocated into schizotypal (cognitive-perceptual, paranoid, negative, disorganized) or control groups according to pre-specified criteria. Participants were also administered a battery of tasks examining working memory, complex selective attention, response inhibition, decision-making and fluid intelligence and their affective counterparts. The outcome measures of each task were reduced to one composite variable thus formulating five cold and five hot cognitive domains. Between-group differences in the cognitive domains were examined with repeated measures analyses of covariance.

RESULTS

For working memory, the control and the cognitive-perceptual groups outperformed negative schizotypes, while for affective working memory controls outperformed the disorganized group. Controls also scored higher compared with the disorganized group in complex selective attention, while both the control and the cognitive-perceptual groups outperformed negative schizotypes in complex affective selective attention. Negative schizotypes also had striking difficulties in response inhibition, as they scored lower compared with all other groups. Despite the lack of differences in fluid intelligence, controls scored higher compared with all schizotypal groups (except from cognitive-perceptual schizotypes) in emotional intelligence; the latter group reported higher emotional intelligence compared with negative schizotypes.

CONCLUSION

Results indicate that there is no categorical association between the different schizotypal dimensions with solely cold or hot executive function processes and support impoverished emotional intelligence as a core feature of schizotypy.

A hippocampo-cortical pathway detects changes in the validity of an action as a predictor of reward

Much research has been dedicated to understanding the psychological and neural bases of goal-directed action, yet the relationship between context and goal-directed action is not well understood. Here, we used excitotoxic lesions, chemogenetics, and circuit-specific manipulations to demonstrate the role of the ventral hippocampus (vHPC) in contextual learning that supports sensitivity to action-outcome contingencies, a hallmark of goal-directed action. We found that chemogenetic inhibition of the ventral, but not dorsal, hippocampus attenuated sensitivity to instrumental contingency degradation. We then tested the hypothesis that this deficit was due to an inability to discern the relative validity of the action compared with the context as a predictor of reward. Using latent inhibition and Pavlovian context conditioning, we confirm that degradation of action-outcome contingencies relies on intact context-outcome learning and show that this learning is dependent on vHPC. Finally, we show that chemogenetic inhibition of vHPC terminals in the medial prefrontal cortex also impairs both instrumental contingency degradation and context-outcome learning. These results implicate a hippocampo-cortical pathway in adapting to changes in instrumental contingencies and indicate that the psychological basis of this deficit is an inability to learn the predictive value of the context. Our findings contribute to a broader understanding of the neural bases of goal-directed action and its contextual regulation.

The behavioral origins of phylogenic responses and ontogenic habits

An examination of innate behavior and its possible origins suggests parallels with the formation of habitual behavior. Inflexible but adaptive responses-innate reflexive behavior, Pavlovian conditioned responses, and operant habits-may have evolved from variable behavior in phylogeny and ontogeny. This form of "plasticity-first" scientific narrative was unpopular post-Darwin but has recently gained credibility in evolutionary biology. The present article seeks to identify originating events and contingencies contributing to such inflexible but adaptive behavior at both phylogenic and ontogenic levels of selection. In ontogeny, the development of inflexible performance (i.e., habit) from variable operant behavior is reminiscent of the genetic accommodation of initially variable phylogenic traits. The effects characteristic of habit (e.g., unresponsiveness to reinforcer devaluation) are explicable as the result of a conflict between behaviors at distinct levels of selection. The present interpretation validates the practice of seeking hard analogies between evolutionary biology and operant behavior. Finding such parallels implies the validity of a claim that organismal behavior, both innate and learned, is a product of selection by consequences. A complete and coherent account of organismal behavior may ultimately focus on functional selective histories in much the same way evolutionary biology does with its subject matter.

The temporal structure of goal-directed and habitual operant behavior

Operant behavior can reflect the influence of goal-directed and habitual processes. These can be distinguished by changes to response rate following devaluation of the reinforcing outcome. Whether a response is goal directed or habitual depends on whether devaluation affects response rate. Response rate can be decomposed into frequencies of bouts and pauses by analyzing the distribution of interresponse times. This study sought to characterize goal-directed and habitual behaviors in terms of bout-initiation rate, within-bout response rate, bout length, and bout duration. Data were taken from three published studies that compared sensitivity to devaluation following brief and extended training with variable-interval schedules. Analyses focused on goal-directed and habitual responding, a comparison of a habitual response to a similarly trained response that had been converted back to goal-directed status after a surprising event, and a demonstration of contextual control of habit and goal direction in the same subjects. Across experiments and despite responses being clearly distinguished as goal directed and habitual by total response rate, analyses of bout-initiation rate, within-bout rate, bout length, and bout duration did not reveal a pattern that distinguished goal-directed from habitual responding.

Making and breaking habits: Revisiting the definitions and behavioral factors that influence habits in animals

Habits have garnered significant interest in studies of associative learning and maladaptive behavior. However, habit research has faced scrutiny and challenges related to the definitions and methods. Differences in the conceptualizations of habits between animal and human studies create difficulties for translational research. Here, we review the definitions and commonly used methods for studying habits in animals and humans and discuss potential alternative ways to assess habits, such as automaticity. To better understand habits, we then focus on the behavioral factors that have been shown to make or break habits in animals, as well as potential mechanisms underlying the influence of these factors. We discuss the evidence that habitual and goal-directed systems learn in parallel and that they seem to interact in competitive and cooperative manners. Finally, we draw parallels between habitual responding and compulsive drug seeking in animals to delineate the similarities and differences in these behaviors.

Habit and persistence

Voluntary behaviors (operants) can come in two varieties: Goal-directed actions, which are emitted based on the remembered value of the reinforcer, and habits, which are evoked by antecedent cues and performed without the reinforcer's value in active memory. The two are perhaps most clearly distinguished with the reinforcer-devaluation test: Goal-directed actions are suppressed when the reinforcer is separately devalued and responding is tested in extinction, and habitual behaviors are not. But what is the function of habit learning? Habits are often thought to be strong and unusually persistent. The present selective review examines this idea by asking whether habits identified by the reinforcer-devaluation test are more resistant to extinction, resistant to the effects of other contingency change, vulnerable to relapse, resistant to the weakening effects of context change, or permanently in place once they are learned. Surprisingly little evidence supports the idea that habits are permanent or more persistent. Habits are more context-specific than goal-directed actions are. Methods that make behavior persistent do not necessarily work by encouraging habit. The function of habit learning may not be to make a behavior strong or more persistent but to make it automatic and efficient in a particular context.

Stimulus control of habits: Evidence for both stimulus specificity and devaluation insensitivity in a dual-response task

Goal-directed and habitual actions are clearly defined by their associative relations. Whereas goal-directed control can be confirmed via tests of outcome devaluation and contingency-degradation sensitivity, a comparable criterion for positively detecting habits has not been established. To confirm habitual responding, a test of control by the stimulus-response association is required while also ruling out goal-directed control. Here we describe an approach to developing such a test in rats using two discriminative stimuli that set the occasion for two different responses that then earn the same outcome. Performance was insensitive to outcome devaluation and showed stimulus-response specificity, indicative of stimulus-controlled behavior. The reliance of stimulus-response associations was further supported by a lack of sensitivity during the single extinction test session used here. These results demonstrate that two concurrently trained responses can come under habitual control when they share a common outcome. By reducing the ability of one stimulus to signal its corresponding response-outcome association, we found evidence for goal-directed control that can be dissociated from habits. Overall, these experiments provide evidence that tests assessing specific stimulus-response associations can be used to investigate habits.

A molar view of goal direction and habit

When we treat behavior within an evolutionary framework and as temporally extended flow, two fundamental questions arise: (a) What is an organism? and (b) What is behavior? An organism is a process that stays intact by constantly exchanging energy with the environment. It takes in resources and puts out waste. The behavior of an organism consists of those process parts of the organism process that make up the exchange. These activities serve the function of reproducing, which generally depends on surviving. Surviving and reproducing depend on responding to phylogenetically important events (PIEs). A PIE induces activities that enhance or mitigate the PIE. Organisms respond not only to a PIE but also to events that covary with the PIE. Both activities and environmental features may covary with a PIE. When either type of covariance is introduced to an organism, behavior adapts over time. The early stages of adaptation constitute what researchers call "goal direction," and the later stages constitute what researchers call "habit." Behavior and environment constitute a dynamic system, and manipulations of the covariances and environmental features of the system allow many experimental interventions. This molar approach allows experiments on goal direction and habit to be understood without appeal to everyday mentalistic terms.

A neural autopilot theory of habit: Evidence from consumer purchases and social media use

This article applies a two-process "neural autopilot" model to field data. The autopilot model hypothesizes that habitual choice occurs when the reward from a behavior has low numerical "doubt" (i.e., reward prediction errors are small). The model toggles between repeating a previous choice (habit) when doubt is low and making a goal-directed choice when doubt is high. The model has ingredients established in animal learning and cognitive neuroscience and is simple enough to make nonobvious predictions. In two empirical applications, we fit the model to field data on purchases of canned tuna and posting on the Chinese social media site Weibo. This style of modeling is called "structural" because there is a theoretical model of how different variables influence choices by agents (the "structure"), which tightly restricts how hidden variables lead to observed choices. There is empirical support for the model, more strongly for tuna purchases than for Weibo posting, relative to a baseline "reduced-form" model in which current choices are correlated with past choices without a mechanistic (structural) explanation. An interesting set of predictions can also be derived about how consumers react to different kinds of changes in prices and qualities of goods (this is called "counterfactual analysis").

Multiple Memory Subsystems: Reconsidering Memory in the Mind and Brain

The multiple-memory-systems framework-that distinct types of memory are supported by distinct brain systems-has guided learning and memory research for decades. However, recent work challenges the one-to-one mapping between brain structures and memory types central to this taxonomy, with key memory-related structures supporting multiple functions across substructures. Here we integrate cross-species findings in the hippocampus, striatum, and amygdala to propose an updated framework of multiple memory subsystems (MMSS). We provide evidence for two organizational principles of the MMSS theory: First, opposing memory representations are colocated in the same brain structures; second, parallel memory representations are supported by distinct structures. We discuss why this burgeoning framework has the potential to provide a useful revision of classic theories of long-term memory, what evidence is needed to further validate the framework, and how this novel perspective on memory organization may guide future research.

Measuring context-response associations that drive habits

People achieve important life outcomes of health, financial security, and productivity by repeating operant behavior. To identify whether such operants reflect goal pursuit or habit, the present research introduces a new paradigm that yields objective measures of learning and controls for the motivations of goal pursuit. In two experiments, participants practiced a sequential task of making sushi and then completed a test of the strength of cue-response (habit) associations in memory. Finally, they repeated the sushi task without instructions while under cognitive load (designed to impede deliberation about goals). As predicted, greater task practice yielded stronger cue-response associations, which in turn promoted task success. Practice did not improve performance by enhancing goal intentions or other task motivations. We conclude that repetition facilitates performance by creating mental associations that automatically activate practiced, habitual responses upon perception of recurring context cues.

Theory of reinforcement schedules

The three principles of reinforcement are (1) events such as incentives and reinforcers increase the activity of an organism; (2) that activity is bounded by competition from other responses; and (3) animals approach incentives and their signs, guided by their temporal and physical conditions, together called the "contingencies of reinforcement." Mathematical models of each of these principles comprised mathematical principles of reinforcement (MPR; Killeen, 1994). Over the ensuing decades, MPR was extended to new experimental contexts. This article reviews the basic theory and its extensions to satiation, warm-up, extinction, sign tracking, pausing, and sequential control in progressive-ratio and multiple schedules. In the latter cases, a single equation balancing target and competing responses governs behavioral contrast and behavioral momentum. Momentum is intrinsic in the fundamental equations, as behavior unspools more slowly from highly aroused responses conditioned by higher rates of incitement than it does from responses from leaner contexts. Habits are responses that have accrued substantial behavioral momentum. Operant responses, being predictors of reinforcement, are approached by making them: The sight and feel of a paw on a lever is approached by placing paw on lever, as attempted for any sign of reinforcement. Behavior in concurrent schedules is governed by approach to momentarily richer patches (melioration). Applications of MPR in behavioral pharmacology and delay discounting are noted.

Contribution of dorsal versus ventral hippocampus to the hierarchical modulation of goal-directed actions in rats

Adaptive behaviour often necessitates that animals learn about events in a manner that is specific to a particular context or environment. These hierarchical organisations allow the animal to decide which action is the most appropriate when faced with ambiguous or conflicting possibilities. This study examined the role of hippocampus in enabling animals to use the context to guide action selection. We used a hierarchical instrumental outcome devaluation task in which male rats learn that the context provides information about the unique action-outcome relations that are in effect. We first confirmed that rats encode and use hierarchical context-(action-outcome) relations. We then show that chemogenetic inhibition of ventral hippocampus impairs both the encoding and retrieval of these associations, while inhibition of dorsal hippocampus impairs only the retrieval. Importantly, neither dorsal nor ventral hippocampus was required for goal-directed behaviour per se as these impairments only emerged when rats were forced to use the context to identify the current action-outcome relationships. These findings are discussed with respect to the role of the hippocampus and its broader circuitry in the contextual modulation of goal-directed behaviour and the importance of hierarchical associations in flexible behaviour.

A comparison of renewal, spontaneous recovery, and reacquisition after punishment and extinction

Punishment and extinction are both effective methods of reducing instrumental responding and may involve similar learning mechanisms. To characterize the similarities and differences between them, we examined three well-established recovery or "relapse" effects -renewal, spontaneous recovery, and reacquisition - following either punishment or extinction of an instrumental response. In Experiment 1a, both punished and extinguished responses renewed to similar degrees following a context change at test (ABA renewal). In Experiment 1b, responding spontaneously recovered to similar degrees following punishment or extinction. In Experiment 2, responding was rapidly reacquired when the response was reinforced again following extinction but not following punishment, as predicted by the idea that the reinforcer delivered in reacquisition is part of the context of punishment, but not extinction. The results collectively suggest that both punishment and extinction produce similar context-dependent retroactive interference effects. More broadly, they also suggest that punished and extinguished responses may be equally likely to return following a change of context despite the intuition that punishment might provide a more extreme and effective means of suppressing behavior. To our knowledge, this is the first direct behavioral comparison of response recovery after punishment and extinction within individual experiments.

Punishment resistance for cocaine is associated with inflexible habits in rats

Addiction is characterized by continued drug use despite negative consequences. In an animal model, a subset of rats continues to self-administer cocaine despite footshock consequences, showing punishment resistance. We sought to test the hypothesis that punishment resistance arises from failure to exert goal-directed control over habitual cocaine seeking. While habits are not inherently permanent or maladaptive, continued use of habits under conditions that should encourage goal-directed control makes them maladaptive and inflexible. We trained male and female Sprague Dawley rats on a seeking-taking chained schedule of cocaine self-administration (2 h/day). We then exposed them to 4 days of punishment testing, in which footshock (0.4 mA, 0.3 s) was delivered randomly on one-third of trials, immediately following completion of seeking and prior to extension of the taking lever. Before and after punishment testing (4 days pre-punishment and ≥4 days post-punishment), we assessed whether cocaine seeking was goal-directed or habitual using outcome devaluation via cocaine satiety. We found that punishment resistance was associated with continued use of habits, whereas punishment sensitivity was associated with increased goal-directed control. Although punishment resistance was not predicted by habitual responding pre-punishment, it was associated with habitual responding post-punishment. In parallel studies of food self-administration, we similarly observed that punishment resistance was associated with habitual responding post-punishment but not pre-punishment. These findings indicate that punishment resistance is related to habits that have become inflexible and persist under conditions that should encourage a transition to goal-directed behavior.

Pathways to the persistence of drug use despite its adverse consequences

The persistence of drug taking despite its adverse consequences plays a central role in the presentation, diagnosis, and impacts of addiction. Eventual recognition and appraisal of these adverse consequences is central to decisions to reduce or cease use. However, the most appropriate ways of conceptualizing persistence in the face of adverse consequences remain unclear. Here we review evidence that there are at least three pathways to persistent use despite the negative consequences of that use. A cognitive pathway for recognition of adverse consequences, a motivational pathway for valuation of these consequences, and a behavioral pathway for responding to these adverse consequences. These pathways are dynamic, not linear, with multiple possible trajectories between them, and each is sufficient to produce persistence. We describe these pathways, their characteristics, brain cellular and circuit substrates, and we highlight their relevance to different pathways to self- and treatment-guided behavior change.

From helplessness to controllability: toward a neuroscience of resilience

"Learned helplessness" refers to debilitating outcomes, such as passivity and increased fear, that follow an uncontrollable adverse event, but do not when that event is controllable. The original explanation argued that when events are uncontrollable the animal learns that outcomes are independent of its behavior, and that this is the active ingredient in producing the effects. Controllable adverse events, in contrast, fail to produce these outcomes because they lack the active uncontrollability element. Recent work on the neural basis of helplessness, however, takes the opposite view. Prolonged exposure to aversive stimulation per se produces the debilitation by potent activation of serotonergic neurons in the brainstem dorsal raphe nucleus. Debilitation is prevented with an instrumental controlling response, which activates prefrontal circuitry detecting control and subsequently blunting the dorsal raphe nucleus response. Furthermore, learning control alters the prefrontal response to future adverse events, thereby preventing debilitation and producing long-term resiliency. The general implications of these neuroscience findings may apply to psychological therapy and prevention, in particular by suggesting the importance of cognitions and control, rather than habits of control.

Learning to stop responding

Learning to stop responding is an important process that allows behavior to adapt to a changing and variable environment. This article reviews recent research in this laboratory and others that has studied how animals learn to stop responding in operant extinction, punishment, and feature-negative learning. Extinction and punishment are shown to be similar in two fundamental ways. First, the response-suppressing effects of both are highly context-specific. Second, the response-suppressing effects of both can be remarkably response-specific: Inhibition of one response transfers little to other responses. Learning to inhibit the response so specifically may result from the correction of "response error," the difference between the level of responding and what the current reinforcer supports. In contrast, the inhibition of responding that develops in feature-negative learning, where the response is reinforced during one discriminative stimulus (A) but not in a compound of A and stimulus B, is less response-specific: The inhibition of responding by stimulus B transfers and inhibits a second response, especially if the second response has itself been inhibited before. The results thus indicate both response-specific and response-general forms of behavioral inhibition. One possibility is that response-specific inhibition is learned when the circumstances encourage the organism to pay attention to the response-to what it is actually doing-as behavioral suppression is learned.

Does stress consistently favor habits over goal-directed behaviors? Data from two preregistered exact replication studies

Instrumental learning is controlled by two distinct parallel systems: goal-directed (action-outcome) and habitual (stimulus-response) processes. Seminal research by Schwabe and Wolf (2009, 2010) has demonstrated that stress renders behavior more habitual by decreasing goal-directed control. More recent studies yielded equivocal evidence for a stress-induced shift towards habitual responding, yet these studies used different paradigms to evaluate instrumental learning or used different stressors. Here, we performed exact replications of the original studies by exposing participants to an acute stressor either before (cf. Schwabe and Wolf, 2009) or directly after (cf. Schwabe and Wolf, 2010) an instrumental learning phase in which they had learned that distinct actions led to distinct, rewarding food outcomes (i.e., instrumental learning). Then, following an outcome devaluation phase in which one of the food outcomes was consumed until participants were satiated, action-outcome associations were tested in extinction. Despite successful instrumental learning and outcome devaluation and increased subjective and physiological stress levels following stress exposure, the stress and no-stress groups in both replication studies responded indifferently to valued and devalued outcomes. That is, non-stressed participants failed to demonstrate goal-directed behavioral control, thereby rendering the critical test of a shift from goal-directed to habitual control in the stress group inapt. Several reasons for these replication failures are discussed, including the rather indiscriminate devaluation of outcomes that may have contributed to indifferent responding during extinction, which emphasize the need to further our understanding of the boundary conditions in research aimed at demonstrating a stress-induced shift towards habitual control.

Hazardous drinking and alcohol use disorders

Alcohol is one of the most widely consumed psychoactive drugs globally. Hazardous drinking, defined by quantity and frequency of consumption, is associated with acute and chronic morbidity. Alcohol use disorders (AUDs) are psychiatric syndromes characterized by impaired control over drinking and other symptoms. Contemporary aetiological perspectives on AUDs apply a biopsychosocial framework that emphasizes the interplay of genetics, neurobiology, psychology, and an individual's social and societal context. There is strong evidence that AUDs are genetically influenced, but with a complex polygenic architecture. Likewise, there is robust evidence for environmental influences, such as adverse childhood exposures and maladaptive developmental trajectories. Well-established biological and psychological determinants of AUDs include neuroadaptive changes following persistent use, differences in brain structure and function, and motivational determinants including overvaluation of alcohol reinforcement, acute effects of environmental triggers and stress, elevations in multiple facets of impulsivity, and lack of alternative reinforcers. Social factors include bidirectional roles of social networks and sociocultural influences, such as public health control strategies and social determinants of health. An array of evidence-based approaches for reducing alcohol harms are available, including screening, pharmacotherapies, psychological interventions and policy strategies, but are substantially underused. Priorities for the field include translating advances in basic biobehavioural research into novel clinical applications and, in turn, promoting widespread implementation of evidence-based clinical approaches in practice and health-care systems.

Lack of action monitoring as a prerequisite for habitual and chunked behavior: Behavioral and neural correlates

We previously reported the rapid development of habitual behavior in a discrete-trials instrumental task in which lever insertion and retraction act as reward-predictive cues delineating sequence execution. Here we asked whether lever cues or performance variables reflective of skill and automaticity might account for habitual behavior in male rats. Behavior in the discrete-trials habit-promoting task was compared with two task variants lacking the sequence-delineating cues of lever extension and retraction. We find that behavior is under goal-directed control in absence of sequence-delineating cues but not in their presence, and that skilled performance does not predict goal-directed vs. habitual behavior. Neural activity recordings revealed an engagement of dorsolateral striatum and a disengagement of dorsomedial striatum during the sequence execution of the habit-promoting task, specifically. Together, these results indicate that sequence delineation cues promote habit and differential engagement of striatal subregions during instrumental responding, a pattern that may reflect cue-elicited behavioral chunking.

Procedures between training and reactivation influence the destabilization of instrumental sucrose memory

Memory destabilization and reconsolidation is hypothesized to be a fundamental mnemonic process that can underpin memory updating. Instrumental memories have been shown recently to be destabilized following a reactivation session that involves a change in instrumental reward contingency. However, the acquisition and performance of an instrumental response occurs in the presence of the learning of other reward-related memories. This may influence the ability of a given reactivation session to destabilize the previously learned instrumental memory. Here we present a series of experiments in male rats involving an instrumental memory trained on an FR1 schedule over 10 days, and then reactivated in a session that imposed a VR5 schedule of reinforcement. When MK-801 was injected prior to the VR5 reactivation session, it reliably impaired subsequent instrumental performance at test only when the reactivation session occurred 48 h, and not 24 h, after the end of training. The interposition between the end of training and the reactivation session of a context extinction session, an additional VR5 reactivation session, or indeed the simple experience of being handled and injected with vehicle, resulted in MK-801 no longer having an amnestic effect on test performance. While we do not have a clear account for the process and mechanism underpinning this apparent selectivity of the effect of the VR5 session to destabilize the instrumental memory, it does additionally highlight the need for greater understanding of the conditions that facilitate reactivation-induced memory destabilization.

Making habits measurable beyond what they are not: A focus on associative dual-process models

Habits are the subject of intense international research. Under the associative dual-process model the outcome devaluation paradigm has been used extensively to classify behaviours as being either goal-directed (sensitive to shifts in the value of associated outcomes) or habitual (triggered by stimuli without anticipation of consequences). This has proven to be a useful framework for studying the neurobiology of habit and relevance of habits in clinical psychopathology. However, in recent years issues have been raised about this rather narrow definition of habits in comparison to habitual behaviour experienced in the real world. Specifically, defining habits as the absence of goal-directed control, the very specific set-ups required to demonstrate habit experimentally and the lack of direct evidence for habits as stimulus-response behaviours are viewed as problematic. In this review paper we address key critiques that have been raised about habit research within the framework of the associative dual-process model. We then highlight novel research approaches studying different features of habits with methods that expand beyond traditional paradigms.

Pilot testing of an adaptive, individualized inhibitory control training for binge drinking: first evidence on feasibility, acceptance, and efficacy

Background

Deficits in inhibitory control seem to promote habit behavior and therefore play an important role in the development and maintenance of addictive diseases. Although several training approaches have been suggested, there is a considerable lack of knowledge about the best way to improve inhibitory control. Based on a literature review regarding shortcomings of existing trainings, an individualized, adaptive inhibitory control training was developed. We aimed to assess feasibility and acceptance of this training and to provide preliminary results on its efficacy regarding inhibitory control and binge drinking.

Methods

Sixty-one individuals (30 female) with binge drinking behavior were randomly allocated to either an experimental group receiving three sessions of the inhibitory control training or a waitlist control group receiving no training. Before and after the training, the participants performed a Go/NoGo task to assess inhibitory control (commission errors and false reaction time), completed a questionnaire on drinking-related self-control, and reported drinking behavior.

Results

Although the training was feasible and accepted by participants, it did not affect self-control over drinking, inhibitory control or drinking behavior. The relationship between session number and false reaction time was linear for alcohol stimuli, but squared for neutral stimuli.

Conclusion

Although our findings have to be interpreted in the light of some shortcomings, they demonstrate that further research is needed to enhance our understanding of how to improve inhibitory control and which factors might moderate this process.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00426-022-01725-4.

Attentional capture by signals of reward persists following outcome devaluation

Attention, the mechanism that prioritizes stimuli in the environment for further processing, plays an important role in behavioral choice. In the present study, we investigated the automatic orienting of attention to cues that signal reward. Such attentional capture occurs despite negative consequences, and we investigated whether this counterproductive and reflexive behavior would persist following outcome devaluation. Thirsty participants completed a visual search task in which the color of a distractor stimulus in the search display signaled whether participants would earn water or potato chips for making a rapid eye movement to a diamond target, but looking at the colored distractor was punished by omission of the signaled reward. Nevertheless, participants looked at the water-signaling distractor more frequently than the chip-signaling distractor. Half the participants then drank water ad libitum before continuing with the visual search task. Although the water was now significantly less desirable for half of the participants, there was no difference between groups in the tendency for the water-signaling distractor to capture attention. These findings suggest that once established, counterproductive attentional bias to signals of reward persists even when those outcomes are no longer valuable. This suggests a "habit-like" attentional mechanism that prioritizes reward stimuli in the environment for further action, regardless of whether those stimuli are aligned with current goals or currently desired.

Cyclic estrogen and progesterone during instrumental acquisition contributes to habit formation in female rats

Habit formation is thought to involve two parallel processes that are mediated by distinct neural substates: one that suppresses goal-directed behavior, and one that facilitates stimulus-response (S-R) learning, which underscores habitual behavior. In previous studies we showed that habitual responding emerges early during instrumental training in gonadally-intact female, compared to male, rats. The present study aimed to determine the role of ovarian hormones during instrumental acquisition in the transition from goal-directed to habitual behavior in female rats. Ovariectomized (OVX) female rats were given subcutaneous silastic capsules that released low levels of 17-β estradiol (E2) to maintain estrogen receptor availability. Rats were assigned to one of three hormone treatment conditions: no additional hormone replacement (Control group), replacement with high E2 (High E2 group), or replacement with high E2 followed by progesterone (High E2 + P4 group). Hormone replacement occurred twice during acquisition to mimic natural hormone fluctuations. At test, the Control and High E2 groups demonstrated responding that was sensitive to devaluation by lithium chloride-induced illness, indicating goal-directed behavior. In contrast, the High E2 + P4 group exhibited a pattern of devaluation-insensitive, habitual responding, that suggested the suppression of goal-directed processes. In a follow-up experiment, similar procedures were conducted, however during acquisition, OVX rats were given cyclic high E2 plus medroxy-progesterone (MPA), a form of progesterone that does not metabolize to neuroactive metabolites. In this group, goal-directed behavior was observed. These data indicate that habit formation is not facilitated in low estrogen states, nor in the presence of cyclic high E2. However, cyclic high E2, together with progesterone during acquisition, appears to facilitate the early emergence of habitual responding. Furthermore, these data suggest that a neuroactive progesterone metabolite, like allopregnanolone, in combination with high cyclic E2, supports this phenomenon.

New functions of the rodent prelimbic and infralimbic cortex in instrumental behavior

The prelimbic and infralimbic cortices of the rodent medial prefrontal cortex mediate the effects of context and goals on instrumental behavior. Recent work from our laboratory has expanded this understanding. Results have shown that the prelimbic cortex is important for the modulation of instrumental behavior by the context in which the behavior is learned (but not other contexts), with context potentially being broadly defined (to include at least previous behaviors). We have also shown that the infralimbic cortex is important in the expression of extensively-trained instrumental behavior, regardless of whether that behavior is expressed as a stimulus-response habit or a goal-directed action. Some of the most recent data suggest that infralimbic cortex may control the currently active behavioral state (e.g., habit vs. action or acquisition vs. extinction) when two states have been learned. We have also begun to examine prelimbic and infralimbic cortex function as key nodes of discrete circuits and have shown that prelimbic cortex projections to an anterior region of the dorsomedial striatum are important for expression of minimally-trained instrumental behavior. Overall, the use of an associative learning perspective on instrumental learning has allowed the research to provide new perspectives on how these two "cognitive" brain regions contribute to instrumental behavior.