Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action

Neural inhibition as implemented by an actor-critic model involves the human dorsal striatum and ventral tegmental area

Inhibition is implicated across virtually all human experiences. As a trade-off of being very efficient, this executive function is also prone to many errors. Rodent and computational studies show that midbrain regions play crucial roles during errors by sending dopaminergic learning signals to the basal ganglia for behavioural adjustment. However, the parallels between animal and human neural anatomy and function are not determined. We scanned human adults while they performed an fMRI inhibitory task requiring trial-and-error learning. Guided by an actor-critic model, our results implicate the dorsal striatum and the ventral tegmental area as the actor and the critic, respectively. Using a multilevel and dimensional approach, we also demonstrate a link between midbrain and striatum circuit activity, inhibitory performance, and self-reported autistic and obsessive-compulsive subclinical traits.

Adaptation of sequential action benefits from timing variability related to lateral basal ganglia circuitry

Streamlined action sequences must remain flexible should stable contingencies in the environment change. By combining analyses of behavioral structure with a circuit-specific manipulation in mice, we report on a relationship between action timing variability and successful adaptation that relates to post-synaptic targets of primary motor cortical (M1) projections to dorsolateral striatum (DLS). In a two-lever instrumental task, mice formed successful action sequences by, first, establishing action scaffolds and, second, smoothly extending action duration to adapt to increased task requirements. Interruption of DLS neurons in M1 projection territories altered this process, evoking higher-rate actions that were more stereotyped in their timing, reducing opportunities for success. Based on evidence from neuronal tracing experiments, we propose that DLS neurons in M1 projection territories supply action timing variability to facilitate adaptation, a function that may involve additional downstream subcortical processing relating to collateralization of descending motor pathways to multiple basal ganglia centers.

Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts

Active reinforcement learning enables dynamic prediction and control, where one should not only maximize rewards but also minimize costs such as of inference, decisions, actions, and time. For an embodied agent such as a human, decisions are also shaped by physical aspects of actions. Beyond the effects of reward outcomes on learning processes, to what extent can modeling of behavior in a reinforcement-learning task be complicated by other sources of variance in sequential action choices? What of the effects of action bias (for actions per se) and action hysteresis determined by the history of actions chosen previously? The present study addressed these questions with incremental assembly of models for the sequential choice data from a task with hierarchical structure for additional complexity in learning. With systematic comparison and falsification of computational models, human choices were tested for signatures of parallel modules representing not only an enhanced form of generalized reinforcement learning but also action bias and hysteresis. We found evidence for substantial differences in bias and hysteresis across participants-even comparable in magnitude to the individual differences in learning. Individuals who did not learn well revealed the greatest biases, but those who did learn accurately were also significantly biased. The direction of hysteresis varied among individuals as repetition or, more commonly, alternation biases persisting from multiple previous actions. Considering that these actions were button presses with trivial motor demands, the idiosyncratic forces biasing sequences of action choices were robust enough to suggest ubiquity across individuals and across tasks requiring various actions. In light of how bias and hysteresis function as a heuristic for efficient control that adapts to uncertainty or low motivation by minimizing the cost of effort, these phenomena broaden the consilient theory of a mixture of experts to encompass a mixture of expert and nonexpert controllers of behavior.

Stimulus awareness is necessary for both instrumental learning and instrumental responding to previously learned stimuli

Instrumental conditioning is a crucial part of adaptive behaviour, allowing agents to selectively interact with stimuli in their environment. Recent evidence suggests that instrumental conditioning cannot proceed without stimulus awareness. However, whether accurate unconscious instrumental responding can emerge from consciously acquired knowledge of the stimulus-action-outcome contingencies is unknown. We studied this question using instrumental trace conditioning, where participants learned to make approach/avoid decisions in two within-subject modes: conscious (stimuli in plain view) and unconscious (visually masked). Both tasks were followed by an unconscious-only instrumental performance task. We show that even when the contingencies are reliably learned in the conscious mode, participants fail to act upon them in the unconscious responding task. We also replicate the previous results that no instrumental learning occurs in the unconscious mode. Consequently, the absence of stimulus awareness not only precludes instrumental conditioning, but also precludes any kind of instrumental responding to already known stimuli. This suggests that instrumental behaviour is entirely supported by conscious awareness of the world, and corroborates the proposals that consciousness may be necessary for adaptive behaviours requiring selective action.

The influence of pitcher handedness on pitch-calling behavior: Insights from fMRI study on baseball umpires

This functional magnetic resonance imaging study delves into the impact of experience and pitcher handedness on the pitch-calling behavior of baseball umpires. Expert and intermediate umpires were asked to make ball/strike calls on videotaped pitches of left- and right-handed pitchers and rate their certainty for the call while undergoing scanning. Behavioral results replicated previous findings that expert umpires were more certain but not more accurate or quicker than intermediate umpires, suggesting that, as sports officials, umpires may learn to project confidence to maintain control of the game. At the neural level, expert umpires exhibited more extensive and pronounced activations within the action observation network, dorsal striatum, and cerebellum. These heightened neural responses were probably associated with their enhanced visual processing abilities for pitching action and ball trajectory, honed over years of officiating. Notably, both expert and intermediate umpires exhibited decreased accuracy when judging pitches from left-handed pitchers compared to right-handed ones. These challenges in accuracy corresponded with weaker neural activations in the aforementioned brain regions, implying difficulties in processing specific visual details of the rarely encountered left-handed pitchers. Moreover, slightly longer reaction times and reduced uncertainty were observed particularly for left-handed ball pitches, as revealed by lower activation in the right premotor cortex, highlighting issues with predictive processing. In summary, our findings shed light on the influence of pitcher handedness on the pitch-calling behavior of baseball umpires and extend the current understanding of the perceptual and decision-making behavior of sports officials.

Comparative Roles of the Caudate and Putamen in the Serial Order of Behavior: Effects of Striatal Glutamate Receptor Blockade on Variable versus Fixed Spatial Self-Ordered Sequencing in Marmosets

Self-ordered sequencing is an important executive function involving planning and executing a series of steps to achieve goal-directed outcomes. The lateral frontal cortex is implicated in this behavior, but downstream striatal outputs remain relatively unexplored. We trained marmosets on a three-stimulus self-ordered spatial sequencing task using a touch-sensitive screen to explore the role of the caudate nucleus and putamen in random and fixed response arrays. By transiently blocking glutamatergic inputs to these regions, using intrastriatal CNQX microinfusions, we demonstrate that the caudate and putamen are both required for, but contribute differently to, flexible and fixed sequencing. CNQX into either the caudate or putamen impaired variable array accuracy, and infusions into both simultaneously elicited greater impairment. We demonstrated that continuous perseverative errors in variable array were caused by putamen infusions, likely due to interference with the putamen's established role in monitoring motor feedback. Caudate infusions, however, did not affect continuous errors, but did cause an upward trend in recurrent perseveration, possibly reflecting interference with the caudate's established role in spatial working memory and goal-directed planning. In contrast to variable array performance, while both caudate and putamen infusions impaired fixed array responding, the combined effects were not additive, suggesting possible competing roles. Infusions into either region individually, but not simultaneously, led to continuous perseveration. Recurrent perseveration in fixed arrays was caused by putamen, but not caudate, infusions. These results are consistent overall with a role of caudate in planning and flexible responding and the putamen in more rigid habitual or automatic responding.

Characterizing Human Habits in the Lab

Habits pose a fundamental puzzle for those aiming to understand human behavior. They pervade our everyday lives and dominate some forms of psychopathology but are extremely hard to elicit in the lab. In this Registered Report, we developed novel experimental paradigms grounded in computational models, which suggest that habit strength should be proportional to the frequency of behavior and, in contrast to previous research, independent of value. Specifically, we manipulated how often participants performed responses in two tasks varying action repetition without, or separately from, variations in value. Moreover, we asked how this frequency-based habitization related to value-based operationalizations of habit and self-reported propensities for habitual behavior in real life. We find that choice frequency during training increases habit strength at test and that this form of habit shows little relation to value-based operationalizations of habit. Our findings empirically ground a novel perspective on the constituents of habits and suggest that habits may arise in the absence of external reinforcement. We further find no evidence for an overlap between different experimental approaches to measuring habits and no associations with self-reported real-life habits. Thus, our findings call for a rigorous reassessment of our understanding and measurement of human habitual behavior in the lab.

Intrinsic Functional Connectivity between the Anterior Insular and Retrosplenial Cortex as a Moderator and Consequence of Cocaine Self-Administration in Rats

While functional brain imaging studies in humans suggest that chronic cocaine use alters functional connectivity (FC) within and between key large-scale brain networks, including the default mode network (DMN), the salience network (SN), and the central executive network (CEN), cross-sectional studies in humans are challenging to obtain brain FC prior to cocaine use. Such information is critical to reveal the relationship between individual's brain FC and the subsequent development of cocaine dependence and brain changes during abstinence. Here, we performed a longitudinal study examining functional magnetic resonance imaging (fMRI) data in male rats (n = 7), acquired before cocaine self-administration (baseline), on 1 d of abstinence following 10 d of cocaine self-administration, and again after 30 d of experimenter-imposed abstinence. Using repeated-measures analysis of variance (ANOVA) with network-based statistics (NBS), significant connectivity changes were found between anterior insular cortex (AI) of the SN, retrosplenial cortex (RSC) of the DMN, somatosensory cortex, and caudate-putamen (CPu), with AI-RSC FC showing the most robust changes between baseline and 1 d of abstinence. Additionally, the level of escalated cocaine intake is associated with AI-RSC and AI-CPu FC changes between 1 d and 30 d of abstinence; further, the subjects' AI-RSC FC prior to cocaine intake is a significant moderator for the AI-RSC changes during abstinence. These results provide novel insights into the roles of AI-RSC FC before and after cocaine intake and suggest this circuit to be a potential target to modulate large-scale network and associated behavioral changes in cocaine use disorders.

Psilocybin-induced changes in neural reactivity to alcohol and emotional cues in patients with alcohol use disorder: an fMRI pilot study

This pilot study investigated psilocybin-induced changes in neural reactivity to alcohol and emotional cues in patients with alcohol use disorder (AUD). Participants were recruited from a phase II, randomized, double-blind, placebo-controlled clinical trial investigating psilocybin-assisted therapy (PAT) for the treatment of AUD (NCT02061293). Eleven adult patients completed task-based blood oxygen dependent functional magnetic resonance imaging (fMRI) approximately 3 days before and 2 days after receiving 25 mg of psilocybin (n = 5) or 50 mg of diphenhydramine (n = 6). Visual alcohol and emotionally valanced (positive, negative, or neutral) stimuli were presented in block design. Across both alcohol and emotional cues, psilocybin increased activity in the medial and lateral prefrontal cortex (PFC) and left caudate, and decreased activity in the insular, motor, temporal, parietal, and occipital cortices, and cerebellum. Unique to negative cues, psilocybin increased supramarginal gyrus activity; unique to positive cues, psilocybin increased right hippocampus activity and decreased left hippocampus activity. Greater PFC and caudate engagement and concomitant insula, motor, and cerebellar disengagement suggests enhanced goal-directed action, improved emotional regulation, and diminished craving. The robust changes in brain activity observed in this pilot study warrant larger neuroimaging studies to elucidate neural mechanisms of PAT.Trial registration: NCT02061293.

Greater reliance on model-free learning in adolescent anorexia nervosa: An examination of dual-system reinforcement learning

Alterations in learning and decision-making systems are thought to contribute to core features of anorexia nervosa (AN), a psychiatric disorder characterized by persistent dietary restriction and weight loss. Instrumental learning theory identifies a dual-system of habit and goal-directed decision-making, linked to model-free and model-based reinforcement learning algorithms. Difficulty arbitrating between these systems, resulting in an over-reliance on one strategy over the other, has been implicated in compulsivity and extreme goal pursuit, both of which are observed in AN. Characterizing alterations in model-free and model-based systems, and their neural correlates, in AN may clarify mechanisms contributing to symptom heterogeneity (e.g., binge/purge symptoms). This study tested whether adolescents with restricting AN (AN-R; n = 36) and binge/purge AN (AN-BP; n = 20) differentially utilized model-based and model-free learning systems compared to a healthy control group (HC; n = 28) during a Markov two-step decision-making task under conditions of reward and punishment. Associations between model-free and model-based learning and resting-state functional connectivity between neural regions of interest, including orbitofrontal cortex (OFC), nucleus accumbens (NAcc), putamen, and sensory motor cortex (SMC) were examined. AN-R showed higher utilization of model-free learning compared to HC for reward, but attenuated model-free and model-based learning for punishment. In AN-R only, higher model-based learning was associated with stronger OFC-to-left NAcc functional connectivity, regions linked to goal-directed behavior. Greater utilization of model-free learning for reward in AN-R may differentiate this group, particularly during adolescence, and facilitate dietary restriction by prioritizing habitual control in rewarding contexts.

The DDHD2-STXBP1 interaction mediates long-term memory via generation of saturated free fatty acids

The phospholipid and free fatty acid (FFA) composition of neuronal membranes plays a crucial role in learning and memory, but the mechanisms through which neuronal activity affects the brain's lipid landscape remain largely unexplored. The levels of saturated FFAs, particularly of myristic acid (C14:0), strongly increase during neuronal stimulation and memory acquisition, suggesting the involvement of phospholipase A1 (PLA1) activity in synaptic plasticity. Here, we show that genetic ablation of the PLA1 isoform DDHD2 in mice dramatically reduces saturated FFA responses to memory acquisition across the brain. Furthermore, DDHD2 loss also decreases memory performance in reward-based learning and spatial memory models prior to the development of neuromuscular deficits that mirror human spastic paraplegia. Via pulldown-mass spectrometry analyses, we find that DDHD2 binds to the key synaptic protein STXBP1. Using STXBP1/2 knockout neurosecretory cells and a haploinsufficient STXBP1+/- mouse model of human early infantile encephalopathy associated with intellectual disability and motor dysfunction, we show that STXBP1 controls targeting of DDHD2 to the plasma membrane and generation of saturated FFAs in the brain. These findings suggest key roles for DDHD2 and STXBP1 in lipid metabolism and in the processes of synaptic plasticity, learning, and memory.

Arbitration between model-free and model-based control is not affected by transient changes in tonic serotonin levels

BACKGROUND

Serotonin has been suggested to modulate decision-making by influencing the arbitration between model-based and model-free control. Disruptions in these control mechanisms are involved in mental disorders such as drug dependence or obsessive-compulsive disorder. While previous reports indicate that lower brain serotonin levels reduce model-based control, it remains unknown whether increases in serotonergic availability might thus increase model-based control. Moreover, the mediating neural mechanisms have not been studied yet.

AIM

The first aim of this study was to investigate whether increased/decreased tonic serotonin levels affect the arbitration between model-free and model-based control. Second, we aimed to identify the underlying neural processes.

METHODS

We employed a sequential two-stage Markov decision-task and measured brain responses during functional magnetic resonance imaging in 98 participants in a randomized, double-blind cross-over within-subject design. To investigate the influence of serotonin on the balance between model-free and model-based control, we used a tryptophan intervention with three intervention levels (loading, balanced, depletion). We hypothesized that model-based behaviour would increase with higher serotonin levels.

RESULTS

We found evidence that neither model-free nor model-based control were affected by changes in tonic serotonin levels. Furthermore, our tryptophan intervention did not elicit relevant changes in Blood-Oxygenation-Level Dependent activity.

The Hippocampus in Pigeons Contributes to the Model-Based Valuation and the Relationship between Temporal Context States

Model-based decision-making guides organism behavior by the representation of the relationships between different states. Previous studies have shown that the mammalian hippocampus (Hp) plays a key role in learning the structure of relationships among experiences. However, the hippocampal neural mechanisms of birds for model-based learning have rarely been reported. Here, we trained six pigeons to perform a two-step task and explore whether their Hp contributes to model-based learning. Behavioral performance and hippocampal multi-channel local field potentials (LFPs) were recorded during the task. We estimated the subjective values using a reinforcement learning model dynamically fitted to the pigeon's choice of behavior. The results show that the model-based learner can capture the behavioral choices of pigeons well throughout the learning process. Neural analysis indicated that high-frequency (12-100 Hz) power in Hp represented the temporal context states. Moreover, dynamic correlation and decoding results provided further support for the high-frequency dependence of model-based valuations. In addition, we observed a significant increase in hippocampal neural similarity at the low-frequency band (1-12 Hz) for common temporal context states after learning. Overall, our findings suggest that pigeons use model-based inferences to learn multi-step tasks, and multiple LFP frequency bands collaboratively contribute to model-based learning. Specifically, the high-frequency (12-100 Hz) oscillations represent model-based valuations, while the low-frequency (1-12 Hz) neural similarity is influenced by the relationship between temporal context states. These results contribute to our understanding of the neural mechanisms underlying model-based learning and broaden the scope of hippocampal contributions to avian behavior.

Neural correlates of prediction error in patients with schizophrenia: evidence from an fMRI meta-analysis

Abnormal processes of learning from prediction errors, i.e. the discrepancies between expectations and outcomes, are thought to underlie motivational impairments in schizophrenia. Although dopaminergic abnormalities in the mesocorticolimbic reward circuit have been found in patients with schizophrenia, the pathway through which prediction error signals are processed in schizophrenia has yet to be elucidated. To determine the neural correlates of prediction error processing in schizophrenia, we conducted a meta-analysis of whole-brain neuroimaging studies that investigated prediction error signal processing in schizophrenia patients and healthy controls. A total of 14 studies (324 schizophrenia patients and 348 healthy controls) using the reinforcement learning paradigm were included. Our meta-analysis showed that, relative to healthy controls, schizophrenia patients showed increased activity in the precentral gyrus and middle frontal gyrus and reduced activity in the mesolimbic circuit, including the striatum, thalamus, amygdala, hippocampus, anterior cingulate cortex, insula, superior temporal gyrus, and cerebellum, when processing prediction errors. We also found hyperactivity in frontal areas and hypoactivity in mesolimbic areas when encoding prediction error signals in schizophrenia patients, potentially indicating abnormal dopamine signaling of reward prediction error and suggesting failure to represent the value of alternative responses during prediction error learning and decision making.

The behavioral origins of phylogenic responses and ontogenic habits

An examination of innate behavior and its possible origins suggests parallels with the formation of habitual behavior. Inflexible but adaptive responses-innate reflexive behavior, Pavlovian conditioned responses, and operant habits-may have evolved from variable behavior in phylogeny and ontogeny. This form of "plasticity-first" scientific narrative was unpopular post-Darwin but has recently gained credibility in evolutionary biology. The present article seeks to identify originating events and contingencies contributing to such inflexible but adaptive behavior at both phylogenic and ontogenic levels of selection. In ontogeny, the development of inflexible performance (i.e., habit) from variable operant behavior is reminiscent of the genetic accommodation of initially variable phylogenic traits. The effects characteristic of habit (e.g., unresponsiveness to reinforcer devaluation) are explicable as the result of a conflict between behaviors at distinct levels of selection. The present interpretation validates the practice of seeking hard analogies between evolutionary biology and operant behavior. Finding such parallels implies the validity of a claim that organismal behavior, both innate and learned, is a product of selection by consequences. A complete and coherent account of organismal behavior may ultimately focus on functional selective histories in much the same way evolutionary biology does with its subject matter.

Making and breaking habits: Revisiting the definitions and behavioral factors that influence habits in animals

Habits have garnered significant interest in studies of associative learning and maladaptive behavior. However, habit research has faced scrutiny and challenges related to the definitions and methods. Differences in the conceptualizations of habits between animal and human studies create difficulties for translational research. Here, we review the definitions and commonly used methods for studying habits in animals and humans and discuss potential alternative ways to assess habits, such as automaticity. To better understand habits, we then focus on the behavioral factors that have been shown to make or break habits in animals, as well as potential mechanisms underlying the influence of these factors. We discuss the evidence that habitual and goal-directed systems learn in parallel and that they seem to interact in competitive and cooperative manners. Finally, we draw parallels between habitual responding and compulsive drug seeking in animals to delineate the similarities and differences in these behaviors.

Habit and persistence

Voluntary behaviors (operants) can come in two varieties: Goal-directed actions, which are emitted based on the remembered value of the reinforcer, and habits, which are evoked by antecedent cues and performed without the reinforcer's value in active memory. The two are perhaps most clearly distinguished with the reinforcer-devaluation test: Goal-directed actions are suppressed when the reinforcer is separately devalued and responding is tested in extinction, and habitual behaviors are not. But what is the function of habit learning? Habits are often thought to be strong and unusually persistent. The present selective review examines this idea by asking whether habits identified by the reinforcer-devaluation test are more resistant to extinction, resistant to the effects of other contingency change, vulnerable to relapse, resistant to the weakening effects of context change, or permanently in place once they are learned. Surprisingly little evidence supports the idea that habits are permanent or more persistent. Habits are more context-specific than goal-directed actions are. Methods that make behavior persistent do not necessarily work by encouraging habit. The function of habit learning may not be to make a behavior strong or more persistent but to make it automatic and efficient in a particular context.

Neurobehavioral Precursors of Compulsive Cocaine Seeking in Dual Frontostriatal Circuits

Background

Only some individuals who use drugs recreationally eventually develop a substance use disorder, characterized in part by the rigid engagement in drug foraging behavior (drug seeking), which is often maintained in the face of adverse consequences (i.e., is compulsive). The neurobehavioral determinants of this individual vulnerability have not been fully elucidated.

Methods

Using a prospective longitudinal study involving 39 male rats, we combined multidimensional characterization of behavioral traits of vulnerability to stimulant use disorder (impulsivity and stickiness) and resilience (sign tracking and sensation seeking/locomotor reactivity to novelty) with magnetic resonance imaging to identify the structural and functional brain correlates of the later emergence of compulsive drug seeking in drug-naïve subjects. We developed a novel behavioral procedure to investigate the individual tendency to persist in drug-seeking behavior in the face of punishment in a drug-free state in subjects with a prolonged history of cocaine seeking under the control of the conditioned reinforcing properties of a drug-paired Pavlovian conditioned stimulus.

Results

In drug-naïve rats, the tendency to develop compulsive cocaine seeking was characterized by behavioral stickiness-related functional hypoconnectivity between the prefrontal cortex and posterior dorsomedial striatum in combination with impulsivity-related structural alterations in the infralimbic cortex, anterior insula, and nucleus accumbens.

Conclusions

These findings show that the vulnerability to developing compulsive cocaine-seeking behavior stems from preexisting structural or functional changes in two distinct corticostriatal systems that underlie deficits in impulse control and goal-directed behavior.

Positive correlation between measures of habitual responding and motivated responding in mice

Habit and motivation are thought to be separate processes, with motivated behavior often considered to be goal directed, whereas habits are defined by the absence of goal-directed control over behavior. However, there has been increasing interrogation of the binary nature of habitual versus goal-directed behavior. Furthermore, although drug and alcohol exposure can promote the formation of habits, drug seeking itself can also be highly flexible, pointing toward the need for complex consideration of the parallel processes that drive behavior. The goal of the current study was to determine whether there was a relation between motivation-as measured by progressive ratio-and habit-as measured by contingency degradation-and whether this relation was affected by ethanol exposure history and sex. The results showed that these measures were positively correlated such that greater contingency insensitivity was associated with achieving higher break points on the progressive-ratio task. However, this relation depended on reinforcement schedule history, ethanol exposure history, and sex. These point to potential relations between measures of habit and motivation and stress the importance of carefully parsing behavioral findings and assays. These findings are also expected to inform future substance use research, as drug history may affect these relations.

Prior experience with behavioral control over stress facilitates social dominance

Dominance status has extensive effects on physical and mental health, and an individual's relative position can be shaped by experiential factors. A variety of considerations suggest that the experience of behavioral control over stressors should produce winning in dominance tests and that winning should blunt the impact of later stressors, as does prior control. To investigate the interplay between competitive success and stressor control, we first examined the impact of stressor controllability on subsequent performance in a warm spot competition test modified for rats. Prior experience of controllable, but not physically identical uncontrollable, stress increased later effortful behavior and occupation of the warm spot. Controllable stress subjects consistently ranked higher than did uncontrollable stress subjects. Pharmacological inactivation of the prelimbic (PL) cortex during behavioral control prevented later facilitation of dominance. Next, we explored whether repeated winning experiences produced later resistance against the typical sequelae of uncontrollable stress. To establish dominance status, triads of rats were given five sessions of warm spot competition. The development of stable dominance was prevented by reversible inactivation of the PL or NMDA receptor blockade in the dorsomedial striatum. Stable winning blunted the later stress-induced increase in dorsal raphe nucleus serotonergic activity, as well as prevented uncontrollable stress-induced social avoidance. In contrast, endocrine and neuroimmune responses to uncontrollable stress were unaffected, indicating a selective impact of prior dominance. Together, these data demonstrate that instrumental control over stress promotes later dominance, but also reveal that winning experiences buffer against the neural and behavioral outcomes of future adversity.

Multiple Memory Subsystems: Reconsidering Memory in the Mind and Brain

The multiple-memory-systems framework-that distinct types of memory are supported by distinct brain systems-has guided learning and memory research for decades. However, recent work challenges the one-to-one mapping between brain structures and memory types central to this taxonomy, with key memory-related structures supporting multiple functions across substructures. Here we integrate cross-species findings in the hippocampus, striatum, and amygdala to propose an updated framework of multiple memory subsystems (MMSS). We provide evidence for two organizational principles of the MMSS theory: First, opposing memory representations are colocated in the same brain structures; second, parallel memory representations are supported by distinct structures. We discuss why this burgeoning framework has the potential to provide a useful revision of classic theories of long-term memory, what evidence is needed to further validate the framework, and how this novel perspective on memory organization may guide future research.

The neurocognitive role of working memory load when Pavlovian motivational control affects instrumental learning

Research suggests that a fast, capacity-limited working memory (WM) system and a slow, incremental reinforcement learning (RL) system jointly contribute to instrumental learning. Thus, situations that strain WM resources alter instrumental learning: under WM loads, learning becomes slow and incremental, the reliance on computationally efficient learning increases, and action selection becomes more random. It is also suggested that Pavlovian learning influences people's behavior during instrumental learning by providing hard-wired instinctive responses including approach to reward predictors and avoidance of punishment predictors. However, it remains unknown how constraints on WM resources affect instrumental learning under Pavlovian influence. Thus, we conducted a functional magnetic resonance imaging (fMRI) study (N = 49) in which participants completed an instrumental learning task with Pavlovian-instrumental conflict (the orthogonalized go/no-go task) both with and without extra WM load. Behavioral and computational modeling analyses revealed that WM load reduced the learning rate and increased random choice, without affecting Pavlovian bias. Model-based fMRI analysis revealed that WM load strengthened RPE signaling in the striatum. Moreover, under WM load, the striatum showed weakened connectivity with the ventromedial and dorsolateral prefrontal cortex when computing reward expectations. These results suggest that the limitation of cognitive resources by WM load promotes slow and incremental learning through the weakened cooperation between WM and RL; such limitation also makes action selection more random, but it does not directly affect the balance between instrumental and Pavlovian systems.

Chronic alcohol exposure differentially alters calcium activity of striatal cell populations during actions

Alcohol Use Disorder (AUD) can induce long lasting alterations to executive function. This includes altered action control, which can manifest as dysfunctional goal-directed control. Cortical and striatal circuits mediate goal-directed control over behavior, and prior research has found chronic alcohol disrupts these circuits. In particular, prior in vivo and ex vivo work have identified alterations to function and activity of dorsal medial striatum (DMS), which is necessary for goal-directed control. However, unknown is whether these alterations manifest as altered activity of select DMS populations during behavior. Here we examine effects of prior chronic alcohol exposure on calcium activity modulation during action-related behaviors via fiber photometry of genetically-identified DMS populations including the direct and indirect output pathways, and fast-spiking interneurons. We find that prior chronic alcohol exposure leads to increased calcium modulation of the direct pathway during action related behavior. In contrast, prior chronic alcohol exposure led to decreased calcium activity modulation of the indirect pathway and the fast-spiking interneuron population around action-related events. Together, our findings suggest an imbalance in striatal activity during action control. This disruption may contribute to the altered goal-directed control previously reported.

Erotic cue exposure increases physiological arousal, biases choices toward immediate rewards, and attenuates model-based reinforcement learning

Computational psychiatry focuses on identifying core cognitive processes that appear altered across distinct psychiatric disorders. Temporal discounting of future rewards and model-based control during reinforcement learning have proven as two promising candidates. Despite its trait-like stability, temporal discounting may be at least partly under contextual control. Highly arousing cues were shown to increase discounting, although evidence to date remains somewhat mixed. Whether model-based reinforcement learning is similarly affected by arousing cues remains unclear. Here, we tested cue-reactivity effects (erotic pictures) on subsequent temporal discounting and model-based reinforcement learning in a within-subjects design in n = 39 healthy heterosexual male participants. Self-reported and physiological arousal (cardiac activity and pupil dilation) were assessed before and during cue exposure. Arousal was increased during exposure of erotic versus neutral cues both on the subjective and autonomic level. Erotic cue exposure increased discounting as reflected by more impatient choices. Hierarchical drift diffusion modeling (DDM) linked increased discounting to a shift in the starting point bias of evidence accumulation toward immediate options. Model-based control during reinforcement learning was reduced following erotic cues according to model-agnostic analysis. Notably, DDM linked this effect to attenuated forgetting rates of unchosen options, leaving the model-based control parameter unchanged. Our findings replicate previous work on cue-reactivity effects in temporal discounting and for the first time show similar effects in model-based reinforcement learning in a heterosexual male sample. This highlights how environmental cues can impact core human decision processes and reveal that comprehensive modeling approaches can yield novel insights in reward-based decision processes.

Pathway-specific alterations in striatal excitability and cholinergic modulation in a SAPAP3 mouse model of compulsive motor behavior

Deletion of the obsessive-compulsive disorder (OCD)-associated gene SAP90/PSD-95-associated protein 3 (Sapap3), which encodes a postsynaptic anchoring protein at corticostriatal synapses, causes OCD-like motor behaviors in mice. While corticostriatal synaptic dysfunction is central to this phenotype, the striatum efficiently adapts to pathological changes, often in ways that expand upon the original circuit impairment. Here, we show that SAPAP3 deletion causes non-synaptic and pathway-specific alterations in dorsolateral striatum circuit function. While somatic excitability was elevated in striatal projection neurons (SPNs), dendritic excitability was exclusively enhanced in direct pathway SPNs. Layered on top of this, cholinergic modulation was altered in opposing ways: striatal cholinergic interneuron density and evoked acetylcholine release were elevated, while basal muscarinic modulation of SPNs was reduced. These data describe how SAPAP3 deletion alters the striatal landscape upon which impaired corticostriatal inputs will act, offering a basis for how pathological synaptic integration and unbalanced striatal output underlying OCD-like behaviors may be shaped.

Propensity for risky choices despite lower cue reactivity in adolescent rats

Adolescence is a time of heightened risk-taking across species. Salient audiovisual cues associated with rewards are a common feature of gambling environments and have been connected to increased risky decision-making. We have previously shown that, in adult male rats, sign tracking - a behavioral measure of cue reactivity - predicts an individual's propensity for suboptimal risky choices in a rodent gambling task (rGT) with win-paired cues. However, adolescents perform less sign tracking than adult animals, suggesting that they are less cue-reactive than adults in some circumstances. Therefore, we investigated the performance of adolescent male rats on the rGT with win cues and examined its relationship with their sign-tracking behavior. We found that adolescents make more risky choices and fewer optimal choices on the rGT compared with adults, evidence of the validity of the rGT as a model of adolescent gambling behavior. We also confirmed that adolescents perform less sign tracking than adults, and we found that, unlike in adults, adolescents' sign tracking was unrelated to their risk-taking in the rGT. This implies that adolescent risk-taking is less likely than that of adults to be driven by reward-related cues. Finally, we found that adults trained on the rGT as adolescents retained an adolescent-like propensity toward risky choices, suggesting that early exposure to a gambling environment may have a long-lasting impact on risk-taking behavior.

Nigrostriatal dopamine modulates the striatal-amygdala pathway in auditory fear conditioning

The auditory striatum, a sensory portion of the dorsal striatum, plays an essential role in learning and memory. In contrast to its roles and underlying mechanisms in operant conditioning, however, little is known about its contribution to classical auditory fear conditioning. Here, we reveal the function of the auditory striatum in auditory-conditioned fear memory. We find that optogenetically inhibiting auditory striatal neurons impairs fear memory formation, which is mediated through the striatal-amygdala pathway. Using calcium imaging in behaving mice, we find that auditory striatal neuronal responses to conditioned tones potentiate across memory acquisition and expression. Furthermore, nigrostriatal dopaminergic projections plays an important role in modulating conditioning-induced striatal potentiation. Together, these findings demonstrate the existence of a nigro-striatal-amygdala circuit for conditioned fear memory formation and expression.

Instant habits versus flexible tenacity: Do implementation intentions accelerate habit formation?

Implementation intentions (strategic "if-then" plans) have been shown to support behaviour change. This may be achieved by mentally forming stimulus-response associations, thereby promoting habit formation. Does this deliberate attempt to instal "strategic automaticity" only offer advantages, or does it also come at the cost of reduced flexibility that characterises learnt habits? To investigate this, we tested healthy, young participants on a computerised instrumental learning task. Critically, we introduced implementation intentions ("if I see stimulus X, then I will respond") versus goal intentions ("for outcome Z, I will respond)" during instrumental acquisition, and subsequently assessed behavioural flexibility in an outcome-revaluation test. In Experiment 1, we conducted a between-subjects manipulation of strategic planning, and in Experiment 2, a within-subject manipulation. We hypothesised that implementation intentions would lead to strong stimulus-response associations and consequently impair performance when the signalled outcome value changed and therefore required a different response, while benefitting performance when the outcome value (and required response) remained the same. We found that implementation intentions supported instrumental learning, but impaired test performance overall (most robustly in Experiment 2), irrespective of whether the signalled outcome value had changed. We argue that this general detrimental effect of implementation intentions on test performance is likely a consequence of their negative effect on stimulus-outcome learning. Our findings warrant caution when applying if-then plans to situations where the agent does not already possess perfect knowledge of behavioural contingencies.While implementation intentions may support efficient and fast behavioural execution, this may come at the expense of behavioural flexibility.

Theory of reinforcement schedules

The three principles of reinforcement are (1) events such as incentives and reinforcers increase the activity of an organism; (2) that activity is bounded by competition from other responses; and (3) animals approach incentives and their signs, guided by their temporal and physical conditions, together called the "contingencies of reinforcement." Mathematical models of each of these principles comprised mathematical principles of reinforcement (MPR; Killeen, 1994). Over the ensuing decades, MPR was extended to new experimental contexts. This article reviews the basic theory and its extensions to satiation, warm-up, extinction, sign tracking, pausing, and sequential control in progressive-ratio and multiple schedules. In the latter cases, a single equation balancing target and competing responses governs behavioral contrast and behavioral momentum. Momentum is intrinsic in the fundamental equations, as behavior unspools more slowly from highly aroused responses conditioned by higher rates of incitement than it does from responses from leaner contexts. Habits are responses that have accrued substantial behavioral momentum. Operant responses, being predictors of reinforcement, are approached by making them: The sight and feel of a paw on a lever is approached by placing paw on lever, as attempted for any sign of reinforcement. Behavior in concurrent schedules is governed by approach to momentarily richer patches (melioration). Applications of MPR in behavioral pharmacology and delay discounting are noted.

Habits and Persistent Food Restriction in Patients with Anorexia Nervosa: A Scoping Review

The aetiology of anorexia nervosa (AN) presents a puzzle for researchers. Recent research has sought to understand the behavioural and neural mechanisms of these patients' persistent choice of calorie restriction. This scoping review aims to map the literature on the contribution of habit-based learning to food restriction in AN. PRISMA-ScR guidelines were adopted. The search strategy was applied to seven databases and to grey literature. A total of 35 studies were included in this review. The results indicate that the habit-based learning model has gained substantial attention in current research, employing neuroimaging methods, scales, and behavioural techniques. Food choices were strongly associated with dorsal striatum activity, and habitual food restriction based on the self-report restriction index was associated with clinical impairment in people chronically ill with restricting AN. High-frequency repetitive transcranial magnetic stimulation (HF-rTMS) and Regulating Emotions and Changing Habits (REaCH) have emerged as potential treatments. Future research should employ longitudinal studies to investigate the time required for habit-based learning and analyse how developmental status, such as adolescence, influences the role of habits in the progression and severity of diet-related illnesses. Ultimately, seeking effective strategies to modify persistent dietary restrictions controlled by habits remains essential.

Development and cadherin-mediated control of prefrontal corticostriatal projections in mice

Action-outcome associations depend on prefrontal cortex (PFC) projections to the dorsal striatum. To assess how these projections form, we measured PFC axon patterning, synapse formation, and functional maturation in the postnatally developing mouse striatum. Using Hotspot analysis, we show that PFC axons form an adult-like pattern of clustered terminations in the first postnatal week that remains largely stable thereafter. PFC-striatal synaptic strength is adult-like by P21, while excitatory synapse density increases until adulthood. We then tested how the targeted deletion of a candidate adhesion/guidance protein, Cadherin-8 (Cdh8), from corticostriatal neurons regulates pathway development. Mutant mice showed diminished PFC axon targeting and reduced spontaneous glutamatergic synaptic activity in the dorsal striatum. They also exhibited impaired behavioral performance in action-outcome learning. The data show that PFC-striatal axons form striatal territories through an early, directed growth model and they highlight essential contributions of Cdh8 to the anatomical and functional features critical for the formation of action-outcome associations.

Goal-directed and habitual control: from circuits and functions to exercise-induced neuroplasticity targets for the treatment of Parkinson's disease

Parkinson's disease (PD) is a neurodegenerative disease characterized by motor and cognitive impairments. The progressive depletion of dopamine (DA) is the pathological basis of dysfunctional goal-directed and habitual control circuits in the basal ganglia. Exercise-induced neuroplasticity could delay disease progression by improving motor and cognitive performance in patients with PD. This paper reviews the research progress on the motor-cognitive basal ganglia circuit and summarizes the current hypotheses for explaining exercise intervention on rehabilitation in PD. Studies on exercise mediated mechanisms will contribute to the understanding of networks that regulate goal-directed and habitual behaviors and deficits in PD, facilitating the development of strategies for treatment of PD.

The Motivational Determinants of Human Action, Their Neural Bases and Functional Impact in Adolescents With Obsessive-Compulsive Disorder

Background

Establishing the motivational influences on human action is essential for understanding choice and decision making in health and disease. Here we used tests of value-based decision making, manipulating both predicted and experienced reward values to assess the motivational control of goal-directed action in healthy adolescents and those with obsessive-compulsive disorder (OCD).

Methods

After instrumental training on a two action-two outcome probabilistic task, adolescents (n = 21) underwent Pavlovian conditioning using distinct stimuli predicting either the instrumental outcomes, a third outcome, or nothing. We then assessed functional magnetic resonance imaging during choice tests in which we varied the predicted value, using specific and general Pavlovian-instrumental transfer, and the experienced value, using outcome devaluation. To establish functional significance, we tested a matched cohort of adolescents with OCD (n = 20).

Results

In healthy adolescents, both predicted and experienced values influenced the performance of goal-directed actions, mediated by distinct orbitofrontal-striatal circuits involving the lateral orbitofrontal cortex (OFC) and medial OFC, respectively. However, in adolescents with OCD, choice was insensitive to changes in either predicted or experienced values. These impairments were related to hypoactivity in the lateral OFC and hyperactivity in the medial OFC during specific Pavlovian-instrumental transfer and hypoactivity in the anterior prefrontal cortex, caudate nucleus, and their connectivity in the devaluation test.

Conclusions

We found that predicted and experienced values exerted a potent influence on the performance of goal-directed actions in adolescents via distinct orbitofrontal- and prefrontal-striatal circuits. Furthermore, the influence of these motivational processes was severely blunted in OCD, as was the functional segregation of circuits involving medial and lateral OFC, producing dysregulated action control.

Attentional Bias Modification Training Based on Virtual Reality and Eye Tracking in Anorexia Nervosa Patients

Anorexia nervosa (AN) patients exhibit attentional bias (AB) related to the body, which is the tendency to pay greater attention to weight-related body areas compared to non-weight-related ones. This phenomenon has been linked to elevated levels of body dissatisfaction (BD) and may potentially reduce the effectiveness of body exposure therapy. The purpose of this pilot study is to assess the efficacy of a single session of a new body-related AB modification task (ABMT) that combines virtual reality with eye tracking in patients with AN. The goals of the ABMT are to reduce body-related AB by balancing attention between weight and non-weight-related body areas and to reduce BD levels. Twenty-three adolescent patients with AN were embodied in a virtual avatar and immersed in a virtual environment where they completed the ABMT. Body-related AB measures and BD levels were assessed before and after the training. A paired samples t-test showed statistically significant differences between pre-assessment and post-assessment; the complete fixation time on weight-related body parts was reduced and BD levels decreased. The initial evidence of the efficacy of this ABMT has important clinical implications, since AB and BD are considered risk factors for developing and maintaining eating disorder symptomatology among patients with AN.

Inhibition of the dorsomedial striatal direct pathway is essential for the execution of action sequences

Contrary to the previous notion that the dorsomedial striatum (DMS) is crucial for acquiring new learning, accumulated evidence has suggested that the DMS also plays a role in the execution of already learned action sequences. Here, we examined how the direct and indirect pathways in the DMS regulate action sequences using a task that requires animals to press a lever consecutively. Cell-type-specific bulk Ca2+ recording revealed that the direct pathway was inhibited at the time of sequence execution. The sequence-related response was blunted in trials where the sequential behaviors were disrupted. Optogenetic activation at the sequence start caused distraction of action sequences without affecting motor function or memory of the task structure. By contrast with the direct pathway, the indirect pathway was slightly activated at the start of the sequence, but the optogenetic suppression of such sequence-related signaling did not impact the behaviors. These results suggest that the inhibition of the DMS direct pathway promotes sequence execution potentially by suppressing the formation of a new association.

Effects of predictive and incentive value manipulation on sign- and goal-tracking behavior

When a neutral stimulus is repeatedly paired with an appetitive reward, two different types of conditioned approach responses may develop: a sign-tracking response directed toward the neutral cue, or a goal-tracking response directed toward the location of impending reward delivery. Sign-tracking responses have been postulated to result from attribution of incentive value to conditioned cues, while goal-tracking reflects the assignment of only predictive value to the cue. We therefore hypothesized that sign-tracking rats would be more sensitive to manipulations of incentive value, while goal-tracking rats would be more responsive to changes in the predictive value of the cue. We tested sign- and goal-tracking before and after devaluation of a food reward using lithium chloride, and tested whether either response could be learned under negative contingency conditions that precluded any serendipitous reinforcement of the behavior that might support instrumental learning. We also tested the effects of blocking the predictive value of a cue using simultaneous presentation of a pre-conditioned cue. We found that sign-tracking was sensitive to outcome devaluation, while goal-tracking was not. We also confirmed that both responses are Pavlovian because they can be learned under negative contingency conditions. Goal-tracking was almost completely blocked by a pre-conditioned cue, while sign-tracking was much less sensitive to such interference. These results indicate that sign- and goal-tracking may follow different rules of reinforcement learning and suggest a need to revise current models of associative learning to account for these differences.

Anatomical and Functional Comparison of the Caudate Tail in Primates and the Tail of the Striatum in Rodents: Implications for Sensory Information Processing and Habitual Behavior

The tail of the striatum (TS) is located at the caudal end in the striatum. Recent studies have advanced our knowledge of the anatomy and function of the TS but also raised questions about the differences between rodent and primate TS. In this review, we compare the anatomy and function of the TS in rodent and primate brains. The primate TS is expanded more caudally during brain development in comparison with the rodent TS. Additionally, five sensory inputs from the cortex and thalamus converge in the rodent TS, but this convergence is not observed in the primate TS. The primate TS, including the caudate tail and putamen tail, primarily receives inputs from the visual areas, implying a specialized function in processing visual inputs for action generation. This anatomical difference leads to further discussion of cellular circuit models to comprehend how the primate brain processes a wider range of complex visual stimuli to produce habitual behavior as compared with the rodent brain. Examining these differences and considering possible neural models may provide better understanding of the anatomy and function of the primate TS.

Immp2l knockdown in male mice increases stimulus-driven instrumental behaviour but does not alter goal-directed learning or neuron density in cortico-striatal circuits in a model of Tourette syndrome and autism spectrum disorder

Cortico-striatal neurocircuits mediate goal-directed and habitual actions which are necessary for adaptive behaviour. It has recently been proposed that some of the core symptoms of autism spectrum disorder (ASD) and Gilles de la Tourette syndrome (GTS), such as tics and other repetitive behaviours, may emerge because of imbalances in these neurocircuits. We have recently developed a model of ASD and GTS by knocking down Immp2l, a mitochondrial gene frequently associated with these disorders. The current study sought to determine whether Immp2l knockdown (KD) in male mice alters flexible, goal- or cue- driven behaviour using procedures specifically designed to examine response-outcome and stimulus-response associations, which underlie goal-directed and habitual behaviour, respectively. Whether Immp2l KD alters neuron density in cortico-striatal neurocircuits known to regulate these behaviours was also examined. Immp2l KD mice and wild type-like mice (WT) were trained on Pavlovian and instrumental learning procedures where auditory cues predicted food delivery and lever-press responses earned a food outcome. It was demonstrated that goal-directed learning was not changed for Immp2l KD mice compared to WT mice, as lever-press responses were sensitive to changes in the value of the food outcome, and to contingency reversal and degradation. There was also no difference in the capacity of KD mice to form habitual behaviours compared to WT mice following extending training of the instrumental action. However, Immp2l KD mice were more responsive to auditory stimuli paired with food as indicated by a non-specific increase in lever response rates during Pavlovian-to-instrumental transfer. Finally, there were no alterations to neuron density in striatum or any prefrontal cortex or limbic brain structures examined. Thus, the current study suggests that Immp2l is not necessary for learned maladaptive goal or stimulus driven behaviours in ASD or GTS, but that it may contribute to increased capacity for external stimuli to drive behaviour. Alterations to stimulus-driven behaviour could potentially influence the expression of tics and repetitive behaviours, suggesting that genetic alterations to Immp2l may contribute to these core symptoms in ASD and GTS. Given that this is the first application of this battery of instrumental learning procedures to a mouse model of ASD or GTS, it is an important initial step in determining the contribution of known risk-genes to goal-directed versus habitual behaviours, which should be more broadly applied to other rodent models of ASD and GTS in the future.

Feasibility of dopamine as a vector-valued feedback signal in the basal ganglia

It is well established that midbrain dopaminergic neurons support reinforcement learning (RL) in the basal ganglia by transmitting a reward prediction error (RPE) to the striatum. In particular, different computational models and experiments have shown that a striatum-wide RPE signal can support RL over a small discrete set of actions (e.g., no/no-go, choose left/right). However, there is accumulating evidence that the basal ganglia functions not as a selector between predefined actions but rather as a dynamical system with graded, continuous outputs. To reconcile this view with RL, there is a need to explain how dopamine could support learning of continuous outputs, rather than discrete action values. Inspired by the recent observations that besides RPE, the firing rates of midbrain dopaminergic neurons correlate with motor and cognitive variables, we propose a model in which dopamine signal in the striatum carries a vector-valued error feedback signal (a loss gradient) instead of a homogeneous scalar error (a loss). We implement a local, "three-factor" corticostriatal plasticity rule involving the presynaptic firing rate, a postsynaptic factor, and the unique dopamine concentration perceived by each striatal neuron. With this learning rule, we show that such a vector-valued feedback signal results in an increased capacity to learn a multidimensional series of real-valued outputs. Crucially, we demonstrate that this plasticity rule does not require precise nigrostriatal synapses but remains compatible with experimental observations of random placement of varicosities and diffuse volume transmission of dopamine.

Exercise therapy for chronic pain: How does exercise change the limbic brain function?

We are exposed to various external and internal threats which might hurt us. The role of taking flexible and appropriate actions against threats is played by "the limbic system" and at the heart of it there is the ventral tegmental area and nucleus accumbens (brain reward system). Pain-related fear causes excessive excitation of amygdala, which in turn causes the suppression of medial prefrontal cortex, leading to chronification of pain. Since the limbic system of chronic pain patients is functionally impaired, they are maladaptive to their situations, unable to take goal-directed behavior and are easily caught by fear-avoidance thinking. We describe the neural mechanisms how exercise activates the brain reward system and enables chronic pain patients to take goal-directed behavior and overcome fear-avoidance thinking. A key to getting out from chronic pain state is to take advantage of the behavioral switching function of the basal nucleus of amygdala. We show that exercise activates positive neurons in this nucleus which project to the nucleus accumbens and promote reward behavior. We also describe fear conditioning and extinction are affected by exercise. In chronic pain patients, the fear response to pain is enhanced and the extinction of fear memories is impaired, so it is difficult to get out of "fear-avoidance thinking". Prolonged avoidance of movement and physical inactivity exacerbate pain and have detrimental effects on the musculoskeletal and cardiovascular systems. Based on the recent findings on multiple bran networks, we propose a well-balanced exercise prescription considering the adherence and pacing of exercise practice. We conclude that therapies targeting the mesocortico-limbic system, such as exercise therapy and cognitive behavioral therapy, may become promising tools in the fight against chronic pain.

Exploring expected reward and efficacy in enhancing cognitive control in patients with depression

BACKGROUND

Depression is associated with impairments in cognitive control. Considering the lack of mechanistic models accounting for cognitive control deficits in depression, the expected value of control (EVC) theory offers a mechanistic view for allocating cognitive control emphasizing motivational components (efficacy, value). Efficacy refers to the possibility that an effort leads to a special outcome and reward refers to the value (amount) associated with the outcome. This study aimed to examine the role of the EVC in depression.

METHOD

This study used a within-between-subject design. Participants with depression (n = 36) and healthy controls (n = 31) completed a clinical diagnostic interview, the Beck Depression Inventory-II, the General Health Questionnaire-12, and a computer-based incentivized Stroop Color-Word Paradigm in which levels of efficacy (high vs. low) and the amount of rewards (high vs. low) were presented as cues before target stimuli.

RESULTS

We found significant interaction effects of group × efficacy and efficacy × reward in terms of reaction time in the Stroop Paradigm. Follow-up analyses indicated the Depressed group were significantly slower than Controls on high efficacy trials, but the two groups did not differ significantly on low efficacy trials. Additionally, on high efficacy trials, reward did not influence performance, but on low efficacy trials, high reward improved performance in both groups.

LIMITATION

Lack of neurological measures and eye tracking techniques.

CONCLUSION

Overall, our findings suggest that reward and efficacy may jointly improve cognitive control allocation and highlight the need for further research examining EVC theory as a mechanistic account of cognitive control deficits in depression.

Dual-process moral judgment beyond fast and slow

De Neys makes a compelling case that the sacrificial moral dilemmas do not elicit competing "fast and slow" processes. But are there even two processes? Or just two intuitions? There remains strong evidence, most notably from lesion studies, that sacrificial dilemmas engage distinct cognitive processes generating conflicting emotional and rational responses. The dual-process theory gets much right, but needs revision.

Cortical glutamate and GABA are related to compulsive behaviour in individuals with obsessive compulsive disorder and healthy controls

There has been little analysis of neurochemical correlates of compulsive behaviour to illuminate its underlying neural mechanisms. We use 7-Tesla proton magnetic resonance spectroscopy (¹H-MRS) to assess the balance of excitatory and inhibitory neurotransmission by measuring glutamate and GABA levels in anterior cingulate cortex (ACC) and supplementary motor area (SMA) of healthy volunteers and participants with Obsessive-Compulsive Disorder (OCD). Within the SMA, trait and clinical measures of compulsive behaviour are related to glutamate levels, whereas a behavioural index of habitual control correlates with the glutamate:GABA ratio. Participants with OCD also show the latter relationship in the ACC while exhibiting elevated glutamate and lower GABA levels in that region. This study highlights SMA mechanisms of habitual control relevant to compulsive behaviour, common to the healthy sub-clinical and OCD populations. The results also demonstrate additional involvement of anterior cingulate in the balance between goal-directed and habitual responding in OCD.

Prosocial learning: Model-based or model-free?

Prosocial learning involves the acquisition of knowledge and skills necessary for making decisions that benefit others. We asked if, in the context of value-based decision-making, there is any difference between learning strategies for oneself vs. for others. We implemented a 2-step reinforcement learning paradigm in which participants learned, in separate blocks, to make decisions for themselves or for a present other confederate who evaluated their performance. We replicated the canonical features of the model-based and model-free reinforcement learning in our results. The behaviour of the majority of participants was best explained by a mixture of the model-based and model-free control, while most participants relied more heavily on MB control, and this strategy enhanced their learning success. Regarding our key self-other hypothesis, we did not find any significant difference between the behavioural performances nor in the model-based parameters of learning when comparing self and other conditions.

The body mass index is associated with increased temporal variability of functional connectivity in brain reward system

The reward system has been proven to be contributed to the vulnerability of obesity. Previous fMRI studies have shown abnormal functional connectivity of the reward system in obesity. However, most studies were based on static index such as resting-state functional connectivity (FC), ignoring the dynamic changes over time. To investigate the dynamic neural correlates of obesity susceptibility, we used a large, demographically well-characterized sample from the Human Connectome Project (HCP) to determine the relationship of body mass index (BMI) with the temporal variability of FC from integrated multilevel perspectives, i.e., regional and within- and between-network levels. Linear regression analysis was used to investigate the association between BMI and temporal variability of FC, adjusting for covariates of no interest. We found that BMI was positively associated with regional FC variability in reward regions, such as the ventral orbitofrontal cortex and visual regions. At the intra-network level, BMI was positively related to the variability of FC within the limbic network (LN) and default mode network (DMN). At the inter-network level, variability of connectivity of LN with DMN, frontoparietal, sensorimotor, and ventral attention networks showed positive correlations with BMI. These findings provided novel evidence for abnormal dynamic functional interaction between the reward network and the rest of the brain in obesity, suggesting a more unstable state and over-frequent interaction of the reward network and other attention and cognitive networks. These findings, thus, provide novel insight into obesity interventions that need to decrease the dynamic interaction between reward networks and other brain networks through behavioral treatment and neural modulation.

Punishment resistance for cocaine is associated with inflexible habits in rats

Addiction is characterized by continued drug use despite negative consequences. In an animal model, a subset of rats continues to self-administer cocaine despite footshock consequences, showing punishment resistance. We sought to test the hypothesis that punishment resistance arises from failure to exert goal-directed control over habitual cocaine seeking. While habits are not inherently permanent or maladaptive, continued use of habits under conditions that should encourage goal-directed control makes them maladaptive and inflexible. We trained male and female Sprague Dawley rats on a seeking-taking chained schedule of cocaine self-administration (2 h/day). We then exposed them to 4 days of punishment testing, in which footshock (0.4 mA, 0.3 s) was delivered randomly on one-third of trials, immediately following completion of seeking and prior to extension of the taking lever. Before and after punishment testing (4 days pre-punishment and ≥4 days post-punishment), we assessed whether cocaine seeking was goal-directed or habitual using outcome devaluation via cocaine satiety. We found that punishment resistance was associated with continued use of habits, whereas punishment sensitivity was associated with increased goal-directed control. Although punishment resistance was not predicted by habitual responding pre-punishment, it was associated with habitual responding post-punishment. In parallel studies of food self-administration, we similarly observed that punishment resistance was associated with habitual responding post-punishment but not pre-punishment. These findings indicate that punishment resistance is related to habits that have become inflexible and persist under conditions that should encourage a transition to goal-directed behavior.

Habits, Goals, and Behavioral Signs of Cognitive Perseveration on Wisconsin Card-Sorting Tasks

Wisconsin card-sorting tasks provide unique opportunities to study cognitive flexibility and its limitations, which express themselves behaviorally as perseverative errors (PE). PE refer to those behavioral errors on Wisconsin card-sorting tasks that are committed when cognitive rules are maintained even though recently received outcomes demand to switch to other rules (i.e., cognitive perseveration). We explored error-suppression effects (ESE) across three Wisconsin card-sorting studies. ESE refer to the phenomenon that PE are reduced on repetitive trials compared to non-repetitive trials. We replicated ESE in all three Wisconsin card-sorting studies. Study 1 revealed that non-associative accounts of ESE, in particular the idea that cognitive inhibition may account for them, are not tenable. Study 2 suggested that models of instrumental learning are among the most promising associative accounts of ESE. Instrumental learning comprises goal-directed control and the formation of corresponding associative memories over and above the formation of habitual memories according to dual-process models of instrumental learning. Study 3 showed that cognitive, rather than motor, representations of responses should be conceptualized as elements entering goal-directed instrumental memories. Collectively, the results imply that ESE on Wisconsin card-sorting tasks are not only a highly replicable phenomenon, but they also indicate that ESE provide an opportunity to study cognitive mechanisms of goal-directed instrumental control. Based on the reported data, we present a novel theory of cognitive perseveration (i.e., the 'goal-directed instrumental control' GIC model), which is outlined in the Concluding Discussion.

Pathways to the persistence of drug use despite its adverse consequences

The persistence of drug taking despite its adverse consequences plays a central role in the presentation, diagnosis, and impacts of addiction. Eventual recognition and appraisal of these adverse consequences is central to decisions to reduce or cease use. However, the most appropriate ways of conceptualizing persistence in the face of adverse consequences remain unclear. Here we review evidence that there are at least three pathways to persistent use despite the negative consequences of that use. A cognitive pathway for recognition of adverse consequences, a motivational pathway for valuation of these consequences, and a behavioral pathway for responding to these adverse consequences. These pathways are dynamic, not linear, with multiple possible trajectories between them, and each is sufficient to produce persistence. We describe these pathways, their characteristics, brain cellular and circuit substrates, and we highlight their relevance to different pathways to self- and treatment-guided behavior change.

Impairment of arbitration between model-based and model-free reinforcement learning in obsessive-compulsive disorder

Introduction

Obsessive-compulsive disorder (OCD) is characterized by an imbalance between goal-directed and habitual learning systems in behavioral control, but it is unclear whether these impairments are due to a single system abnormality of the goal-directed system or due to an impairment in a separate arbitration mechanism that selects which system controls behavior at each point in time.

Methods

A total of 30 OCD patients and 120 healthy controls performed a 2-choice, 3-stage Markov decision-making paradigm. Reinforcement learning models were used to estimate goal-directed learning (as model-based reinforcement learning) and habitual learning (as model-free reinforcement learning). In general, 29 high Obsessive-Compulsive Inventory-Revised (OCI-R) score controls, 31 low OCI-R score controls, and all 30 OCD patients were selected for the analysis.

Results

Obsessive-compulsive disorder (OCD) patients showed less appropriate strategy choices than controls regardless of whether the OCI-R scores in the control subjects were high (p = 0.012) or low (p < 0.001), specifically showing a greater model-free strategy use in task conditions where the model-based strategy was optimal. Furthermore, OCD patients (p = 0.001) and control subjects with high OCI-R scores (H-OCI-R; p = 0.009) both showed greater system switching rather than consistent strategy use in task conditions where model-free use was optimal.

Conclusion

These findings indicated an impaired arbitration mechanism for flexible adaptation to environmental demands in both OCD patients and healthy individuals reporting high OCI-R scores.