The Meaning of Behavior: Discriminating Reflex and Volition in the Brain

The neuroscience of active learning and direct instruction

Throughout the educational system, students experiencing active learning pedagogy perform better and fail less than those taught through direct instruction. Can this be ascribed to differences in learning from a neuroscientific perspective? This review examines mechanistic, neuroscientific evidence that might explain differences in cognitive engagement contributing to learning outcomes between these instructional approaches. In classrooms, direct instruction comprehensively describes academic content, while active learning provides structured opportunities for learners to explore, apply, and manipulate content. Synaptic plasticity and its modulation by arousal or novelty are central to all learning and both approaches. As a form of social learning, direct instruction relies upon working memory. The reinforcement learning circuit, associated agency, curiosity, and peer-to-peer social interactions combine to enhance motivation, improve retention, and build higher-order-thinking skills in active learning environments. When working memory becomes overwhelmed, additionally engaging the reinforcement learning circuit improves retention, providing an explanation for the benefits of active learning. This analysis provides a mechanistic examination of how emerging neuroscience principles might inform pedagogical choices at all educational levels.

Cortico-striatal white-matter connectivity underlies the ability to exert goal-directed control

The balance between goal-directed and habitual control has been proposed to determine the flexibility of instrumental behaviour, in both humans and animals. This view is supported by neuroscientific studies that have implicated dissociable neural pathways in the ability to flexibly adjust behaviour when outcome values change. A previous Diffusion Tensor Imaging study provided preliminary evidence that flexible instrumental performance depends on the strength of parallel cortico-striatal white-matter pathways previously implicated in goal-directed and habitual control. Specifically, estimated white-matter strength between caudate and ventromedial prefrontal cortex correlated positively with behavioural flexibility, and posterior putamen-premotor cortex connectivity correlated negatively, in line with the notion that these pathways compete for control. However, the sample size of the original study was limited, and so far, there have been no attempts to replicate these findings. In the present study, we aimed to conceptually replicate these findings by testing a large sample of 205 young adults to relate cortico-striatal connectivity to performance on the slips-of-action task. In short, we found only positive neural correlates of goal-directed performance, including striatal connectivity (caudate and anterior putamen) with the dorsolateral prefrontal cortex. However, we failed to provide converging evidence for the existence of a neural habit system that puts limits on the capacity for flexible, goal-directed action. We discuss the implications of our findings for dual-process theories of instrumental action.

Neural interactions in the human frontal cortex dissociate reward and punishment learning

How human prefrontal and insular regions interact while maximizing rewards and minimizing punishments is unknown. Capitalizing on human intracranial recordings, we demonstrate that the functional specificity toward reward or punishment learning is better disentangled by interactions compared to local representations. Prefrontal and insular cortices display non-selective neural populations to rewards and punishments. Non-selective responses, however, give rise to context-specific interareal interactions. We identify a reward subsystem with redundant interactions between the orbitofrontal and ventromedial prefrontal cortices, with a driving role of the latter. In addition, we find a punishment subsystem with redundant interactions between the insular and dorsolateral cortices, with a driving role of the insula. Finally, switching between reward and punishment learning is mediated by synergistic interactions between the two subsystems. These results provide a unifying explanation of distributed cortical representations and interactions supporting reward and punishment learning.

A ventral pallidal-thalamocortical circuit mediates the cognitive control of instrumental action

Predictive learning can engage a selective form of cognitive control that biases choice between actions based on information about future outcomes that the learning provides. This influence has been hypothesized to depend on a feedback circuit in the brain through which the basal ganglia modulate activity in the prefrontal cortex; however, direct evidence for this functional circuit has proven elusive. Here, using an animal model of cognitive control, we found that the influence of predictive learning on decision making is mediated by an inhibitory feedback circuit linking the medial ventral pallidum and the mediodorsal thalamus, the activation of which causes disinhibition of the orbitofrontal cortex via reduced activation of inhibitory parvalbumin interneurons during choice. Thus, we found that, for this function, the mediodorsal thalamus serves as a pallidal-cortical relay through which predictive learning controls action selection, which has important implications for understanding cognitive control and its vicissitudes in various psychiatric disorders and addiction.

What Role Does Striatal Dopamine Play in Goal-directed Action?

Evidence suggests that dopamine activity provides a US-related prediction error for Pavlovian conditioning and the reinforcement signal supporting the acquisition of habits. However, its role in goal-directed action is less clear. There are currently few studies that have assessed dopamine release as animals acquire and perform self-paced instrumental actions. Here we briefly review the literature documenting the psychological, behavioral and neural bases of goal-directed actions in rats and mice, before turning to describe recent studies investigating the role of dopamine in instrumental learning and performance. Plasticity in dorsomedial striatum, a central node in the network supporting goal-directed action, clearly requires dopamine release, the timing of which, relative to cortical and thalamic inputs, determines the degree and form of that plasticity. Beyond this, bilateral release appears to reflect reward prediction errors as animals experience the consequences of an action. Such signals feedforward to update the value of the specific action associated with that outcome during subsequent performance, with dopamine release at the time of action reflecting the updated predicted action value. More recently, evidence has also emerged for a hemispherically lateralised signal associated with the action; dopamine release is greater in the hemisphere contralateral to the spatial target of the action. This effect emerges over the course of acquisition and appears to reflect the strength of the action-outcome association. Thus, during goal-directed action, dopamine release signals the action, the outcome and their association to shape the learning and performance processes necessary to support this form of behavioral control.

Goal-directed and flexible modulation of syllable sequence within birdsong

Songs constitute a complex system of vocal signals for inter-individual communication in songbirds. Here, we elucidate the flexibility which songbirds exhibit in the organizing and sequencing of syllables within their songs. Utilizing a newly devised song decoder for quasi-real-time annotation, we execute an operant conditioning paradigm, with rewards contingent upon specific syllable syntax. Our analysis reveals that birds possess the capacity to modify the contents of their songs, adjust the repetition length of particular syllables and employing specific motifs. Notably, birds altered their syllable sequence in a goal-directed manner to obtain rewards. We demonstrate that such modulation occurs within a distinct song segment, with adjustments made within 10 minutes after cue presentation. Additionally, we identify the involvement of the parietal-basal ganglia pathway in orchestrating these flexible modulations of syllable sequences. Our findings unveil an unappreciated aspect of songbird communication, drawing parallels with human speech.

A novel free-operant framework enables experimental habit induction in humans

Habits are a prominent feature of both adaptive and maladaptive behavior. Yet, despite substantial research efforts, there are currently no well-established experimental procedures for habit induction in humans. It is likely that laboratory experimental settings, as well as the session-based structure typically used in controlled experiments (also outside the lab), impose serious constraints on studying habits and other effects that are sensitive to context, motivation, and training duration and frequency. To overcome these challenges, we devised a unique real-world free-operant task structure, implemented through a novel smartphone application, whereby participants could freely enter the app (24 hours a day, 7 days a week) to win rewards. This procedure is free of typical laboratory constraints, yet well controlled. Using the canonical sensitivity to outcome devaluation criterion, we successfully demonstrated habit formation as a function of training duration, a long-standing challenge in the field. Additionally, we show a positive relationship between multiple facets of engagement/motivation and goal-directedness. We suggest that our novel paradigm can be used to study the neurobehavioral and psychological mechanism underlying habits in humans. Moreover, the real-world free-operant framework can potentially be used to examine other instrumental behavior-related questions, with greater face validity in naturalistic conditions.

Striatal dopamine release tracks the relationship between actions and their consequences

The acquisition and performance of goal-directed actions has long been argued to depend on the integration of glutamatergic inputs to the posterior dorsomedial striatum (pDMS) under the modulatory influence of dopamine. Nevertheless, relatively little is known about the dynamics of striatal dopamine during goal-directed actions. To investigate this, we chronically recorded dopamine release in the pDMS as rats acquired two actions for distinct outcomes as these action-outcome associations were incremented and then subsequently degraded or reversed. We found that bilateral dopamine release scaled with action value, whereas the lateralized dopamine signal, i.e., the difference in dopamine release ipsilaterally and contralaterally to the direction of the goal-directed action, reflected the strength of the action-outcome association independently of changes in movement. Our results establish, therefore, that striatal dopamine activity during goal-directed action reflects both bilateral moment-to-moment changes in action value and the long-term action-outcome association.

The mediodorsal thalamus in executive control

Executive control, the ability to organize thoughts and action plans in real time, is a defining feature of higher cognition. Classical theories have emphasized cortical contributions to this process, but recent studies have reinvigorated interest in the role of the thalamus. Although it is well established that local thalamic damage diminishes cognitive capacity, such observations have been difficult to inform functional models. Recent progress in experimental techniques is beginning to enrich our understanding of the anatomical, physiological, and computational substrates underlying thalamic engagement in executive control. In this review, we discuss this progress and particularly focus on the mediodorsal thalamus, which regulates the activity within and across frontal cortical areas. We end with a synthesis that highlights frontal thalamocortical interactions in cognitive computations and discusses its functional implications in normal and pathological conditions.

Adaptation of sequential action benefits from timing variability related to lateral basal ganglia circuitry

Streamlined action sequences must remain flexible should stable contingencies in the environment change. By combining analyses of behavioral structure with a circuit-specific manipulation in mice, we report on a relationship between action timing variability and successful adaptation that relates to post-synaptic targets of primary motor cortical (M1) projections to dorsolateral striatum (DLS). In a two-lever instrumental task, mice formed successful action sequences by, first, establishing action scaffolds and, second, smoothly extending action duration to adapt to increased task requirements. Interruption of DLS neurons in M1 projection territories altered this process, evoking higher-rate actions that were more stereotyped in their timing, reducing opportunities for success. Based on evidence from neuronal tracing experiments, we propose that DLS neurons in M1 projection territories supply action timing variability to facilitate adaptation, a function that may involve additional downstream subcortical processing relating to collateralization of descending motor pathways to multiple basal ganglia centers.

Separate gut-brain circuits for fat and sugar reinforcement combine to promote overeating

Food is a powerful natural reinforcer that guides feeding decisions. The vagus nerve conveys internal sensory information from the gut to the brain about nutritional value; however, the cellular and molecular basis of macronutrient-specific reward circuits is poorly understood. Here, we monitor in vivo calcium dynamics to provide direct evidence of independent vagal sensing pathways for the detection of dietary fats and sugars. Using activity-dependent genetic capture of vagal neurons activated in response to gut infusions of nutrients, we demonstrate the existence of separate gut-brain circuits for fat and sugar sensing that are necessary and sufficient for nutrient-specific reinforcement. Even when controlling for calories, combined activation of fat and sugar circuits increases nigrostriatal dopamine release and overeating compared with fat or sugar alone. This work provides new insights into the complex sensory circuitry that mediates motivated behavior and suggests that a subconscious internal drive to consume obesogenic diets (e.g., those high in both fat and sugar) may impede conscious dieting efforts.

The DDHD2-STXBP1 interaction mediates long-term memory via generation of saturated free fatty acids

The phospholipid and free fatty acid (FFA) composition of neuronal membranes plays a crucial role in learning and memory, but the mechanisms through which neuronal activity affects the brain's lipid landscape remain largely unexplored. The levels of saturated FFAs, particularly of myristic acid (C14:0), strongly increase during neuronal stimulation and memory acquisition, suggesting the involvement of phospholipase A1 (PLA1) activity in synaptic plasticity. Here, we show that genetic ablation of the PLA1 isoform DDHD2 in mice dramatically reduces saturated FFA responses to memory acquisition across the brain. Furthermore, DDHD2 loss also decreases memory performance in reward-based learning and spatial memory models prior to the development of neuromuscular deficits that mirror human spastic paraplegia. Via pulldown-mass spectrometry analyses, we find that DDHD2 binds to the key synaptic protein STXBP1. Using STXBP1/2 knockout neurosecretory cells and a haploinsufficient STXBP1+/- mouse model of human early infantile encephalopathy associated with intellectual disability and motor dysfunction, we show that STXBP1 controls targeting of DDHD2 to the plasma membrane and generation of saturated FFAs in the brain. These findings suggest key roles for DDHD2 and STXBP1 in lipid metabolism and in the processes of synaptic plasticity, learning, and memory.

The behavioral origins of phylogenic responses and ontogenic habits

An examination of innate behavior and its possible origins suggests parallels with the formation of habitual behavior. Inflexible but adaptive responses-innate reflexive behavior, Pavlovian conditioned responses, and operant habits-may have evolved from variable behavior in phylogeny and ontogeny. This form of "plasticity-first" scientific narrative was unpopular post-Darwin but has recently gained credibility in evolutionary biology. The present article seeks to identify originating events and contingencies contributing to such inflexible but adaptive behavior at both phylogenic and ontogenic levels of selection. In ontogeny, the development of inflexible performance (i.e., habit) from variable operant behavior is reminiscent of the genetic accommodation of initially variable phylogenic traits. The effects characteristic of habit (e.g., unresponsiveness to reinforcer devaluation) are explicable as the result of a conflict between behaviors at distinct levels of selection. The present interpretation validates the practice of seeking hard analogies between evolutionary biology and operant behavior. Finding such parallels implies the validity of a claim that organismal behavior, both innate and learned, is a product of selection by consequences. A complete and coherent account of organismal behavior may ultimately focus on functional selective histories in much the same way evolutionary biology does with its subject matter.

The temporal structure of goal-directed and habitual operant behavior

Operant behavior can reflect the influence of goal-directed and habitual processes. These can be distinguished by changes to response rate following devaluation of the reinforcing outcome. Whether a response is goal directed or habitual depends on whether devaluation affects response rate. Response rate can be decomposed into frequencies of bouts and pauses by analyzing the distribution of interresponse times. This study sought to characterize goal-directed and habitual behaviors in terms of bout-initiation rate, within-bout response rate, bout length, and bout duration. Data were taken from three published studies that compared sensitivity to devaluation following brief and extended training with variable-interval schedules. Analyses focused on goal-directed and habitual responding, a comparison of a habitual response to a similarly trained response that had been converted back to goal-directed status after a surprising event, and a demonstration of contextual control of habit and goal direction in the same subjects. Across experiments and despite responses being clearly distinguished as goal directed and habitual by total response rate, analyses of bout-initiation rate, within-bout rate, bout length, and bout duration did not reveal a pattern that distinguished goal-directed from habitual responding.

Habit and persistence

Voluntary behaviors (operants) can come in two varieties: Goal-directed actions, which are emitted based on the remembered value of the reinforcer, and habits, which are evoked by antecedent cues and performed without the reinforcer's value in active memory. The two are perhaps most clearly distinguished with the reinforcer-devaluation test: Goal-directed actions are suppressed when the reinforcer is separately devalued and responding is tested in extinction, and habitual behaviors are not. But what is the function of habit learning? Habits are often thought to be strong and unusually persistent. The present selective review examines this idea by asking whether habits identified by the reinforcer-devaluation test are more resistant to extinction, resistant to the effects of other contingency change, vulnerable to relapse, resistant to the weakening effects of context change, or permanently in place once they are learned. Surprisingly little evidence supports the idea that habits are permanent or more persistent. Habits are more context-specific than goal-directed actions are. Methods that make behavior persistent do not necessarily work by encouraging habit. The function of habit learning may not be to make a behavior strong or more persistent but to make it automatic and efficient in a particular context.

A neural mechanism for conserved value computations integrating information and rewards

Behavioral and economic theory dictate that we decide between options based on their values. However, humans and animals eagerly seek information about uncertain future rewards, even when this does not provide any objective value. This implies that decisions are made by endowing information with subjective value and integrating it with the value of extrinsic rewards, but the mechanism is unknown. Here, we show that human and monkey value judgements obey strikingly conserved computational principles during multi-attribute decisions trading off information and extrinsic reward. We then identify a neural substrate in a highly conserved ancient structure, the lateral habenula (LHb). LHb neurons signal subjective value, integrating information's value with extrinsic rewards, and the LHb predicts and causally influences ongoing decisions. Neurons in key input areas to the LHb largely signal components of these computations, not integrated value signals. Thus, our data uncover neural mechanisms of conserved computations underlying decisions to seek information about the future.

Multiple Memory Subsystems: Reconsidering Memory in the Mind and Brain

The multiple-memory-systems framework-that distinct types of memory are supported by distinct brain systems-has guided learning and memory research for decades. However, recent work challenges the one-to-one mapping between brain structures and memory types central to this taxonomy, with key memory-related structures supporting multiple functions across substructures. Here we integrate cross-species findings in the hippocampus, striatum, and amygdala to propose an updated framework of multiple memory subsystems (MMSS). We provide evidence for two organizational principles of the MMSS theory: First, opposing memory representations are colocated in the same brain structures; second, parallel memory representations are supported by distinct structures. We discuss why this burgeoning framework has the potential to provide a useful revision of classic theories of long-term memory, what evidence is needed to further validate the framework, and how this novel perspective on memory organization may guide future research.

Striatal Acetylcholine and Dopamine Interactions Produce Situationappropriate Action Selection

Individuals often learn how to perform new actions for particular outcomes against a complex background of existing action-outcome associations. As such, this new knowledge can interfere or even compete with existing knowledge, such that individuals must use internal and external cues to determine which action is appropriate to the current situation. The question thus remains as to how this problem is solved at a neural level. Research over the last decade or so has begun to determine how the brain achieves situation-appropriate action selection. Several converging lines of evidence suggest that it is achieved through the complex interactions of acetylcholine and dopamine within the striatum in a manner that relies on glutamatergic inputs from the cortex and thalamus. Here we briefly review this evidence, then relate it to several very recent findings to provide new, speculative insights regarding the precise nature of striatal acetylcholine/dopamine interaction dynamics and their relation to situation- appropriate action selection.

Chronic alcohol exposure differentially alters calcium activity of striatal cell populations during actions

Alcohol Use Disorder (AUD) can induce long lasting alterations to executive function. This includes altered action control, which can manifest as dysfunctional goal-directed control. Cortical and striatal circuits mediate goal-directed control over behavior, and prior research has found chronic alcohol disrupts these circuits. In particular, prior in vivo and ex vivo work have identified alterations to function and activity of dorsal medial striatum (DMS), which is necessary for goal-directed control. However, unknown is whether these alterations manifest as altered activity of select DMS populations during behavior. Here we examine effects of prior chronic alcohol exposure on calcium activity modulation during action-related behaviors via fiber photometry of genetically-identified DMS populations including the direct and indirect output pathways, and fast-spiking interneurons. We find that prior chronic alcohol exposure leads to increased calcium modulation of the direct pathway during action related behavior. In contrast, prior chronic alcohol exposure led to decreased calcium activity modulation of the indirect pathway and the fast-spiking interneuron population around action-related events. Together, our findings suggest an imbalance in striatal activity during action control. This disruption may contribute to the altered goal-directed control previously reported.

Distinct value computations support rapid sequential decisions

The value of the environment determines animals' motivational states and sets expectations for error-based learning1-3. How are values computed? Reinforcement learning systems can store or cache values of states or actions that are learned from experience, or they can compute values using a model of the environment to simulate possible futures3. These value computations have distinct trade-offs, and a central question is how neural systems decide which computations to use or whether/how to combine them4-8. Here we show that rats use distinct value computations for sequential decisions within single trials. We used high-throughput training to collect statistically powerful datasets from 291 rats performing a temporal wagering task with hidden reward states. Rats adjusted how quickly they initiated trials and how long they waited for rewards across states, balancing effort and time costs against expected rewards. Statistical modeling revealed that animals computed the value of the environment differently when initiating trials versus when deciding how long to wait for rewards, even though these decisions were only seconds apart. Moreover, value estimates interacted via a dynamic learning rate. Our results reveal how distinct value computations interact on rapid timescales, and demonstrate the power of using high-throughput training to understand rich, cognitive behaviors.

Theory of reinforcement schedules

The three principles of reinforcement are (1) events such as incentives and reinforcers increase the activity of an organism; (2) that activity is bounded by competition from other responses; and (3) animals approach incentives and their signs, guided by their temporal and physical conditions, together called the "contingencies of reinforcement." Mathematical models of each of these principles comprised mathematical principles of reinforcement (MPR; Killeen, 1994). Over the ensuing decades, MPR was extended to new experimental contexts. This article reviews the basic theory and its extensions to satiation, warm-up, extinction, sign tracking, pausing, and sequential control in progressive-ratio and multiple schedules. In the latter cases, a single equation balancing target and competing responses governs behavioral contrast and behavioral momentum. Momentum is intrinsic in the fundamental equations, as behavior unspools more slowly from highly aroused responses conditioned by higher rates of incitement than it does from responses from leaner contexts. Habits are responses that have accrued substantial behavioral momentum. Operant responses, being predictors of reinforcement, are approached by making them: The sight and feel of a paw on a lever is approached by placing paw on lever, as attempted for any sign of reinforcement. Behavior in concurrent schedules is governed by approach to momentarily richer patches (melioration). Applications of MPR in behavioral pharmacology and delay discounting are noted.

A dual-pathway architecture enables chronic stress to promote habit formation

Chronic stress can change how we learn and, thus, how we make decisions by promoting the formation of inflexible, potentially maladaptive, habits. Here we investigated the neuronal circuit mechanisms that enable this. Using a multifaceted approach in male and female mice, we reveal a dual pathway, amygdala-striatal, neuronal circuit architecture by which a recent history of chronic stress shapes learning to disrupt flexible goal-directed behavior in favor of inflexible habits. Chronic stress inhibits activity of basolateral amygdala projections to the dorsomedial striatum to impede the action-outcome learning that supports flexible, goal-directed decisions. Stress also increases activity in direct central amygdala projections to the dorsomedial striatum to promote the formation of rigid, inflexible habits. Thus, stress exerts opposing effects on two amygdala-striatal pathways to promote premature habit formation. These data provide neuronal circuit insights into how chronic stress shapes learning and decision making, and help understand how stress can lead to the disrupted decision making and pathological habits that characterize substance use disorders and other psychiatric conditions.

The Motivational Determinants of Human Action, Their Neural Bases and Functional Impact in Adolescents With Obsessive-Compulsive Disorder

Background

Establishing the motivational influences on human action is essential for understanding choice and decision making in health and disease. Here we used tests of value-based decision making, manipulating both predicted and experienced reward values to assess the motivational control of goal-directed action in healthy adolescents and those with obsessive-compulsive disorder (OCD).

Methods

After instrumental training on a two action-two outcome probabilistic task, adolescents (n = 21) underwent Pavlovian conditioning using distinct stimuli predicting either the instrumental outcomes, a third outcome, or nothing. We then assessed functional magnetic resonance imaging during choice tests in which we varied the predicted value, using specific and general Pavlovian-instrumental transfer, and the experienced value, using outcome devaluation. To establish functional significance, we tested a matched cohort of adolescents with OCD (n = 20).

Results

In healthy adolescents, both predicted and experienced values influenced the performance of goal-directed actions, mediated by distinct orbitofrontal-striatal circuits involving the lateral orbitofrontal cortex (OFC) and medial OFC, respectively. However, in adolescents with OCD, choice was insensitive to changes in either predicted or experienced values. These impairments were related to hypoactivity in the lateral OFC and hyperactivity in the medial OFC during specific Pavlovian-instrumental transfer and hypoactivity in the anterior prefrontal cortex, caudate nucleus, and their connectivity in the devaluation test.

Conclusions

We found that predicted and experienced values exerted a potent influence on the performance of goal-directed actions in adolescents via distinct orbitofrontal- and prefrontal-striatal circuits. Furthermore, the influence of these motivational processes was severely blunted in OCD, as was the functional segregation of circuits involving medial and lateral OFC, producing dysregulated action control.

Distinct value computations support rapid sequential decisions

The value of the environment determines animals' motivational states and sets expectations for error-based learning1-3. How are values computed? Reinforcement learning systems can store or "cache" values of states or actions that are learned from experience, or they can compute values using a model of the environment to simulate possible futures3. These value computations have distinct trade-offs, and a central question is how neural systems decide which computations to use or whether/how to combine them4-8. Here we show that rats use distinct value computations for sequential decisions within single trials. We used high-throughput training to collect statistically powerful datasets from 291 rats performing a temporal wagering task with hidden reward states. Rats adjusted how quickly they initiated trials and how long they waited for rewards across states, balancing effort and time costs against expected rewards. Statistical modeling revealed that animals computed the value of the environment differently when initiating trials versus when deciding how long to wait for rewards, even though these decisions were only seconds apart. Moreover, value estimates interacted via a dynamic learning rate. Our results reveal how distinct value computations interact on rapid timescales, and demonstrate the power of using high-throughput training to understand rich, cognitive behaviors.

Pathways to the persistence of drug use despite its adverse consequences

The persistence of drug taking despite its adverse consequences plays a central role in the presentation, diagnosis, and impacts of addiction. Eventual recognition and appraisal of these adverse consequences is central to decisions to reduce or cease use. However, the most appropriate ways of conceptualizing persistence in the face of adverse consequences remain unclear. Here we review evidence that there are at least three pathways to persistent use despite the negative consequences of that use. A cognitive pathway for recognition of adverse consequences, a motivational pathway for valuation of these consequences, and a behavioral pathway for responding to these adverse consequences. These pathways are dynamic, not linear, with multiple possible trajectories between them, and each is sufficient to produce persistence. We describe these pathways, their characteristics, brain cellular and circuit substrates, and we highlight their relevance to different pathways to self- and treatment-guided behavior change.

Dopaminergic and prefrontal dynamics co-determine mouse decisions in a spatial gambling task

The neural mechanisms by which animals initiate goal-directed actions, choose between options, or explore opportunities remain unknown. Here, we develop a spatial gambling task in which mice, to obtain intracranial self-stimulation rewards, self-determine the initiation, direction, vigor, and pace of their actions based on their knowledge of the outcomes. Using electrophysiological recordings, pharmacology, and optogenetics, we identify a sequence of oscillations and firings in the ventral tegmental area (VTA), orbitofrontal cortex (OFC), and prefrontal cortex (PFC) that co-encodes and co-determines self-initiation and choices. This sequence appeared with learning as an uncued realignment of spontaneous dynamics. Interactions between the structures varied with the reward context, particularly the uncertainty associated with the different options. We suggest that self-generated choices arise from a distributed circuit based on an OFC-VTA core determining whether to wait for or initiate actions, while the PFC is specifically engaged by reward uncertainty in action selection and pace.

Emotion, motivation, decision-making, the orbitofrontal cortex, anterior cingulate cortex, and the amygdala

The orbitofrontal cortex and amygdala are involved in emotion and in motivation, but the relationship between these functions performed by these brain structures is not clear. To address this, a unified theory of emotion and motivation is described in which motivational states are states in which instrumental goal-directed actions are performed to obtain rewards or avoid punishers, and emotional states are states that are elicited when the reward or punisher is or is not received. This greatly simplifies our understanding of emotion and motivation, for the same set of genes and associated brain systems can define the primary or unlearned rewards and punishers such as sweet taste or pain. Recent evidence on the connectivity of human brain systems involved in emotion and motivation indicates that the orbitofrontal cortex is involved in reward value and experienced emotion with outputs to cortical regions including those involved in language, and is a key brain region involved in depression and the associated changes in motivation. The amygdala has weak effective connectivity back to the cortex in humans, and is implicated in brainstem-mediated responses to stimuli such as freezing and autonomic activity, rather than in declarative emotion. The anterior cingulate cortex is involved in learning actions to obtain rewards, and with the orbitofrontal cortex and ventromedial prefrontal cortex in providing the goals for navigation and in reward-related effects on memory consolidation mediated partly via the cholinergic system.

Long-term effects of THC exposure on reward learning and motivated behavior in adolescent and adult male rats

RATIONALE

The endocannabinoid system makes critical contributions to reward processing, motivation, and behavioral control. Repeated exposure to THC or other cannabinoid drugs can cause persistent adaptions in the endocannabinoid system and associated neural circuitry. It remains unclear how such treatments affect the way rewards are processed and pursued.

OBJECTIVE AND METHODS

We examined if repeated THC exposure (5 mg/kg/day for 14 days) during adolescence or adulthood led to long-term changes in rats' capacity to flexibly encode and use action-outcome associations for goal-directed decision making. Effects on hedonic feeding and progressive ratio responding were also assessed.

RESULTS

THC exposure had no effect on rats' ability to flexibly select actions following reward devaluation. However, instrumental contingency degradation learning, which involves avoiding an action that is unnecessary for reward delivery, was augmented in rats with a history of adult but not adolescent THC exposure. THC-exposed rats also displayed more vigorous instrumental behavior in this study, suggesting a motivational enhancement. A separate experiment found that while THC exposure had no effect on hedonic feeding behavior, it increased rats' willingness to work for food on a progressive ratio schedule, an effect that was more pronounced when THC was administered to adults. Adolescent and adult THC exposure had opposing effects on the CB1 receptor dependence of progressive ratio performance, decreasing and increasing sensitivity to rimonabant-induced behavioral suppression, respectively.

CONCLUSIONS

Our findings reveal that exposure to a translationally relevant THC exposure regimen induces long-lasting, age-dependent alterations in cognitive and motivational processes that regulate the pursuit of rewards.

Restoring the youthful state of striatal plasticity in aged mice re-enables cognitive control of action

Multidisciplinary evidence suggests that the control of voluntary action arbitrates between two major forms of behavioral processing: cognitively guided (or goal directed) and autonomously guided (or habitual). Brain-state irregularities affecting the striatum-such as aging-commonly shift control toward the latter, although the responsible neural mechanisms remain unknown. Combining instrumental conditioning with cell-specific mapping and chemogenetics in striatal neurons, we explored strategies that invigorate goal-directed capacity in aged mice. We found that, under conditions favoring goal-directed control, aged animals resiliently expressed autonomously guided behavior, a response that was underpinned by a characteristic one-to-one functional engagement of the two main neuronal populations in the striatum-D1- and D2-dopamine receptor-expressing spiny projection neurons (SPNs). Chemogenetically induced desensitization of D2-SPN signaling in aged transgenic mice recapitulated the striatal plasticity state observed in young mice, an effect that shifted behavior toward vigorous, goal-directed action. Our findings contribute to the understanding of the neural bases of behavioral control and propose neural system interventions that enhance cognitive functioning in habit-prone brains.

Error-related signaling in nucleus accumbens D2 receptor-expressing neurons guides inhibition-based choice behavior in mice

Learned associations between environmental cues and the outcomes they predict (cue-outcome associations) play a major role in behavioral control, guiding not only which responses we should perform, but also which we should inhibit, in order to achieve a specific goal. The encoding of such cue-outcome associations, as well as the performance of cue-guided choice behavior, is thought to involve dopamine D1 and D2 receptor-expressing medium spiny neurons (D1-/D2-MSNs) of the nucleus accumbens (NAc). Here, using a visual discrimination task in male mice, we assessed the role of NAc D1-/D2-MSNs in cue-guided inhibition of inappropriate responding. Cell-type specific neuronal silencing and in-vivo imaging revealed NAc D2-MSNs to contribute to inhibiting behavioral responses, with activation of NAc D2-MSNs following response errors playing an important role in optimizing future choice behavior. Our findings indicate that error-signaling by NAc D2-MSNs contributes to the ability to use environmental cues to inhibit inappropriate behavior.

Computationally Informed Interventions for Targeting Compulsive Behaviors

Compulsive behaviors are central to addiction and obsessive-compulsive disorder and can be understood as a failure of adaptive decision making. Particularly, they can be conceptualized as an imbalance in behavioral control, such that behavior is guided predominantly by learned rather than inferred outcome expectations. Inference is a computational process required for adaptive behavior, and recent work across species has identified the neural circuitry that supports inference-based decision making. This includes the orbitofrontal cortex, which has long been implicated in disorders of compulsive behavior. Inspired by evidence that modulating orbitofrontal cortex activity can alter inference-based behaviors, here we discuss noninvasive approaches to target these circuits in humans. Specifically, we discuss the potential of network-targeted transcranial magnetic stimulation and real-time neurofeedback to modulate the neural underpinnings of inference. Both interventions leverage recent advances in our understanding of the neurocomputational mechanisms of inference-based behavior and may be used to complement current treatment approaches for behavioral disorders.

Circuit dysfunctions of associative and sensorimotor basal ganglia loops in alcohol use disorder: insights from animal models

Persons that develop Alcohol Use Disorder (AUD) experience behavioral changes that include compulsion to seek and take alcohol despite its negative consequences on the person's psychosocial, health and economic spheres, inability to limit alcohol intake and a negative emotional/ motivational state that emerges during withdrawal. During all the stages of AUD executive functions, i.e. the person's ability to direct their behavior towards a goal, working memory and cognitive flexibility are eroded. Animal models of AUD recapitulate aspects of action selection impairment and offer the opportunity to benchmark the underlying circuit mechanisms. Here we propose a circuit-based approach to AUD research focusing on recent advances in behavioral analysis, neuroanatomy, genetics, and physiology to guide future research in the field.

The positive valence system, adaptive behaviour and the origins of reward

Although the hey-day of motivation as an area of study is long past, the issues with which motivational theorists grappled have not grown less important: i.e. the development of deterministic explanations for the particular tuning of the nervous system to specific changes in the internal and external environment and the organisation of adaptive behavioural responses to those changes. Here, we briefly elaborate these issues in describing the structure and function of the 'positive valence system'. We describe the origins of adaptive behaviour in an ascending arousal system, sensitive to peripheral regulatory changes, that modulates and activates various central motivational states. Associations between these motivational states and sensory inputs underlie evaluative conditioning and generate the representation of the 'unconditioned' stimuli fundamental to Pavlovian conditioning. As a consequence, associations with these stimuli can generate Pavlovian conditioned responses through the motivational control of stimulus event associations with sensory and affective components of the valence system to elicit conditioned orienting, consummatory and preparatory responses, particularly the affective responses reflecting Pavlovian excitation and inhibition, arousal and reinforcement, the latter used to control the formation of habits. These affective processes also provoke emotional responses, allowing the externalisation of positive valence in hedonic experience to generate the goal or reward values that mediate goal-directed action. Together these processes form the positive valence system, ensure the maintenance of adaptive behaviour and, through the association of sensory events and emotional responses through consummatory experience, provide the origins of reward.

The role of the bed nucleus of the stria terminalis in the motivational control of instrumental action

We review recent studies assessing the role of the bed nucleus of the stria terminalis (BNST) in the motivational control of instrumental conditioning. This evidence suggests that the BNST and central nucleus of the amygdala (CeA) form a circuit that modulates the ventral tegmental area (VTA) input to the nucleus accumbens core (NAc core) to control the influence of Pavlovian cues on instrumental performance. In support of these claims, we found that activity in the oval region of BNST was increased by instrumental conditioning, as indexed by phosphorylated ERK activity (Experiment 1), but that this increase was not due to exposure to the instrumental contingency or to the instrumental outcome per se (Experiment 2). Instead, BNST activity was most significantly incremented in a test conducted when the instrumental outcome was anticipated but not delivered, suggesting a role for BNST in the motivational effects of anticipated outcomes on instrumental performance. To test this claim, we examined the effect of NMDA-induced cell body lesions of the BNST on general Pavlovian-to-instrumental transfer (Experiment 3). These lesions had no effect on instrumental performance or on conditioned responding during Pavlovian conditioning to either an excitory conditioned stimulus (CS) or a neutral CS (CS0) but significantly attenuated the excitatory effect of the Pavlovian CS on instrumental performance. These data are consistent with the claim that the BNST mediates the general excitatory influence of Pavlovian cues on instrumental performance and suggest BNST activity may be central to CeA-BNST modulation of a VTA-NAc core circuit in incentive motivation.

Orbitofrontal cortex populations are differentially recruited to support actions

The ability to use information from one's prior actions is necessary for decision-making. While orbitofrontal cortex (OFC) has been hypothesized as key for inferences made using cue and value-related information, whether OFC populations contribute to the use of information from volitional actions to guide behavior is not clear. Here, we used a self-paced lever-press hold-down task in which mice infer prior lever-press durations to guide subsequent action performance. We show that the activity of genetically identified lateral OFC (lOFC) subpopulations differentially instantiate current and prior action information during ongoing action execution. Transient state-dependent lOFC circuit disruptions of specified subpopulations reduced the encoding of ongoing press durations but did not disrupt the use of prior action information to guide future action performance. In contrast, a chronic functional loss of lOFC circuit activity resulted in increased reliance on recently executed lever-press durations and impaired contingency reversal, suggesting the recruitment of compensatory mechanisms that resulted in repetitive action control. Our results identify a novel role for lOFC in the integration of action information to guide adaptive behavior.

The effects of induced positive and negative affect on Pavlovian-instrumental interactions

Across species, animals have an intrinsic drive to approach appetitive stimuli and to withdraw from aversive stimuli. In affective science, influential theories of emotion link positive affect with strengthened behavioural approach and negative affect with avoidance. Based on these theories, we predicted that individuals' positive and negative affect levels should particularly influence their behaviour when innate Pavlovian approach/avoidance tendencies conflict with learned instrumental behaviours. Here, across two experiments - exploratory Experiment 1 (N = 91) and a preregistered confirmatory Experiment 2 (N = 335) - we assessed how induced positive and negative affect influenced Pavlovian-instrumental interactions in a reward/punishment Go/No-Go task. Contrary to our hypotheses, we found no evidence for a main effect of positive/negative affect on either approach/avoidance behaviour or Pavlovian-instrumental interactions. However, we did find evidence that the effects of induced affect on behaviour were moderated by individual differences in self-reported behavioural inhibition and gender. Exploratory computational modelling analyses explained these demographic moderating effects as arising from positive correlations between demographic factors and individual differences in the strength of Pavlovian-instrumental interactions. These findings serve to sharpen our understanding of the effects of positive and negative affect on instrumental behaviour.

Making habits measurable beyond what they are not: A focus on associative dual-process models

Habits are the subject of intense international research. Under the associative dual-process model the outcome devaluation paradigm has been used extensively to classify behaviours as being either goal-directed (sensitive to shifts in the value of associated outcomes) or habitual (triggered by stimuli without anticipation of consequences). This has proven to be a useful framework for studying the neurobiology of habit and relevance of habits in clinical psychopathology. However, in recent years issues have been raised about this rather narrow definition of habits in comparison to habitual behaviour experienced in the real world. Specifically, defining habits as the absence of goal-directed control, the very specific set-ups required to demonstrate habit experimentally and the lack of direct evidence for habits as stimulus-response behaviours are viewed as problematic. In this review paper we address key critiques that have been raised about habit research within the framework of the associative dual-process model. We then highlight novel research approaches studying different features of habits with methods that expand beyond traditional paradigms.

Amygdala-cortical collaboration in reward learning and decision making

Adaptive reward-related decision making requires accurate prospective consideration of the specific outcome of each option and its current desirability. These mental simulations are informed by stored memories of the associative relationships that exist within an environment. In this review, I discuss recent investigations of the function of circuitry between the basolateral amygdala (BLA) and lateral (lOFC) and medial (mOFC) orbitofrontal cortex in the learning and use of associative reward memories. I draw conclusions from data collected using sophisticated behavioral approaches to diagnose the content of appetitive memory in combination with modern circuit dissection tools. I propose that, via their direct bidirectional connections, the BLA and OFC collaborate to help us encode detailed, outcome-specific, state-dependent reward memories and to use those memories to enable the predictions and inferences that support adaptive decision making. Whereas lOFC→BLA projections mediate the encoding of outcome-specific reward memories, mOFC→BLA projections regulate the ability to use these memories to inform reward pursuit decisions. BLA projections to lOFC and mOFC both contribute to using reward memories to guide decision making. The BLA→lOFC pathway mediates the ability to represent the identity of a specific predicted reward and the BLA→mOFC pathway facilitates understanding of the value of predicted events. Thus, I outline a neuronal circuit architecture for reward learning and decision making and provide new testable hypotheses as well as implications for both adaptive and maladaptive decision making.

Ideomotor learning: Time to generalize a longstanding principle

The ideomotor principle holds that anticipating the sensory consequences of a movement triggers an associated motor response. Even though this framework dates back to the 19th century, it continues to lie at the heart of many contemporary approaches to human action control. Here we specifically focus on the ideomotor learning mechanism that has to precede action initiation via effect anticipation. Traditional approaches to this learning mechanism focused on establishing novel action-effect (or response-effect) associations. Here we apply the theoretical concept of common coding for action and perception to argue that the same learning principle should result in response-response and stimulus-stimulus associations just as well. Generalizing ideomotor learning in such a way results in a powerful and general framework of ideomotor action control, and it allows for integrating the two seemingly separate fields of ideomotor approaches and hierarchical learning.

Nucleus Accumbens Core Dopamine D2 Receptor-Expressing Neurons Control Reversal Learning but Not Set-Shifting in Behavioral Flexibility in Male Mice

The ability to use environmental cues to flexibly guide responses is crucial for adaptive behavior and is thought to be controlled within a series of cortico-basal ganglia-thalamo-cortical loops. Previous evidence has indicated that different prefrontal cortical regions control dissociable aspects of behavioral flexibility, with the medial prefrontal cortex (mPFC) necessary for the ability to shift attention to a novel strategy (set-shifting) and the orbitofrontal cortex (OFC) necessary for shifting attention between learned stimulus-outcome associations (reversal learning). The nucleus accumbens (NAc) is a major downstream target of both the mPFC and the OFC; however, its role in controlling reversal learning and set-shifting abilities is still unclear. Here we investigated the contribution of the two major NAc neuronal populations, medium spiny neurons expressing either dopamine D1 or D2 receptors (D1-/D2-MSNs), in guiding reversal learning and set-shifting in an attentional set-shifting task (ASST). Persistent inhibition of neurotransmitter release from NAc D2-MSNs, but not D1-MSNs, resulted in an impaired ability for reversal learning, but not set-shifting in male mice. These findings suggest that NAc D2-MSNs play a critical role in suppressing responding toward specific learned cues that are now associated with unfavorable outcomes (i.e., in reversal stages), but not in the suppression of more general learned strategies (i.e., in set-shifting). This study provides further evidence for the anatomical separation of reversal learning and set-shifting abilities within cortico-basal ganglia-thalamo-cortical loops.

A neuromuscular model of human locomotion combines spinal reflex circuits with voluntary movements

Existing models of human walking use low-level reflexes or neural oscillators to generate movement. While appropriate to generate the stable, rhythmic movement patterns of steady-state walking, these models lack the ability to change their movement patterns or spontaneously generate new movements in the specific, goal-directed way characteristic of voluntary movements. Here we present a neuromuscular model of human locomotion that bridges this gap and combines the ability to execute goal directed movements with the generation of stable, rhythmic movement patterns that are required for robust locomotion. The model represents goals for voluntary movements of the swing leg on the task level of swing leg joint kinematics. Smooth movements plans towards the goal configuration are generated on the task level and transformed into descending motor commands that execute the planned movements, using internal models. The movement goals and plans are updated in real time based on sensory feedback and task constraints. On the spinal level, the descending commands during the swing phase are integrated with a generic stretch reflex for each muscle. Stance leg control solely relies on dedicated spinal reflex pathways. Spinal reflexes stimulate Hill-type muscles that actuate a biomechanical model with eight internal joints and six free-body degrees of freedom. The model is able to generate voluntary, goal-directed reaching movements with the swing leg and combine multiple movements in a rhythmic sequence. During walking, the swing leg is moved in a goal-directed manner to a target that is updated in real-time based on sensory feedback to maintain upright balance, while the stance leg is stabilized by low-level reflexes and a behavioral organization switching between swing and stance control for each leg. With this combination of reflex-based stance leg and voluntary, goal-directed control of the swing leg, the model controller generates rhythmic, stable walking patterns in which the swing leg movement can be flexibly updated in real-time to step over or around obstacles.

The Neural Bases of Action-Outcome Learning in Humans

From an associative perspective the acquisition of new goal-directed actions requires the encoding of specific action-outcome (AO) associations and, therefore, sensitivity to the validity of an action as a predictor of a specific outcome relative to other events. Although competitive architectures have been proposed within associative learning theory to achieve this kind of identity-based selection, whether and how these architectures are implemented by the brain is still a matter of conjecture. To investigate this issue, we trained human participants to encode various AO associations while undergoing functional neuroimaging (fMRI). We then degraded one AO contingency by increasing the probability of the outcome in the absence of its associated action while keeping other AO contingencies intact. We found that this treatment selectively reduced performance of the degraded action. Furthermore, when a signal predicted the unpaired outcome, performance of the action was restored, suggesting that the degradation effect reflects competition between the action and the context for prediction of the specific outcome. We used a Kalman filter to model the contribution of different causal variables to AO learning and found that activity in the medial prefrontal cortex (mPFC) and the dorsal anterior cingulate cortex (dACC) tracked changes in the association of the action and context, respectively, with regard to the specific outcome. Furthermore, we found the mPFC participated in a network with the striatum and posterior parietal cortex to segregate the influence of the various competing predictors to establish specific AO associations.SIGNIFICANCE STATEMENT Humans and other animals learn the consequences of their actions, allowing them to control their environment in a goal-directed manner. Nevertheless, it is unknown how we parse environmental causes from the effects of our own actions to establish these specific action-outcome (AO) relationships. Here, we show that the brain learns the causal structure of the environment by segregating the unique influence of actions from other causes in the medial prefrontal and anterior cingulate cortices and, through a network of structures, including the caudate nucleus and posterior parietal cortex, establishes the distinct causal relationships from which specific AO associations are formed.

Information normally considered task-irrelevant drives decision-making and affects premotor circuit recruitment

Decision-making is a continuous and dynamic process with prior experience reflected in and used by the brain to guide adaptive behavior. However, most neurobiological studies constrain behavior and/or analyses to task-related variables, not accounting for the continuous internal and temporal space in which they occur. We show mice rely on information learned through recent and longer-term experience beyond just prior actions and reward - including checking behavior and the passage of time - to guide self-initiated, self-paced, and self-generated actions. These experiences are represented in secondary motor cortex (M2) activity and its projections into dorsal medial striatum (DMS). M2 integrates this information to bias strategy-level decision-making, and DMS projections reflect specific aspects of this recent experience to guide actions. This suggests diverse aspects of experience drive decision-making and its neural representation, and shows premotor corticostriatal circuits are crucial for using selective aspects of experiential information to guide adaptive behavior.

Relapse after incentivized choice treatment in humans: A laboratory model for studying behavior change

Behavior change interventions that incentivize desired behavior are highly effective for improving personal health, but difficult to maintain long term. Relapse is common and examining the mechanisms that contribute to relapse in experimental settings can identify processes relevant to substance abuse treatment. We developed a laboratory task that parallels a recent operant model of relapse after incentivized choice reported in the rodent laboratory. In two experiments, undergraduate participants first learned to make an operant response (keyboard button; R1) to earn a reinforcer consisting of an image of a preferred snack food (O1). In a second phase (Phase 2), R1 was still reinforced, but a new response (R2) was introduced and reinforced with a different reinforcer (a coin; O2). In a test phase, contingent incentives for R2 were removed (extinction) and relapse of R1 was assessed. Experiment 1 found that the O2 contingency suppressed R1 during Phase 2, and R1 relapsed rapidly in the test. Neither effect was consistently related to O2 value. Experiment 2 examined whether noncontingent presentations of O1 or O2 during the test could weaken relapse. Here, we found that noncontingent reinforcers did little to reduce or slow the increase in R1 responding. The present experiments highlight a laboratory approach to studying variables that may influence relapse after incentivized treatment. We identify and discuss areas for development to address differences between the present results and prior observations from animal and clinical studies. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Animal models of action control and cognitive dysfunction in Parkinson's disease

Parkinson's disease (PD) has historically been considered a motor disorder induced by a loss of dopaminergic neurons in the substantia nigra pars compacta. More recently, it has been recognized to have significant non-motor symptoms, most prominently cognitive symptoms associated with a dysexecutive syndrome. It is common in the literature to see motor and cognitive symptoms treated separately and, indeed, there has been a general call for specialized treatment of the latter, particularly in the more severe cases of PD with mild cognitive impairment and dementia. Animal studies have similarly been developed to model the motor or non-motor symptoms. Nevertheless, considerable research has established that segregating consideration of cognition from the precursors to motor movement, particularly movement associated with goal-directed action, is difficult if not impossible. Indeed, on some contemporary views cognition is embodied in action control, something that is particularly prevalent in theory and evidence relating to the integration of goal-directed and habitual control processes. The current paper addresses these issues within the literature detailing animal models of cognitive dysfunction in PD and their neural and neurochemical bases. Generally, studies using animal models of PD provide some of the clearest evidence for the integration of these action control processes at multiple levels of analysis and imply that consideration of this integrative process may have significant benefits for developing new approaches to the treatment of PD.

As soon as there was life, there was danger: the deep history of survival behaviours and the shallower history of consciousness

It is often said that fear is a universal innate emotion that we humans have inherited from our mammalian ancestors by virtue of having inherited conserved features of their nervous systems. Contrary to this common sense-based scientific point of view, I have argued that what we have inherited from our mammalian ancestors, and they from their distal vertebrate ancestors, and they from their chordate ancestors, and so forth, is not a fear circuit. It is, instead, a defensive survival circuit that detects threats, and in response, initiates defensive survival behaviours and supporting physiological adjustments. Seen in this light, the defensive survival circuits of humans and other mammals can be conceptualized as manifestations of an ancient survival function-the ability to detect danger and respond to it-that may in fact predate animals and their nervous systems, and perhaps may go back to the beginning of life. Fear, on the other hand, from my perspective, is a product of cortical cognitive circuits. This conception is not just of academic interest. It also has practical implications, offering clues as to why efforts to treat problems related to fear and anxiety are not more effective, and what might make them better. This article is part of the theme issue 'Systems neuroscience through the lens of evolutionary theory'.

The nervous system as a solution for implementing closed negative feedback control loops

Behavior can be regarded as the output of a system (action), as a function linking stimulus to response (reaction), or as an abstraction of the bidirectional relationship between the environment and the organism (interaction). When considering the latter possibility, a relevant question arises concerning how an organism can materially and continuously implement such a relationship during its lifetime in order to perpetuate itself. The feedback control approach has taken up the task of answering just that question. During the last several decades, said approach has been progressing and has started to be recognized as a paradigm shift, superseding certain canonical notions in mainstream behavior analysis, cognitive psychology, and even neuroscience. In this paper, we describe the main features of feedback control theory and its associated techniques, concentrating on its critiques of behavior analysis, as well as the commonalities they share. While some of feedback control theory's major critiques of behavior analysis arise from the fact that they focus on different levels of organization, we believe that some are legitimate and meaningful. Moreover, feedback control theory seems to blend with neurobiology more smoothly as compared to canonical behavior analysis, which only subsists in a scattered handful of fields. If this paradigm shift truly takes place, behavior analysts-whether they accept or reject this new currency-should be mindful of the basics of the feedback control approach.

Decreased putamen activation in balancing goal-directed and habitual behavior in binge eating disorder

Acute stress is associated with a shift from goal-directed to habitual behavior. This stress-induced preference for habitual behavior has been suggested as a potential mechanism by which binge eating disorder (BED) patients succumb to eating large amounts of high-caloric foods in an uncontrolled manner (i.e., binge episodes). While in healthy subjects the balance between goal-directed and habitual behavior is subserved by the anterior cingulate cortex (ACC), insular cortex, orbitofrontal cortex (OFC), anterior caudate nucleus, and posterior putamen, the brain mechanism that underlies this (possibly amplified) stress-induced behavioral shift in BED patients is currently unknown. In the current study, 76 participants (38 BED, 38 healthy controls (HCs)) learned six stimulus-response-outcome associations in a well-established instrumental learning task. Subsequently, three outcomes were selectively devalued, after which participants underwent either a stress induction procedure (Maastricht Acute Stress Test; MAST) or a no-stress control procedure. Next, the balance between goal-directed and habitual behavior was assessed during functional magnetic resonance imaging. Findings show that the balance between goal-directed and habitual behavior was associated with activity in the ACC, insula, and OFC in no-stress HCs. Although stress and BED did not modulate the balance between goal-directed and habitual behavior, BED participants displayed a smaller difference in putamen activation between trials probing goal-directed and habitual behavior compared with HCs when using a ROI approach. We conclude that putamen activity differences between BED and HC could reflect changes in monitoring of response accuracy or reward value, albeit perhaps not sufficiently to induce a measurable shift from goal-directed to habitual behavior. Future research could clarify potential boundary conditions of stress-induced shifts in instrumental behavior in BED patients.

CRF-receptor1 modulation of the dopamine projection to prelimbic cortex facilitates cognitive flexibility after acute and chronic stress

Stress reduces cognitive flexibility and dopamine D1 receptor-related activity in the prelimbic cortex (PL), effects hypothesized to depend on reduced corticotropic releasing factor receptor type 1 (CRFr1) regulation of dopamine neurons in the ventral tegmental area (VTA). We assessed this hypothesis in rats by examining the effect of chronic unpredictable restraint stress (CUS), mild acute stress, or their combination on cognitive flexibility, CRFr1 expression in the VTA and D1-related activity in PL. In Experiment 1, rats received either CUS or equivalent handling for 14 days before being trained to press two levers to earn distinct food outcomes. Initial learning was assessed using an outcome devaluation test after which cognitive flexibility was assessed by reversing the outcomes earned by the actions. Prior to each reversal training session, half the CUS and controls receiving acute stress with action-outcome updating assessed using a second devaluation test and CRFr1 expression in the VTA assessed using in-situ hybridisation. Although CUS did not itself affect action-outcome learning, its combination with acute stress blocked reversal learning and decreased VTA CRFr1 expression after acute shock. The relationship between these latter two effects was assessed in Experiment 2 by pharmacologically disconnecting the VTA and PL, unilaterally blocking neurons expressing CRFr1 in the VTA and D1 receptors in the contralateral PL during reversal learning after acute stress. Acute stress again blocked reversal learning but only in the group with VTA-PL disconnection, demonstrating that VTA CRFr1-induced facilitation of dopaminergic activity in the PL is necessary for maintaining cognitive flexibility after acute stress. [250].

•

Acute stress increased CRF receptor1 expression in the VTA.

•

Chronic stress attenuated the effect of acute stress on CRFr1 expression.

•

Chronic stress plus acute stress produced a loss of cognitive flexibility.

•

Blocking VTA CFRr1 and dopamine D1r in PL reduced cognitive flexibility following stress.