Agency and the Calibration of Motivated Behavior

Neural Computations of Threat

A host of learning, memory, and decision-making processes form the individual's response to threat and may be disrupted in anxiety and post-trauma psychopathology. Here we review the neural computations of threat, from the first encounter with a dangerous situation, through learning, storing, and updating cues that predict it, to making decisions about the optimal course of action. The overview highlights the interconnected nature of these processes and their reliance on shared neural and computational mechanisms. We propose an integrative approach to the study of threat-related processes, in which specific computations are studied across the various stages of threat experience rather than in isolation. This approach can generate new insights about the evolution, diagnosis, and treatment of threat-related psychopathology.

Controllability over stressor decreases responses in key threat-related brain areas

Controllability over stressors has major impacts on brain and behavior. In humans, however, the effect of controllability on responses to stressors is poorly understood. Using functional magnetic resonance imaging (fMRI), we investigated how controllability altered responses to a shock-plus-sound stressor with a between-group yoked design, where participants in controllable and uncontrollable groups experienced matched stressor exposure. Employing Bayesian multilevel analysis at the level of regions of interest and voxels in the insula, and standard voxelwise analysis, we found that controllability decreased stressor-related responses across threat-related regions, notably in the bed nucleus of the stria terminalis and anterior insula. Posterior cingulate cortex, posterior insula, and possibly medial frontal gyrus showed increased responses during control over stressor. Our findings support the idea that the aversiveness of stressors is reduced when controllable, leading to decreased responses across key regions involved in anxiety-related processing, even at the level of the extended amygdala.

The rational use of causal inference to guide reinforcement learning strengthens with age

Beliefs about the controllability of positive or negative events in the environment can shape learning throughout the lifespan. Previous research has shown that adults' learning is modulated by beliefs about the causal structure of the environment such that they update their value estimates to a lesser extent when the outcomes can be attributed to hidden causes. This study examined whether external causes similarly influenced outcome attributions and learning across development. Ninety participants, ages 7 to 25 years, completed a reinforcement learning task in which they chose between two options with fixed reward probabilities. Choices were made in three distinct environments in which different hidden agents occasionally intervened to generate positive, negative, or random outcomes. Participants' beliefs about hidden-agent intervention aligned with the true probabilities of the positive, negative, or random outcome manipulation in each of the three environments. Computational modeling of the learning data revealed that while the choices made by both adults (ages 18-25) and adolescents (ages 13-17) were best fit by Bayesian reinforcement learning models that incorporate beliefs about hidden-agent intervention, those of children (ages 7-12) were best fit by a one learning rate model that updates value estimates based on choice outcomes alone. Together, these results suggest that while children demonstrate explicit awareness of the causal structure of the task environment, they do not implicitly use beliefs about the causal structure of the environment to guide reinforcement learning in the same manner as adolescents and adults.

Motivational factors underlying aversive Pavlovian-instrumental transfer

While interest in active avoidance has recently been resurgent, many concerns relating to the nature of this form of learning remain unresolved. By separating stimulus and response acquisition, aversive Pavlovian-instrumental transfer can be used to measure the effect of avoidance learning on threat processing with more control than typical avoidance procedures. However, the motivational substrates that contribute to the aversive transfer effect have not been thoroughly examined. In three studies using rodents, the impact of a variety of aversive signals on shock-avoidance responding (i.e., two-way shuttling) was evaluated. Fox urine, as well as a tone paired with the delivery of the predator odor were insufficient modulatory stimuli for the avoidance response. Similarly, a signal for the absence of food did not generate appropriate aversive motivation to enhance shuttling. Only conditioned Pavlovian stimuli that had been paired with unconditioned threats were capable of augmenting shock-avoidance responding. This was true whether the signaled outcome was the same (e.g., shock) or different (e.g., klaxon) from the avoidance outcome (i.e., shock). These findings help to characterize the aversive transfer effect and provide a more thorough analysis of its generalization to warning signals for different kinds of threats. This feature of aversive motivation has not been demonstrated using conventional avoidance procedures and could be potentially useful for applying avoidance in treatment settings.

Adolescents exhibit reduced Pavlovian biases on instrumental learning

Multiple learning systems allow individuals to flexibly respond to opportunities and challenges present in the environment. An evolutionarily conserved "Pavlovian" learning mechanism couples valence and action, promoting a tendency to approach cues associated with reward and to inhibit action in the face of anticipated punishment. Although this default response system may be adaptive, these hard-wired reactions can hinder the ability to learn flexible "instrumental" actions in pursuit of a goal. Such constraints on behavioral flexibility have been studied extensively in adults. However, the extent to which these valence-specific response tendencies bias instrumental learning across development remains poorly characterized. Here, we show that while Pavlovian response biases constrain flexible action learning in children and adults, these biases are attenuated in adolescents. This adolescent-specific reduction in Pavlovian bias may promote unbiased exploration of approach and avoidance responses, facilitating the discovery of rewarding behavior in the many novel contexts that adolescents encounter.

Psychological mechanisms and functions of 5-HT and SSRIs in potential therapeutic change: Lessons from the serotonergic modulation of action selection, learning, affect, and social cognition

Uncertainty regarding which psychological mechanisms are fundamental in mediating SSRI treatment outcomes and wide-ranging variability in their efficacy has raised more questions than it has solved. Since subjective mood states are an abstract scientific construct, only available through self-report in humans, and likely involving input from multiple top-down and bottom-up signals, it has been difficult to model at what level SSRIs interact with this process. Converging translational evidence indicates a role for serotonin in modulating context-dependent parameters of action selection, affect, and social cognition; and concurrently supporting learning mechanisms, which promote adaptability and behavioural flexibility. We examine the theoretical basis, ecological validity, and interaction of these constructs and how they may or may not exert a clinical benefit. Specifically, we bridge crucial gaps between disparate lines of research, particularly findings from animal models and human clinical trials, which often seem to present irreconcilable differences. In determining how SSRIs exert their effects, our approach examines the endogenous functions of 5-HT neurons, how 5-HT manipulations affect behaviour in different contexts, and how their therapeutic effects may be exerted in humans - which may illuminate issues of translational models, hierarchical mechanisms, idiographic variables, and social cognition.

A Translational Paradigm to Study the Effects of Uncontrollable Stress in Humans

Theories on the aetiology of depression in humans are intimately linked to animal research on stressor controllability effects. However, explicit translations of established animal designs are lacking. In two consecutive studies, we developed a translational paradigm to study stressor controllability effects in humans. In the first study, we compared three groups of participants, one exposed to escapable stress, one yoked inescapable stress group, and a control group not exposed to stress. Although group differences indicated successful stress induction, the manipulation failed to differentiate groups according to controllability. In the second study, we employed an improved paradigm and contrasted only an escapable stress group to a yoked inescapable stress group. The final design successfully induced differential effects on self-reported perceived control, exhaustion, helplessness, and behavioural indices of adaptation to stress. The latter were examined in a new escape behaviour test which was modelled after the classic shuttle box animal paradigm. Contrary to the learned helplessness literature, exposure to uncontrollable stress led to more activity and exploration; however, these behaviours were ultimately not adaptive. We discuss the results and possible applications in light of the findings on learning and agency beliefs, inter-individual differences, and interventions aimed at improving resilience to stress-induced mental dysfunction.

The role of mPFC and MTL neurons in human choice under goal-conflict

Resolving approach-avoidance conflicts relies on encoding motivation outcomes and learning from past experiences. Accumulating evidence points to the role of the Medial Temporal Lobe (MTL) and Medial Prefrontal Cortex (mPFC) in these processes, but their differential contributions have not been convincingly deciphered in humans. We detect 310 neurons from mPFC and MTL from patients with epilepsy undergoing intracranial recordings and participating in a goal-conflict task where rewards and punishments could be controlled or not. mPFC neurons are more selective to punishments than rewards when controlled. However, only MTL firing following punishment is linked to a lower probability for subsequent approach behavior. mPFC response to punishment precedes a similar MTL response and affects subsequent behavior via an interaction with MTL firing. We thus propose a model where approach-avoidance conflict resolution in humans depends on outcome value tagging in mPFC neurons influencing encoding of such value in MTL to affect subsequent choice.

Reducing shock imminence eliminates poor avoidance in rats

In signaled active avoidance (SigAA), rats learn to suppress Pavlovian freezing and emit actions to remove threats and prevent footshocks. SigAA is critical for understanding aversively motivated instrumental behavior and anxiety-related active coping. However, with standard protocols ∼25% of rats exhibit high freezing and poor avoidance. This has dampened enthusiasm for the paradigm and stalled progress. We demonstrate that reducing shock imminence with long-duration warning signals leads to greater freezing suppression and perfect avoidance in all subjects. This suggests that instrumental SigAA mechanisms evolved to cope with distant harm and protocols that promote inflexible Pavlovian reactions are poorly designed to study avoidance.

Influences of early-life stress on frontolimbic circuitry: Harnessing a dimensional approach to elucidate the effects of heterogeneity in stress exposure

Early-life stress confers profound and lasting risk for developing cognitive, social, emotional, and physical health problems. The effects of stress on the developing brain contribute to this risk, with frontolimbic circuitry particularly susceptible to early experiences, possibly due to its innervation with glucocorticoid receptors and the timing of frontolimbic circuit maturation. To date, the majority of studies on stress and frontolimbic circuitry have employed a categorical approach, comparing stress-exposed versus non-stress-exposed youth. However, there is vast heterogeneity in the nature of stress exposure and in outcomes. Recent forays into understanding the psychobiological effects of stress have employed a dimensional approach focused on experiential, environmental, and temporal factors that influence the association between stress and subsequent vulnerability. This review highlights empirical findings that inform a dimensional approach to understanding the effects of stress on frontolimbic circuitry. We identify the timing, type, severity, controllability, and predictability of stress, and the degree to which a caregiver is involved, as specific features of stress that may play a substantial role in differential outcomes. We propose a framework for the effects of these features of stress on frontolimbic development that may partially determine how heterogeneity in stress exposure influences this circuitry and, ultimately, mental health.

Prefrontal cortex projections to the nucleus reuniens suppress freezing following two-way signaled avoidance training

Signaled active avoidance (SAA) behavior requires the suppression of defensive reactions, such as freezing, that conflict with the avoidance response. The neural mechanisms of this inhibitory process are not well understood. Here, we demonstrate that ventromedial prefrontal cortex projections to the nucleus reuniens of the thalamus are recruited following SAA training to suppress freezing in rats. This projection may serve as a crucial common pathway for the inhibition of innate defensive reactions that interfere with proactive behavior, thus facilitating adaptive coping.

Modeling the evolution of sensitive periods

In the past decade, there has been monumental progress in our understanding of the neurobiological basis of sensitive periods. Little is known, however, about the evolution of sensitive periods. Recent studies have started to address this gap. Biologists have built mathematical models exploring the environmental conditions in which sensitive periods are likely to evolve. These models investigate how mechanisms of plasticity can respond optimally to experience during an individual's lifetime. This paper discusses the central tenets, insights, and predictions of these models, in relation to empirical work on humans and other animals. We also discuss which future models are needed to improve the bridge between theory and data, advancing their synergy. Our paper is written in an accessible manner and for a broad audience. We hope our work will contribute to recently emerging connections between the fields of developmental neuroscience and evolutionary biology.

Controllability governs the balance between Pavlovian and instrumental action selection

A Pavlovian bias to approach reward-predictive cues and avoid punishment-predictive cues can conflict with instrumentally-optimal actions. Here, we propose that the brain arbitrates between Pavlovian and instrumental control by inferring which is a better predictor of reward. The instrumental predictor is more flexible; it can learn values that depend on both stimuli and actions, whereas the Pavlovian predictor learns values that depend only on stimuli. The arbitration theory predicts that the Pavlovian predictor will be favored when rewards are relatively uncontrollable, because the additional flexibility of the instrumental predictor is not useful. Consistent with this hypothesis, we find that the Pavlovian approach bias is stronger under low control compared to high control contexts.

Pavlovian and instrumentally driven actions often conflict when determining the best outcome. Here, the authors present an arbitration theory supported by human behavioral data where Pavlovian predictors drive action selection in an uncontrollable environment, while more flexible instrumental prediction dominates under conditions of high controllability.

The study of active avoidance: A platform for discussion

Traditional active avoidance tasks have advanced the field of aversive learning and memory for decades and are useful for studying simple avoidance responses in isolation; however, these tasks have limited clinical relevance because they do not model several key features of clinical avoidance. In contrast, platform-mediated avoidance (PMA) more closely resembles clinical avoidance because the response i) is associated with an unambiguous safe location, ii) is not associated with an artificial termination of the warning signal, and iii) is associated with a decision-based appetitive cost. Recent findings on the neuronal circuits of PMA have confirmed that amygdala-striatal circuits are essential for avoidance. In PMA, however, the prelimbic cortex facilitates the avoidance response early during the warning signal, perhaps through disinhibition of the striatum. Future studies on avoidance should account for additional factors such as sex differences and social interactions that will advance our understanding of maladaptive avoidance contributing to neuropsychiatric disorders.

Motivation and cognitive control in depression

Depression is linked to deficits in cognitive control and a host of other cognitive impairments arise as a consequence of these deficits. Despite of their important role in depression, there are no mechanistic models of cognitive control deficits in depression. In this paper we propose how these deficits can emerge from the interaction between motivational and cognitive processes. We review depression-related impairments in key components of motivation along with new cognitive neuroscience models that focus on the role of motivation in the decision-making about cognitive control allocation. Based on this review we propose a unifying framework which connects motivational and cognitive control deficits in depression. This framework is rooted in computational models of cognitive control and offers a mechanistic understanding of cognitive control deficits in depression.

Controllability affects endocrine response of adolescent male rats to stress as well as impulsivity and behavioral flexibility during adulthood

Exposure to stress during adolescence exerts a long-term impact on behavior and might contribute to the development of several neuropsychiatric disorders. In adults, control over stress has been found to protect from the negative consequences of stress, but the influence of controllability at early ages has not been extensively studied. Here, we evaluated in a rodent model the effects of repeated exposure in adolescent male rats to controllable versus uncontrollable foot-shock stress (CST or UST, respectively). Rats were assigned to three groups: non-stress (stress-naïve), CST (exposed to 8 sessions of a two-way shuttle active avoidance task over a period of 22 days) and UST (receiving the same amount of shocks as CST, regardless of their actual behavior). During adulthood, different cohorts were tested in several tasks evaluating inhibitory control and cognitive flexibility: 5-choice serial reaction time, delay-discounting, gambling test and probabilistic reversal learning. Results showed that the hypothalamic-pituitary-adrenal response to the first shock session was similar in CST and UST animals, but the response to the 8^th session was lower in CST animals. In adulthood, the UST animals presented impaired motor (but not cognitive) impulsivity and more perseverative behavior. The behavioral effects of UST were associated with increased number of D2 dopamine receptors in dorsomedial striatum, but not in other striatal regions. In summary, UST exposure during adolescence induced long-term impairments in impulsivity and compulsivity, whereas CST had only minor effects. These data support a critical role of stress uncontrollability on the long-lasting consequences of stress, as a risk factor for mental illnesses.

A Reward-Based Framework of Perceived Control

Perceived control can be broadly defined as the belief in one's ability to exert control over situations or events. It has long been known that perceived control is a major contributor toward mental and physical health as well as a strong predictor of achievements in life. However, one issue that limits a mechanistic understanding of perceived control is the heterogeneity of how the term is defined in models in psychology and neuroscience, and used in experimental settings across a wide spectrum of studies. Here, we propose a framework for studying perceived control by integrating the ideas from traditionally separate work on perceived control. Specifically, we discuss key properties of perceived control from a reward-based framework, including choice opportunity, instrumental contingency, and success/reward rate. We argue that these separate reward-related processes are integral to fostering an enhanced perception of control and influencing an individual's behavior and well-being. We draw on select studies to elucidate how these reward-related elements are implicated separately and collectively in the investigation of perceived control. We highlight the role of dopamine within corticostriatal pathways shared by reward-related processes and perceived control. Finally, through the lens of this reward-based framework of perceived control, we consider the implications of perceived control in clinical deficits and how these insights could help us better understand psychopathology and treatment options.

Individual differences in blink rate modulate the effect of instrumental control on subsequent Pavlovian responding

RATIONALE

Pavlovian conditioned responses to cues that signal threat are rapidly acquired and tend to persist over time. However, recent research suggests that the ability to actively avoid or exert control over an anticipated threat can diminish the subsequent expression of Pavlovian responses. Studies in animal models suggest that active avoidance behavior and its consequences may be mediated by dopaminergic function. In the present study, we sought to replicate the finding that active control over threat can attenuate subsequent Pavlovian responding in humans and conducted exploratory analyses testing whether individual differences in blink rate, a putative index of dopaminergic function, might modulate this effect.

METHODS

Participants underwent Pavlovian aversive conditioning, followed immediately by one of two conditions. In the active avoidance condition, participants had the opportunity to actively prevent the occurrence of an anticipated shock, whereas in a yoked extinction condition, participants passively observed the cessation of shocks, but with no ability to influence their occurrence. The following day, the conditioned stimuli were presented without shock, but both groups of participants had no opportunity to employ active instrumental responses. Blink rate was measured throughout the task, and skin conductance responses served as our index of Pavlovian conditioned responding.

RESULTS

Consistent with our previous findings, we observed that the group that could actively avoid the shock on day 1 exhibited attenuated recovery of Pavlovian conditioned responses. Further, we found that individuals in the active avoidance group with higher blink rates exhibited a more robust attenuation of spontaneous recovery.

CONCLUSION

This finding suggests that individual variation in dopaminergic function may modulate the efficacy with which active avoidance strategies can attenuate reactive Pavlovian responses.

Rethinking avoidance: Toward a balanced approach to avoidance in treating anxiety disorders

Avoidance is typically considered a maladaptive behavioral response to excessive fear and anxiety, leading to the maintenance of anxiety disorders. Exposure is a core element of cognitive-behavioral therapy for anxiety disorders. One important aspect of this treatment is repeated and prolonged exposure to a threat while discouraging patients from using avoidance strategies, such as escape or safety behaviors. We will first revisit the role of avoidance learning in the development and maintenance of anxiety disorders, including important insights from the neuroscience literature. Next, we will consider both the negative and positive aspects of avoidance for therapeutic interventions. Finally, we will explore the application of adaptive avoidance in exposure therapy for anxiety disorders. We will argue that there are occasions when avoidance behaviors can serve as effective coping strategies to enhance the person's perception of control over the environment and the potential threat. We conclude that avoidance behaviors can be a valuable therapeutic element, depending on the function of these behaviors.