The interaction of acute and chronic stress impairs model-based behavioral control

Negative affect-driven impulsivity as hierarchical model-based overgeneralization

'If your mouth is burned by milk, you blow before you eat yogurt' ('Sütten ağzı yanan yoğurdu üfleyerek yer'). This Turkish proverb advises caution based on past experiences when similar situations are encountered. However, although we may infer similarities across experiences, each situation is a complex combination of many features, and generalizing across situations based on perceived similarities may not achieve desired outcomes when obtaining them depends on more subtle or overlooked features. Here, we examine how models of generalization can uncover the model-based (MB) processes underlying reactive and rigid behaviors traditionally considered model-free (MF). Our novel conceptualization suggests that emotionally driven impulsive behaviors stem from a propensity to overgeneralize based on surface-level similarities, hindering the incorporation of other informative, discriminant cues.

Impact of provoked stress on model-free and model-based reinforcement learning in individuals with alcohol use disorder

Background

From both clinical and theoretical perspectives, understanding the functionality of evaluative reinforcement learning mechanisms (Model-Free, MF, and Model-Based, MB) under provoked stress, particularly in Alcohol Use Disorder (AUD), is crucial yet underexplored. This study aims to evaluate whether individuals with AUD who do not seek treatment show a greater tendency towards retrospective behaviors (MF) rather than prospective and deliberative simulations (MB) compared to controls. Additionally, it examines the impact of induced social stress on these decision-making processes.

Methods

A cohort comprising 117 participants, including 55 individuals with AUD and 62 controls, was examined. Acute social stress was induced through the socially evaluated cold pressor task (SECPT), followed by engagement in a Two-Step Markov task to assess MB and MF learning tendencies. We measured hypothalamic-pituitary-adrenal axis stress response using salivary cortisol levels.

Results

Both groups showed similar baseline cortisol levels and responses to the SECPT. Our findings indicate that participants with AUD exhibit a reduced reliance on MB strategies compared to those without AUD. Furthermore, stress decreases reliance on MB strategies in healthy participants, but this effect is not observed in those with AUD.

Conclusion

An atypical pattern of stress modulation impacting the balance between MB and MF reinforcement learning was identified in individuals with AUD who are not seeking treatment. Potential explanations for these findings and their clinical implications are explored.

Model-free decision-making underlies motor errors in rapid sequential movements under threat

Our movements, especially sequential ones, are usually goal-directed, i.e., coupled with task-level goals. Consequently, cognitive strategies for decision-making and motor performance are likely to influence each other. However, evidence linking decision-making strategies and motor performance remains elusive. Here, we designed a modified version of the two-step task, named the two-step sequential movement task, where participants had to conduct rapid sequential finger movements to obtain rewards (n = 40). In the shock session, participants received an electrical shock if they made an erroneous or slow movement, while in the no-shock session, they only received zero reward. We found that participants who prioritised model-free decision-making committed more motor errors in the presence of the shock stimulus (shock sessions) than those who prioritised model-based decision-making. Using a mediation analysis, we also revealed a strong link between the balance of the model-based and the model-free learning strategies and sequential movement performances. These results suggested that model-free decision-making produces more motor errors than model-based decision-making in rapid sequential movements under the threat of stressful stimuli.

Prosocial learning: Model-based or model-free?

Prosocial learning involves the acquisition of knowledge and skills necessary for making decisions that benefit others. We asked if, in the context of value-based decision-making, there is any difference between learning strategies for oneself vs. for others. We implemented a 2-step reinforcement learning paradigm in which participants learned, in separate blocks, to make decisions for themselves or for a present other confederate who evaluated their performance. We replicated the canonical features of the model-based and model-free reinforcement learning in our results. The behaviour of the majority of participants was best explained by a mixture of the model-based and model-free control, while most participants relied more heavily on MB control, and this strategy enhanced their learning success. Regarding our key self-other hypothesis, we did not find any significant difference between the behavioural performances nor in the model-based parameters of learning when comparing self and other conditions.

Goal-directed and habitual decision making under stress in gambling disorder: An fMRI study

The development of addictive behaviors has been suggested to be related to a transition from goal-directed to habitual decision making. Stress is a factor known to prompt habitual behavior and to increase the risk for addiction and relapse. In the current study, we therefore used functional MRI to investigate the balance between goal-directed 'model-based' and habitual 'model-free' control systems and whether acute stress would differentially shift this balance in gambling disorder (GD) patients compared to healthy controls (HCs). Using a within-subject design, 22 patients with GD and 20 HCs underwent stress induction or a control condition before performing a multistep decision-making task during fMRI. Salivary cortisol levels showed that the stress induction was successful. Contrary to our hypothesis, GD patients did not show impaired goal-directed 'model-based' decision making, which remained similar to HCs after stress induction. Bayes factors provided three times more evidence against a difference between the groups or a group-by-stress interaction on the balance between model-based and model-free decision making. Similarly, no differences were found between groups and conditions on the neural estimates of model-based or model-free decision making. These results challenge the notion that GD is related to an increased reliance on habitual (or decreased goal-directed) control, even during stress.

Striatopallidal adenosine A2A receptor modulation of goal-directed behavior: Homeostatic control with cognitive flexibility

Dysfunction of goal-directed behaviors under stressful or pathological conditions results in impaired decision-making and loss of flexibility of thoughts and behaviors, which underlie behavioral deficits ranging from depression, obsessive-compulsive disorders and drug addiction. Tackling the neuromodulators fine-tuning this core behavioral element may facilitate the development of effective strategies to control these deficits present in multiple psychiatric disorders. The current investigation of goal-directed behaviors has concentrated on dopamine and glutamate signaling in the corticostriatal pathway. In accordance with the beneficial effects of caffeine intake on mood and cognitive dysfunction, we now propose that caffeine's main site of action - adenosine A2A receptors (A2AR) - represent a novel target to homeostatically control goal-directed behavior and cognitive flexibility. A2AR are abundantly expressed in striatopallidal neurons and colocalize and interact with dopamine D2, NMDA and metabotropic glutamate 5 receptors to integrate dopamine and glutamate signaling. Specifically, striatopallidal A2AR (i) exert an overall "break" control of a variety of cognitive processes, making A2AR antagonists a novel strategy for improving goal-directed behavior; (ii) confer homeostatic control of goal-directed behavior by acting at multiple sites with often opposite effects, to enhance cognitive flexibility; (iii) integrate dopamine and adenosine signaling through multimeric A2AR-D2R heterocomplexes allowing a temporally precise fine-tuning in response to local signaling changes. As the U.S. Food and Drug Administration recently approved the A2AR antagonist Nourianz® (istradefylline) to treat Parkinson's disease, striatal A2AR-mediated control of goal-directed behavior may offer a new and real opportunity for improving deficits of goal-directed behavior and enhance cognitive flexibility under various neuropsychiatric conditions. This article is part of the Special Issue on "Purinergic Signaling: 50 years".

Acute stress alters probabilistic reversal learning in healthy male adults

Behavioural adaptation is a fundamental cognitive ability, ensuring survival by allowing for flexible adjustment to changing environments. In laboratory settings, behavioural adaptation can be measured with reversal learning paradigms requiring agents to adjust reward learning to stimulus-action-outcome contingency changes. Stress is found to alter flexibility of reward learning, but effect directionality is mixed across studies. Here, we used model-based functional MRI (fMRI) in a within-subjects design to investigate the effect of acute psychosocial stress on flexible behavioural adaptation. Healthy male volunteers (n = 28) did a reversal learning task during fMRI in two sessions, once after the Trier Social Stress Test (TSST), a validated psychosocial stress induction method, and once after a control condition. Stress effects on choice behaviour were investigated using multilevel generalized linear models and computational models describing different learning processes that potentially generated the data. Computational models were fitted using a hierarchical Bayesian approach, and model-derived reward prediction errors (RPE) were used as fMRI regressors. We found that acute psychosocial stress slightly increased correct response rates. Model comparison revealed that double-update learning with altered choice temperature under stress best explained the observed behaviour. In the brain, model-derived RPEs were correlated with BOLD signals in striatum and ventromedial prefrontal cortex (vmPFC). Striatal RPE signals for win trials were stronger during stress compared with the control condition. Our study suggests that acute psychosocial stress could enhance reversal learning and RPE brain responses in healthy male participants and provides a starting point to explore these effects further in a more diverse population.

Does stress consistently favor habits over goal-directed behaviors? Data from two preregistered exact replication studies

Instrumental learning is controlled by two distinct parallel systems: goal-directed (action-outcome) and habitual (stimulus-response) processes. Seminal research by Schwabe and Wolf (2009, 2010) has demonstrated that stress renders behavior more habitual by decreasing goal-directed control. More recent studies yielded equivocal evidence for a stress-induced shift towards habitual responding, yet these studies used different paradigms to evaluate instrumental learning or used different stressors. Here, we performed exact replications of the original studies by exposing participants to an acute stressor either before (cf. Schwabe and Wolf, 2009) or directly after (cf. Schwabe and Wolf, 2010) an instrumental learning phase in which they had learned that distinct actions led to distinct, rewarding food outcomes (i.e., instrumental learning). Then, following an outcome devaluation phase in which one of the food outcomes was consumed until participants were satiated, action-outcome associations were tested in extinction. Despite successful instrumental learning and outcome devaluation and increased subjective and physiological stress levels following stress exposure, the stress and no-stress groups in both replication studies responded indifferently to valued and devalued outcomes. That is, non-stressed participants failed to demonstrate goal-directed behavioral control, thereby rendering the critical test of a shift from goal-directed to habitual control in the stress group inapt. Several reasons for these replication failures are discussed, including the rather indiscriminate devaluation of outcomes that may have contributed to indifferent responding during extinction, which emphasize the need to further our understanding of the boundary conditions in research aimed at demonstrating a stress-induced shift towards habitual control.

Negative symptoms, striatal dopamine and model-free reward decision-making in schizophrenia

Negative symptoms, such as lack of motivation or social withdrawal, are highly prevalent and debilitating in patients with schizophrenia. Underlying mechanisms of negative symptoms are incompletely understood, thereby preventing the development of targeted treatments. We hypothesized that in patients with schizophrenia during psychotic remission, impaired influences of both model-based and model-free reward predictions on decision-making ('reward prediction influence', RPI) underlie negative symptoms. We focused on psychotic remission, because psychotic symptoms might confound reward-based decision-making. Moreover, we hypothesized that impaired model-based/model-free RPIs depend on alterations of both associative striatum dopamine synthesis and storage (DSS) and executive functioning. Both factors influence RPI in healthy subjects and are typically impaired in schizophrenia. Twenty-five patients with schizophrenia with pronounced negative symptoms during psychotic remission and 24 healthy controls were included in the study. Negative symptom severity was measured by the Positive and Negative Syndrome Scale negative subscale, model-based/model-free RPI by the two-stage decision task, associative striatum DSS by 18F-DOPA positron emission tomography and executive functioning by the symbol coding task. Model-free RPI was selectively reduced in patients and associated with negative symptom severity as well as with reduced associative striatum DSS (in patients only) and executive functions (both in patients and controls). In contrast, model-based RPI was not altered in patients. Results provide evidence for impaired model-free reward prediction influence as a mechanism for negative symptoms in schizophrenia as well as for reduced associative striatum dopamine and executive dysfunction as relevant factors. Data suggest potential treatment targets for patients with schizophrenia and pronounced negative symptoms.

The modulation of acute stress on model-free and model-based reinforcement learning in gambling disorder

BACKGROUND AND AIMS

Experiencing acute stress is common in behavioral addictions such as gambling disorder. Additionally, like most substance-induced addictions, aberrant decision-making wherein a reactive habit-induced response (conceptualized as a Model-free [MF] in reinforcement learning) suppresses a flexible goal-directed response (conceptualized as a Model-based [MB]) is also common in gambling disorder. In the current study we investigated the influence of acute stress on the balance between habitual response and the goal-directed system.

METHODS

A sample of N = 116 problem gamblers (PG) and healthy controls (HC) performed an acute stress task - the Socially Evaluated Cold pressure task (SECPT) - or a control task. Self-reported stress and salivary cortisol were collected as measures of acute stress. Following the SECPT, participants performed the Two-Step Markov Task to account for the relative contribution of MB and MF strategies. Additionally, verbal working memory and IQ measures were collected to account for their mediating effects on the orchestration between MB/MF and the impact of stress.

RESULTS

Both groups had comparable baseline and stress-induced cortisol response to the SECPT. Non-stressed PG displayed lower MB learning than HC. MANOVA and regression analyses showed a deleterious effect of stress-induced cortisol response on the orchestration between MB and MF learning in HC but not in PG. These effects remained when controlling for working memory and IQ.

DISCUSSION AND CONCLUSIONS

We found an abnormal pattern of modulation of stress on the orchestration between MB and MF learning among PG. Several interpretations and future research directions are discussed.

Effects of psychological stress and cortisol on decision making and modulating factors: a systematic review

Evidence suggests that psychological stress has effects on decision making, but the results are inconsistent, and the influence of cortisol and other modulating factors remains unclear. Based on the PRISMA criteria, 18 studies carried out between 2015 and 2020 that examined the effects of psychological stress on decision making and measured cortisol levels were selected. Eight studies employed uncertainty-based economic tasks, five studies used decision-making tasks in hypothetical situations that can be encountered in real life or in a specific setting, and five studies employed prosocial decision tasks. Seventeen studies assessed acute stress, and two assessed chronic stress; eight evaluated the influence of sex. Most of the studies that explored the association between stress and decision making using uncertainty-based economic tasks found statistically significant differences as a function of stress exposure and the cortisol response to stress, whereas most of the studies that employed non-economic decision-making tasks in hypothetical situations did not find statistically significant differences. When prosocial decision making was evaluated, more altruistic decisions were found after acute stress, and these decisions were positively associated with cortisol. Half of the studies that assessed the role of sex observed a greater impact on decision making after stress in women. Results suggest that it is important to consider modulating factors - the type of decision-making task, the cortisol response to stress, the characteristics of the psychological stressor, or the subject's sex - when trying to understand psychosocial stress phenomena.

Explicit knowledge of task structure is a primary determinant of human model-based action

Explicit information obtained through instruction profoundly shapes human choice behaviour. However, this has been studied in computationally simple tasks, and it is unknown how model-based and model-free systems, respectively generating goal-directed and habitual actions, are affected by the absence or presence of instructions. We assessed behaviour in a variant of a computationally more complex decision-making task, before and after providing information about task structure, both in healthy volunteers and in individuals suffering from obsessive-compulsive or other disorders. Initial behaviour was model-free, with rewards directly reinforcing preceding actions. Model-based control, employing predictions of states resulting from each action, emerged with experience in a minority of participants, and less in those with obsessive-compulsive disorder. Providing task structure information strongly increased model-based control, similarly across all groups. Thus, in humans, explicit task structural knowledge is a primary determinant of model-based reinforcement learning and is most readily acquired from instruction rather than experience.

Gender moderates the association between chronic academic stress with top-down and bottom-up attention

Research on the relationship between chronic stress and cognition is limited by a lack of concurrent measurement of state-anxiety, physiological arousal, and gender. For the first time, we assessed the impact of these factors on top-down/conscious (simple and choice reaction time) and bottom-up/reflexive (saccadic reaction time) measures of attention using CONVIRT virtual-reality cognitive tests. Participants (N = 163) completed measures of academic stress (effort-reward imbalance; ERI) and state-anxiety while heart-rate variability was recorded continuously throughout the experiment. Gender moderated the association between academic stress with the top-down measures (b = -0.002, t = -2.023, p = .045; b = -0.063, t = -3.080, p = .002) and higher academic stress was associated with poorer/slower reaction times only for male participants. For bottom-up attention, heart rate variability moderated the relationship between academic stress and saccadic reaction time (b = 0.092, t = 1.991, p = .048), and only female participants who were more stressed (i.e., ERI ≥ 1) and displayed stronger sympathetic dominance had slower reaction times. Our findings align with emerging evidence that chronic stress is related to hyperarousal in women and cognitive decrements in men. Our findings suggest that higher ERI and sympathetic dominance during cognitive testing was associated with poorer bottom-up attention in women, whereas for men, academic stress was related with poorer top-down attention irrespective of sympathovagal balance.

Determining the effects of training duration on the behavioral expression of habitual control in humans: a multilaboratory investigation

It has been suggested that there are two distinct and parallel mechanisms for controlling instrumental behavior in mammals: goal-directed actions and habits. To gain an understanding of how these two systems interact to control behavior, it is essential to characterize the mechanisms by which the balance between these systems is influenced by experience. Studies in rodents have shown that the amount of training governs the relative expression of these two systems: Behavior is goal-directed following moderate training, but the more extensively an instrumental action is trained, the more it becomes habitual. It is less clear whether humans exhibit similar training effects on the expression of goal-directed and habitual behavior, as human studies have reported contradictory findings. To tackle these contradictory findings, we formed a consortium, where four laboratories undertook a preregistered experimental induction of habits by manipulating the amount of training. There was no statistical evidence for a main effect of the amount of training on the formation and expression of habits. However, exploratory analyses suggest a moderating effect of the affective component of stress on the impact of training over habit expression. Participants who were lower in affective stress appeared to be initially goal-directed, but became habitual with increased training, whereas participants who were high in affective stress were already habitual even after moderate training, thereby manifesting insensitivity to overtraining effects. Our findings highlight the importance of the role of moderating variables such as individual differences in stress and anxiety when studying the experimental induction of habits in humans.

Stress Watch: The Use of Heart Rate and Heart Rate Variability to Detect Stress: A Pilot Study Using Smart Watch Wearables

Stress is an inherent part of the normal human experience. Although, for the most part, this stress response is advantageous, chronic, heightened, or inappropriate stress responses can have deleterious effects on the human body. It has been suggested that individuals who experience repeated or prolonged stress exhibit blunted biological stress responses when compared to the general population. Thus, when assessing whether a ubiquitous stress response exists, it is important to stratify based on resting levels in the absence of stress. Research has shown that stress that causes symptomatic responses requires early intervention in order to mitigate possible associated mental health decline and personal risks. Given this, real-time monitoring of stress may provide immediate biofeedback to the individual and allow for early self-intervention. This study aimed to determine if the change in heart rate variability could predict, in two different cohorts, the quality of response to acute stress when exposed to an acute stressor and, in turn, contribute to the development of a physiological algorithm for stress which could be utilized in future smartwatch technologies. This study also aimed to assess whether baseline stress levels may affect the changes seen in heart rate variability at baseline and following stress tasks. A total of 30 student doctor participants and 30 participants from the general population were recruited for the study. The Trier Stress Test was utilized to induce stress, with resting and stress phase ECGs recorded, as well as inter-second heart rate (recorded using a FitBit). Although the present study failed to identify ubiquitous patterns of HRV and HR changes during stress, it did identify novel changes in these parameters between resting and stress states. This study has shown that the utilization of HRV as a measure of stress should be calculated with consideration of resting (baseline) anxiety and stress states in order to ensure an accurate measure of the effects of additive acute stress.

Perceived chronic stress influences the effect of acute stress on cognitive flexibility

Executive functions are cognitive processes that facilitate goal-directed behavior by enabling us to direct and control our thoughts. Cognitive flexibility is an executive function characterized by the ability to mentally shift between rules, strategies, or tasks. Several studies have reported that acute (brief) stress impairs cognitive flexibility. Even though an individual's perception of their chronic stress levels is shown to influence effects of future stressors, the interactive effect of acute and perceived chronic stress on cognitive flexibility is not known. We conducted two experiments to address this gap. In both studies, perceived chronic stress was measured using the Perceived Stress Scale. Acute stress was induced using the Cold Pressor Test. Number of perseverative errors on the Wisconsin Card Sorting Test was used as an indicator of cognitive flexibility. In Study 2, we also measured salivary alpha amylase as a marker of the physiological stress response. Data from our two studies are consistent with the hypothesis that an individual's perception of their chronic stress level may impact the effect of acute stress on perseveration. In Study 1, we observed a significant interaction between acute and perceived chronic stress on perseverative errors, such that only individuals who reported high levels of perceived chronic stress prior to acute stress exposure showed no change in perseveration following the acute stress manipulation. This effect did not differ based on participant sex. In Study 2, we found a similar interaction effect of acute and perceived chronic stress on perseverative errors in an all-woman sample. After identifying salivary alpha amylase responders and non-responders, we observed a strong, negative correlation between perceived chronic stress and perseverative errors amongst the responders only. Our data highlight the value in studying salivary alpha amylase in response to acute stress exposure. Additionally, perceived chronic stress emerged as a key variable in the relationship between acute stress and cognitive flexibility. Overall, our work suggests that future research interested in interrogating moderators in the relationship between acute stress and cognition would benefit from inclusion of measures of chronic stress.

Hunger improves reinforcement-driven but not planned action

Human decisions can be reflexive or planned, being governed respectively by model-free and model-based learning systems. These two systems might differ in their responsiveness to our needs. Hunger drives us to specifically seek food rewards, but here we ask whether it might have more general effects on these two decision systems. On one hand, the model-based system is often considered flexible and context-sensitive, and might therefore be modulated by metabolic needs. On the other hand, the model-free system's primitive reinforcement mechanisms may have closer ties to biological drives. Here, we tested participants on a well-established two-stage sequential decision-making task that dissociates the contribution of model-based and model-free control. Hunger enhanced overall performance by increasing model-free control, without affecting model-based control. These results demonstrate a generalized effect of hunger on decision-making that enhances reliance on primitive reinforcement learning, which in some situations translates into adaptive benefits.

The psychobiological links between chronic stress-related diseases, periodontal/peri-implant diseases, and wound healing

Chronic stress is a relevant disease to periodontal practice, encompassing 25%-28% of the US population (American Psychological Association 2015). While it is well established that chronic psychologic stress can have significant deleterious systemic effects, only in recent decades have we begun to explore the biochemical, microbial, and physiologic impacts of chronic stress diseases on oral tissues. Currently, chronic stress is classified as a "risk indicator" for periodontal disease. However, as the evidence in this field matures with additional clinically controlled trials, more homogeneous data collection methods, and a better grasp of the biologic underpinnings of stress-mediated dysbiosis, emerging evidence suggests that chronic stress and related diseases (depression, anxiety) may be significant contributing factors in periodontal/peri-implant disease progression and inconsistent wound healing following periodontal-related therapeutics. Ideal solutions for these patients include classification of the disease process and de-escalation of chronic stress conditions through coping strategies. This paper also summarizes periodontal/implant-related therapeutic approaches to ensure predictable results for this specific patient subpopulation.

Acute Psychosocial Stress Increases Cognitive-Effort Avoidance

Adverse effects following acute stress are traditionally thought to reflect functional impairments of central executive-dependent cognitive-control processes. However, recent evidence demonstrates that cognitive-control application is perceived as effortful and aversive, indicating that stress-related decrements in cognitive performance could denote decreased motivation to expend effort instead. To investigate this hypothesis, we tested 40 young, healthy individuals (20 female, 20 male) under both stress and control conditions in a 2-day study that had a within-subjects design. Cognitive-effort avoidance was assessed using the demand-selection task, in which participants chose between performing low-demand and high-demand variants of a task-switching paradigm. We found that acute stress indeed increased participants' preference for less demanding behavior, whereas task-switching performance remained intact. Additional Bayesian and multiverse analyses confirmed the robustness of this effect. Our findings provide novel insights into how stressful experiences shape behavior by modulating our motivation to employ cognitive control.

Neural systems and the emotion-memory link

The present brief review for this Special Issue summarizes some of the original research on the emotional modulation of memory. The review begins by highlighting the pioneering research from James L. McGaugh and colleagues demonstrating modulatory effects of post-training drug administration on memory consolidation, in particular the stress hormone epinephrine. The subsequent discovery of a critical role for the basolateral amygdala in emotional modulation of memory is described. Within the context of a multiple systems approach to memory focusing on selective roles for the hippocampus and dorsolateral striatum in cognitive and habit memory, the original studies indicating that robust emotional arousal can bias animals and humans toward the predominant use of habit memory are reviewed. This effect of emotional arousal on the relative use of multiple memory systems depends on a modulatory role of the basolateral amygdala. Finally, we briefly consider how an emotion-induced enhancement of dorsolateral striatal-dependent memory may be relevant to understanding maladaptive habitual behaviors in certain human psychopathologies.

Experimentally induced and real-world anxiety have no demonstrable effect on goal-directed behaviour

BACKGROUND

Goal-directed control guides optimal decision-making and it is an important cognitive faculty that protects against developing habits. Previous studies have found some evidence of goal-directed deficits when healthy individuals are stressed, and in psychiatric conditions characterised by compulsive behaviours and anxiety. Here, we tested if goal-directed control is affected by state anxiety, which might explain the former results.

METHODS

We carried out a causal test of this hypothesis in two experiments (between-subject N = 88; within-subject N = 50) that used the inhalation of hypercapnic gas (7.5% CO2) to induce an acute state of anxiety in healthy volunteers. In a third experiment (N = 1413), we used a correlational design to test if real-life anxiety-provoking events (panic attacks, stressful events) are associated with impaired goal-directed control.

RESULTS

In the former two causal experiments, we induced a profoundly anxious state, both physiologically and psychologically, but this did not affect goal-directed performance. In the third, correlational, study, we found no evidence for an association between goal-directed control, panic attacks or stressful life eventsover and above variance accounted for by trait differences in compulsivity.

CONCLUSIONS

In sum, three complementary experiments found no evidence that anxiety impairs goal-directed control in human subjects.

Effects of subclinical depression on prefrontal-striatal model-based and model-free learning

Depression is characterized by deficits in the reinforcement learning (RL) process. Although many computational and neural studies have extended our knowledge of the impact of depression on RL, most focus on habitual control (model-free RL), yielding a relatively poor understanding of goal-directed control (model-based RL) and arbitration control to find a balance between the two. We investigated the effects of subclinical depression on model-based and model-free learning in the prefrontal-striatal circuitry. First, we found that subclinical depression is associated with the attenuated state and reward prediction error representation in the insula and caudate. Critically, we found that it accompanies the disrupted arbitration control between model-based and model-free learning in the predominantly inferior lateral prefrontal cortex and frontopolar cortex. We also found that depression undermines the ability to exploit viable options, called exploitation sensitivity. These findings characterize how subclinical depression influences different levels of the decision-making hierarchy, advancing previous conflicting views that depression simply influences either habitual or goal-directed control. Our study creates possibilities for various clinical applications, such as early diagnosis and behavioral therapy design.

Anxiety and depressive symptoms effects on cortisol trajectories from pregnancy to postpartum: Differences and similarities between women and men

Anxiety and depressive symptoms may influence cortisol trajectories in women and men during pregnancy and the postpartum period. Using a multilevel approach, anxiety and depressive symptoms effects on 24-hour urinary free cortisol trajectories from the 2nd trimester to 3-months postpartum were examined in a sample of 66 women and 65 men with no known psychosocial or medical risk (N = 131; 33 (50%) of them were couples that participated in the same assessment waves). Results showed that both anxiety and depressive symptoms influence women's and men's cortisol trajectories from mid-pregnancy to 3-months postpartum. Women with high depressive symptoms and men with high anxiety or high depressive symptoms exhibited less accentuated variations in the 24-hour urinary free cortisol trajectories compared with women with low depressive symptoms and men with low anxiety or depressive symptoms, respectively. These effects were significant for women's cortisol trajectories from the 2nd to the 3rd pregnancy trimester and for men's cortisol trajectories throughout the entire period. The effect of anxiety and depressive symptoms on HPA axis functioning and cortisol production during pregnancy and postpartum, seems to be sex-specific. Reproductive-related alterations (associated with gestation, parturition and lactation) in women's HPA axis functioning may explain these sex-specific effects.

Neural Computations of Threat

A host of learning, memory, and decision-making processes form the individual's response to threat and may be disrupted in anxiety and post-trauma psychopathology. Here we review the neural computations of threat, from the first encounter with a dangerous situation, through learning, storing, and updating cues that predict it, to making decisions about the optimal course of action. The overview highlights the interconnected nature of these processes and their reliance on shared neural and computational mechanisms. We propose an integrative approach to the study of threat-related processes, in which specific computations are studied across the various stages of threat experience rather than in isolation. This approach can generate new insights about the evolution, diagnosis, and treatment of threat-related psychopathology.

Stress reduces both model-based and model-free neural computations during flexible learning

Stressful events are thought to impair the flexible adaptation to changing environments, yet the underlying mechanisms are largely unknown. Here, we combined computational modeling and functional magnetic resonance imaging (fMRI) to elucidate the neurocomputational mechanisms underlying stress-induced deficits in flexible learning. Healthy participants underwent a stress or control manipulation before they completed, in the MRI scanner, a Markov decision task, frequently used to dissociate model-based and model-free contributions to choice, with repeated reversals of reward contingencies. Our results showed that stress attenuated the behavioral sensitivity to reversals in reward contingencies. Computational modeling further indicated that stress specifically affected the use of value computations for subsequent action selection. This reduced application of learned information on subsequent behavior was paralleled by a stress-induced reduction in inferolateral prefrontal cortex activity during model-free computations. For model-based learning, stress decreased specifically posterior, but not anterior, hippocampal activity, pointing to a functional segregation of model-based processing and its modulation by stress along the hippocampal longitudinal axis. Our findings shed light on the mechanisms underlying deficits in flexible learning under stress and indicate that, in highly dynamic environments, stress may hamper both model-based and model-free contributions to adaptive behavior.

Learning to make smart choices in the context of risk: The roles of internal affective feedback and life events

Autonomic arousal may facilitate beneficial decision-making when the link between choices and outcomes is uncertain. However, it is unknown whether greater risk-specific autonomic arousal is linearly associated with faster learning to avoid risky decisions. Furthermore, although the influence of stress on decision-making is well documented, it is unknown whether recent life stress might moderate the relationship between this internal affective feedback and decision-making. We report two studies using the Iowa Gambling Task with diverse community samples. Each study demonstrated a linear relationship between the level of autonomic arousal prior to risky decision-making and the rate of learning to avoid risk. Additionally, participants' recent life events conditionally moderated this association. Specifically, the relationship between risk-specific arousal and advantageous learning was strongest for participants who experienced relatively more positive and fewer negative life events in the previous four months. These findings suggest that autonomic arousal may generally inform decision-making, but less so when life circumstances are relatively poor.

Model based planners reflect on their model-free propensities

Dual-reinforcement learning theory proposes behaviour is under the tutelage of a retrospective, value-caching, model-free (MF) system and a prospective-planning, model-based (MB), system. This architecture raises a question as to the degree to which, when devising a plan, a MB controller takes account of influences from its MF counterpart. We present evidence that such a sophisticated self-reflective MB planner incorporates an anticipation of the influences its own MF-proclivities exerts on the execution of its planned future actions. Using a novel bandit task, wherein subjects were periodically allowed to design their environment, we show that reward-assignments were constructed in a manner consistent with a MB system taking account of its MF propensities. Thus, in the task participants assigned higher rewards to bandits that were momentarily associated with stronger MF tendencies. Our findings have implications for a range of decision making domains that includes drug abuse, pre-commitment, and the tension between short and long-term decision horizons in economics.

Acute stress impairs reward learning in men

Acute stress is ubiquitous in everyday life, but the extent to which acute stress affects how people learn from the outcomes of their choices is still poorly understood. Here, we investigate how acute stress impacts reward and punishment learning in men using a reinforcement-learning task. Sixty-two male participants performed the task whilst under stress and control conditions. We observed that acute stress impaired participants' choice performance towards monetary gains, but not losses. To unravel the mechanism(s) underlying such impairment, we fitted a reinforcement-learning model to participants' trial-by-trial choices. Computational modeling indicated that under acute stress participants learned more slowly from positive prediction errors - when the outcomes were better than expected - consistent with stress-induced dopamine disruptions. Such mechanistic understanding of how acute stress impairs reward learning is particularly important given the pervasiveness of stress in our daily life and the impact that stress can have on our wellbeing and mental health.

A Translational Paradigm to Study the Effects of Uncontrollable Stress in Humans

Theories on the aetiology of depression in humans are intimately linked to animal research on stressor controllability effects. However, explicit translations of established animal designs are lacking. In two consecutive studies, we developed a translational paradigm to study stressor controllability effects in humans. In the first study, we compared three groups of participants, one exposed to escapable stress, one yoked inescapable stress group, and a control group not exposed to stress. Although group differences indicated successful stress induction, the manipulation failed to differentiate groups according to controllability. In the second study, we employed an improved paradigm and contrasted only an escapable stress group to a yoked inescapable stress group. The final design successfully induced differential effects on self-reported perceived control, exhaustion, helplessness, and behavioural indices of adaptation to stress. The latter were examined in a new escape behaviour test which was modelled after the classic shuttle box animal paradigm. Contrary to the learned helplessness literature, exposure to uncontrollable stress led to more activity and exploration; however, these behaviours were ultimately not adaptive. We discuss the results and possible applications in light of the findings on learning and agency beliefs, inter-individual differences, and interventions aimed at improving resilience to stress-induced mental dysfunction.

Trait impulsivity and acute stress interact to influence choice and decision speed during multi-stage decision-making

Impulsivity and stress exposure are two factors that are associated with changes in reward-related behavior in ways that are relevant to both healthy and maladaptive decision-making. Nonetheless, little empirical work has examined the possible independent and joint effects of these factors upon reward learning. Here, we sought to examine how trait impulsivity and acute stress exposure affect participants' choice behavior and decision speed in a two-stage sequential reinforcement-learning task. We found that more impulsive participants were more likely to repeat second-stage choices after previous reward, irrespective of stress condition. Exposure to stress, on the other hand, was associated with an increased tendency to repeat second-stage choices independent of whether these choices previously led to a reward, and this tendency was exacerbated in more impulsive individuals. Such interaction effects between stress and impulsivity were also found on decision speed. Stress and impulsivity levels interacted to drive faster choices overall (again irrespective of reward) at both task stages, while reward received on the previous trial slowed subsequent first-stage choices, particularly among impulsive individuals under stress. Collectively, our results reveal novel, largely interactive effects of trait impulsivity and stress exposure and suggest that stress may reveal individual differences in decision-making tied to impulsivity that are not readily apparent in the absence of stress.

Goal-oriented and habitual decisions: Neural signatures of model-based and model-free learning

Human decision-making is mainly driven by two fundamental learning processes: a slow, deliberative, goal-directed model-based process that maps out the potential outcomes of all options and a rapid habitual model-free process that enables reflexive repetition of previously successful choices. Although many model-informed neuroimaging studies have examined the neural correlates of model-based and model-free learning, the concordant activity among these two processes remains unclear. We used quantitative meta-analyses of functional magnetic resonance imaging experiments to identify the concordant activity pertaining to model-based and model-free learning over a range of reward-related paradigms. We found that: 1) both processes yielded concordant ventral striatum activity, 2) model-based learning activated the medial prefrontal cortex and orbital frontal cortex, and 3) model-free learning specifically activated the left globus pallidus and right caudate head. Our findings suggest that model-free and model-based decision making engage overlapping yet distinct neural regions. These stereotaxic maps improve our understanding of how deliberative goal-directed and reflexive habitual learning are implemented in the brain.

Assessing the association of university stress and physiological reactivity with decision-making among students

Compared with age-matched employees, university students report higher levels of chronic stress and this may affect their decision-making. The impact of chronic stress and physiological reactivity upon cognitive function is receiving more attention, but few studies have empirically assessed the associations of these variables concurrently. Our aim was to investigate if chronic student stress, as assessed by effort-reward imbalance (ERI) and overcommitment, and physiological reactivity, were related to decision-making. As measures of physiological reactivity, we collected salivary alpha-amylase (sAA) and continuously recorded heart rate variability (HRV) data from male students (n = 79) at pretest and immediately after some computerized decision-making tasks (simple and choice- reaction times). Our findings suggest that students who are higher in overcommitment and who are more physiologically reactive (sAA and HRV indices) at the pretest stage may be more "at-risk" of poor decision-making than others. If others can replicate our findings in more diverse samples, this will contribute to an evidence base for interventions targeted at reducing overcommitment, ERI, and dysregulated autonomic reactivity to improve decision-making.

Minimally-invasive methods for examining biological changes in response to chronic stress: A scoping review

BACKGROUND

Nurse researchers are increasingly interested in incorporating biological indicators related to chronic stress, or repeated or constant exposure to psychological stressors. Minimally invasive collection methods may improve access to vulnerable populations.

OBJECTIVE

To map biological indicators measured through minimally invasive methods investigating biological changes in response to chronic stress.

DESIGN, DATA SOURCES, AND METHODS

The paper seeks to answer two questions: What are the characteristics of the minimally-invasive methods used to measure the biological correlates of chronic stress? What are the limitations regarding the use of the minimally-invasive methods and/or biological indicators identified above? Authors completed a scoping review following guidelines from the Joanna Briggs Institute Manual and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) extension for Scoping Reviews. A literature search was completed in PubMed, PsycINFO, and Scopus. 2518 articles were screened and 145 studies were included. Data were extracted using a standardized extraction tool, compiled, and coded.

RESULTS

Studies included minimally-invasive methods to measure the hypothalamic-adrenal-pituitary axis (N = 173), immune and inflammatory markers (N = 118), and adult neurogenesis (N = 6). Cortisol was most frequently measured (N = 136), usually in saliva (N = 86). Studies included a variety of limitations for the methods and indicators, including concerns about timing and accuracy of collection, frequency of sampling, and controlling for acute stressors.

CONCLUSIONS

Nurse researchers have access to many minimally-invasive methods to measure altered biological processes related to chronic stress. A gap identified by this review is the paucity of minimally-invasive methods for investigating neurogenesis; the measurement of brain derived neurotrophic factor in plasma is a distal proxy and further research is needed to test the response of peripheral levels to psychosocial stress interventions. Additionally, while this scoping review allows nurse researchers to consider possible biological indicators to include in their research, future research is still needed on some of the basic premises of stress research, including agreement on the conceptualization of chronic stress.

Striatopallidal adenosine A2A receptors in the nucleus accumbens confer motivational control of goal-directed behavior

The ability to learn the reward-value and action-outcome contingencies in dynamic environment is critical for flexible adaptive behavior and development of effective pharmacological control of goal-directed behaviors represents an important challenge for improving the deficits in goal-directed behavior which may underlie seemingly disparate symptoms across psychiatric disorders. Adenosine A_2A receptor (A_2AR) is emerging as a novel neuromodulatory target for controlling goal-directed behavior for its unique neuromodulatory features: the ability to integrate dopamine and glutamate signaling, the "brake" constraint of various cognitive processes and the balanced control of goal-directed and habit actions. However, the contribution and circuit mechanisms of the striatopallidal A_2ARs in nucleus accumbens (NAc) to control of goal-directed behavior remain to be determined. Here, we employed newly developed opto-A_2AR and the focal A_2AR knockdown strategies to demonstrate the causal role of NAc A_2AR in control of goal-directed behavior. Furthermore, we dissected out multiple distinct behavioral mechanisms underlying which NAc A_2ARs control goal-directed behavior: (i) NAc A_2ARs preferentially control goal-directed behavior at the expense of habit formation. (ii) NAc A_2ARs modify the animals' sensitivity to the value of the reward without affecting the action-outcome contingency. (iii) A_2AR antagonist KW6002 promotes instrumental actions by invigorating motivation. (iv) NAc A_2ARs facilitate Pavlovian incentive value transferring to instrumental action. (v) NAc A_2ARs control goal-directed behavior probably not through NAc-VP pathway. These insights into the behavioral and circuit mechanisms for NAc A_2AR control of goal-directed behavior facilitate translational potential for A_2AR antagonists in reversal of deficits in goal-directed decision-making associated with multiple neuropsychiatric disorders.

Comparison of the Association Between Goal-Directed Planning and Self-reported Compulsivity vs Obsessive-Compulsive Disorder Diagnosis

IMPORTANCE

Dimensional definitions of transdiagnostic mental health problems have been suggested as an alternative to categorical diagnoses, having the advantage of capturing heterogeneity within diagnostic categories and similarity across them and bridging more naturally psychological and neural substrates.

OBJECTIVE

To examine whether a self-reported compulsivity dimension has a stronger association with goal-directed and related higher-order cognitive deficits compared with a diagnosis of obsessive-compulsive disorder (OCD).

DESIGN, SETTING, AND PARTICIPANTS

In this cross-sectional study, patients with OCD and/or generalized anxiety disorder (GAD) from across the United States completed a telephone-based diagnostic interview by a trained rater, internet-based cognitive testing, and self-reported clinical assessments from October 8, 2015, to October 1, 2017. Follow-up data were collected to test for replicability.

MAIN OUTCOMES AND MEASURES

Performance was measured on a test of goal-directed planning and cognitive flexibility (Wisconsin Card Sorting Test [WCST]) and a test of abstract reasoning. Clinical variables included DSM-5 diagnosis of OCD and GAD and 3 psychiatric symptom dimensions (general distress, compulsivity, and obsessionality) derived from a factor analysis.

RESULTS

Of 285 individuals in the analysis (mean [SD] age, 32 [12] years; age range, 18-77 years; 219 [76.8%] female), 111 had OCD; 82, GAD; and 92, OCD and GAD. A diagnosis of OCD was not associated with goal-directed performance compared with GAD at baseline (β [SE], -0.02 [0.02]; P = .18). In contrast, a compulsivity dimension was negatively associated with goal-directed performance (β [SE], -0.05 [0.02]; P = .003). Results for abstract reasoning task and WCST mirrored this pattern; the compulsivity dimension was associated with abstract reasoning (β [SE], 2.99 [0.63]; P < .001) and several indicators of WCST performance (eg, categories completed: β [SE], -0.57 [0.09]; P < .001), whereas OCD diagnosis was not (abstract reasoning: β [SE], 0.39 [0.66]; P = .56; categories completed: β [SE], -0.09 [0.10]; P = .38). Other symptom dimensions relevant to OCD, obsessionality, and general distress had no reliable association with goal-directed performance, WCST, or abstract reasoning. Obsessionality had a positive association with requiring more trials to reach the first category on the WCST at baseline (β [SE], 2.92 [1.39]; P = .04), and general distress was associated with impaired goal-directed performance at baseline (β [SE],-0.04 [0.02]; P = .01). However, unlike the key results of this study, neither survived correction for multiple comparisons or was replicated at follow-up testing.

CONCLUSIONS AND RELEVANCE

Deficits in goal-directed planning in OCD may be more strongly associated with a compulsivity dimension than with OCD diagnosis. This result may have implications for research assessing the association between brain mechanisms and clinical manifestations and for understanding the structure of mental illness.

Social anxiety and dynamic social reinforcement learning in a volatile environment

Adaptive social behavior requires learning probabilities of social reward and punishment, and updating these probabilities when they change. Given prior research on aberrant reinforcement learning in affective disorders, this study examines how social anxiety affects probabilistic social reinforcement learning and dynamic updating of learned probabilities in a volatile environment. N=222 online participants completed questionnaires and a computerized ball-catching game with changing probabilities of reward and punishment. Dynamic learning rates were estimated to assess the relative importance ascribed to new information in response to volatility. Mixed-effects regression was used to analyze throw patterns as a function of social anxiety symptoms. Higher social anxiety predicted fewer throws to the previously punishing avatar and different learning rates after certain role changes, suggesting that social anxiety may be characterized by difficulty updating learned social probabilities. Socially anxious individuals may miss the chance to learn that a once-punishing situation no longer poses a threat.

Addiction as Learned Behavior Patterns

Individuals with substance use disorders (SUDs) have to cope with drug-related cues and contexts which can affect instrumental drug seeking, as shown with Pavlovian-to-instrumental transfer (PIT) tasks among humans and animals. Our review addresses two potential mechanisms that may contribute to habitual or even compulsive drug seeking and taking. One mechanism is represented by Pavlovian and PIT effects on drug intake. The other is a shift from goal-directed to habitual drug intake, which can be accessed via model-based versus model-free decision-making in respective learning tasks. We discuss the impact of these learning mechanisms on drug consumption. First, we describe how Pavlovian and instrumental learning mechanisms interact in drug addiction. Secondly, we address the effects of acute and chronic stress exposure on behavioral and neural PIT effects in alcohol use disorder (AUD). Thirdly, we discuss how these learning mechanisms and their respective neurobiological correlates can contribute to losing versus regaining control over drug intake. Utilizing mobile technology (mobile applications on smartphones including games that measure learning mechanisms, activity bracelets), computational models, and real-world data may help to better identify patients with a high relapse risk and to offer targeted behavioral and pharmacotherapeutic interventions for vulnerable patients.

Reward and avoidance learning in the context of aversive environments and possible implications for depressive symptoms

BACKGROUND

Aversive stimuli in the environment influence human actions. This includes valence-dependent influences on action selection, e.g., increased avoidance but decreased approach behavior. However, it is yet unclear how aversive stimuli interact with complex learning and decision-making in the reward and avoidance domain. Moreover, the underlying computational mechanisms of these decision-making biases are unknown.

METHODS

To elucidate these mechanisms, 54 healthy young male subjects performed a two-step sequential decision-making task, which allows to computationally model different aspects of learning, e.g., model-free, habitual, and model-based, goal-directed learning. We used a within-subject design, crossing task valence (reward vs. punishment learning) with emotional context (aversive vs. neutral background stimuli). We analyzed choice data, applied a computational model, and performed simulations.

RESULTS

Whereas model-based learning was not affected, aversive stimuli interacted with model-free learning in a way that depended on task valence. Thus, aversive stimuli increased model-free avoidance learning but decreased model-free reward learning. The computational model confirmed this effect: the parameter lambda that indicates the influence of reward prediction errors on decision values was increased in the punishment condition but decreased in the reward condition when aversive stimuli were present. Further, by using the inferred computational parameters to simulate choice data, our effects were captured. Exploratory analyses revealed that the observed biases were associated with subclinical depressive symptoms.

CONCLUSION

Our data show that aversive environmental stimuli affect complex learning and decision-making, which depends on task valence. Further, we provide a model of the underlying computations of this affective modulation. Finally, our finding of increased decision-making biases in subjects reporting subclinical depressive symptoms matches recent reports of amplified Pavlovian influences on action selection in depression and suggests a potential vulnerability factor for mood disorders. We discuss our findings in the light of the involvement of the neuromodulators serotonin and dopamine.

Acute and past subjective stress influence working memory and related neural substrates

Stress has been proposed to affect cognitive control capacities, including working memory (WM) maintenance. This effect may depend on variability in stress reactivity and past subjective stress. However, as most studies employed between-subjects designs, evidence for within-subject stress effects remains scarce. To understand the role of intra-individual stress effects on WM, we adopted a within-subject design to study how acute stress, variability in stress reactivity, and past subjective stress influence behavioral and neural WM mechanisms. Thirty-four healthy males performed a WM task during functional magnetic resonance imaging (fMRI) in a control versus acute stress condition following the Trier Social Stress Test, a validated psychosocial stressor method. We tested for stress effects on WM performance and related neural activation by associating them with individual acute stress responsivity and past subjective stress experience using retrospective self-report questionnaires. We found no evidence of an effect of acute stress or related stress-reactivity on intra-individual WM performance. However, past subjective stress negatively influenced acute stress-induced changes to WM. On the neural level, acute stress reduced WM-related activation in the dorsolateral prefrontal cortex (dlPFC). The observed negative influence of inter-individual variability in past subjective stress experience on changes in WM performance, suggests that past subjective stress might induce vulnerability for impairing effects of acute stress on cognitive functioning. Because acute stress reduced WM-related dlPFC activation while WM performance remained unaffected, acute stress might boost neural processing efficiency in this group of high performing healthy individuals. Our study suggests that measures of past subjective stress should be considered when studying and interpreting the effects of acute stress on cognition.

Chronic Stress and Moral Decision-Making: An Exploration With the CNI Model

Stress is prevalent in our daily life, and people often make moral decision-making in a stressful state. Several studies indicated the influence of acute stress on moral decision-making and behavior. The present study extended the investigation to chronic stress, and employed a new approach, the CNI model, to add new insights regarding the mechanism underlying the association between chronic stress and moral decision-making. A total of 197 undergraduates completed the Perceived Stress Scale and made moral decision-making on a series of deliberately designed moral dilemmas. The results indicated that higher chronic stress was related to more deontological moral choices. The process-dissociation analyses revealed that chronic stress was marginally significantly associated with deontological inclinations but not with utilitarian inclinations. And the CNI model analyses suggested that the high-stress group (above the median) showed a stronger general preference for inaction than the low-stress group (below the median) did, but there were no significant differences in sensitivity to consequences or sensitivity to moral norms between the two groups. Finally, the implications of the findings were discussed.

Model-based learning and individual differences in depression: The moderating role of stress

Inflexible decision-making has been proposed as a transdiagnostic risk factor for mood disorders. Evidence suggests that inflexible decision-making may emerge only when individuals are experiencing increased negative affect or stress. 151 participants completed symptom measures of depression and anxiety, followed by a two-stage decision-making task that distinguishes between habitual and goal-directed choice. An experimental manipulation to induce stress was introduced halfway through the task. Individuals with higher depression levels became less model-based after the manipulation than those with lower depression levels. There was no relationship between trait anxiety and the impact of the manipulation on decision-making. Controlling for main effects of anxiety did not attenuate the association between depression and impact of stress. Anhedonia was associated with the impact of the manipulation on model-based decision-making. These results suggest that risk for depression is associated with reflexive decision-making, but these effects may only emerge under conditions of stress.

Deliberation and Procedural Automation on a Two-Step Task for Rats

Current theories suggest that decision-making arises from multiple, competing action-selection systems. Rodent studies dissociate deliberation and procedural behavior, and find a transition from procedural to deliberative behavior with experience. However, it remains unknown how this transition from deliberative to procedural control evolves within single trials, or within blocks of repeated choices. We adapted for rats a two-step task which has been used to dissociate model-based from model-free decisions in humans. We found that a mixture of model-based and model-free algorithms was more likely to explain rat choice strategies on the task than either model-based or model-free algorithms alone. This task contained two choices per trial, which provides a more complex and non-discrete per-trial choice structure. This task structure enabled us to evaluate how deliberative and procedural behavior evolved within-trial and within blocks of repeated choice sequences. We found that vicarious trial and error (VTE), a behavioral correlate of deliberation in rodents, was correlated between the two choice points on a given lap. We also found that behavioral stereotypy, a correlate of procedural automation, increased with the number of repeated choices. While VTE at the first choice point decreased [corrected] with the number of repeated choices, VTE at the second choice point did not, and only increased after unexpected transitions within the task. This suggests that deliberation at the beginning of trials may correspond to changes in choice patterns, while mid-trial deliberation may correspond to an interruption of a procedural process.

Stress-induced reliance on habitual behavior is moderated by cortisol reactivity

Instrumental learning, i.e., learning that specific behaviors lead to desired outcomes, occurs through goal-directed and habit memory systems. Exposure to acute stress has been shown to result in less goal-directed control, thus rendering behavior more habitual. The aim of the current studies was to replicate and extend findings on stress-induced prompting of habitual responding and specifically focused on the role of stress-induced cortisol reactivity. Study 1 used an established outcome devaluation paradigm to assess goal-directed and habitual control. Study 2 utilized a modified version of this paradigm that was intended to establish stronger habitual responding through more extensive reward training and applying a relevant behavioral devaluation procedure (i.e., eating to satiety). Both studies failed to replicate that stress overall, i.e., independent of cortisol reactivity, shifted behavior from goal-directed to habitual control. However, both studies found that relative to stress-exposed cortisol non-responders and no-stress controls, participants displaying stress-induced cortisol reactivity displayed prominent habitual responding. These findings highlight the importance of stress-induced cortisol reactivity in facilitating habits.

Developmental Changes in Learning: Computational Mechanisms and Social Influences

Our ability to learn from the outcomes of our actions and to adapt our decisions accordingly changes over the course of the human lifespan. In recent years, there has been an increasing interest in using computational models to understand developmental changes in learning and decision-making. Moreover, extensions of these models are currently applied to study socio-emotional influences on learning in different age groups, a topic that is of great relevance for applications in education and health psychology. In this article, we aim to provide an introduction to basic ideas underlying computational models of reinforcement learning and focus on parameters and model variants that might be of interest to developmental scientists. We then highlight recent attempts to use reinforcement learning models to study the influence of social information on learning across development. The aim of this review is to illustrate how computational models can be applied in developmental science, what they can add to our understanding of developmental mechanisms and how they can be used to bridge the gap between psychological and neurobiological theories of development.

Model-Based Control in Dimensional Psychiatry

We use parallel interacting goal-directed and habitual strategies to make our daily decisions. The arbitration between these strategies is relevant to inflexible repetitive behaviors in psychiatric disorders. Goal-directed control, also known as model-based control, is based on an affective outcome relying on a learned internal model to prospectively make decisions. In contrast, habit control, also known as model-free control, is based on an integration of previous reinforced learning autonomous of the current outcome value and is implicit and more efficient but at the cost of greater inflexibility. The concept of model-based control can be further extended into pavlovian processes. Here we describe and compare tasks that tap into these constructs and emphasize the clinical relevance and translation of these tasks in psychiatric disorders. Together, these findings highlight a role for model-based control as a transdiagnostic impairment underlying compulsive behaviors and representing a promising therapeutic target.

Computational approaches to schizophrenia: A perspective on negative symptoms

Schizophrenia is a heterogeneous spectrum disorder often associated with detrimental negative symptoms. In recent years, computational approaches to psychiatry have attracted growing attention. Negative symptoms have shown some overlap with general cognitive impairments and were also linked to impaired motivational processing in brain circuits implementing reward prediction. In this review, we outline how computational approaches may help to provide a better understanding of negative symptoms in terms of the potentially underlying behavioural and biological mechanisms. First, we describe the idea that negative symptoms could arise from a failure to represent reward expectations to enable flexible behavioural adaptation. It has been proposed that these impairments arise from a failure to use prediction errors to update expectations. Important previous studies focused on processing of so-called model-free prediction errors where learning is determined by past rewards only. However, learning and decision-making arise from multiple cognitive mechanisms functioning simultaneously, and dissecting them via well-designed tasks in conjunction with computational modelling is a promising avenue. Second, we move on to a proof-of-concept example on how generative models of functional imaging data from a cognitive task enable the identification of subgroups of patients mapping on different levels of negative symptoms. Combining the latter approach with behavioural studies regarding learning and decision-making may allow the identification of key behavioural and biological parameters distinctive for different dimensions of negative symptoms versus a general cognitive impairment. We conclude with an outlook on how this computational framework could, at some point, enrich future clinical studies.

Stress enhances model-free reinforcement learning only after negative outcome

Previous studies found that stress shifts behavioral control by promoting habits while decreasing goal-directed behaviors during reward-based decision-making. It is, however, unclear how stress disrupts the relative contribution of the two systems controlling reward-seeking behavior, i.e. model-free (or habit) and model-based (or goal-directed). Here, we investigated whether stress biases the contribution of model-free and model-based reinforcement learning processes differently depending on the valence of outcome, and whether stress alters the learning rate, i.e., how quickly information from the new environment is incorporated into choices. Participants were randomly assigned to either a stress or a control condition, and performed a two-stage Markov decision-making task in which the reward probabilities underwent periodic reversals without notice. We found that stress increased the contribution of model-free reinforcement learning only after negative outcome. Furthermore, stress decreased the learning rate. The results suggest that stress diminishes one's ability to make adaptive choices in multiple aspects of reinforcement learning. This finding has implications for understanding how stress facilitates maladaptive habits, such as addictive behavior, and other dysfunctional behaviors associated with stress in clinical and educational contexts.