Maintaining homeostasis by decision-making

Anxiety as a disorder of uncertainty: implications for understanding maladaptive anxiety, anxious avoidance, and exposure therapy

In cognitive-behavioral conceptualizations of anxiety, exaggerated threat expectancies underlie maladaptive anxiety. This view has led to successful treatments, notably exposure therapy, but is not consistent with the empirical literature on learning and choice alterations in anxiety. Empirically, anxiety is better described as a disorder of uncertainty learning. How disruptions in uncertainty lead to impairing avoidance and are treated with exposure-based methods, however, is unclear. Here, we integrate concepts from neurocomputational learning models with clinical literature on exposure therapy to propose a new framework for understanding maladaptive uncertainty functioning in anxiety. Specifically, we propose that anxiety disorders are fundamentally disorders of uncertainty learning and that successful treatments, particularly exposure therapy, work by remediating maladaptive avoidance from dysfunctional explore/exploit decisions in uncertain, potentially aversive situations. This framework reconciles several inconsistencies in the literature and provides a path forward to better understand and treat anxiety.

The effect of apathy and compulsivity on planning and stopping in sequential decision-making

Real-life decision-making often comprises sequences of successive decisions about whether to take opportunities as they are encountered or keep searching for better ones instead. We investigated individual differences related to such sequential decision-making and link them especially to apathy and compulsivity in a large online sample (discovery sample: n = 449 and confirmation sample: n = 756). Our cognitive model revealed distinct changes in the way participants evaluated their environments and planned their own future behaviour. Apathy was linked to decision inertia, i.e., automatically persisting with a sequence of searches for longer than appropriate given the value of searching. Thus, despite being less motivated, they did not avoid the effort associated with longer searches. In contrast, compulsivity was linked to self-reported insensitivity to the cost of continuing with a sequence of searches. The objective measures of behavioural cost insensitivity were clearly linked to compulsivity only in the discovery sample. While the confirmation sample showed a similar effect, it did not reach significance. Nevertheless, in both samples, participants reported awareness of such bias (experienced as "overchasing"). In addition, this awareness made them report preemptively avoiding situations related to the bias. However, we found no evidence of them actually preempting more in the task, which might mean a misalignment of their metacognitive beliefs or that our behavioural measures were incomplete. In summary, individual variation in distinct, fundamental aspects of sequential decision-making can be linked to variation in 2 measures of behavioural traits associated with psychological illness in the normal population.

The Hunger Games: Homeostatic State-Dependent Fluctuations in Disinhibition Measured with a Novel Gamified Test Battery

Food homeostatic states (hunger and satiety) influence the cognitive systems regulating impulsive responses, but the direction and specific mechanisms involved in this effect remain elusive. We examined how fasting, and satiety, affect cognitive mechanisms underpinning disinhibition using a novel framework and a gamified test-battery. Thirty-four participants completed the test-battery measuring three cognitive facets of disinhibition: attentional control, information gathering and monitoring of feedback, across two experimental sessions: one after overnight fasting and another after a standardised meal. Homeostatic state was assessed using subjective self-reports and biological markers (i.e., blood-derived liver-expressed antimicrobial protein 2 (LEAP-2), insulin and leptin). We found that participants who experienced greater subjective hunger during the satiety session were more impulsive in the information gathering task; results were not confounded by changes in mood or anxiety. Homeostatic state did not significantly influence disinhibition mechanisms linked to attentional control or feedback monitoring. However, we found a significant interaction between homeostatic state and LEAP-2 on attentional control, with higher LEAP-2 associated with faster reaction times in the fasted condition only. Our findings indicate lingering hunger after eating increases impulsive behaviour via reduced information gathering. These findings identify a novel mechanism that may underpin the tendency to overeat and/or engage in broader impulsive behaviours.

Wolbachia manipulate fitness benefits of olfactory associative learning in a parasitoid wasp

Upon encountering a host, a female parasitoid wasp has to decide whether to learn positive or negative cues related to the host. The optimal female decision will depend on the fitness costs and benefits of learned stimuli. Reward quality is positively related to the rate of behavioral acquisition in processes such as associative learning. Wolbachia, an endosymbiotic bacterium, often plays an impressive role in the manipulation of its arthropod host's biology. Here, we studied the responses of two natural Wolbachia infected/uninfected Trichogramma brassicae wasp populations to theoretically high- and low-reward values during a conditioning process and the consequences of their responses in terms of memory duration. According to our results, uninfected wasps showed an attraction response to high-value rewards, but showed aversive learning in response to low-value rewards. The memory span of uninfected wasps after conditioning by low-value rewards was significantly shorter than that for high-value rewards. As our results revealed, responses to high-quality hosts will bring more benefits (bigger size, increased fecundity and enhanced survival) than those to low-quality hosts for uninfected wasps. Infected wasps were attracted to conditioned stimuli with the same memory duration after conditioning by both types of hosts. This was linked to the fact that parasitoids emerging from both types of hosts present the same life-history traits. Therefore, these hosts represent the same quality reward for infected wasps. According to the obtained results, it can be concluded that Wolbachia manipulates the learning ability of its host, resulting in the wasp responding to all reward values similarly.

Hippocampal Representation of Threat Features and Behavior in a Human Approach-Avoidance Conflict Anxiety Task

Decisions under threat are crucial to survival and require integration of distinct situational features, such as threat probability and magnitude. Recent evidence from human lesion and neuroimaging studies implicated anterior hippocampus (aHC) and amygdala in approach-avoidance decisions under threat, and linked their integrity to cautious behavior. Here we sought to elucidate how threat dimensions and behavior are represented in these structures. Twenty human participants (11 female) completed an approach-avoidance conflict task during high-resolution fMRI. Participants could gather tokens under threat of capture by a virtual predator, which would lead to token loss. Threat probability (predator wake-up rate) and magnitude (amount of token loss) varied on each trial. To disentangle effects of threat features, and ensuing behavior, we performed a multifold parametric analysis. We found that high threat probability and magnitude related to BOLD signal in left aHC/entorhinal cortex. However, BOLD signal in this region was better explained by avoidance behavior than by these threat features. A priori ROI analysis confirmed the relation of aHC BOLD response with avoidance. Exploratory subfield analysis revealed that this relation was specific to anterior CA2/3 but not CA1. Left lateral amygdala responded to low and high, but not intermediate, threat probability. Our results suggest that aHC BOLD signal is better explained by avoidance behavior than by threat features in approach-avoidance conflict. Rather than representing threat features in a monotonic manner, it appears that aHC may compute approach-avoidance decisions based on integration of situational threat features represented in other neural structures.SIGNIFICANCE STATEMENT An effective threat anticipation system is crucial to survival across species. Natural threats, however, are diverse and have distinct features. To be able to adapt to different modes of danger, the brain needs to recognize these features, integrate them, and use them to modify behavior. Our results disclose the human anterior hippocampus as a likely arbiter of approach-avoidance decisions harnessing compound environmental information while partially replicating previous findings and blending into recent efforts to illuminate the neural basis of approach-avoidance conflict in humans.

Predictors of risky foraging behaviour in healthy young people

During adolescence and early adulthood, learning when to avoid threats and when to pursue rewards becomes crucial. Using a risky foraging task, we investigated individual differences in this dynamic across 781 individuals aged 14-24 years who were split into a hypothesis-generating discovery sample and a hold-out confirmation sample. Sex was the most important predictor of cautious behaviour and performance. Males earned one standard deviation (or 20%) more reward than females, collected more reward when there was little to lose and reduced foraging to the same level as females when potential losses became high. Other independent predictors of cautiousness and performance were self-reported daringness, IQ and self-reported cognitive complexity. We found no evidence for an impact of age or maturation. Thus, maleness, a high IQ or self-reported cognitive complexity, and self-reported daringness predicted greater success in risky foraging, possibly due to better exploitation of low-risk opportunities in high-risk environments.

Stimulation of the vagus nerve reduces learning in a go/no-go reinforcement learning task

When facing decisions to approach rewards or to avoid punishments, we often figuratively go with our gut, and the impact of metabolic states such as hunger on motivation are well documented. However, whether and how vagal feedback signals from the gut influence instrumental actions is unknown. Here, we investigated the effect of non-invasive transcutaneous auricular vagus nerve stimulation (taVNS) vs. sham (randomized cross-over design) on approach and avoidance behavior using an established go/no-go reinforcement learning paradigm in 39 healthy human participants (23 female) after an overnight fast. First, mixed-effects logistic regression analysis of choice accuracy showed that taVNS acutely impaired decision-making, p = .041. Computational reinforcement learning models identified the cause of this as a reduction in the learning rate through taVNS (∆α = -0.092, p_boot = .002), particularly after punishment (∆α_Pun = -0.081, p_boot = .012 vs. ∆α_Rew =-0.031, p_boot = .22). However, taVNS had no effect on go biases, Pavlovian response biases or response time. Hence, taVNS appeared to influence learning rather than action execution. These results highlight a novel role of vagal afferent input in modulating reinforcement learning by tuning the learning rate according to homeostatic needs.

Representation of probabilistic outcomes during risky decision-making

Goal-directed behaviour requires prospectively retrieving and evaluating multiple possible action outcomes. While a plethora of studies suggested sequential retrieval for deterministic choice outcomes, it remains unclear whether this is also the case when integrating multiple probabilistic outcomes of the same action. We address this question by capitalising on magnetoencephalography (MEG) in humans who made choices in a risky foraging task. We train classifiers to distinguish MEG field patterns during presentation of two probabilistic outcomes (reward, loss), and then apply these to decode such patterns during deliberation. First, decoded outcome representations have a temporal structure, suggesting alternating retrieval of the outcomes. Moreover, the probability that one or the other outcome is being represented depends on loss magnitude, but not on loss probability, and it predicts the chosen action. In summary, we demonstrate decodable outcome representations during probabilistic decision-making, which are sequentially structured, depend on task features, and predict subsequent action.

Hunger increases delay discounting of food and non-food rewards

How do our valuation systems change to homeostatically correct undesirable psychological or physiological states, such as those caused by hunger? There is evidence that hunger increases discounting for food rewards, biasing choices towards smaller but sooner food reward over larger but later reward. However, it is not understood how hunger modulates delay discounting for non-food items. We outline and quantitatively evaluate six possible models of how our valuation systems modulate discounting of various commodities in the face of the undesirable state of being hungry. With a repeated-measures design, an experimental hunger manipulation, and quantitative modeling, we find strong evidence that hunger causes large increases in delay discounting for food, with an approximately 25% spillover effect to non-food commodities. The results provide evidence that in the face of hunger, our valuation systems increase discounting for commodities, which cannot achieve a desired state change as well as for those commodities that can. Given that strong delay discounting can cause negative outcomes in many non-food (consumer, investment, medical, or inter-personal) domains, the present findings suggest caution may be necessary when making decisions involving non-food outcomes while hungry.

Minimizing threat via heuristic and optimal policies recruits hippocampus and medial prefrontal cortex

Jointly minimizing multiple threats over extended time horizons enhances survival. Consequently, many tests of approach-avoidance conflicts incorporate multiple threats for probing corollaries of animal and human anxiety. To facilitate computations necessary for threat minimization, the human brain may concurrently harness multiple decision policies and associated neural controllers, but it is unclear which. We combine a task that mimics foraging under predation with behavioural modelling and functional neuroimaging. Human choices rely on immediate predator probability-a myopic heuristic policy-and on the optimal policy, which integrates all relevant variables. Predator probability relates positively and the associated choice uncertainty relates negatively to activations in the anterior hippocampus, amygdala and dorsolateral prefrontal cortex. The optimal policy is positively associated with dorsomedial prefrontal cortex activity. We thus provide a decision-theoretic outlook on the role of the human hippocampus, amygdala and prefrontal cortex in resolving approach-avoidance conflicts relevant for anxiety and integral for survival.

Heuristic and optimal policy computations in the human brain during sequential decision-making

Optimal decisions across extended time horizons require value calculations over multiple probabilistic future states. Humans may circumvent such complex computations by resorting to easy-to-compute heuristics that approximate optimal solutions. To probe the potential interplay between heuristic and optimal computations, we develop a novel sequential decision-making task, framed as virtual foraging in which participants have to avoid virtual starvation. Rewards depend only on final outcomes over five-trial blocks, necessitating planning over five sequential decisions and probabilistic outcomes. Here, we report model comparisons demonstrating that participants primarily rely on the best available heuristic but also use the normatively optimal policy. FMRI signals in medial prefrontal cortex (MPFC) relate to heuristic and optimal policies and associated choice uncertainties. Crucially, reaction times and dorsal MPFC activity scale with discrepancies between heuristic and optimal policies. Thus, sequential decision-making in humans may emerge from integration between heuristic and optimal policies, implemented by controllers in MPFC.

Alhough humans often make a series of related decisions, it is unknown whether this is done by relying on optimal or heuristic strategies. Here, the authors show that humans rely on both the best heuristic and the optimal policy, and that these strategies are controlled by parts of the medial prefrontal cortex.

A Primer on Foraging and the Explore/Exploit Trade-Off for Psychiatry Research

Foraging is a fundamental behavior, and many types of animals appear to have solved foraging problems using a shared set of mechanisms. Perhaps the most common foraging problem is the choice between exploiting a familiar option for a known reward and exploring unfamiliar options for unknown rewards-the so-called explore/exploit trade-off. This trade-off has been studied extensively in behavioral ecology and computational neuroscience, but is relatively new to the field of psychiatry. Explore/exploit paradigms can offer psychiatry research a new approach to studying motivation, outcome valuation, and effort-related processes, which are disrupted in many mental and emotional disorders. In addition, the explore/exploit trade-off encompasses elements of risk-taking and impulsivity-common behaviors in psychiatric disorders-and provides a novel framework for understanding these behaviors within an ecological context. Here we explain relevant concepts and some common paradigms used to measure explore/exploit decisions in the laboratory, review clinically relevant research on the neurobiology and neuroanatomy of explore/exploit decision making, and discuss how computational psychiatry can benefit from foraging theory.

Algorithms for survival: a comparative perspective on emotions

The nature and neural implementation of emotions is the subject of vigorous debate. Here, we use Bayesian decision theory to address key complexities in this field and conceptualize emotions in terms of their relationship to survival-relevant behavioural choices. Decision theory indicates which behaviours are optimal in a given situation; however, the calculations required are radically intractable. We therefore conjecture that the brain uses a range of pre-programmed algorithms that provide approximate solutions. These solutions seem to produce specific behavioural manifestations of emotions and can also be associated with core affective dimensions. We identify principles according to which these algorithms are implemented in the brain and illustrate our approach by considering decision making in the face of proximal threat.

The cognitive architecture of anxiety-like behavioral inhibition

The combination of reward and potential threat is termed approach/avoidance conflict and elicits specific behaviors, including passive avoidance and behavioral inhibition (BI). Anxiety-relieving drugs reduce these behaviors, and a rich psychological literature has addressed how personality traits dominated by BI predispose for anxiety disorders. Yet, a formal understanding of the cognitive inference and planning processes underlying anxiety-like BI is lacking. Here, we present and empirically test such formalization in the terminology of reinforcement learning. We capitalize on a human computer game in which participants collect sequentially appearing monetary tokens while under threat of virtual “predation.” First, we demonstrate that humans modulate BI according to experienced consequences. This suggests an instrumental implementation of BI generation rather than a Pavlovian mechanism that is agnostic about action outcomes. Second, an internal model that would make BI adaptive is expressed in an independent task that involves no threat. The existence of such internal model is a necessary condition to conclude that BI is under model-based control. These findings relate a plethora of human and nonhuman observations on BI to reinforcement learning theory, and crucially constrain the quest for its neural implementation.

To work or not to work: Neural representation of cost and benefit of instrumental action

By definition, instrumental actions are performed in order to obtain certain goals. Nevertheless, the attainment of goals typically implies obstacles, and response vigor is known to reflect an integration of subjective benefit and cost. Whereas several brain regions have been associated with cost/benefit ratio decision-making, trial-by-trial fluctuations in motivation are not well understood. We review recent evidence supporting the motivational implications of signal fluctuations in the mesocorticolimbic system. As an extension of "set-point" theories of instrumental action, we propose that response vigor is determined by a rapid integration of brain signals that reflect value and cost on a trial-by-trial basis giving rise to an online estimate of utility. Critically, we posit that fluctuations in key nodes of the network can predict deviations in response vigor and that variability in instrumental behavior can be accounted for by models devised from optimal control theory, which incorporate the effortful control of noise. Notwithstanding, the post hoc analysis of signaling dynamics has caveats that can effectively be addressed in future research with the help of two novel fMRI imaging techniques. First, adaptive fMRI paradigms can be used to establish a time-order relationship, which is a prerequisite for causality, by using observed signal fluctuations as triggers for stimulus presentation. Second, real-time fMRI neurofeedback can be employed to induce predefined brain states that may facilitate benefit or cost aspects of instrumental actions. Ultimately, understanding temporal dynamics in brain networks subserving response vigor holds the promise for targeted interventions that could help to readjust the motivational balance of behavior.