Dopaminergic modulation of the exploration/exploitation trade-off in human decision-making

Electrophysiological signatures of the effect of context on exploration: Greater attentional and learning signals when exploration is costly

Humans are excellent at modifying our behaviour depending on context. For example, we will change how we explore when losses are possible compared to when losses are not possible. However, it remains unclear what specific cognitive and neural processes are modulated when exploring in different contexts. Here, we had participants learn within two different contexts: in one the participants could lose points while in the other the participants could not. Our goal was to determine how the inclusion of losses impacted human exploratory behaviour (experiment one), and whether we could explain the neural basis of these effects using EEG (experiment two). In experiment one, we found that participants preferred less-variable choices and explored less often when losses were possible. In addition, computational modelling revealed that participants engaged in less random exploration, had a lower rate of learning, and showed lower choice stickiness when losses were possible. In experiment two, we replicated these effects while examining a series of neural signals involved in exploration. During exploration, signals tied to working memory and learning (P3b), attention orienting (P3a) and motivation (late positive potential; an exploratory analysis) were enhanced when losses were possible. These neural differences contribute to why exploratory behaviour is changed by different learning contexts and can be explained by the theoretical claim that losses recruit attention and lead to increased task focus. These results provide insight into the cognitive processes that underlie exploration, and how exploratory behaviour changes across contexts.

Pharmacological Modulation of Dopamine Receptors Reveals Distinct Brain-Wide Networks Associated with Learning and Motivation in Nonhuman Primates

The neurotransmitter dopamine (DA) has a multifaceted role in healthy and disordered brains through its action on multiple subtypes of dopaminergic receptors. How the modulation of these receptors influences learning and motivation by altering intrinsic brain-wide networks remains unclear. Here, we performed parallel behavioral and resting-state functional MRI experiments after administration of two different DA receptor antagonists in male and female macaque monkeys. Systemic administration of SCH-23390 (D1 antagonist) slowed probabilistic learning when subjects had to learn new stimulus-reward associations and diminished functional connectivity (FC) in corticocortical and frontostriatal connections. In contrast, haloperidol (D2 antagonist) improved learning and broadly enhanced FC in cortical connections. Further comparisons between the effect of SCH-23390/haloperidol on behavioral and resting-state FC revealed specific cortical and subcortical networks associated with the cognitive and motivational effects of DA manipulation, respectively. Thus, we reveal distinct brain-wide networks that are associated with the dopaminergic control of learning and motivation via DA receptors.

Distinct computational mechanisms of uncertainty processing explain opposing exploratory behaviors in anxiety and apathy

BACKGROUND

Decision-making in uncertain environments can lead to varied outcomes, and how we process those outcomes may depend on our emotional state. Understanding how individuals interpret the sources of uncertainty is crucial for understanding adaptive behavior and mental well-being. Uncertainty can be broadly categorized into two components: volatility and stochasticity. Volatility describes how quickly conditions change. Stochasticity, on the other hand, refers to outcome randomness. We investigated how anxiety and apathy influenced people's perceptions of uncertainty, and how uncertainty perception shaped explore-exploit decisions.

METHODS

Participants (N = 1001, non-clinical sample) completed a restless three-armed bandit task that was analyzed using both latent state and process models.

RESULTS

Anxious individuals perceived uncertainty as resulting more from volatility, leading to increased exploration and learning rates, especially after reward omission. Conversely, apathetic individuals viewed uncertainty as more stochastic, resulting in decreased exploration and learning rates. The perceived volatility-to-stochasticity ratio mediated the anxiety-exploration relationship post-adverse outcomes. Dimensionality reduction showed exploration and uncertainty estimation to be distinct but related latent factors shaping a manifold of adaptive behavior that is modulated by anxiety and apathy.

CONCLUSIONS

These findings reveal distinct computational mechanisms for how anxiety and apathy influence decision-making, providing a framework for understanding cognitive and affective processes in neuropsychiatric disorders.

The influence of anxiety on exploration: A review of computational modeling studies

Exploratory behaviors can serve an adaptive role within novel or changing environments. Namely, they facilitate information gain, allowing an organism to maintain accurate beliefs about the environment and select actions that better maximize reward. However, finding the optimal balance between exploration and reward-seeking behavior - the so-called explore-exploit dilemma - can be challenging, as it requires sensitivity to one's own uncertainty and to the predictability of one's surroundings. Given the close relationship between uncertainty and anxiety, a body of work has now also emerged identifying associated effects on exploration. In particular, the field of computational psychiatry has begun to use cognitive computational models to characterize how anxiety may modulate underlying information processing mechanisms, such as estimation of uncertainty and the value of information, and how this might contribute to psychopathology. Here, we review computational modeling studies investigating how exploration is influenced by anxiety. While some apparent inconsistencies remain to be resolved, studies using reinforcement learning tasks suggest that directed (but not random) forms of exploration may be elevated by trait and/or cognitive anxiety, but reduced by state and/or somatic anxiety. Anxiety is also consistently associated with less exploration in foraging tasks. Some differences in exploration may further stem from how anxiety modulates changes in uncertainty over time (learning rates). Jointly, these results highlight important directions for future work in refining choice of tasks and anxiety measures and maintaining consistent methodology across studies.

Pharmacological modulation of dopamine receptors reveals distinct brain-wide networks associated with learning and motivation in non-human primates

The neurotransmitter dopamine (DA) has a multifaceted role in healthy and disordered brains through its action on multiple subtypes of dopaminergic receptors. How modulation of these receptors influences learning and motivation by altering intrinsic brain-wide networks remains unclear. Here we performed parallel behavioral and resting-state functional MRI experiments after administration of two different DA receptor antagonists in macaque monkeys. Systemic administration of SCH-23390 (D1 antagonist) slowed probabilistic learning when subjects had to learn new stimulus-reward associations and diminished functional connectivity (FC) in cortico-cortical and fronto-striatal connections. By contrast, haloperidol (D2 antagonist) improved learning and broadly enhanced FC in cortical connections. Further comparisons between the effect of SCH-23390/haloperidol on behavioral and resting-state FC revealed specific cortical and subcortical networks associated with the cognitive and motivational effects of DA manipulation, respectively. Thus, we reveal distinct brain-wide networks that are associated with the dopaminergic control of learning and motivation via DA receptors.

Individual differences in information demand have a low dimensional structure predicted by some curiosity traits

To understand human learning and progress, it is crucial to understand curiosity. But how consistent is curiosity's conception and assessment across scientific research disciplines? We present the results of a large collaborative project assessing the correspondence between curiosity measures in personality psychology and cognitive science. A total of 820 participants completed 15 personality trait measures and 9 cognitive tasks that tested multiple aspects of information demand. We show that shared variance across the cognitive tasks was captured by a dimension reflecting directed (uncertainty-driven) versus random (stochasticity-driven) exploration and individual differences along this axis were significantly and consistently predicted by personality traits. However, the personality metrics that best predicted information demand were not the central curiosity traits of openness to experience, deprivation sensitivity, and joyous exploration, but instead included more peripheral curiosity traits (need for cognition, thrill seeking, and stress tolerance) and measures not traditionally associated with curiosity (extraversion and behavioral inhibition). The results suggest that the umbrella term "curiosity" reflects a constellation of cognitive and emotional processes, only some of which are shared between personality measures and cognitive tasks. The results reflect the distinct methods that are used in these fields, indicating a need for caution in comparing results across fields and for future interdisciplinary collaborations to strengthen our emerging understanding of curiosity.

Decrease in decision noise from adolescence into adulthood mediates an increase in more sophisticated choice behaviors and performance gain

Learning and decision-making undergo substantial developmental changes, with adolescence being a particular vulnerable window of opportunity. In adolescents, developmental changes in specific choice behaviors have been observed (e.g., goal-directed behavior, motivational influences over choice). Elevated levels of decision noise, i.e., choosing suboptimal options, were reported consistently in adolescents. However, it remains unknown whether these observations, the development of specific and more sophisticated choice processes and higher decision noise, are independent or related. It is conceivable, but has not yet been investigated, that the development of specific choice processes might be impacted by age-dependent changes in decision noise. To answer this, we examined 93 participants (12 to 42 years) who completed 3 reinforcement learning (RL) tasks: a motivational Go/NoGo task assessing motivational influences over choices, a reversal learning task capturing adaptive decision-making in response to environmental changes, and a sequential choice task measuring goal-directed behavior. This allowed testing of (1) cross-task generalization of computational parameters focusing on decision noise; and (2) assessment of mediation effects of noise on specific choice behaviors. Firstly, we found only noise levels to be strongly correlated across RL tasks. Second, and critically, noise levels mediated age-dependent increases in more sophisticated choice behaviors and performance gain. Our findings provide novel insights into the computational processes underlying developmental changes in decision-making: namely a vital role of seemingly unspecific changes in noise in the specific development of more complex choice components. Studying the neurocomputational mechanisms of how varying levels of noise impact distinct aspects of learning and decision processes may also be key to better understand the developmental onset of psychiatric diseases.

A causal role of the right dorsolateral prefrontal cortex in random exploration

Decision to explore new options with uncertain outcomes or exploit familiar options with known outcomes is a fundamental challenge that the brain faces in almost all real-life decisions. Previous studies have shown that humans use two main explorative strategies to negotiate this explore-exploit tradeoff. Exploring for the sake of information is called directed exploration, and exploration driven by behavioral variability is known as random exploration. While previous neuroimaging studies have shown different neural correlates for these explorative strategies, including right frontopolar cortex (FPC), right dorsolateral prefrontal cortex (DLPFC), and dorsal anterior cingulate cortex (dACC), there is still a lack of causal evidence for most of these brain regions. Here, we focused on the right DLPFC, which was previously supported to be involved in exploration. Using the continuous theta burst stimulation (cTBS) and Horizon task on twenty-five healthy right-handed adult participants, we showed that inhibiting rDLPFC did not change directed exploration but selectively reduced random exploration, by increasing reward sensitivity over the average reward of each option. This suggests a causal role for rDLPFC in random exploration, and further supports dissociable neural implementations for these two explorative strategies.

Reviewing explore/exploit decision-making as a transdiagnostic target for psychosis, depression, and anxiety

In many everyday decisions, individuals choose between trialling something novel or something they know well. Deciding when to try a new option or stick with an option that is already known to you, known as the "explore/exploit" dilemma, is an important feature of cognition that characterises a range of decision-making contexts encountered by humans. Recent evidence has suggested preferences in explore/exploit biases are associated with psychopathology, although this has typically been examined within individual disorders. The current review examined whether explore/exploit decision-making represents a promising transdiagnostic target for psychosis, depression, and anxiety. A systematic search of academic databases was conducted, yielding a total of 29 studies. Studies examining psychosis were mostly consistent in showing that individuals with psychosis explored more compared with individuals without psychosis. The literature on anxiety and depression was more heterogenous; some studies found that anxiety and depression were associated with more exploration, whereas other studies demonstrated reduced exploration in anxiety and depression. However, examining a subset of studies that employed case-control methods, there was some evidence that both anxiety and depression also were associated with increased exploration. Due to the heterogeneity across the literature, we suggest that there is insufficient evidence to conclude whether explore/exploit decision-making is a transdiagnostic target for psychosis, depression, and anxiety. However, alongside our advisory groups of lived experience advisors, we suggest that this context of decision-making is a promising candidate that merits further investigation using well-powered, longitudinal designs. Such work also should examine whether biases in explore/exploit choices are amenable to intervention.

Meta-Analysis Reveals That Explore-Exploit Decisions are Dissociable by Activation in the Dorsal Lateral Prefrontal Cortex, Anterior Insula, and the Anterior Cingulate Cortex

Explore-exploit research faces challenges in generalizability due to a limited theoretical basis for exploration and exploitation. Neuroimaging can help identify whether explore-exploit decisions involve an opponent processing system to address this issue. Thus, we conducted a coordinate-based meta-analysis (N=23 studies) finding activation in the dorsal lateral prefrontal cortex, anterior insula, and anterior cingulate cortex during exploration versus exploitation, which provides some evidence for opponent processing. However, the conjunction of explore-exploit decisions was associated with activation in the dorsal anterior cingulate cortex and dorsal medial prefrontal cortex, suggesting that these brain regions do not engage in opponent processing. Furthermore, exploratory analyses revealed heterogeneity in brain responses between task types during exploration and exploitation respectively. Coupled with results suggesting that activation during exploration and exploitation decisions is generally more similar than it is different suggests, there remain significant challenges in characterizing explore-exploit decision making. Nonetheless, dlPFC, AI, and ACC activation differentiate explore and exploit decisions and identifying these responses can aid in targeted interventions aimed at manipulating these decisions.

Causal computations of supplementary motor area on spatial impulsivity

Spatial proximity to important stimuli often induces impulsive behaviour. How we overcome impulsive tendencies is what determines behaviour to be adaptive. Here, we used virtual reality to investigate whether the spatial proximity of stimuli is causally related to the supplementary motor area (SMA) functions. In two experiments, we set out to investigate these processes using a virtual environment that recreates close and distant spaces to test the causal contributions of the SMA in spatial impulsivity. In an online first experiment (N = 93) we validated and measured the influence of distant stimuli using a go/no-go task with close (21 cm) or distant stimuli (360 cm). In experiment 2 (N = 28), we applied transcranial static magnetic stimulation (tSMS) over the SMA (double-blind, crossover, sham-controlled design) to test its computations in controlling impulsive tendencies towards close vs distant stimuli. Reaction times and error rates (omission and commission) were analysed. In addition, the EZ Model parameters (a, v, Ter and MDT) were computed. Close stimuli elicited faster responses compared to distant stimuli but also exhibited higher error rates, specifically in commission errors (experiment 1). Real stimulation over SMA slowed response latencies (experiment 2), an effect mediated by an increase in decision thresholds (a). Current findings suggest that impulsivity might be modulated by spatial proximity, resulting in accelerated actions that may lead to an increase of inaccurate responses to nearby objects. Our study also provides a first starting point on the role of the SMA in regulating spatial impulsivity.

Distinct computational mechanisms of uncertainty processing explain opposing exploratory behaviors in anxiety and apathy

Decision-making in uncertain environments often leads to varied outcomes. Understanding how individuals interpret the causes of unexpected feedback is crucial for adaptive behavior and mental well-being. Uncertainty can be broadly categorized into two components: volatility and stochasticity. Volatility is about how quickly conditions change, impacting results. Stochasticity, on the other hand, refers to outcomes affected by random chance or "luck". Understanding these factors enables individuals to have more effective environmental analysis and strategy implementation (explore or exploit) for future decisions. This study investigates how anxiety and apathy, two prevalent affective states, influence the perceptions of uncertainty and exploratory behavior. Participants (N = 1001) completed a restless three-armed bandit task that was analyzed using latent state models. Anxious individuals perceived uncertainty as more volatile, leading to increased exploration and learning rates, especially after reward omission. Conversely, apathetic individuals viewed uncertainty as more stochastic, resulting in decreased exploration and learning rates. The perceived volatility-to-stochasticity ratio mediated the anxiety-exploration relationship post-adverse outcomes. Dimensionality reduction showed exploration and uncertainty estimation to be distinct but related latent factors shaping a manifold of adaptive behavior that is modulated by anxiety and apathy. These findings reveal distinct computational mechanisms for how anxiety and apathy influence decision-making, providing a framework for understanding cognitive and affective processes in neuropsychiatric disorders.

Assessing the influence of dopamine and mindfulness on the formation of routines in visual search

Given experience in cluttered but stable visual environments, our eye-movements form stereotyped routines that sample task-relevant locations, while not mixing-up routines between similar task-settings. Both dopamine signaling and mindfulness have been posited as factors that influence the formation of such routines, yet quantification of their impact remains to be tested in healthy humans. Over two sessions, participants searched through grids of doors to find hidden targets, using a gaze-contingent display. Within each session, door scenes appeared in either one of two colors, with each color signaling a differing set of likely target locations. We derived measures for how well target locations were learned (target-accuracy), how routine were sets of eye-movements (stereotypy), and the extent of interference between the two scenes (setting-accuracy). Participants completed two sessions, where they were administered either levodopa (dopamine precursor) or placebo (vitamin C), under double-blind counterbalanced conditions. Dopamine and trait mindfulness (assessed by questionnaire) interacted to influence both target-accuracy and stereotypy. Increasing dopamine improved accuracy and reduced stereotypy for high mindfulness scorers, but induced the opposite pattern for low mindfulness scorers. Dopamine also disrupted setting-accuracy invariant to mindfulness. Our findings show that mindfulness modulates the impact of dopamine on the target-accuracy and stereotypy of eye-movement routines, whereas increasing dopamine promotes interference between task-settings, regardless of mindfulness. These findings provide a link between non-human and human models regarding the influence of dopamine on the formation of task-relevant eye-movement routines and provide novel insights into behavior-trait factors that modulate the use of experience when building adaptive repertoires.

Neural mechanisms of information seeking

We ubiquitously seek information to make better decisions. Particularly in the modern age, when more information is available at our fingertips than ever, the information we choose to collect determines the quality of our decisions. Decision neuroscience has long adopted empirical approaches where the information available to decision-makers is fully controlled by the researchers, leaving neural mechanisms of information seeking less understood. Although information seeking has long been studied in the context of the exploration-exploitation trade-off, recent studies have widened the scope to investigate more overt information seeking in a way distinct from other decision processes. Insights gained from these studies, accumulated over the last few years, raise the possibility that information seeking is driven by the reward system signaling the subjective value of information. In this piece, we review findings from the recent studies, highlighting the conceptual and empirical relationships between distinct literatures, and discuss future research directions necessary to establish a more comprehensive understanding of how individuals seek information as a part of value-based decision-making.

Behavioral Bias for Exploration Is Associated with Enhanced Signaling in the Lateral and Medial Frontopolar Cortex

Should we keep doing what we know works for us, or should we risk trying something new as it could work even better? The exploration-exploitation dilemma is ubiquitous in daily life decision-making, and balancing between the two is crucial for adaptive behavior. Yet, we only have started to unravel the neurocognitive mechanisms that help us to find this balance in practice. Analyzing BOLD signals of healthy young adults during virtual foraging, we could show that a behavioral tendency for prolonged exploitation was associated with weakened signaling during exploration in central node points of the frontoparietal attention network, plus the frontopolar cortex. These results provide an important link between behavioral heuristics that we use to balance between exploitation and exploration and the brain function that supports shifts from one tendency to the other. Importantly, they stress that interindividual differences in behavioral strategies are reflected in differences in brain activity during exploration and should thus be more in the focus of basic research that aims at delineating general laws governing visual attention.

Bayesian reinforcement learning: A basic overview

We and other animals learn because there is some aspect of the world about which we are uncertain. This uncertainty arises from initial ignorance, and from changes in the world that we do not perfectly know; the uncertainty often becomes evident when our predictions about the world are found to be erroneous. The Rescorla-Wagner learning rule, which specifies one way that prediction errors can occasion learning, has been hugely influential as a characterization of Pavlovian conditioning and, through its equivalence to the delta rule in engineering, in a much wider class of learning problems. Here, we review the embedding of the Rescorla-Wagner rule in a Bayesian context that is precise about the link between uncertainty and learning, and thereby discuss extensions to such suggestions as the Kalman filter, structure learning, and beyond, that collectively encompass a wider range of uncertainties and accommodate a wider assortment of phenomena in conditioning.

Impaired value-based decision-making in Parkinson's disease apathy

Apathy is a common and disabling complication of Parkinson's disease characterized by reduced goal-directed behaviour. Several studies have reported dysfunction within prefrontal cortical regions and projections from brainstem nuclei whose neuromodulators include dopamine, serotonin and noradrenaline. Work in animal and human neuroscience have confirmed contributions of these neuromodulators on aspects of motivated decision-making. Specifically, these neuromodulators have overlapping contributions to encoding the value of decisions, and influence whether to explore alternative courses of action or persist in an existing strategy to achieve a rewarding goal. Building upon this work, we hypothesized that apathy in Parkinson's disease should be associated with an impairment in value-based learning. Using a four-armed restless bandit reinforcement learning task, we studied decision-making in 75 volunteers; 53 patients with Parkinson's disease, with and without clinical apathy, and 22 age-matched healthy control subjects. Patients with apathy exhibited impaired ability to choose the highest value bandit. Task performance predicted an individual patient's apathy severity measured using the Lille Apathy Rating Scale (R = -0.46, P < 0.001). Computational modelling of the patient's choices confirmed the apathy group made decisions that were indifferent to the learnt value of the options, consistent with previous reports of reward insensitivity. Further analysis demonstrated a shift away from exploiting the highest value option and a reduction in perseveration, which also correlated with apathy scores (R = -0.5, P < 0.001). We went on to acquire functional MRI in 59 volunteers; a group of 19 patients with and 20 without apathy and 20 age-matched controls performing the Restless Bandit Task. Analysis of the functional MRI signal at the point of reward feedback confirmed diminished signal within ventromedial prefrontal cortex in Parkinson's disease, which was more marked in apathy, but not predictive of their individual apathy severity. Using a model-based categorization of choice type, decisions to explore lower value bandits in the apathy group activated prefrontal cortex to a similar degree to the age-matched controls. In contrast, Parkinson's patients without apathy demonstrated significantly increased activation across a distributed thalamo-cortical network. Enhanced activity in the thalamus predicted individual apathy severity across both patient groups and exhibited functional connectivity with dorsal anterior cingulate cortex and anterior insula. Given that task performance in patients without apathy was no different to the age-matched control subjects, we interpret the recruitment of this network as a possible compensatory mechanism, which compensates against symptomatic manifestation of apathy in Parkinson's disease.

Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts

Active reinforcement learning enables dynamic prediction and control, where one should not only maximize rewards but also minimize costs such as of inference, decisions, actions, and time. For an embodied agent such as a human, decisions are also shaped by physical aspects of actions. Beyond the effects of reward outcomes on learning processes, to what extent can modeling of behavior in a reinforcement-learning task be complicated by other sources of variance in sequential action choices? What of the effects of action bias (for actions per se) and action hysteresis determined by the history of actions chosen previously? The present study addressed these questions with incremental assembly of models for the sequential choice data from a task with hierarchical structure for additional complexity in learning. With systematic comparison and falsification of computational models, human choices were tested for signatures of parallel modules representing not only an enhanced form of generalized reinforcement learning but also action bias and hysteresis. We found evidence for substantial differences in bias and hysteresis across participants-even comparable in magnitude to the individual differences in learning. Individuals who did not learn well revealed the greatest biases, but those who did learn accurately were also significantly biased. The direction of hysteresis varied among individuals as repetition or, more commonly, alternation biases persisting from multiple previous actions. Considering that these actions were button presses with trivial motor demands, the idiosyncratic forces biasing sequences of action choices were robust enough to suggest ubiquity across individuals and across tasks requiring various actions. In light of how bias and hysteresis function as a heuristic for efficient control that adapts to uncertainty or low motivation by minimizing the cost of effort, these phenomena broaden the consilient theory of a mixture of experts to encompass a mixture of expert and nonexpert controllers of behavior.

Motivation as a Lens for Understanding Information-seeking Behaviors

Most prior research characterizes information-seeking behaviors as serving utilitarian purposes, such as whether the obtained information can help solve practical problems. However, information-seeking behaviors are sensitive to different contexts (i.e., threat vs. curiosity), despite having equivalent utility. Furthermore, these search behaviors can be modulated by individuals' life history and personality traits. Yet the emphasis on utilitarian utility has precluded the development of a unified model, which explains when and how individuals actively seek information. To account for this variability and flexibility, we propose a unified information-seeking framework that examines information-seeking through the lens of motivation. This unified model accounts for integration across individuals' internal goal states and the salient features of the environment to influence information-seeking behavior. We propose that information-seeking is determined by motivation for information, invigorated either by instrumental utility or hedonic utility, wherein one's personal or environmental context moderates this relationship. Furthermore, we speculate that the final common denominator in guiding information-seeking is the engagement of different neuromodulatory circuits centered on dopaminergic and noradrenergic tone. Our framework provides a unified framework for information-seeking behaviors and generates several testable predictions for future studies.

Exploration versus exploitation decisions in the human brain: A systematic review of functional neuroimaging and neuropsychological studies

Thoughts and actions are often driven by a decision to either explore new avenues with unknown outcomes, or to exploit known options with predictable outcomes. Yet, the neural mechanisms underlying this exploration-exploitation trade-off in humans remain poorly understood. This is attributable to variability in the operationalization of exploration and exploitation as psychological constructs, as well as the heterogeneity of experimental protocols and paradigms used to study these choice behaviours. To address this gap, here we present a comprehensive review of the literature to investigate the neural basis of explore-exploit decision-making in humans. We first conducted a systematic review of functional magnetic resonance imaging (fMRI) studies of exploration-versus exploitation-based decision-making in healthy adult humans during foraging, reinforcement learning, and information search. Eleven fMRI studies met inclusion criterion for this review. Adopting a network neuroscience framework, synthesis of the findings across these studies revealed that exploration-based choice was associated with the engagement of attentional, control, and salience networks. In contrast, exploitation-based choice was associated with engagement of default network brain regions. We interpret these results in the context of a network architecture that supports the flexible switching between externally and internally directed cognitive processes, necessary for adaptive, goal-directed behaviour. To further investigate potential neural mechanisms underlying the exploration-exploitation trade-off we next surveyed studies involving neurodevelopmental, neuropsychological, and neuropsychiatric disorders, as well as lifespan development, and neurodegenerative diseases. We observed striking differences in patterns of explore-exploit decision-making across these populations, again suggesting that these two decision-making modes are supported by independent neural circuits. Taken together, our review highlights the need for precision-mapping of the neural circuitry and behavioural correlates associated with exploration and exploitation in humans. Characterizing exploration versus exploitation decision-making biases may offer a novel, trans-diagnostic approach to assessment, surveillance, and intervention for cognitive decline and dysfunction in normal development and clinical populations.

Erotic cue exposure increases physiological arousal, biases choices toward immediate rewards, and attenuates model-based reinforcement learning

Computational psychiatry focuses on identifying core cognitive processes that appear altered across distinct psychiatric disorders. Temporal discounting of future rewards and model-based control during reinforcement learning have proven as two promising candidates. Despite its trait-like stability, temporal discounting may be at least partly under contextual control. Highly arousing cues were shown to increase discounting, although evidence to date remains somewhat mixed. Whether model-based reinforcement learning is similarly affected by arousing cues remains unclear. Here, we tested cue-reactivity effects (erotic pictures) on subsequent temporal discounting and model-based reinforcement learning in a within-subjects design in n = 39 healthy heterosexual male participants. Self-reported and physiological arousal (cardiac activity and pupil dilation) were assessed before and during cue exposure. Arousal was increased during exposure of erotic versus neutral cues both on the subjective and autonomic level. Erotic cue exposure increased discounting as reflected by more impatient choices. Hierarchical drift diffusion modeling (DDM) linked increased discounting to a shift in the starting point bias of evidence accumulation toward immediate options. Model-based control during reinforcement learning was reduced following erotic cues according to model-agnostic analysis. Notably, DDM linked this effect to attenuated forgetting rates of unchosen options, leaving the model-based control parameter unchanged. Our findings replicate previous work on cue-reactivity effects in temporal discounting and for the first time show similar effects in model-based reinforcement learning in a heterosexual male sample. This highlights how environmental cues can impact core human decision processes and reveal that comprehensive modeling approaches can yield novel insights in reward-based decision processes.

How does apathy impact exploration-exploitation decision-making in older patients with neurocognitive disorders?

Apathy is a pervasive clinical syndrome in neurocognitive disorders, characterized by a quantitative reduction in goal-directed behaviors. The brain structures involved in the physiopathology of apathy have also been connected to the brain structures involved in probabilistic reward learning in the exploration-exploitation dilemma. This dilemma in question involves the challenge of selecting between a familiar option with a more predictable outcome, and another option whose outcome is uncertain and may yield potentially greater rewards compared to the known option. The aim of this study was to combine experimental procedures and computational modeling to examine whether, in older adults with mild neurocognitive disorders, apathy affects performance in the exploration-exploitation dilemma. Through using a four-armed bandit reinforcement-learning task, we showed that apathetic older adults explored more and performed worse than non-apathetic subjects. Moreover, the mental flexibility assessed by the Trail-making test-B was negatively associated with the percentage of exploration. These results suggest that apathy is characterized by an increased explorative behavior and inefficient decision-making, possibly due to weak mental flexibility to switch toward the exploitation of the more rewarding options. Apathetic participants also took longer to make a choice and failed more often to respond in the allotted time, which could reflect the difficulties in action initiation and selection. In conclusion, the present results suggest that apathy in participants with neurocognitive disorders is associated with specific disturbances in the exploration-exploitation trade-off and sheds light on the disturbances in reward processing in patients with apathy.

Interindividual Brain and Behavior Differences in Adaptation to Unexpected Uncertainty

To adapt to a new environment, individuals must alternate between exploiting previously learned "action-consequence" combinations and exploring new actions for which the consequences are unknown: they face an exploration/exploitation trade-off. The neural substrates of these behaviors and the factors that may relate to the interindividual variability in their expression remain overlooked, in particular when considering neural connectivity patterns. Here, to trigger environmental uncertainty, false feedbacks were introduced in the second phase of an associative learning task. Indices reflecting exploitation and cost of uncertainty were computed. Changes in the intrinsic connectivity were determined using resting-state functional connectivity (rFC) analyses before and after performing the "cheated" phase of the task in the MRI. We explored their links with behavioral and psychological factors. Dispersion in the participants' cost of uncertainty was used to categorize two groups. These groups showed different patterns of rFC changes. Moreover, in the overall sample, exploitation was correlated with rFC changes between (1) the anterior cingulate cortex and the cerebellum region 3, and (2) the left frontal inferior gyrus (orbital part) and the right frontal inferior gyrus (triangular part). Anxiety and doubt about action propensity were weakly correlated with some rFC changes. These results demonstrate that the exploration/exploitation trade-off involves the modulation of cortico-cerebellar intrinsic connectivity.

PET-measured human dopamine synthesis capacity and receptor availability predict trading rewards and time-costs during foraging

Foraging behavior requires weighing costs of time to decide when to leave one reward patch to search for another. Computational and animal studies suggest that striatal dopamine is key to this process; however, the specific role of dopamine in foraging behavior in humans is not well characterized. We use positron emission tomography (PET) imaging to directly measure dopamine synthesis capacity and D1 and D2/3 receptor availability in 57 healthy adults who complete a computerized foraging task. Using voxelwise data and principal component analysis to identify patterns of variation across PET measures, we show that striatal D1 and D2/3 receptor availability and a pattern of mesolimbic and anterior cingulate cortex dopamine function are important for adjusting the threshold for leaving a patch to explore, with specific sensitivity to changes in travel time. These findings suggest a key role for dopamine in trading reward benefits against temporal costs to modulate behavioral adaptions to changes in the reward environment critical for foraging.

Acetylcholine and noradrenaline enhance foraging optimality in humans

Foraging theory prescribes when optimal foragers should leave the current option for more rewarding alternatives. Actual foragers often exploit options longer than prescribed by the theory, but it is unclear how this foraging suboptimality arises. We investigated whether the upregulation of cholinergic, noradrenergic, and dopaminergic systems increases foraging optimality. In a double-blind, between-subject design, participants (N = 160) received placebo, the nicotinic acetylcholine receptor agonist nicotine, a noradrenaline reuptake inhibitor reboxetine, or a preferential dopamine reuptake inhibitor methylphenidate, and played the role of a farmer who collected milk from patches with different yield. Across all groups, participants on average overharvested. While methylphenidate had no effects on this bias, nicotine, and to some extent also reboxetine, significantly reduced deviation from foraging optimality, which resulted in better performance compared to placebo. Concurring with amplified goal-directedness and excluding heuristic explanations, nicotine independently also improved trial initiation and time perception. Our findings elucidate the neurochemical basis of behavioral flexibility and decision optimality and open unique perspectives on psychiatric disorders affecting these functions.

Dopamine regulates decision thresholds in human reinforcement learning in males

Dopamine fundamentally contributes to reinforcement learning, but recent accounts also suggest a contribution to specific action selection mechanisms and the regulation of response vigour. Here, we examine dopaminergic mechanisms underlying human reinforcement learning and action selection via a combined pharmacological neuroimaging approach in male human volunteers (n = 31, within-subjects; Placebo, 150 mg of the dopamine precursor L-dopa, 2 mg of the D2 receptor antagonist Haloperidol). We found little credible evidence for previously reported beneficial effects of L-dopa vs. Haloperidol on learning from gains and altered neural prediction error signals, which may be partly due to differences experimental design and/or drug dosages. Reinforcement learning drift diffusion models account for learning-related changes in accuracy and response times, and reveal consistent decision threshold reductions under both drugs, in line with the idea that lower dosages of D2 receptor antagonists increase striatal DA release via an autoreceptor-mediated feedback mechanism. These results are in line with the idea that dopamine regulates decision thresholds during reinforcement learning, and may help to bridge action selection and response vigor accounts of dopamine.

Enhancing reinforcement learning models by including direct and indirect pathways improves performance on striatal dependent tasks

A major advance in understanding learning behavior stems from experiments showing that reward learning requires dopamine inputs to striatal neurons and arises from synaptic plasticity of cortico-striatal synapses. Numerous reinforcement learning models mimic this dopamine-dependent synaptic plasticity by using the reward prediction error, which resembles dopamine neuron firing, to learn the best action in response to a set of cues. Though these models can explain many facets of behavior, reproducing some types of goal-directed behavior, such as renewal and reversal, require additional model components. Here we present a reinforcement learning model, TD2Q, which better corresponds to the basal ganglia with two Q matrices, one representing direct pathway neurons (G) and another representing indirect pathway neurons (N). Unlike previous two-Q architectures, a novel and critical aspect of TD2Q is to update the G and N matrices utilizing the temporal difference reward prediction error. A best action is selected for N and G using a softmax with a reward-dependent adaptive exploration parameter, and then differences are resolved using a second selection step applied to the two action probabilities. The model is tested on a range of multi-step tasks including extinction, renewal, discrimination; switching reward probability learning; and sequence learning. Simulations show that TD2Q produces behaviors similar to rodents in choice and sequence learning tasks, and that use of the temporal difference reward prediction error is required to learn multi-step tasks. Blocking the update rule on the N matrix blocks discrimination learning, as observed experimentally. Performance in the sequence learning task is dramatically improved with two matrices. These results suggest that including additional aspects of basal ganglia physiology can improve the performance of reinforcement learning models, better reproduce animal behaviors, and provide insight as to the role of direct- and indirect-pathway striatal neurons.

Instructed motivational states bias reinforcement learning and memory formation

Motivation influences goals, decisions, and memory formation. Imperative motivation links urgent goals to actions, narrowing the focus of attention and memory. Conversely, interrogative motivation integrates goals over time and space, supporting rich memory encoding for flexible future use. We manipulated motivational states via cover stories for a reinforcement learning task: The imperative group imagined executing a museum heist, whereas the interrogative group imagined planning a future heist. Participants repeatedly chose among four doors, representing different museum rooms, to sample trial-unique paintings with variable rewards (later converted to bonus payments). The next day, participants performed a surprise memory test. Crucially, only the cover stories differed between the imperative and interrogative groups; the reinforcement learning task was identical, and all participants had the same expectations about how and when bonus payments would be awarded. In an initial sample and a preregistered replication, we demonstrated that imperative motivation increased exploitation during reinforcement learning. Conversely, interrogative motivation increased directed (but not random) exploration, despite the cost to participants' earnings. At test, the interrogative group was more accurate at recognizing paintings and recalling associated values. In the interrogative group, higher value paintings were more likely to be remembered; imperative motivation disrupted this effect of reward modulating memory. Overall, we demonstrate that a prelearning motivational manipulation can bias learning and memory, bearing implications for education, behavior change, clinical interventions, and communication.

Integrative Brain Network and Salience Models of Psychopathology and Cognitive Dysfunction in Schizophrenia

Brain network models of cognitive control are central to advancing our understanding of psychopathology and cognitive dysfunction in schizophrenia. This review examines the role of large-scale brain organization in schizophrenia, with a particular focus on a triple-network model of cognitive control and its role in aberrant salience processing. First, we provide an overview of the triple network involving the salience, frontoparietal, and default mode networks and highlight the central role of the insula-anchored salience network in the aberrant mapping of salient external and internal events in schizophrenia. We summarize the extensive evidence that has emerged from structural, neurochemical, and functional brain imaging studies for aberrancies in these networks and their dynamic temporal interactions in schizophrenia. Next, we consider the hypothesis that atypical striatal dopamine release results in misattribution of salience to irrelevant external stimuli and self-referential mental events. We propose an integrated triple-network salience-based model incorporating striatal dysfunction and sensitivity to perceptual and cognitive prediction errors in the insula node of the salience network and postulate that dysregulated dopamine modulation of salience network-centered processes contributes to the core clinical phenotype of schizophrenia. Thus, a powerful paradigm to characterize the neurobiology of schizophrenia emerges when we combine conceptual models of salience with large-scale cognitive control networks in a unified manner. We conclude by discussing potential therapeutic leads on restoring brain network dysfunction in schizophrenia.

Dopamine and norepinephrine differentially mediate the exploration-exploitation tradeoff

The catecholamines dopamine (DA) and norepinephrine (NE) have been repeatedly implicated in neuropsychiatric vulnerability, in part via their roles in mediating the decision making processes. Although the two neuromodulators share a synthesis pathway and are co-activated under states of arousal, they engage in distinct circuits and roles in modulating neural activity across the brain. However, in the computational neuroscience literature, they have been assigned similar roles in modulating the latent cognitive processes of decision making, in particular the exploration-exploitation tradeoff. Revealing how each neuromodulator contributes to this explore-exploit process will be important in guiding mechanistic hypotheses emerging from computational psychiatric approaches. To understand the differences and overlaps of the roles of these two catecholamine systems in regulating exploration and exploitation, a direct comparison using the same dynamic decision making task is needed. Here, we ran mice in a restless two-armed bandit task, which encourages both exploration and exploitation. We systemically administered a nonselective DA receptor antagonist (flupenthixol), a nonselective DA receptor agonist (apomorphine), a NE beta-receptor antagonist (propranolol), and a NE beta-receptor agonist (isoproterenol), and examined changes in exploration within subjects across sessions. We found a bidirectional modulatory effect of dopamine receptor activity on the level of exploration. Increasing dopamine activity decreased exploration and decreasing dopamine activity increased exploration. Beta-noradrenergic receptor activity also modulated exploration, but the modulatory effect was mediated by sex. Reinforcement learning model parameters suggested that dopamine modulation affected exploration via decision noise and norepinephrine modulation affected exploration via outcome sensitivity. Together, these findings suggested that the mechanisms that govern the transition between exploration and exploitation are sensitive to changes in both catecholamine functions and revealed differential roles for NE and DA in mediating exploration.

Pupillary response is associated with the reset and switching of functional brain networks during salience processing

The interface between processing internal goals and salient events in the environment involves various top-down processes. Previous studies have identified multiple brain areas for salience processing, including the salience network (SN), dorsal attention network, and the locus coeruleus-norepinephrine (LC-NE) system. However, interactions among these systems in salience processing remain unclear. Here, we simultaneously recorded pupillometry, EEG, and fMRI during an auditory oddball paradigm. The analyses of EEG and fMRI data uncovered spatiotemporally organized target-associated neural correlates. By modeling the target-modulated effective connectivity, we found that the target-evoked pupillary response is associated with the network directional couplings from late to early subsystems in the trial, as well as the network switching initiated by the SN. These findings indicate that the SN might cooperate with the pupil-indexed LC-NE system in the reset and switching of cortical networks, and shed light on their implications in various cognitive processes and neurological diseases.

l-DOPA and oxytocin influence the neurocomputational mechanisms of self-benefitting and prosocial reinforcement learning

Humans learn through reinforcement, particularly when outcomes are unexpected. Recent research suggests similar mechanisms drive how we learn to benefit other people, that is, how we learn to be prosocial. Yet the neurochemical mechanisms underlying such prosocial computations remain poorly understood. Here, we investigated whether pharmacological manipulation of oxytocin and dopamine influence the neurocomputational mechanisms underlying self-benefitting and prosocial reinforcement learning. Using a double-blind placebo-controlled cross-over design, we administered intranasal oxytocin (24 IU), dopamine precursor l-DOPA (100 mg + 25 mg carbidopa), or placebo over three sessions. Participants performed a probabilistic reinforcement learning task with potential rewards for themselves, another participant, or no one, during functional magnetic resonance imaging. Computational models of reinforcement learning were used to calculate prediction errors (PEs) and learning rates. Participants behavior was best explained by a model with different learning rates for each recipient, but these were unaffected by either drug. On the neural level, however, both drugs blunted PE signaling in the ventral striatum and led to negative signaling of PEs in the anterior mid-cingulate cortex, dorsolateral prefrontal cortex, inferior parietal gyrus, and precentral gyrus, compared to placebo, and regardless of recipient. Oxytocin (versus placebo) administration was additionally associated with opposing tracking of self-benefitting versus prosocial PEs in dorsal anterior cingulate cortex, insula and superior temporal gyrus. These findings suggest that both l-DOPA and oxytocin induce a context-independent shift from positive towards negative tracking of PEs during learning. Moreover, oxytocin may have opposing effects on PE signaling when learning to benefit oneself versus another.

Maladaptive explore/exploit trade-offs in schizophrenia

Schizophrenia is a complex disorder that remains poorly understood, particularly at the systems level. In this opinion article we argue that the explore/exploit trade-off concept provides a holistic and ecologically valid framework to resolve some of the apparent paradoxes that have emerged within schizophrenia research. We review recent evidence suggesting that fundamental explore/exploit behaviors may be maladaptive in schizophrenia during physical, visual, and cognitive foraging. We also describe how theories from the broader optimal foraging literature, such as the marginal value theorem (MVT), could provide valuable insight into how aberrant processing of reward, context, and cost/effort evaluations interact to produce maladaptive responses.

Recurrent networks endowed with structural priors explain suboptimal animal behavior

The strategies found by animals facing a new task are determined both by individual experience and by structural priors evolved to leverage the statistics of natural environments. Rats quickly learn to capitalize on the trial sequence correlations of two-alternative forced choice (2AFC) tasks after correct trials but consistently deviate from optimal behavior after error trials. To understand this outcome-dependent gating, we first show that recurrent neural networks (RNNs) trained in the same 2AFC task outperform rats as they can readily learn to use across-trial information both after correct and error trials. We hypothesize that, although RNNs can optimize their behavior in the 2AFC task without any a priori restrictions, rats' strategy is constrained by a structural prior adapted to a natural environment in which rewarded and non-rewarded actions provide largely asymmetric information. When pre-training RNNs in a more ecological task with more than two possible choices, networks develop a strategy by which they gate off the across-trial evidence after errors, mimicking rats' behavior. Population analyses show that the pre-trained networks form an accurate representation of the sequence statistics independently of the outcome in the previous trial. After error trials, gating is implemented by a change in the network dynamics that temporarily decouple the categorization of the stimulus from the across-trial accumulated evidence. Our results suggest that the rats' suboptimal behavior reflects the influence of a structural prior that reacts to errors by isolating the network decision dynamics from the context, ultimately constraining the performance in a 2AFC laboratory task.

Pallidal neuromodulation of the explore/exploit trade-off in decision-making

Every decision that we make involves a conflict between exploiting our current knowledge of an action's value or exploring alternative courses of action that might lead to a better, or worse outcome. The sub-cortical nuclei that make up the basal ganglia have been proposed as a neural circuit that may contribute to resolving this explore-exploit 'dilemma'. To test this hypothesis, we examined the effects of neuromodulating the basal ganglia's output nucleus, the globus pallidus interna, in patients who had undergone deep brain stimulation (DBS) for isolated dystonia. Neuromodulation enhanced the number of exploratory choices to the lower value option in a two-armed bandit probabilistic reversal-learning task. Enhanced exploration was explained by a reduction in the rate of evidence accumulation (drift rate) in a reinforcement learning drift diffusion model. We estimated the functional connectivity profile between the stimulating DBS electrode and the rest of the brain using a normative functional connectome derived from heathy controls. Variation in the extent of neuromodulation induced exploration between patients was associated with functional connectivity from the stimulation electrode site to a distributed brain functional network. We conclude that the basal ganglia's output nucleus, the globus pallidus interna, can adaptively modify decision choice when faced with the dilemma to explore or exploit.

Disentangling the roles of dopamine and noradrenaline in the exploration-exploitation tradeoff during human decision-making

Balancing the exploration of new options and the exploitation of known options is a fundamental challenge in decision-making, yet the mechanisms involved in this balance are not fully understood. Here, we aimed to elucidate the distinct roles of dopamine and noradrenaline in the exploration-exploitation tradeoff during human choice. To this end, we used a double-blind, placebo-controlled design in which participants received either a placebo, 400 mg of the D2/D3 receptor antagonist amisulpride, or 40 mg of the β-adrenergic receptor antagonist propranolol before they completed a virtual patch-foraging task probing exploration and exploitation. We systematically varied the rewards associated with choice options, the rate by which rewards decreased over time, and the opportunity costs it took to switch to the next option to disentangle the contributions of dopamine and noradrenaline to specific choice aspects. Our data show that amisulpride increased the sensitivity to all of these three critical choice features, whereas propranolol was associated with a reduced tendency to use value information. Our findings provide novel insights into the specific roles of dopamine and noradrenaline in the regulation of human choice behavior, suggesting a critical involvement of dopamine in directed exploration and a role of noradrenaline in more random exploration.

The catecholamine precursor Tyrosine reduces autonomic arousal and decreases decision thresholds in reinforcement learning and temporal discounting

Supplementation with the catecholamine precursor L-Tyrosine might enhance cognitive performance, but overall findings are mixed. Here, we investigate the effect of a single dose of tyrosine (2g) vs. placebo on two catecholamine-dependent trans-diagnostic traits: model-based control during reinforcement learning (2-step task) and temporal discounting, using a double-blind, placebo-controlled, within-subject design (n = 28 healthy male participants). We leveraged drift diffusion models in a hierarchical Bayesian framework to jointly model participants' choices and response times (RTS) in both tasks. Furthermore, comprehensive autonomic monitoring (heart rate, heart rate variability, pupillometry, spontaneous eye blink rate) was performed both pre- and post-supplementation, to explore potential physiological effects of supplementation. Across tasks, tyrosine consistently reduced participants' RTs without deteriorating task-performance. Diffusion modeling linked this effect to attenuated decision-thresholds in both tasks and further revealed increased model-based control (2-step task) and (if anything) attenuated temporal discounting. On the physiological level, participants' pupil dilation was predictive of the individual degree of temporal discounting. Tyrosine supplementation reduced physiological arousal as revealed by increases in pupil dilation variability and reductions in heart rate. Supplementation-related changes in physiological arousal predicted individual changes in temporal discounting. Our findings provide first evidence that tyrosine supplementation might impact psychophysiological parameters, and suggest that modeling approaches based on sequential sampling models can yield novel insights into latent cognitive processes modulated by amino-acid supplementation.

Striatal D1 and D2 receptor availability are selectively associated with eye-blink rates after methylphenidate treatment

Eye-blink rate has been proposed as a biomarker of the brain dopamine system, however, findings have not been consistent. This study assessed the relationship between blink rates, measured after oral placebo) (PL) and after a challenge with oral methylphenidate (MP; 60 mg) and striatal D1 receptor (D1R) (measured at baseline) and D2 receptor (D2R) availability (measured after PL and after MP) in healthy participants. PET measures of baseline D1R ([¹¹C]NNC112) (BL-D1R) and D2R availability ([¹¹C]raclopride) after PL (PL-D2R) and after MP (MP-D2R) were quantified in the striatum as non-displaceable binding potential. MP reduced the number of blinks and increased the time participants kept their eyes open. Correlations with dopamine receptors were only significant for the eye blink measures obtained after MP; being positive for BL-D1R in putamen and MP-D2R in caudate (PL-D2R were not significant). MP-induced changes in blink rates (PL minus MP) were negatively correlated with BL-D1R in caudate and putamen. Our findings suggest that eye blink measures obtained while stressing the dopamine system might provide a more sensitive behavioral biomarker of striatal D1R or D2R in healthy volunteers than that obtained at baseline or after placebo.

PET imaging in human participants revealed that D1 and D2 dopamine receptor availability was associated with eye-blink rates following treatment with oral methylphenidate, but not a placebo.

Different brain systems support learning from received and avoided pain during human pain-avoidance learning

Both unexpected pain and unexpected pain absence can drive avoidance learning, but whether they do so via shared or separate neural and neurochemical systems is largely unknown. To address this issue, we combined an instrumental pain-avoidance learning task with computational modeling, functional magnetic resonance imaging (fMRI), and pharmacological manipulations of the dopaminergic (100 mg levodopa) and opioidergic (50 mg naltrexone) systems (N = 83). Computational modeling provided evidence that untreated participants learned more from received than avoided pain. Our dopamine and opioid manipulations negated this learning asymmetry by selectively increasing learning rates for avoided pain. Furthermore, our fMRI analyses revealed that pain prediction errors were encoded in subcortical and limbic brain regions, whereas no-pain prediction errors were encoded in frontal and parietal cortical regions. However, we found no effects of our pharmacological manipulations on the neural encoding of prediction errors. Together, our results suggest that human pain-avoidance learning is supported by separate threat- and safety-learning systems, and that dopamine and endogenous opioids specifically regulate learning from successfully avoided pain.

Motor response vigour and visual fixation patterns reflect subjective valuation during intertemporal choice

Value-based decision-making is of central interest in cognitive neuroscience and psychology, as well as in the context of neuropsychiatric disorders characterised by decision-making impairments. Studies examining (neuro-)computational mechanisms underlying choice behaviour typically focus on participants’ decisions. However, there is increasing evidence that option valuation might also be reflected in motor response vigour and eye movements, implicit measures of subjective utility. To examine motor response vigour and visual fixation correlates of option valuation in intertemporal choice, we set up a task where the participants selected an option by pressing a grip force transducer, simultaneously tracking fixation shifts between options. As outlined in our preregistration (https://osf.io/k6jct), we used hierarchical Bayesian parameter estimation to model the choices assuming hyperbolic discounting, compared variants of the softmax and drift diffusion model, and assessed the relationship between response vigour and the estimated model parameters. The behavioural data were best explained by a drift diffusion model specifying a non-linear scaling of the drift rate by the subjective value differences. Replicating previous findings, we found a magnitude effect for temporal discounting, such that higher rewards were discounted less. This magnitude effect was further reflected in motor response vigour, such that stronger forces were exerted in the high vs. the low magnitude condition. Bayesian hierarchical linear regression further revealed higher grip forces, faster response times and a lower number of fixation shifts for trials with higher subjective value differences. An exploratory analysis revealed that subjective value sums across options showed an even more pronounced association with trial-wise grip force amplitudes. Our data suggest that subjective utility or implicit valuation is reflected in motor response vigour and visual fixation patterns during intertemporal choice. Taking into account response vigour might thus provide deeper insight into decision-making, reward valuation and maladaptive changes in these processes, e.g. in the context of neuropsychiatric disorders.

Decision neuroscience and neuroeconomics: Recent progress and ongoing challenges

In the past decade, decision neuroscience and neuroeconomics have developed many new insights in the study of decision making. This review provides an overarching update on how the field has advanced in this time period. Although our initial review a decade ago outlined several theoretical, conceptual, methodological, empirical, and practical challenges, there has only been limited progress in resolving these challenges. We summarize significant trends in decision neuroscience through the lens of the challenges outlined for the field and review examples where the field has had significant, direct, and applicable impacts across economics and psychology. First, we review progress on topics including reward learning, explore-exploit decisions, risk and ambiguity, intertemporal choice, and valuation. Next, we assess the impacts of emotion, social rewards, and social context on decision making. Then, we follow up with how individual differences impact choices and new exciting developments in the prediction and neuroforecasting of future decisions. Finally, we consider how trends in decision-neuroscience research reflect progress toward resolving past challenges, discuss new and exciting applications of recent research, and identify new challenges for the field. This article is categorized under: Psychology > Reasoning and Decision Making Psychology > Emotion and Motivation.

Effects of theta transcranial alternating current stimulation (tACS) on exploration and exploitation during uncertain decision-making

Exploring ones surroundings may yield unexpected rewards, but is associated with uncertainty and risk. Alternatively, exploitation of certain outcomes is related to low risk, yet potentially better outcomes remain unexamined. As such, risk-taking behavior depends on perceived uncertainty and a trade-off between exploration-exploitation. Previously, it has been suggested that risk-taking may relate to theta activity in the prefrontal cortex. Furthermore, previous studies hinted at a relationship between a right-hemispheric bias in frontal theta asymmetry and risky behavior. In the present double-blind sham-controlled within-subject study, we applied bifrontal transcranial alternating current stimulation (tACS) at the theta frequency (5 Hz) on eighteen healthy volunteers during a gambling task. Two tACS montages with either left-right or posterior-anterior current flow were employed at an intensity of 1 mA. Results showed that, compared to sham, theta tACS increased perceived uncertainty irrespective of current flow direction. Despite this observation, no direct effect of tACS on exploration behavior and general risk-taking was observed. Furthermore, frontal theta asymmetry was more right-hemispherically biased after posterior-anterior tACS, compared to sham. Finally, we used electric field simulation to identify which regions were targeted by the tACS montages as an overlap in regions may explain why the two montages resulted in comparable outcomes. Our findings provide a first step towards understanding the relationship between frontal theta oscillations and different features of risk-taking.

A neural and behavioral trade-off between value and uncertainty underlies exploratory decisions in normative anxiety

Exploration reduces uncertainty about the environment and improves the quality of future decisions, but at the cost of provisional uncertain and suboptimal outcomes. Although anxiety promotes intolerance to uncertainty, it remains unclear whether and by which mechanisms anxiety relates to exploratory decision-making. We use a dynamic three-armed-bandit task and find that higher trait-anxiety is associated with increased exploration, which in turn harms overall performance. We identify two distinct behavioral sources: first, decisions made by anxious individuals are guided toward reduction of uncertainty; and second, decisions are less guided by immediate value gains. These findings are similar in both loss and gain domains, and further demonstrate that an affective trait relates to exploration and results in an inverse-U-shaped relationship between anxiety and overall performance. Additional imaging data (fMRI) suggests that normative anxiety correlates negatively with the representation of expected-value in the dorsal-anterior-cingulate-cortex, and in contrast, positively with the representation of uncertainty in the anterior-insula. We conclude that a trade-off between value-gains and uncertainty-reduction entails maladaptive decision-making in individuals with higher normal-range anxiety.

A bias toward the unknown: individual and environmental factors influencing exploratory behavior

With limited resources, exploring new opportunities is crucial for survival. Exploring novel options, however, comes at the cost of uncertainty. Therefore, there is a trade‐off between exploiting options with a known beneficial outcome and exploring novel options with a potentially higher gain. Computational models have suggested that novelty may promote exploratory behavior by inducing a so‐called novelty bonus through reward‐related processes. So far, few studies have provided behavioral evidence for such a novelty bonus. In this study, we aimed to investigate whether spatial novelty can stimulate exploratory behavior (Experiment 1), and whether age, novelty‐seeking, and reduced action radius or social interactions due to COVID‐19 restrictions influenced the exploration–exploitation trade‐off (Experiment 2). In both experiments, we employed a novel paradigm in which participants made binary decisions between food items, while on rare trials, a surprise option was presented. Results from Experiment 1 are in line with a novelty bonus, with spatial novelty promoting exploratory behavior. In Experiment 2, we found that exploratory behavior declined with age, high novelty seekers made more exploratory choices than low novelty seekers, and participants with a smaller action radius made fewer exploratory choices. These findings are consistent with previous findings in animals and predictions from computational models.

Computational analysis of probabilistic reversal learning deficits in male subjects with alcohol use disorder

BACKGROUND

Alcohol use disorder is characterized by perseverative alcohol use despite negative consequences. This hallmark feature of addiction potentially relates to impairments in behavioral flexibility, which can be measured by probabilistic reversal learning (PRL) paradigms. We here aimed to examine the cognitive mechanisms underlying impaired PRL task performance in patients with alcohol use disorder (AUDP) using computational models of reinforcement learning.

METHODS

Twenty-eight early abstinent AUDP and 27 healthy controls (HC) performed an extensive PRL paradigm. We compared conventional behavioral variables of choices (perseveration; correct responses) between groups. Moreover, we fitted Bayesian computational models to the task data to compare differences in latent cognitive variables including reward and punishment learning and choice consistency between groups.

RESULTS

AUDP and HC did not significantly differ with regard to direct perseveration rates after reversals. However, AUDP made overall less correct responses and specifically showed decreased win-stay behavior compared to HC. Interestingly, AUDP showed premature switching after no or little negative feedback but elevated proneness to stay when accumulation of negative feedback would make switching a more optimal option. Computational modeling revealed that AUDP compared to HC showed enhanced learning from punishment, a tendency to learn less from positive feedback and lower choice consistency.

CONCLUSION

Our data do not support the assumption that AUDP are characterized by increased perseveration behavior. Instead our findings provide evidence that enhanced negative reinforcement and decreased non-drug-related reward learning as well as diminished choice consistency underlie dysfunctional choice behavior in AUDP.

Dynamic decision policy reconfiguration under outcome uncertainty

In uncertain or unstable environments, sometimes the best decision is to change your mind. To shed light on this flexibility, we evaluated how the underlying decision policy adapts when the most rewarding action changes. Human participants performed a dynamic two-armed bandit task that manipulated the certainty in relative reward (conflict) and the reliability of action-outcomes (volatility). Continuous estimates of conflict and volatility contributed to shifts in exploratory states by changing both the rate of evidence accumulation (drift rate) and the amount of evidence needed to make a decision (boundary height), respectively. At the trialwise level, following a switch in the optimal choice, the drift rate plummets and the boundary height weakly spikes, leading to a slow exploratory state. We find that the drift rate drives most of this response, with an unreliable contribution of boundary height across experiments. Surprisingly, we find no evidence that pupillary responses associated with decision policy changes. We conclude that humans show a stereotypical shift in their decision policies in response to environmental changes.

From exploration to exploitation: a shifting mental mode in late life development

Changes in cognition, affect, and brain function combine to promote a shift in the nature of mentation in older adulthood, favoring exploitation of prior knowledge over exploratory search as the starting point for thought and action. Age-related exploitation biases result from the accumulation of prior knowledge, reduced cognitive control, and a shift toward affective goals. These are accompanied by changes in cortical networks, as well as attention and reward circuits. By incorporating these factors into a unified account, the exploration-to-exploitation shift offers an integrative model of cognitive, affective, and brain aging. Here, we review evidence for this model, identify determinants and consequences, and survey the challenges and opportunities posed by an exploitation-biased mental mode in later life.

Better living through understanding the insula: Why subregions can make all the difference

Insula function is considered critical for many motivated behaviors, with proposed functions ranging from attention, behavioral control, emotional regulation, goal-directed and aversion-resistant responding. Further, the insula is implicated in many neuropsychiatric conditions including substance abuse. More recently, multiple insula subregions have been distinguished based on anatomy, connectivity, and functional contributions. Generally, posterior insula is thought to encode more somatosensory inputs, which integrate with limbic/emotional information in middle insula, that in turn integrate with cognitive processes in anterior insula. Together, these regions provide rapid interoceptive information about the current or predicted situation, facilitating autonomic recruitment and quick, flexible action. Here, we seek to create a robust foundation from which to understand potential subregion differences, and provide direction for future studies. We address subregion differences across humans and rodents, so that the latter's mechanistic interventions can best mesh with clinical relevance of human conditions. We first consider the insula's suggested roles in humans, then compare subregional studies, and finally describe rodent work. One primary goal is to encourage precision in describing insula subregions, since imprecision (e.g. including both posterior and anterior studies when describing insula work) does a disservice to a larger understanding of insula contributions. Additionally, we note that specific task details can greatly impact recruitment of various subregions, requiring care and nuance in design and interpretation of studies. Nonetheless, the central ethological importance of the insula makes continued research to uncover mechanistic, mood, and behavioral contributions of paramount importance and interest. This article is part of the special Issue on 'Neurocircuitry Modulating Drug and Alcohol Abuse'.

To learn or to gain: neural signatures of exploration in human decision-making

Individuals not only take actions to obtain immediate rewards but also to gain more information to guide future choices. An ideal exploration-exploitation balance is crucial for maximizing reward over the long run. However, the neural signatures of exploration in humans remain unclear. Using quantitative meta-analyses of functional magnetic resonance imaging experiments on exploratory behaviors, we sought to identify the concordant activity pertaining to exploration over a range of experiments. The results revealed that exploration activates concordant brain activity associated with risk (e.g., dorsal medial prefrontal cortex and anterior insula), cognitive control (e.g., dorsolateral prefrontal cortex and inferior frontal gyrus), and motor processing (e.g., premotor cortex). These stereotaxic maps of exploration may indicate that exploration is highly linked to risk processing, but is also specifically associated with regions involved in executive control processes. Although this explanation should be treated as exploratory, these findings support theories positing an important role for the prefrontal-insular-motor cortical network in exploration.