Brain mechanism of foraging: Reward-dependent synaptic plasticity versus neural integration of values

During foraging behavior, action values are persistently encoded in neural activity and updated depending on the history of choice outcomes. What is the neural mechanism for action value maintenance and updating? Here, we explore two contrasting network models: synaptic learning of action value versus neural integration. We show that both models can reproduce extant experimental data, but they yield distinct predictions about the underlying biological neural circuits. In particular, the neural integrator model but not the synaptic model requires that reward signals are mediated by neural pools selective for action alternatives and their projections are aligned with linear attractor axes in the valuation system. We demonstrate experimentally observable neural dynamical signatures and feasible perturbations to differentiate the two contrasting scenarios, suggesting that the synaptic model is a more robust candidate mechanism. Overall, this work provides a modeling framework to guide future experimental research on probabilistic foraging.

Acetylcholine and noradrenaline enhance foraging optimality in humans

Foraging theory prescribes when optimal foragers should leave the current option for more rewarding alternatives. Actual foragers often exploit options longer than prescribed by the theory, but it is unclear how this foraging suboptimality arises. We investigated whether the upregulation of cholinergic, noradrenergic, and dopaminergic systems increases foraging optimality. In a double-blind, between-subject design, participants (N = 160) received placebo, the nicotinic acetylcholine receptor agonist nicotine, a noradrenaline reuptake inhibitor reboxetine, or a preferential dopamine reuptake inhibitor methylphenidate, and played the role of a farmer who collected milk from patches with different yield. Across all groups, participants on average overharvested. While methylphenidate had no effects on this bias, nicotine, and to some extent also reboxetine, significantly reduced deviation from foraging optimality, which resulted in better performance compared to placebo. Concurring with amplified goal-directedness and excluding heuristic explanations, nicotine independently also improved trial initiation and time perception. Our findings elucidate the neurochemical basis of behavioral flexibility and decision optimality and open unique perspectives on psychiatric disorders affecting these functions.

Reduced loss aversion in value-based decision-making and edge-centric functional connectivity in patients with internet gaming disorder

Background and aims

Impaired value-based decision-making is a feature of substance and behavioral addictions. Loss aversion is a core of value-based decision-making and its alteration plays an important role in addiction. However, few studies explored it in internet gaming disorder patients (IGD).

Methods

In this study, IGD patients (PIGD) and healthy controls (Con-PIGD) performed the Iowa gambling task (IGT), under functional magnetic resonance imaging (fMRI). We investigated group differences in loss aversion, brain functional networks of node-centric functional connectivity (nFC) and the overlapping community features of edge-centric functional connectivity (eFC) in IGT.

Results

PIGD performed worse with lower average net score in IGT. The computational model results showed that PIGD significantly reduced loss aversion. There was no group difference in nFC. However, there were significant group differences in the overlapping community features of eFC1. Furthermore, in Con-PIGD, loss aversion was positively correlated with the edge community profile similarity of the edge2 between left IFG and right hippocampus at right caudate. This relationship was suppressed by response consistency3 in PIGD. In addition, reduced loss aversion was negatively correlated with the promoted bottom-to-up neuromodulation from the right hippocampus to the left IFG in PIGD.

Discussion and conclusions

The reduced loss aversion in value-based decision making and their related edge-centric functional connectivity support that the IGD showed the same value-based decision-making deficit as the substance use and other behavioral addictive disorders. These findings may have important significance for understanding the definition and mechanism of IGD in the future.

Attempts to Influence the Value of Alcohol by Manipulating Social Influence and Context

Background: Recent cognitive neuroscience models of value-based decision-making suggest value-based choices for alcohol are sensitive to various inputs, such as context and social influence. In two online experiments, we tested whether manipulating these inputs influenced proxies for alcohol value. Experiment 1: 157 social drinkers were presented with 4 hypothetical scenarios (drinking alone, with friends who are also drinking, with friends but trying to "cut-down" for health reasons, with friends who aren't drinking) in a within-subjects design, and completed the Brief Assessment of Alcohol Demand after each as a measure of value. Value for alcohol (number of drinks purchased) was greatest when drinking with friends who were also drinking compared to drinking alone (d = 0.95), friends not drinking (d = 1.49) and friends drinking/health related (d = 1.59). Value for alcohol was also greater when drinking alone compared to with friends who were not drinking (d = 0.55), and also with friends drinking/health related (d = 0.62). Experiment 2: 241 participants were randomly allocated to see one of four categories of images in a 2 (context: bar vs house) x 2 (social influence: enjoy vs not enjoy) design, before completing a Concurrent Choice Task for alcohol and Visual Analog Scales. There were no significant effects found on either task, both taken as proxies for value. Conclusion: There was inconclusive evidence that the value for alcohol could be manipulated by social context. This could be explained by greater saliency of the manipulation in asking participants to imagine themselves in a hypothetical situation as opposed to presenting images depicting drinking scenarios.

The role of reinforcement learning and value-based decision-making frameworks in understanding food choice and eating behaviors

The obesogenic food environment includes easy access to highly-palatable, energy-dense, "ultra-processed" foods that are heavily marketed to consumers; therefore, it is critical to understand the neurocognitive processes the underlie overeating in response to environmental food-cues (e.g., food images, food branding/advertisements). Eating habits are learned through reinforcement, which is the process through which environmental food cues become valued and influence behavior. This process is supported by multiple behavioral control systems (e.g., Pavlovian, Habitual, Goal-Directed). Therefore, using neurocognitive frameworks for reinforcement learning and value-based decision-making can improve our understanding of food-choice and eating behaviors. Specifically, the role of reinforcement learning in eating behaviors was considered using the frameworks of (1) Sign-versus Goal-Tracking Phenotypes; (2) Model-Free versus Model-Based; and (3) the Utility or Value-Based Model. The sign-and goal-tracking phenotypes may contribute a mechanistic insight on the role of food-cue incentive salience in two prevailing models of overconsumption-the Extended Behavioral Susceptibility Theory and the Reactivity to Embedded Food Cues in Advertising Model. Similarly, the model-free versus model-based framework may contribute insight to the Extended Behavioral Susceptibility Theory and the Healthy Food Promotion Model. Finally, the value-based model provides a framework for understanding how all three learning systems are integrated to influence food choice. Together, these frameworks can provide mechanistic insight to existing models of food choice and overconsumption and may contribute to the development of future prevention and treatment efforts.

Fast and slow contributions to decision-making in corticostriatal circuits

We make complex decisions using both fast judgments and slower, more deliberative reasoning. For example, during value-based decision-making, animals make rapid value-guided orienting eye movements after stimulus presentation that bias the upcoming decision. The neural mechanisms underlying these processes remain unclear. To address this, we recorded from the caudate nucleus and orbitofrontal cortex while animals made value-guided decisions. Using population-level decoding, we found a rapid, phasic signal in caudate that predicted the choice response and closely aligned with animals' initial orienting eye movements. In contrast, the dynamics in orbitofrontal cortex were more consistent with a deliberative system serially representing the value of each available option. The phasic caudate value signal and the deliberative orbitofrontal value signal were largely independent from each other, consistent with value-guided orienting and value-guided decision-making being independent processes.

Gustatory Cortex Is Involved in Evidence Accumulation during Food Choice

Food choice is one of the most fundamental and most frequent value-based decisions for all animals including humans. However, the neural circuitry involved in food-based decisions is only recently being addressed. Given the relatively fast dynamics of decision formation, electroencephalography (EEG)-informed fMRI analysis is highly beneficial for localizing this circuitry in humans. Here, by using the EEG correlates of evidence accumulation in a simultaneously recorded EEG-fMRI dataset, we found a significant role for the right temporal-parietal operculum (PO) and medial insula including gustatory cortex (GC) in binary choice between food items. These activations were uncovered by using the “EEG energy” (power 2 of EEG) as the BOLD regressor and were missed if conventional analysis with the EEG signal itself were to be used, in agreement with theoretical predictions for EEG and BOLD relations. No significant positive correlations were found with higher powers of EEG (powers 3 or 4) pointing to specificity and sufficiency of EEG energy as the main correlate of the BOLD response. This finding extends the role of cortical areas traditionally involved in palatability processing to value-based decision-making and offers the “EEG energy” as a key regressor of BOLD response in simultaneous EEG-fMRI designs.

Cognitive Effort Exertion Enhances Electrophysiological Responses to Rewarding Outcomes

Recent work has highlighted neural mechanisms underlying cognitive effort-related discounting of anticipated rewards. However, findings on whether effort exertion alters the subjective value of obtained rewards are inconsistent. Here, we provide a more nuanced account of how cognitive effort affects subsequent reward processing in a novel task designed to assess effort-induced modulations of the Reward Positivity, an event-related potential indexing reward-related neural activity. We found that neural responses to both gains and losses were significantly elevated in trials requiring more versus less cognitive effort. Moreover, time-frequency analysis revealed that these effects were mirrored in gain-related delta, but not in loss-related theta band activity, suggesting that people ascribed more value to high-effort outcomes. In addition, we also explored whether individual differences in behavioral effort discounting rates and reward sensitivity in the absence of effort may affect the relationship between effort exertion and subsequent reward processing. Together, our findings provide evidence that cognitive effort exertion can increase the subjective value of subsequent outcomes and that this effect may primarily rely on modulations of delta band activity.

No Differences in Value-Based Decision-Making Due to Use of Oral Contraceptives

Fluctuating ovarian hormones have been shown to affect decision-making processes in women. While emerging evidence suggests effects of endogenous ovarian hormones such as estradiol and progesterone on value-based decision-making in women, the impact of exogenous synthetic hormones, as in most oral contraceptives, is not clear. In a between-subjects design, we assessed measures of value-based decision-making in three groups of women aged 18 to 29 years, during (1) active oral contraceptive intake (N = 22), (2) the early follicular phase of the natural menstrual cycle (N = 20), and (3) the periovulatory phase of the natural menstrual cycle (N = 20). Estradiol, progesterone, testosterone, and sex-hormone binding globulin levels were assessed in all groups via blood samples. We used a test battery which measured different facets of value-based decision-making: delay discounting, risk-aversion, risk-seeking, and loss aversion. While hormonal levels did show the expected patterns for the three groups, there were no differences in value-based decision-making parameters. Consequently, Bayes factors showed conclusive evidence in support of the null hypothesis. We conclude that women on oral contraceptives show no differences in value-based decision-making compared to the early follicular and periovulatory natural menstrual cycle phases.

Neural responses reveal associations between personal values and value-based decisions

Personal values are thought to modulate value-based decisions, but the neural mechanisms underlying this influence remain unclear. Using a Lottery Choice Task functional brain imaging experiment, we examined the associations between personal value for hedonism and security (based on the Schwartz Value Survey) and subjective neurocognitive processing of reward and loss probability and magnitude objectively coded in stimuli. Hedonistic individuals accepted more losing stakes and showed increased right dorsolateral prefrontal and striatal and left parietal responses with increasing probability of losing. Individuals prioritizing security rejected more stakes and showed reduced right inferior frontal and amygdala responses with increasing stake magnitude, but increased precuneus responses for high-magnitude high-winning probability. With higher hedonism, task-related functional connectivity with the whole brain was higher in right insula and lower in bilateral habenula. For those with higher security ratings, whole-brain functional connectivity was higher in bilateral insula, supplementary motor areas, right superior frontal gyrus, dorsal anterior cingulate cortex, and lower in right middle occipital gyrus. These findings highlight distinct neural engagement across brain systems involved in reward and affective processing, and cognitive control that subserves how individual differences in personal value for gaining rewards or maintaining status quo modulate value-based decisions

Action in auctions: neural and computational mechanisms of bidding behaviour

Competition for resources is a fundamental characteristic of evolution. Auctions have been widely used to model competition of individuals for resources, and bidding behaviour plays a major role in social competition. Yet, how humans learn to bid efficiently remains an open question. We used model‐based neuroimaging to investigate the neural mechanisms of bidding behaviour under different types of competition. Twenty‐seven subjects (nine male) played a prototypical bidding game: a double action, with three “market” types, which differed in the number of competitors. We compared different computational learning models of bidding: directional learning models (DL), where the model bid is “nudged” depending on whether it was accepted or rejected, along with standard reinforcement learning models (RL). We found that DL fit the behaviour best and resulted in higher payoffs. We found the binary learning signal associated with DL to be represented by neural activity in the striatum distinctly posterior to a weaker reward prediction error signal. We posited that DL is an efficient heuristic for valuation when the action (bid) space is continuous. Indeed, we found that the posterior parietal cortex represents the continuous action space of the task, and the frontopolar prefrontal cortex distinguishes among conditions of social competition. Based on our findings, we proposed a conceptual model that accounts for a sequence of processes that are required to perform successful and flexible bidding under different types of competition.

Neural structure mapping in human probabilistic reward learning

Humans can learn abstract concepts that describe invariances over relational patterns in data. One such concept, known as magnitude, allows stimuli to be compactly represented on a single dimension (i.e. on a mental line). Here, we measured representations of magnitude in humans by recording neural signals whilst they viewed symbolic numbers. During a subsequent reward-guided learning task, the neural patterns elicited by novel complex visual images reflected their payout probability in a way that suggested they were encoded onto the same mental number line, with 'bad' bandits sharing neural representation with 'small' numbers and 'good' bandits with 'large' numbers. Using neural network simulations, we provide a mechanistic model that explains our findings and shows how structural alignment can promote transfer learning. Our findings suggest that in humans, learning about reward probability is accompanied by structural alignment of value representations with neural codes for the abstract concept of magnitude.

Transcranial Stimulation Over the Dorsolateral Prefrontal Cortex Increases the Impact of Past Expenses on Decision-Making

Goal-directed choices should be guided by the expected value of the available options. However, people are often influenced by past costs in their decisions, thus succumbing to a bias known as the "sunk-cost effect." Recent functional magnetic resonance imaging data show that the sunk-cost effect is associated with increased activity in dorsolateral prefrontal cortex (dlPFC) and altered crosstalk of the dlPFC with other prefrontal areas. Are these correlated neural processes causally involved in the sunk-cost effect? Here, we employed transcranial direct current stimulation (tDCS) to examine the role of the dlPFC for biasing choices in line with the cost of past expenses. Specifically, we applied different types of tDCS over the right dlPFC while participants performed an investment task designed to assess the impact of past investments on current choices. Our results show a pronounced sunk-cost effect that was significantly increased by anodal tDCS, but left unaltered by cathodal or sham stimulation. Importantly, choices were not affected by stimulation when no prior investments had been made, underlining the specificity of the obtained effect. Our findings suggest a critical role of the dlPFC in the sunk-cost effect and thus elucidate neural mechanisms by which past investments may influence current decision-making.

Visualization of Global Disease Burden for the Optimization of Patient Management and Treatment

Background

The assessment and treatment of complex disorders is challenged by the multiple domains and instruments used to evaluate clinical outcome. With the large number of assessment tools typically used in complex disorders comes the challenge of obtaining an integrative view of disease status to further evaluate treatment outcome both at the individual level and at the group level. Radar plots appear as an attractive visual tool to display multivariate data on a two-dimensional graphical illustration. Here, we describe the use of radar plots for the visualization of disease characteristics applied in the context of tinnitus, a complex and heterogeneous condition, the treatment of which has shown mixed success.

Methods

Data from two different cohorts, the Swedish Tinnitus Outreach Project (STOP) and the Tinnitus Research Initiative (TRI) database, were used. STOP is a population-based cohort where cross-sectional data from 1,223 non-tinnitus and 933 tinnitus subjects were analyzed. By contrast, the TRI contained data from 571 patients who underwent various treatments and whose Clinical Global Impression (CGI) score was accessible to infer treatment outcome. In the latter, 34,560 permutations were tested to evaluate whether a particular ordering of the instruments could reflect better the treatment outcome measured with the CGI.

Results

Radar plots confirmed that tinnitus subtypes such as occasional and chronic tinnitus from the STOP cohort could be strikingly different, and helped appreciate a gender bias in tinnitus severity. Radar plots with greater surface areas were consistent with greater burden, and enabled a rapid appreciation of the global distress associated with tinnitus in patients categorized according to tinnitus severity. Permutations in the arrangement of instruments allowed to identify a configuration with minimal variance and maximized surface difference between CGI groups from the TRI database, thus affording a means of optimally evaluating the outcomes in individual patients.

Conclusion

We anticipate such a tool to become a starting point for more sophisticated measures in clinical outcomes, applicable not only in the context of tinnitus but also in other complex diseases where the integration of multiple variables is needed for a comprehensive evaluation of treatment response.

Interaction of insular cortex and ventral striatum mediates the effect of incentive memory on choice between goal-directed actions

The anterior insular cortex (IC) and the nucleus accumbens (NAc) core have been separately implicated in the selection and performance of actions based on the incentive value of the instrumental outcome. Here, we examined the role of connections between the IC and the NAc core in the performance of goal-directed actions. Rats were trained on two actions for distinct outcomes, after which one of the two outcomes was devalued by specific satiety immediately before a choice extinction test. We first confirmed the projection from the IC to the NAc core and then disconnected these structures via asymmetrical excitotoxic lesions before training. Contralateral, but not ipsilateral, disconnection of the IC and NAc core disrupted outcome devaluation. We hypothesized that communication between the IC and NAc core is necessary for the retrieval of incentive value at test. To test this, we infused the GABAA agonist muscimol into the IC and the μ-opioid receptor antagonist CTAP into the contralateral NAc before the choice extinction test. As expected, inactivation of the IC in one hemisphere and blocking μ-opioid receptors in the contralateral NAc core abolished outcome-selective devaluation. These results suggest that the IC and NAc core form part of a circuit mediating the retrieval of outcome values and the subsequent choice between goal-directed actions based on those values.

The suggestible brain: posthypnotic effects on value-based decision-making

Hypnosis can affect perception, motor function and memory. However, so far no study using neuroimaging has investigated whether hypnosis can influence reward processing and decision-making. Here, we assessed whether posthypnotic suggestions can diminish the attractiveness of unhealthy food and whether this is more effective than diminishing attractiveness by one's own effort via autosuggestion. In total, 16 participants were hypnotized and 16 others were instructed to associate a color cue (blue or green) with disgust regarding specific snacks (sweet or salty). Afterwards, participants bid for snack items shown on an either blue or green background during functional magnetic resonance imaging measurement. Both hypnosis and autosuggestion successfully devalued snacks. This was indicated by participants' decision-making, their self-report and by decreased blood oxygen level-dependent signal in the ventromedial prefrontal cortex (vmPFC), a region known to represent value. Different vmPFC subregions coded for cue and snack type. The cue had significantly stronger effects on vmPFC after hypnosis than after autosuggestion, indicating that hypnosis was more effective in genuinely reducing value. Supporting previous findings, the precuneus was involved in the hypnotic effects by encoding whether a snack was sweet or salty during hypnotic cue presentation. Our results demonstrate that posthypnotic suggestions can influence valuation and decision-making.

Predeliberation activity in prefrontal cortex and striatum and the prediction of subsequent value judgment

Rational, value-based decision-making mandates selecting the option with highest subjective expected value after appropriate deliberation. We examined activity in the dorsolateral prefrontal cortex (DLPFC) and striatum of monkeys deciding between smaller, immediate rewards and larger, delayed ones. We previously found neurons that modulated their activity in this task according to the animal's choice, while it deliberated (choice neurons). Here we found neurons whose spiking activities were predictive of the spatial location of the selected target (spatial-bias neurons) or the size of the chosen reward (reward-bias neurons) before the onset of the cue presenting the decision-alternatives, and thus before rational deliberation could begin. Their predictive power increased as the values the animals associated with the two decision alternatives became more similar. The ventral striatum (VS) preferentially contained spatial-bias neurons; the caudate nucleus (CD) preferentially contained choice neurons. In contrast, the DLPFC contained significant numbers of all three neuron types, but choice neurons were not preferentially also bias neurons of either kind there, nor were spatial-bias neurons preferentially also choice neurons, and vice versa. We suggest a simple winner-take-all (WTA) circuit model to account for the dissociation of choice and bias neurons. The model reproduced our results and made additional predictions that were borne out empirically. Our data are compatible with the hypothesis that the DLPFC and striatum harbor dissociated neural populations that represent choices and predeliberation biases that are combined after cue onset; the bias neurons have a weaker effect on the ultimate decision than the choice neurons, so their influence is progressively apparent for trials where the values associated with the decision alternatives are increasingly similar.