Dopamine modulates novelty seeking behavior during decision making

Temperament and Offending Behaviors in Male Adolescents

The aim of the current paper was to examine temperament profiles and temperament dimensions as risk factors for persistent criminal behavior in juveniles who offended (JOs). A sample of 137 male adolescents from a Swiss detention center and 137 age and sex matched community controls were included in the present study. Temperament was measured with the Junior Temperament and Character Inventory (JTCI). Using Latent Profile Analysis (LPA), three temperament profiles were found, a "moderate," an "adventurous-disinhibited" (higher levels of novelty seeking, lower levels of harm avoidance, reward dependence, and persistence), and a "worried-passive" profile (higher levels of harm avoidance, low persistence). None of the profiles and dimensions were associated with detention sample (i.e., JO) status. In JOs, the "novelty seeking" scale predicted recidivism after release from detention even when controlling for other covariates. Further research should address temperament profiles and temperament dimensions in larger samples of JOs to elaborate their relation to previous and future offending behavior.

Adolescent-to-adult gains in cognitive flexibility are adaptively supported by reward sensitivity, exploration, and neural variability

Cognitive flexibility exhibits dynamic changes throughout development, with different forms of flexibility showing dissociable developmental trajectories. In this review, we propose that an adolescent-specific mode of flexibility in the face of changing environmental contingencies supports the emergence of adolescent-to-adult gains in cognitive shifting efficiency. We first describe how cognitive shifting abilities monotonically improve from childhood to adulthood, accompanied by increases in brain state flexibility, neural variability, and excitatory/inhibitory balance. We next summarize evidence supporting the existence of a dopamine-driven, adolescent peak in flexible behavior that results in reward seeking, undirected exploration, and environmental sampling. We propose a neurodevelopmental framework that relates these adolescent behaviors to the refinement of neural phenotypes relevant to mature cognitive flexibility, and thus highlight the importance of the adolescent period in fostering healthy neurocognitive trajectories.

Phasic Stimulation of Dopaminergic Neurons of the Lateral Substantia Nigra Increases Open Field Exploratory Behaviour and Reduces Habituation Over Time

Transient nigrostriatal dopaminergic signalling is well known for its role in reinforcement learning and increasingly so for its role in the initiation of voluntary movement. However, how transient bursts of dopamine modulate voluntary movement remains unclear, likely due to the heterogeneity of the nigrostriatal system, the focus of optogenetic studies on locomotion at sub-sec time intervals, and the overlapping roles of phasic dopamine in behaviour and novelty signalling. In this study we investigated how phasic activity in the lateral substantia nigra pars compacta (lateral SNc) over time affects voluntary behaviours during exploration. Using a transgenic mouse model of both sexes expressing channelrhodopsin (ChR2) in dopamine transporter-expressing cells, we stimulated the lateral SNc while mice explored an open field over two consecutive days. We found that phasic activation of the lateral SNc induced an increase in exploratory behaviours including horizontal movement activity, locomotion initiation, and rearing specifically on the first open field exposure, but not on the second day. In addition, stimulated animals did not habituate to the same extent as their ChR2-negative counterparts, as indicated by a lack of decrease in baseline activity. These findings suggest that rather than prompting voluntary movement in general, phasic nigrostriatal dopamine prompts context-appropriate behaviours. In addition, dopamine signalling that modulates movement acts over longer timescales than the transient signal, affecting behaviour even after the signal has ended.

Cognitive asymmetry in rats in response to emergent vs. disappearing affordances

This study examines the effects of novel environmental changes on the behavior of rats in an experimental chamber. We hypothesized that newly discovered opportunities, detected by the animal's cognitive system, would motivate greater investigation of environmental changes than comparable changes that prevent a given behavior. Three experiments differed in the emergence vs. elimination of affordances represented by open or closed tunnels. In Experiment 1, rats were habituated to a chamber with all four tunnels closed, and then two tunnels were opened. In Experiment 2, rats were habituated to a chamber where all four tunnels were open, and then two tunnels were closed. In Experiment 3, rats were habituated to a chamber with two open tunnels on one side, and two closed tunnels on the other. Then, the arrangement of open and closed tunnels was swapped. Results of the Exp. 1 show that the rats responded by spending more time near the newly opened tunnels and less time near the closed tunnels, the central zone, and the transporter. This suggests that rats are more motivated to investigate the environmental change combined with the emergent affordance (opening of the tunnels) than the environmental change alone. In Exp. 2, the rats responded by spending more time near the open tunnels and less time in the central zone. This suggests that the rats are more triggered by the available affordances (open tunnels) than by the environmental change (closed tunnels). Finally, in Exp. 3, the rats responded by spending more time near the newly opened tunnels and less near the central zone. However, they did not spend less time near the newly closed tunnels. These results suggest that rats process both the novelty itself and the emergence/disappearance of available affordances. The results are discussed regarding the cognitive asymmetry in the perception of emergent vs. disappearing affordances. It is proposed that the rat's cognitive system is specialized for detecting newly emergent environmental opportunities/affordances rather than novelty in general.

Rethinking dopamine-guided action sequence learning

As opposed to those requiring a single action for reward acquisition, tasks necessitating action sequences demand that animals learn action elements and their sequential order and sustain the behaviour until the sequence is completed. With repeated learning, animals not only exhibit precise execution of these sequences but also demonstrate enhanced smoothness and efficiency. Previous research has demonstrated that midbrain dopamine and its major projection target, the striatum, play crucial roles in these processes. Recent studies have shown that dopamine from the substantia nigra pars compacta (SNc) and the ventral tegmental area (VTA) serve distinct functions in action sequence learning. The distinct contributions of dopamine also depend on the striatal subregions, namely the ventral, dorsomedial and dorsolateral striatum. Here, we have reviewed recent findings on the role of striatal dopamine in action sequence learning, with a focus on recent rodent studies.

Family-based genome-wide association analysis of novelty seeking in a Korean schizophrenic population: A pilot study

Schizophrenia (SPR) is the most devastating mental illness that causes severe deterioration in social and occupational functioning, but, the etiology remains unknown. The objective of this study is to explore the genetic underpinnings of novelty seeking behavior in schizophrenic family within the Korean population. By conducting a family-based genome-wide association study, we aim to identify potential genetic markers and variations associated with novelty seeking traits in the context of SPR. We have recruited 27 probands (with SPR) with their parents and siblings whenever possible. DNA was extracted from blood sampling of 58 individuals in 27 families and analyzed in an Illumina core exome single nucleotide polymorphism (SNP) array. A family-based association test (qFAM) was used to derive SNP association values across all chromosomes. Although none of the final 800,000 SNPs reached the genome-wide significant threshold of 8.45 × 10-7, the most significant 4 SNPs were within the 10-5 to 10-7. This study identifies genetic associations between novelty seeking behavior and SPR within families. RAPGEF5 emerges as a significant gene, along with other neuropsychiatric-related genes. Noteworthy genes like DRD4 and COMT did not show associations, possibly due to the focus on schizophrenic family. While shedding light on this complex relationship, larger studies are needed for robust conclusions and deeper mechanistic insights.

Callous-unemotional traits and pre-ejection period in response to reward

Callous-unemotional (CU) traits have important utility in distinguishing individuals exhibiting more severe and persistent antisocial behavior, and our understanding of reward processing and CU traits contributes to behavioral modification. However, research on CU traits often investigated reward alongside punishment and examined solely on average reward reactivity, neglecting the reward response pattern over time such as habituation. This study assessed individuals' pre-ejection period (PEP), a sympathetic nervous system cardiac-linked biomarker with specificity to reward, during a simple reward task to investigate the association between CU traits and both average reward reactivity and reward response pattern over time (captured as responding trajectory). A heterogeneous sample of 126 adult males was recruited from a major metropolitan area in the US. Participants reported their CU traits using the Inventory of Callous-Unemotional Traits and completed a simple reward task while impedance cardiography and electrocardiogram were recorded to derive PEP. The results revealed no significant association between average PEP reward reactivity and CU traits. However, CU traits predicted both linear and quadratic slopes of the PEP reactivity trajectory: individuals with higher CU traits had slower habituation initially, followed by a rapid habituation in later blocks. Findings highlight the importance of modeling the trajectory of PEP reward response when studying CU traits. We discussed the implications of individuals with high CU traits having the responding pattern of slower initial habituation followed by rapid habituation to reward and the possible mechanisms.

Neural mechanisms of information seeking

We ubiquitously seek information to make better decisions. Particularly in the modern age, when more information is available at our fingertips than ever, the information we choose to collect determines the quality of our decisions. Decision neuroscience has long adopted empirical approaches where the information available to decision-makers is fully controlled by the researchers, leaving neural mechanisms of information seeking less understood. Although information seeking has long been studied in the context of the exploration-exploitation trade-off, recent studies have widened the scope to investigate more overt information seeking in a way distinct from other decision processes. Insights gained from these studies, accumulated over the last few years, raise the possibility that information seeking is driven by the reward system signaling the subjective value of information. In this piece, we review findings from the recent studies, highlighting the conceptual and empirical relationships between distinct literatures, and discuss future research directions necessary to establish a more comprehensive understanding of how individuals seek information as a part of value-based decision-making.

Navigating Exploitative Traps: Unveiling the Uncontrollable Reward Seeking of Individuals With Internet Gaming Disorder

BACKGROUND

Internet gaming disorder (IGD) involves an imbalance in the brain's dual system, characterized by heightened reward seeking and diminished cognitive control, which lead to decision-making challenges. The exploration-exploitation strategy is key to decision making, but how IGD affects this process is unclear.

METHODS

To investigate the impact of IGD on decision making, a modified version of the 2-armed bandit task was employed. Participants included 41 individuals with IGD and 44 healthy control individuals. The study assessed the strategies used by participants in the task, particularly focusing on the exploitation-exploration strategy. Additionally, functional magnetic resonance imaging was used to examine brain activation patterns during decision-making and estimation phases.

RESULTS

The study found that individuals with IGD demonstrated greater reliance on exploitative strategies in decision making due to their elevated value-seeking tendencies and decreased cognitive control. Individuals with IGD also displayed heightened activation in the presupplementary motor area and the ventral striatum compared with the healthy control group in both decision-making and estimation phases. Meanwhile, the prefrontal cortex showed more inhibition in individuals with IGD than in the healthy control group during exploitative strategies. This inhibition decreased as cognitive control diminished.

CONCLUSIONS

The imbalance in the development of the dual system in individuals with IGD may lead to an overreliance on exploitative strategies. This imbalance, marked by increased reward seeking and reduced cognitive control, contributes to difficulties in decision making and value-related behavioral processes in individuals with IGD.

Addiction-Prone Personality and Creative Cognitive Styles: A Moderated Mediation Model of Novelty Seeking and Depression Tendency

This study attempted to examine the mechanism of the impact of Addiction-Prone Personality (APP) on creative cognitive styles (idea generation, idea selection), especially to explore the mediating role of novelty seeking and the moderating role of depression tendency on the relationship between APP and creative cognitive styles. College students (N = 576, 79% female) participated in and completed measures of APP, idea generation and selection, novelty seeking, and depression tendency. Results showed that (1) APP was positively related with idea generation while negatively related with idea selection; (2) novelty seeking played a partial mediating role in the relationship between APP and idea generation and a suppressing effect between APP and idea selection; (3) depression tendency moderated the indirect relationship between APP and creative cognitive styles through novelty seeking. Therefore, APP has different indirect effects on idea generation and idea selection via novelty seeking. When there was a higher depression tendency, there was a stronger indirect effect. The study highlights the significant importance of the underlying processes between APP and creative cognitive styles and offers implications for rethinking the relationship between addiction and creativity.

Dopamine encoding of novelty facilitates efficient uncertainty-driven exploration

When facing an unfamiliar environment, animals need to explore to gain new knowledge about which actions provide reward, but also put the newly acquired knowledge to use as quickly as possible. Optimal reinforcement learning strategies should therefore assess the uncertainties of these action-reward associations and utilise them to inform decision making. We propose a novel model whereby direct and indirect striatal pathways act together to estimate both the mean and variance of reward distributions, and mesolimbic dopaminergic neurons provide transient novelty signals, facilitating effective uncertainty-driven exploration. We utilised electrophysiological recording data to verify our model of the basal ganglia, and we fitted exploration strategies derived from the neural model to data from behavioural experiments. We also compared the performance of directed exploration strategies inspired by our basal ganglia model with other exploration algorithms including classic variants of upper confidence bound (UCB) strategy in simulation. The exploration strategies inspired by the basal ganglia model can achieve overall superior performance in simulation, and we found qualitatively similar results in fitting model to behavioural data compared with the fitting of more idealised normative models with less implementation level detail. Overall, our results suggest that transient dopamine levels in the basal ganglia that encode novelty could contribute to an uncertainty representation which efficiently drives exploration in reinforcement learning.

Curiosity: primate neural circuits for novelty and information seeking

For many years, neuroscientists have investigated the behavioural, computational and neurobiological mechanisms that support value-based decisions, revealing how humans and animals make choices to obtain rewards. However, many decisions are influenced by factors other than the value of physical rewards or second-order reinforcers (such as money). For instance, animals (including humans) frequently explore novel objects that have no intrinsic value solely because they are novel and they exhibit the desire to gain information to reduce their uncertainties about the future, even if this information cannot lead to reward or assist them in accomplishing upcoming tasks. In this Review, I discuss how circuits in the primate brain responsible for detecting, predicting and assessing novelty and uncertainty regulate behaviour and give rise to these behavioural components of curiosity. I also briefly discuss how curiosity-related behaviours arise during postnatal development and point out some important reasons for the persistence of curiosity across generations.

The impact of poor fetal growth and chronic hyperpalatable diet exposure in adulthood on hippocampal function and feeding patterns in male rats

Poor fetal growth affects eating behavior and the mesocorticolimbic system; however, its influence on the hippocampus has been less explored. Brain insulin sensitivity has been linked to developmental plasticity in response to fetal adversity and to cognitive performance following high-fat diet intake. We investigated whether poor fetal growth and exposure to chronic hyperpalatable food in adulthood could influence the recognition of environmental and food cues, eating behavior patterns, and hippocampal insulin signaling. At 60 days of life, we assigned male offspring from a prenatal animal model of 50% food restriction (FR) to receive either a high-fat and -sugar (HFS) diet or standard chow (CON) diet. Behavioral tests were conducted at 140 days, then tissues were collected. HFS groups showed a diminished hippocampal pAkt/Akt ratio. FR-CON and FR-HFS groups had higher levels of suppressor of cytokine signaling 3, compared to control groups. FR groups showed increased exploration of a novel hyperpalatable food, independent of their diet, and HFS groups exhibited overall lower entropy (less random, more predictable eating behavior) when the environment changed. Poor fetal growth and chronic HFS diet in adulthood altered hippocampal insulin signaling and eating patterns, diminishing the flexibility associated with eating behavior in response to extrinsic changes in food availability in the environment.

Motor System-Dependent Effects of Amygdala and Ventral Striatum Lesions on Explore-Exploit Behaviors

Deciding whether to forego immediate rewards or explore new opportunities is a key component of flexible behavior and is critical for the survival of the species. Although previous studies have shown that different cortical and subcortical areas, including the amygdala and ventral striatum (VS), are implicated in representing the immediate (exploitative) and future (explorative) value of choices, the effect of the motor system used to make choices has not been examined. Here, we tested male rhesus macaques with amygdala or VS lesions on two versions of a three-arm bandit task where choices were registered with either a saccade or an arm movement. In both tasks we presented the monkeys with explore-exploit tradeoffs by periodically replacing familiar options with novel options that had unknown reward probabilities. We found that monkeys explored more with saccades but showed better learning with arm movements. VS lesions caused the monkeys to be more explorative with arm movements and less explorative with saccades, although this may have been due to an overall decrease in performance. VS lesions affected the monkeys' ability to learn novel stimulus-reward associations in both tasks, while after amygdala lesions this effect was stronger when choices were made with saccades. Further, on average, VS and amygdala lesions reduced the monkeys' ability to choose better options only when choices were made with a saccade. These results show that learning reward value associations to manage explore-exploit behaviors is motor system dependent and they further define the contributions of amygdala and VS to reinforcement learning.

The amygdala is not necessary for the familiarity aspect of recognition memory

Dual-process accounts of item recognition posit two memory processes: slow but detailed recollection, and quick but vague familiarity. It has been proposed, based on prior rodent work, that the amygdala is critical for the familiarity aspect of item recognition. Here, we evaluated this proposal in male rhesus monkeys (Macaca mulatta) with selective bilateral excitotoxic amygdala damage. We used four established visual memory tests designed to assess different aspects of familiarity, all administered on touchscreen computers. Specifically, we assessed monkeys' tendencies to make low-latency false alarms, to make false alarms to recently seen lures, to produce curvilinear ROC curves, and to discriminate stimuli based on repetition across days. Three of the four tests showed no familiarity impairment and the fourth was explained by a deficit in reward processing. Consistent with this, amygdala damage did produce an anticipated deficit in reward processing in a three-arm-bandit gambling task, verifying the effectiveness of the lesions. Together, these results contradict prior rodent work and suggest that the amygdala is not critical for the familiarity aspect of item recognition.

PET-measured human dopamine synthesis capacity and receptor availability predict trading rewards and time-costs during foraging

Foraging behavior requires weighing costs of time to decide when to leave one reward patch to search for another. Computational and animal studies suggest that striatal dopamine is key to this process; however, the specific role of dopamine in foraging behavior in humans is not well characterized. We use positron emission tomography (PET) imaging to directly measure dopamine synthesis capacity and D1 and D2/3 receptor availability in 57 healthy adults who complete a computerized foraging task. Using voxelwise data and principal component analysis to identify patterns of variation across PET measures, we show that striatal D1 and D2/3 receptor availability and a pattern of mesolimbic and anterior cingulate cortex dopamine function are important for adjusting the threshold for leaving a patch to explore, with specific sensitivity to changes in travel time. These findings suggest a key role for dopamine in trading reward benefits against temporal costs to modulate behavioral adaptions to changes in the reward environment critical for foraging.

Art Value Creation and Destruction

I present a theory of creative and destructive value state referring to abstract art. Value is a probabilistic state held as a mixture of its expectation and information forces that coexist in a give-and-take relationship. Expectations are driven by the disclosure of novel information about the value state of various events of desire. Each bit of accumulated information contributes to the improvement of perception up to a threshold level, beyond which begin conscious states. The desire to disclose a value state triggers a triadic system of evaluation which uses concepts, observables and approaches. While the triadic valuation mechanisms can be used to assess various commodities, the scope of this work is limited to the case of artworks, in particular abstract paintings. I assume that art value is basically mediated by the interplay between these value state mechanisms of creation and destruction. Expectations in artwork develop attraction by challenging its contemplator to evaluate (predict) its meaning. Once the relevant information, corresponding to its creative expectations, is acquired (and conditioned), emotional states of indifference, disinterest and desensitization develop.

Electrophysiological Markers of Aberrant Cue-Specific Exploration in Hazardous Drinkers

Background

Hazardous drinking is associated with maladaptive alcohol-related decision-making. Existing studies have often focused on how participants learn to exploit familiar cues based on prior reinforcement, but little is known about the mechanisms that drive hazardous drinkers to explore novel alcohol cues when their value is not known.

Methods

We investigated exploration of novel alcohol and non-alcohol cues in hazardous drinkers (N = 27) and control participants (N = 26) during electroencephalography (EEG). A normative computational model with two free parameters was fit to estimate participants' weighting of the future value of exploration and immediate value of exploitation.

Results

Hazardous drinkers demonstrated increased exploration of novel alcohol cues, and conversely, increased probability of exploiting familiar alternatives instead of exploring novel non-alcohol cues. The motivation to explore novel alcohol stimuli in hazardous drinkers was driven by an elevated relative future valuation of uncertain alcohol cues. P3a predicted more exploratory decision policies driven by an enhanced relative future valuation of novel alcohol cues. P3b did not predict choice behavior, but computational parameter estimates suggested that hazardous drinkers with enhanced P3b to alcohol cues were likely to learn to exploit their immediate expected value.

Conclusions

Hazardous drinkers did not display atypical choice behavior, different P3a/P3b amplitudes, or computational estimates to novel non-alcohol cues-diverging from previous studies in addiction showing atypical generalized explore-exploit decisions with non-drug-related cues. These findings reveal that cue-specific neural computations may drive aberrant alcohol-related decision-making in hazardous drinkers-highlighting the importance of drug-relevant cues in studies of decision-making in addiction.

Semaglutide reduces alcohol intake and relapse-like drinking in male and female rats

BACKGROUND

Glucagon-like peptide1 receptor (GLP-1R) agonists have been found to reduce alcohol drinking in rodents and overweight patients with alcohol use disorder (AUD). However, the probability of low semaglutide doses, an agonist with higher potency and affinity for GLP-1R, to attenuate alcohol-related responses in rodents and the underlying neuronal mechanisms is unknown.

METHODS

In the intermittent access model, we examined the ability of semaglutide to decrease alcohol intake and block relapse-like drinking, as well as imaging the binding of fluorescently marked semaglutide to nucleus accumbens (NAc) in both male and female rats. The suppressive effect of semaglutide on alcohol-induced locomotor stimulation and in vivo dopamine release in NAc was tested in male mice. We evaluated effect of semaglutide on the in vivo release of dopamine metabolites (DOPAC and HVA) and gene expression of enzymes metabolising dopamine (MAOA and COMT) in male mice.

FINDINGS

In male and female rats, acute and repeated semaglutide administration reduced alcohol intake and prevented relapse-like drinking. Moreover, fluorescently labelled semaglutide was detected in NAc of alcohol-drinking male and female rats. Further, semaglutide attenuated the ability of alcohol to cause hyperlocomotion and to elevate dopamine in NAc in male mice. As further shown in male mice, semaglutide enhanced DOPAC and HVA in NAc when alcohol was onboard and increased the gene expression of COMT and MAOA.

INTERPRETATION

Altogether, this indicates that semaglutide reduces alcohol drinking behaviours, possibly via a reduction in alcohol-induced reward and NAc dependent mechanisms. As semaglutide also decreased body weight of alcohol-drinking rats of both sexes, upcoming clinical studies should test the plausibility that semaglutide reduces alcohol intake and body weight in overweight AUD patients.

FUNDING

Swedish Research Council (2019-01676), LUA/ALF (723941) from the Sahlgrenska University Hospital and the Swedish brain foundation.

Exploratory preferences explain the human fascination for imaginary worlds in fictional stories

Imaginary worlds are present and often central in many of the most culturally successful modern narrative fictions, be it in novels (e.g., Harry Potter), movies (e.g., Star Wars), video games (e.g., The Legend of Zelda), graphic novels (e.g., One Piece) and TV series (e.g., Game of Thrones). We propose that imaginary worlds are popular because they activate exploratory preferences that evolved to help us navigate the real world and find new fitness-relevant information. Therefore, we hypothesize that the attraction to imaginary worlds is intrinsically linked to the desire to explore novel environments and that both are influenced by the same underlying factors. Notably, the inter-individual and cross-cultural variability of the preference for imaginary worlds should follow the inter-individual and cross-cultural variability of exploratory preferences (with the personality trait Openness-to-experience, age, sex, and ecological conditions). We test these predictions with both experimental and computational methods. For experimental tests, we run a pre-registered online experiment about movie preferences (N = 230). For computational tests, we leverage two large cultural datasets, namely the Internet Movie Database (N = 9424 movies) and the Movie Personality Dataset (N = 3.5 million participants), and use machine-learning algorithms (i.e., random forest and topic modeling). In all, consistent with how the human preference for spatial exploration adaptively varies, we provide empirical evidence that imaginary worlds appeal more to more explorative people, people higher in Openness-to-experience, younger individuals, males, and individuals living in more affluent environments. We discuss the implications of these findings for our understanding of the cultural evolution of narrative fiction and, more broadly, the evolution of human exploratory preferences.

Parasubthalamic calretinin neurons modulate wakefulness associated with exploration in male mice

The parasubthalamic nucleus (PSTN) is considered to be involved in motivation, feeding and hunting, all of which are highly depending on wakefulness. However, the roles and underlying neural circuits of the PSTN in wakefulness remain unclear. Neurons expressing calretinin (CR) account for the majority of PSTN neurons. In this study in male mice, fiber photometry recordings showed that the activity of PSTN^CR neurons increased at the transitions from non-rapid eye movement (non-REM, NREM) sleep to either wakefulness or REM sleep, as well as exploratory behavior. Chemogenetic and optogenetic experiments demonstrated that PSTN^CR neurons were necessary for initiating and/or maintaining arousal associated with exploration. Photoactivation of projections of PSTN^CR neurons revealed that they regulated exploration-related wakefulness by innervating the ventral tegmental area. Collectively, our findings indicate that PSTN^CR circuitry is essential for the induction and maintenance of the awake state associated with exploration.

Maladaptive explore/exploit trade-offs in schizophrenia

Schizophrenia is a complex disorder that remains poorly understood, particularly at the systems level. In this opinion article we argue that the explore/exploit trade-off concept provides a holistic and ecologically valid framework to resolve some of the apparent paradoxes that have emerged within schizophrenia research. We review recent evidence suggesting that fundamental explore/exploit behaviors may be maladaptive in schizophrenia during physical, visual, and cognitive foraging. We also describe how theories from the broader optimal foraging literature, such as the marginal value theorem (MVT), could provide valuable insight into how aberrant processing of reward, context, and cost/effort evaluations interact to produce maladaptive responses.

Low-Affinity/High-Selectivity Dopamine Transport Inhibition Sufficient to Rescue Cognitive Functions in the Aging Rat

The worldwide increase in cognitive decline, both in aging and with psychiatric disorders, warrants a search for pharmacological treatment. Although dopaminergic treatment approaches represent a major step forward, current dopamine transporter (DAT) inhibitors are not sufficiently specific as they also target other transporters and receptors, thus showing unwanted side effects. Herein, we describe an enantiomerically pure, highly specific DAT inhibitor, S-CE-123, synthetized in our laboratory. Following binding studies to DAT, NET and SERT, GPCR and kinome screening, pharmacokinetics and a basic neurotoxic screen, S-CE-123 was tested for its potential to enhance and/or rescue cognitive functions in young and in aged rats in the non-invasive reward-motivated paradigm of a hole-board test for spatial learning. In addition, an open field study with young rats was carried out. We demonstrated that S-CE-123 is a low-affinity but highly selective dopamine reuptake inhibitor with good bioavailability. S-CE-123 did not induce hyperlocomotion or anxiogenic or stereotypic behaviour in young rats. Our compound improved the performance of aged but not young rats in a reward-motivated task. The well-described impairment of the dopaminergic system in aging may underlie the age-specific effect. We propose S-CE-123 as a possible candidate for developing a tentative therapeutic strategy for age-related cognitive decline and cognitive dysfunction in psychiatric disorders.

Pallidal neuromodulation of the explore/exploit trade-off in decision-making

Every decision that we make involves a conflict between exploiting our current knowledge of an action's value or exploring alternative courses of action that might lead to a better, or worse outcome. The sub-cortical nuclei that make up the basal ganglia have been proposed as a neural circuit that may contribute to resolving this explore-exploit 'dilemma'. To test this hypothesis, we examined the effects of neuromodulating the basal ganglia's output nucleus, the globus pallidus interna, in patients who had undergone deep brain stimulation (DBS) for isolated dystonia. Neuromodulation enhanced the number of exploratory choices to the lower value option in a two-armed bandit probabilistic reversal-learning task. Enhanced exploration was explained by a reduction in the rate of evidence accumulation (drift rate) in a reinforcement learning drift diffusion model. We estimated the functional connectivity profile between the stimulating DBS electrode and the rest of the brain using a normative functional connectome derived from heathy controls. Variation in the extent of neuromodulation induced exploration between patients was associated with functional connectivity from the stimulation electrode site to a distributed brain functional network. We conclude that the basal ganglia's output nucleus, the globus pallidus interna, can adaptively modify decision choice when faced with the dilemma to explore or exploit.

Environmental uncertainty and the advantage of impulsive choice strategies

Choice impulsivity is characterized by the choice of immediate, smaller reward options over future, larger reward options, and is often thought to be associated with negative life outcomes. However, some environments make future rewards more uncertain, and in these environments impulsive choices can be beneficial. Here we examined the conditions under which impulsive vs. non-impulsive decision strategies would be advantageous. We used Markov Decision Processes (MDPs) to model three common decision-making tasks: Temporal Discounting, Information Sampling, and an Explore-Exploit task. We manipulated environmental variables to create circumstances where future outcomes were relatively uncertain. We then manipulated the discount factor of an MDP agent, which affects the value of immediate versus future rewards, to model impulsive and non-impulsive behavior. This allowed us to examine the performance of impulsive and non-impulsive agents in more or less predictable environments. In Temporal Discounting, we manipulated the transition probability to delayed rewards and found that the agent with the lower discount factor (i.e. the impulsive agent) collected more average reward than the agent with a higher discount factor (the non-impulsive agent) by selecting immediate reward options when the probability of receiving the future reward was low. In the Information Sampling task, we manipulated the amount of information obtained with each sample. When sampling led to small information gains, the impulsive MDP agent collected more average reward than the non-impulsive agent. Third, in the Explore-Exploit task, we manipulated the substitution rate for novel options. When the substitution rate was high, the impulsive agent again performed better than the non-impulsive agent, as it explored the novel options less and instead exploited options with known reward values. The results of these analyses show that impulsivity can be advantageous in environments that are unexpectedly uncertain.

Disentangling the roles of dopamine and noradrenaline in the exploration-exploitation tradeoff during human decision-making

Balancing the exploration of new options and the exploitation of known options is a fundamental challenge in decision-making, yet the mechanisms involved in this balance are not fully understood. Here, we aimed to elucidate the distinct roles of dopamine and noradrenaline in the exploration-exploitation tradeoff during human choice. To this end, we used a double-blind, placebo-controlled design in which participants received either a placebo, 400 mg of the D2/D3 receptor antagonist amisulpride, or 40 mg of the β-adrenergic receptor antagonist propranolol before they completed a virtual patch-foraging task probing exploration and exploitation. We systematically varied the rewards associated with choice options, the rate by which rewards decreased over time, and the opportunity costs it took to switch to the next option to disentangle the contributions of dopamine and noradrenaline to specific choice aspects. Our data show that amisulpride increased the sensitivity to all of these three critical choice features, whereas propranolol was associated with a reduced tendency to use value information. Our findings provide novel insights into the specific roles of dopamine and noradrenaline in the regulation of human choice behavior, suggesting a critical involvement of dopamine in directed exploration and a role of noradrenaline in more random exploration.

Clinical Markers of 6 Pre-dominant Coping Behaviors in Living With Parkinson Disease: A Convergent Mixed Methods Study

People with Parkinson's disease (PwP) experience a variety of symptoms and fluctuations in these, which they have to cope with every day. In tailoring a person-centered treatment to PwP there is a lack of knowledge about the association between pre-dominant coping behaviors and clinical markers among PwP. To describe and compare specific clinical markers between 6 suggested coping behaviors. Thirty-four PwP, who previously had been classified into 6 different pre-dominant coping behaviors, were included in this mixed methods study. Six primary variables were included in the descriptive analysis; motor function (UPDRS-III), non-motor symptoms score (NMS-Quest), change in bradykinesia score, apathy score (LARS), personality traits (NEO-FFI), and cognitive status (evaluated by a neuropsychologist). The merged results of this mixed methods study indicate that clinical markers as apathy, burden of non-motor symptoms, cognitive impairments and personality traits, have the potential to impact the coping behavior in PwP. In a clinical setting the markers; NMS-burden, degree of apathy, cognition, and personality traits may indicate specific coping behavior. Three of the six suggested typologies of coping behaviors differed from the other groups when comparing descriptive data. In order to improve patient care and guide the development of person-centered therapies, each PwP should be approached based on those typologies.

Reactivating hippocampal-mediated memories during reconsolidation to disrupt fear

Memories are stored in the brain as cellular ensembles activated during learning and reactivated during retrieval. Using the Tet-tag system in mice, we label dorsal dentate gyrus neurons activated by positive, neutral or negative experiences with channelrhodopsin-2. Following fear-conditioning, these cells are artificially reactivated during fear memory recall. Optical stimulation of a competing positive memory is sufficient to update the memory during reconsolidation, thereby reducing conditioned fear acutely and enduringly. Moreover, mice demonstrate operant responding for reactivation of a positive memory, confirming its rewarding properties. These results show that interference from a rewarding experience can counteract negative affective states. While memory-updating, induced by memory reactivation, involves a relatively small set of neurons, we also find that activating a large population of randomly labeled dorsal dentate gyrus neurons is effective in promoting reconsolidation. Importantly, memory-updating is specific to the fear memory. These findings implicate the dorsal dentate gyrus as a potential therapeutic node for modulating memories to suppress fear.

Personality and brain contribute to academic achievements of medical students

There are many factors that influence the academic achievements of medical students, but how personality and brain modulate the academic achievements of medical students remains unclear. The study collected the personality, brain imaging, and academic data from 448 medical students at Tianjin Medical University with admission time between 2008 and 2017. Four types of academic achievements, including behavioral and social sciences, clinical sciences and skills, basic biomedical sciences, and scientific methods, were assessed by the academic records of 58 courses. Personality was evaluated by Tridimensional Personality Questionnaire and Neuroticism Extraversion Openness Personality Inventory. Brain structural and functional properties, including gray matter volume, spontaneous brain activity and functional connectivity, were computed based on magnetic resonance imaging (MRI). Linear regression was used to evaluate the associations between personality and academic achievements. A voxel-wise correlation was used to identify areas of the brain where structural and functional properties were associated with academic achievements. Mediation analysis was used to test whether brain properties and personality independently contribute to academic achievements. Our results showed that novelty seeking (NS) was negatively correlated, and conscientiousness was positively correlated with all types of academic achievements. Brain functional properties showed negatively correlated with academic achievement in basic biomedical sciences. However, we did not find any mediation effect of the brain functional properties on the association between personality (NS and conscientiousness) and academic achievement in basic biomedical sciences, nor mediation effect of the personality (NS and conscientiousness) on the association between brain functional properties and academic achievement in basic biomedical sciences. These findings suggest that specific personality (NS and conscientiousness) and brain functional properties independently contribute to academic achievements in basic biomedical sciences, and that modulation of these properties may benefit academic achievements among medical students.

The dopamine circuit as a reward-taxis navigation system

Studying the brain circuits that control behavior is challenging, since in addition to their structural complexity there are continuous feedback interactions between actions and sensed inputs from the environment. It is therefore important to identify mathematical principles that can be used to develop testable hypotheses. In this study, we use ideas and concepts from systems biology to study the dopamine system, which controls learning, motivation, and movement. Using data from neuronal recordings in behavioral experiments, we developed a mathematical model for dopamine responses and the effect of dopamine on movement. We show that the dopamine system shares core functional analogies with bacterial chemotaxis. Just as chemotaxis robustly climbs chemical attractant gradients, the dopamine circuit performs ‘reward-taxis’ where the attractant is the expected value of reward. The reward-taxis mechanism provides a simple explanation for scale-invariant dopaminergic responses and for matching in free operant settings, and makes testable quantitative predictions. We propose that reward-taxis is a simple and robust navigation strategy that complements other, more goal-directed navigation mechanisms.

The neurocomputational bases of explore-exploit decision-making

Flexible decision-making requires animals to forego immediate rewards (exploitation) and try novel choice options (exploration) to discover if they are preferable to familiar alternatives. Using the same task and a partially observable Markov decision process (POMDP) model to quantify the value of choices, we first determined that the computational basis for managing explore-exploit tradeoffs is conserved across monkeys and humans. We then used fMRI to identify where in the human brain the immediate value of exploitative choices and relative uncertainty about the value of exploratory choices were encoded. Consistent with prior neurophysiological evidence in monkeys, we observed divergent encoding of reward value and uncertainty in prefrontal and parietal regions, including frontopolar cortex, and parallel encoding of these computations in motivational regions including the amygdala, ventral striatum, and orbitofrontal cortex. These results clarify the interplay between prefrontal and motivational circuits that supports adaptive explore-exploit decisions in humans and nonhuman primates.

Liking as a balance between synchronization, complexity and novelty

Synchronization has been identified as a key aspect in social bonding. While synchronization could be maximized by increasing the predictability of an interaction, such predictability is in tension with individuals’ level of interest, which is tied to the interaction’s complexity and novelty. In this study, we tested the interplay between synchronization and interest. We asked 104 female dyads to play the Mirror Game, in which they had to move their hands as coordinately as possible, and then report how much they liked each other. Utilizing information theory and video processing tools, we found that a combination of movement synchronization and complexity explained liking almost two times better than movement synchronization alone. Moreover, we found that people initiated novel and challenging interactions, even though they paid a price—being less synchronized. Examining the interactions’ dynamics, we found that people who liked each other moved in a more synchronized, complex, and novel manner during most of the interaction. This suggests that in addition to synchronization, maintaining interest may be critical for positive social bonding. Thus, we propose a new framework in which balancing synchronization and interest, rather than merely maximizing synchronization, optimizes the interaction quality.

Differential coding of goals and actions in ventral and dorsal corticostriatal circuits during goal-directed behavior

Goal-directed behavior requires identifying objects in the environment that can satisfy internal needs and executing actions to obtain those objects. The current study examines ventral and dorsal corticostriatal circuits that support complementary aspects of goal-directed behavior. We analyze activity from the amygdala, ventral striatum, orbitofrontal cortex, and lateral prefrontal cortex (LPFC) while monkeys perform a three-armed bandit task. Information about chosen stimuli and their value is primarily encoded in the amygdala, ventral striatum, and orbitofrontal cortex, while the spatial information is primarily encoded in the LPFC. Before the options are presented, information about the to-be-chosen stimulus is represented in the amygdala, ventral striatum, and orbitofrontal cortex; at the time of choice, the information is passed to the LPFC to direct a saccade. Thus, learned value information specifying behavioral goals is maintained throughout the ventral corticostriatal circuit, and it is routed through the dorsal circuit at the time actions are selected.

A primate temporal cortex-zona incerta pathway for novelty seeking

Primates interact with the world by exploring visual objects; they seek opportunities to view novel objects even when these have no extrinsic reward value. How the brain controls this novelty seeking is unknown. Here we show that novelty seeking in monkeys is regulated by the zona incerta (ZI). As monkeys made eye movements to familiar objects to trigger an opportunity to view novel objects, many ZI neurons were preferentially activated by predictions of novel objects before the gaze shift. Low-intensity ZI stimulation facilitated gaze shifts, whereas ZI inactivation reduced novelty seeking. ZI-dependent novelty seeking was not regulated by neurons in the lateral habenula or by many dopamine neurons in the substantia nigra, traditionally associated with reward seeking. But the anterior ventral medial temporal cortex, an area important for object vision and memory, was a prominent source of novelty predictions. These data uncover a functional pathway in the primate brain that regulates novelty seeking.

DAT1 polymorphism associated with poor decision-making in males with antisocial personality disorder and high psychopathic traits

Studies suggest that abnormalities of the dopaminergic system underlie decision-making deficits, a hallmark of antisocial personality disorder (ASPD) and psychopathy. The dopamine transporter gene (DAT1) is of particular interest due to a polymorphism that controls dopamine transporter (DAT) activity. However, the association between DAT1 genotypes and decision-making in ASPD has never been studied. The current study investigated the effect of DAT1 genotype on decision-making, as measured by the Iowa Gambling Task (IGT), in ASPD and healthy controls. A total of 17 participants with ASPD and 16 healthy control participants without ASPD were sampled. The Hare Psychopathy Checklist-Revised and the IGT were administered to all participants. All participants provided blood samples for genotyping. Data revealed a novel interaction effect between DAT1 genotype and diagnosis, whereby ASPD participants with low DAT activity genotypes performed significantly worse on the IGT and selected from disadvantageous decks more often, whereas the low DAT activity genotype in the healthy control group was associated with better performance on the IGT, and they selected from disadvantageous decks less often. We demonstrate, for the first time, that low DAT activity genotypes in ASPD with high psychopathic traits contribute to poor decision-making.

Distinct motivations to seek out information in healthy individuals and problem gamblers

As massive amounts of information are becoming available to people, understanding the mechanisms underlying information-seeking is more pertinent today than ever. In this study, we investigate the underlying motivations to seek out information in healthy and addicted individuals. We developed a novel decision-making task and a novel computational model which allows dissociating the relative contribution of two motivating factors to seek out information: a desire for novelty and a general desire for knowledge. To investigate whether/how the motivations to seek out information vary between healthy and addicted individuals, in addition to healthy controls we included a sample of individuals with gambling disorder-a form of addiction without the confound of substance consumption and characterized by compulsive gambling. Our results indicate that healthy subjects and problem gamblers adopt distinct information-seeking "modes". Healthy information-seeking behavior was mostly motivated by a desire for novelty. Problem gamblers, on the contrary, displayed reduced novelty-seeking and an increased desire for accumulating knowledge compared to healthy controls. Our findings not only shed new light on the motivations driving healthy and addicted individuals to seek out information, but they also have important implications for the treatment and diagnosis of behavioral addiction.

Reinforcement learning links spontaneous cortical dopamine impulses to reward

In their pioneering study on dopamine release, Romo and Schultz speculated "...that the amount of dopamine released by unmodulated spontaneous impulse activity exerts a tonic, permissive influence on neuronal processes more actively engaged in preparation of self-initiated movements...."¹ Motivated by the suggestion of "spontaneous impulses," as well as by the "ramp up" of dopaminergic neuronal activity that occurs when rodents navigate to a reward,^2-5 we asked two questions. First, are there spontaneous impulses of dopamine that are released in cortex? Using cell-based optical sensors of extrasynaptic dopamine, [DA]_ex,⁶ we found that spontaneous dopamine impulses in cortex of naive mice occur at a rate of ∼0.01 per second. Next, can mice be trained to change the amplitude and/or timing of dopamine events triggered by internal brain dynamics, much as they can change the amplitude and timing of dopamine impulses based on an external cue?^7-9 Using a reinforcement learning paradigm based solely on rewards that were gated by feedback from real-time measurements of [DA]_ex, we found that mice can volitionally modulate their spontaneous [DA]_ex. In particular, by only the second session of daily, hour-long training, mice increased the rate of impulses of [DA]_ex, increased the amplitude of the impulses, and increased their tonic level of [DA]_ex for a reward. Critically, mice learned to reliably elicit [DA]_ex impulses prior to receiving a reward. These effects reversed when the reward was removed. We posit that spontaneous dopamine impulses may serve as a salient cognitive event in behavioral planning.

Understanding Variations in the Health Consequences of Sedentary Behavior: A Taxonomy of Social Interaction, Novelty, Choice, and Cognition

The study of sedentary behaviors requires taxonomies (classification schemes) to standardize data collection, measurements, and outcomes. Three taxonomies of sedentary behaviors have been identified, but none address an important challenge in sedentary behavior research, which is to distinguish between beneficial and detrimental health effects of various sedentary behaviors. Some sedentary behaviors (e.g., reading) are associated with positive health outcomes, whereas other sedentary behaviors (e.g., television viewing) are associated with adverse health outcomes. To address directly this complexity and present a different conception and understanding of discrepant findings related to health outcomes, a new taxonomy is needed. The development of the new taxonomy is guided by analysis of literature and selection of a relevant and informative behavioral sciences theoretical framework (i.e., self-determination theory). Because older adults are an increasing percentage of the population and report a high prevalence of sedentary behaviors, the new taxonomy was designed for older adults with potential application to all age groups. Taylor's taxonomy of sedentary behaviors is parsimonious with four domains: social interaction (i.e., not solitary, companionship, interacting, and connecting with others); novelty (i.e., refreshingly new, unusual, or different); choice (i.e., volition, preferred option or alternative, the power, freedom, or decision to choose); and cognition (i.e., mentally stimulating and engaging).

Teacher Online Informal Learning as a Means to Innovative Teaching During Home Quarantine in the COVID-19 Pandemic

The home quarantine in the COVID-19 pandemic has created challenges for teaching across the world and called for innovative teaching, as well as teachers' learning. Given the rapid development of teachers' online learning and teaching, identifying effective ways to facilitate innovative teaching under such challenging conditions is a critical issue. Although researchers have realized that workplace informal learning (IL) increasingly reveals its potential value to individual development, the relationship between IL and innovation has been under-explored. The purpose of this study was to evaluate the impact of IL on innovative teaching, through the mediating roles of three types of teaching-related efficacy, with a particular focus on college teachers and online context. A sample of 479 Chinese college teachers was randomly selected to participate in the survey. The results showed that teachers' online IL in pandemic improved their personal teaching efficacy and ICT efficacy (information and communication technology efficacy), and then facilitated their innovative teaching without differences of gender and teaching-age effect. Whereas, general teaching efficacy was not a mediator between online IL and innovative teaching. Hence, we proposed a can-do motivating model of teacher efficacy in fostering innovative teaching through informal learning. It implies three properties of teachers' online IL: social interaction, autonomous learning and novelty-seeking. It also revealed that innovative teaching can be driven in COVID-19 pandemic, mainly by learning domain-specific knowledge and skills, thus enhancing personal teaching efficacy and ICT efficacy in online teaching context.

Understanding the Role of Incentive Salience in Sexual Decision-Making

In the search for understanding female sexual decision-making, progress has been made in uncovering a variety of perceptual biases and most of these concern the animal's sensory biology and cognitive processes. We are now poised to dig deeper into the female's decision-making and ask if incentive salience, which plays a role in all types of appetitive behaviors, also influences a female's "taste for the beautiful." The incentive salience hypothesis suggests that dopamine assigns value or salience to objects or actions. After value is assigned to all potential actions, an action selection system then chooses among potential options to select the most valuable action. In this view, dopamine stimulates reward-seeking behavior by assigning incentive salience to specific behavioral actions, which in turn, increases pursuit and focus on objects or stimuli that represent the valuable action. Here, we apply this framework to understand why females are compelled to respond maximally to some male courtship signals over others and how this process may reveal a female's hidden mate preferences. We examine studies of dopamine and the mesolimbic reward system because these may play a role in expanding the female's perceptual landscape for novelty in male courtship signals and establishing novel hidden preferences. We review three avenues of research that may identify signatures of incentive salience in females during sexual decision-making. This review includes studies of dopamine agonist or antagonist administration in females during mate choice or partner preference tests, measures of neural activity in dopaminergic neural circuits during mate choice or partner preference tests, and social regulation of dopamine in females when entering reproductive contexts and/or exposure to mate signals. By applying the incentive salience hypothesis to female reproductive decision-making, it redefines how we see the female's role in sexual encounters. Females cannot be considered passive during reproductive encounters; rather they are seeking sexual encounters, particularly with males that tap into their perceptual biases and initiate a reward-seeking response. Incentive salience applied to reproductive behavior requires considering females as viewing sexual stimuli as rewarding and initiating action to seek out this reward, all of which indicates females are driving sexual encounters.

Tonic dopamine, uncertainty and basal ganglia action selection

To make optimal decisions in uncertain circumstances flexible adaption of behaviour is required; exploring alternatives when the best choice is unknown, exploiting what is known when that is best. Using a computational model of the basal ganglia, we propose that switches between exploratory and exploitative decisions are mediated by the interaction between tonic dopamine and cortical input to the basal ganglia. We show that a biologically detailed action selection circuit model, endowed with dopamine dependant striatal plasticity, can optimally solve the explore-exploit problem, estimating the true underlying state of a noisy Gaussian diffusion process. Critical to the model's performance was a fluctuating level of tonic dopamine which increased under conditions of uncertainty. With an optimal range of tonic dopamine, explore-exploit decisions were mediated by the effects of tonic dopamine on the precision of the model action selection mechanism. Under conditions of uncertain reward pay-out, the model's reduced selectivity allowed disinhibition of multiple alternative actions to be explored at random. Conversely, when uncertainly about reward pay-out was low, enhanced selectivity of the action selection circuit facilitated exploitation of the high value choice. Model performance was at the level of a Kalman filter which provides an optimal solution for the task. These simulations support the idea that this subcortical neural circuit may have evolved to facilitate decision making in non-stationary reward environments. The model generates several experimental predictions with relevance to abnormal decision making in neuropsychiatric and neurological disease.

A New Paradigm for Training Hyperactive Dopamine Transporter Knockout Rats: Influence of Novel Stimuli on Object Recognition

Attention deficit hyperactivity disorder (ADHD) is believed to be connected with a high level of hyperactivity caused by alterations of the control of dopaminergic transmission in the brain. The strain of hyperdopaminergic dopamine transporter knockout (DAT-KO) rats represents an optimal model for investigating ADHD-related pathological mechanisms. The goal of this work was to study the influence of the overactivated dopamine system in the brain on a motor cognitive task fulfillment. The DAT-KO rats were trained to learn an object recognition task and store it in long-term memory. We found that DAT-KO rats can learn to move an object and retrieve food from the rewarded familiar objects and not to move the non-rewarded novel objects. However, we observed that the time of task performance and the distances traveled were significantly increased in DAT-KO rats in comparison with wild-type controls. Both groups of rats explored the novel objects longer than the familiar cubes. However, unlike controls, DAT-KO rats explored novel objects significantly longer and with fewer errors, since they preferred not to move the non-rewarded novel objects. After a 3 months' interval that followed the training period, they were able to retain the learned skills in memory and to efficiently retrieve them. The data obtained indicate that DAT-KO rats have a deficiency in learning the cognitive task, but their hyperactivity does not prevent the ability to learn a non-spatial cognitive task under the presentation of novel stimuli. The longer exploration of novel objects during training may ensure persistent learning of the task paradigm. These findings may serve as a base for developing new ADHD learning paradigms.

What is dopamine doing in model-based reinforcement learning?

Experiments have implicated dopamine in model-based reinforcement learning (RL). These findings are unexpected as dopamine is thought to encode a reward prediction error (RPE), which is the key teaching signal in model-free RL. Here we examine two possible accounts for dopamine's involvement in model-based RL: the first that dopamine neurons carry a prediction error used to update a type of predictive state representation called a successor representation, the second that two well established aspects of dopaminergic activity, RPEs and surprise signals, can together explain dopamine's involvement in model-based RL.

Balancing exploration and exploitation with information and randomization

Explore-exploit decisions require us to trade off the benefits of exploring unknown options to learn more about them, with exploiting known options, for immediate reward. Such decisions are ubiquitous in nature, but from a computational perspective, they are notoriously hard. There is therefore much interest in how humans and animals make these decisions and recently there has been an explosion of research in this area. Here we provide a biased and incomplete snapshot of this field focusing on the major finding that many organisms use two distinct strategies to solve the explore-exploit dilemma: a bias for information ('directed exploration') and the randomization of choice ('random exploration'). We review evidence for the existence of these strategies, their computational properties, their neural implementations, as well as how directed and random exploration vary over the lifespan. We conclude by highlighting open questions in this field that are ripe to both explore and exploit.

Individual associations of adolescent alcohol use disorder versus cannabis use disorder symptoms in neural prediction error signaling and the response to novelty

Two of the most commonly used illegal substances by adolescents are alcohol and cannabis. Alcohol use disorder (AUD) and cannabis use disorder (CUD) are associated with poorer decision-making in adolescents. In adolescents, level of AUD symptomatology has been negatively associated with striatal reward responsivity. However, little work has explored the relationship with striatal reward prediction error (RPE) representation and the extent to which any augmentation of RPE by novel stimuli is impacted. One-hundred fifty-one adolescents participated in the Novelty Task while undergoing functional magnetic resonance imaging (fMRI). In this task, participants learn to choose novel or non-novel stimuli to gain monetary reward. Level of AUD symptomatology was negatively associated with both optimal decision-making and BOLD response modulation by RPE within striatum and regions of prefrontal cortex. The neural alterations in RPE representation were particularly pronounced when participants were exploring novel stimuli. Level of CUD symptomatology moderated the relationship between novelty propensity and RPE representation within inferior parietal lobule and dorsomedial prefrontal cortex. These data expand on an emerging literature investigating individual associations of AUD symptomatology levels versus CUD symptomatology levels and RPE representation during reinforcement processing and provide insight on the role of neuro-computational processes underlying reinforcement learning/decision-making in adolescents.

Human complex exploration strategies are enriched by noradrenaline-modulated heuristics

An exploration-exploitation trade-off, the arbitration between sampling a lesser-known against a known rich option, is thought to be solved using computationally demanding exploration algorithms. Given known limitations in human cognitive resources, we hypothesised the presence of additional cheaper strategies. We examined for such heuristics in choice behaviour where we show this involves a value-free random exploration, that ignores all prior knowledge, and a novelty exploration that targets novel options alone. In a double-blind, placebo-controlled drug study, assessing contributions of dopamine (400 mg amisulpride) and noradrenaline (40 mg propranolol), we show that value-free random exploration is attenuated under the influence of propranolol, but not under amisulpride. Our findings demonstrate that humans deploy distinct computationally cheap exploration strategies and that value-free random exploration is under noradrenergic control.