Striatal activity underlies novelty-based choice in humans

The neuroscience of active learning and direct instruction

Throughout the educational system, students experiencing active learning pedagogy perform better and fail less than those taught through direct instruction. Can this be ascribed to differences in learning from a neuroscientific perspective? This review examines mechanistic, neuroscientific evidence that might explain differences in cognitive engagement contributing to learning outcomes between these instructional approaches. In classrooms, direct instruction comprehensively describes academic content, while active learning provides structured opportunities for learners to explore, apply, and manipulate content. Synaptic plasticity and its modulation by arousal or novelty are central to all learning and both approaches. As a form of social learning, direct instruction relies upon working memory. The reinforcement learning circuit, associated agency, curiosity, and peer-to-peer social interactions combine to enhance motivation, improve retention, and build higher-order-thinking skills in active learning environments. When working memory becomes overwhelmed, additionally engaging the reinforcement learning circuit improves retention, providing an explanation for the benefits of active learning. This analysis provides a mechanistic examination of how emerging neuroscience principles might inform pedagogical choices at all educational levels.

Neural mechanisms of information seeking

We ubiquitously seek information to make better decisions. Particularly in the modern age, when more information is available at our fingertips than ever, the information we choose to collect determines the quality of our decisions. Decision neuroscience has long adopted empirical approaches where the information available to decision-makers is fully controlled by the researchers, leaving neural mechanisms of information seeking less understood. Although information seeking has long been studied in the context of the exploration-exploitation trade-off, recent studies have widened the scope to investigate more overt information seeking in a way distinct from other decision processes. Insights gained from these studies, accumulated over the last few years, raise the possibility that information seeking is driven by the reward system signaling the subjective value of information. In this piece, we review findings from the recent studies, highlighting the conceptual and empirical relationships between distinct literatures, and discuss future research directions necessary to establish a more comprehensive understanding of how individuals seek information as a part of value-based decision-making.

Expanding horizons in reinforcement learning for curious exploration and creative planning

Curiosity and creativity are expressions of the trade-off between leveraging that with which we are familiar or seeking out novelty. Through the computational lens of reinforcement learning, we describe how formulating the value of information seeking and generation via their complementary effects on planning horizons formally captures a range of solutions to striking this balance.

Curiosity is more than novelty seeking

The novelty-seeking model (NSM) does not offer a compelling unifying framework for understanding creativity and curiosity. It fails to explain important manifestations and features of curiosity. Moreover, the arguments offered to support a curiosity-creativity link - a shared association with a common core process and various superficial associations between them - are neither convincing nor do they yield useful predictions.

The neurocomputational link between defensive cardiac states and approach-avoidance arbitration under threat

Avoidance, a hallmark of anxiety-related psychopathology, often comes at a cost; avoiding threat may forgo the possibility of a reward. Theories predict that optimal approach-avoidance arbitration depends on threat-induced psychophysiological states, like freezing-related bradycardia. Here we used model-based fMRI analyses to investigate whether and how bradycardia states are linked to the neurocomputational underpinnings of approach-avoidance arbitration under varying reward and threat magnitudes. We show that bradycardia states are associated with increased threat-induced avoidance and more pronounced reward-threat value comparison (i.e., a stronger tendency to approach vs. avoid when expected reward outweighs threat). An amygdala-striatal-prefrontal circuit supports approach-avoidance arbitration under threat, with specific involvement of the amygdala and dorsal anterior cingulate (dACC) in integrating reward-threat value and bradycardia states. These findings highlight the role of human freezing states in value-based decision making, relevant for optimal threat coping. They point to a specific role for amygdala/dACC in state-value integration under threat.

Trichotomy revisited: A monolithic theory of attentional control

The control of attention was long held to reflect the influence of two competing mechanisms of assigning priority, one goal-directed and the other stimulus-driven. Learning-dependent influences on the control of attention that could not be attributed to either of those two established mechanisms of control gave rise to the concept of selection history and a corresponding third mechanism of attentional control. The trichotomy framework that ensued has come to dominate theories of attentional control over the past decade, replacing the historical dichotomy. In this theoretical review, I readily affirm that distinctions between the influence of goals, salience, and selection history are substantive and meaningful, and that abandoning the dichotomy between goal-directed and stimulus-driven mechanisms of control was appropriate. I do, however, question whether a theoretical trichotomy is the right answer to the problem posed by selection history. If we reframe the influence of goals and selection history as different flavors of memory-dependent modulations of attentional priority and if we characterize the influence of salience as a consequence of insufficient competition from such memory-dependent sources of priority, it is possible to account for a wide range of attention-related phenomena with only one mechanism of control. The monolithic framework for the control of attention that I propose offers several concrete advantages over a trichotomy framework, which I explore here.

Prefrontal signals precede striatal signals for biased credit assignment in motivational learning biases

Actions are biased by the outcomes they can produce: Humans are more likely to show action under reward prospect, but hold back under punishment prospect. Such motivational biases derive not only from biased response selection, but also from biased learning: humans tend to attribute rewards to their own actions, but are reluctant to attribute punishments to having held back. The neural origin of these biases is unclear. Specifically, it remains open whether motivational biases arise primarily from the architecture of subcortical regions or also reflect cortical influences, the latter being typically associated with increased behavioral flexibility and control beyond stereotyped behaviors. Simultaneous EEG-fMRI allowed us to track which regions encoded biased prediction errors in which order. Biased prediction errors occurred in cortical regions (dorsal anterior and posterior cingulate cortices) before subcortical regions (striatum). These results highlight that biased learning is not a mere feature of the basal ganglia, but arises through prefrontal cortical contributions, revealing motivational biases to be a potentially flexible, sophisticated mechanism.

Curiosity-driven exploration: foundations in neuroscience and computational modeling

Curiosity refers to the intrinsic desire of humans and animals to explore the unknown, even when there is no apparent reason to do so. Thus far, no single, widely accepted definition or framework for curiosity has emerged, but there is growing consensus that curious behavior is not goal-directed but related to seeking or reacting to information. In this review, we take a phenomenological approach and group behavioral and neurophysiological studies which meet these criteria into three categories according to the type of information seeking observed. We then review recent computational models of curiosity from the field of machine learning and discuss how they enable integrating different types of information seeking into one theoretical framework. Combinations of behavioral and neurophysiological studies along with computational modeling will be instrumental in demystifying the notion of curiosity.

Novelty and uncertainty differentially drive exploration across development

Across the lifespan, individuals frequently choose between exploiting known rewarding options or exploring unknown alternatives. A large body of work has suggested that children may explore more than adults. However, because novelty and reward uncertainty are often correlated, it is unclear how they differentially influence decision-making across development. Here, children, adolescents, and adults (ages 8-27 years, N = 122) completed an adapted version of a recently developed value-guided decision-making task that decouples novelty and uncertainty. In line with prior studies, we found that exploration decreased with increasing age. Critically, participants of all ages demonstrated a similar bias to select choice options with greater novelty, whereas aversion to reward uncertainty increased into adulthood. Computational modeling of participant choices revealed that whereas adolescents and adults demonstrated attenuated uncertainty aversion for more novel choice options, children's choices were not influenced by reward uncertainty.

Gene expression genetics of the striatum of Diversity Outbred mice

Brain transcriptional variation is a heritable trait that mediates complex behaviors, including addiction. Expression quantitative trait locus (eQTL) mapping reveals genomic regions harboring genetic variants that influence transcript abundance. In this study, we profiled transcript abundance in the striatum of 386 Diversity Outbred (J:DO) mice of both sexes using RNA-Seq. All mice were characterized using a behavioral battery of widely-used exploratory and risk-taking assays prior to transcriptional profiling. We performed eQTL mapping, incorporated the results into a browser-based eQTL viewer, and deposited co-expression network members in GeneWeaver. The eQTL viewer allows researchers to query specific genes to obtain allelic effect plots, analyze SNP associations, assess gene expression correlations, and apply mediation analysis to evaluate whether the regulatory variant is acting through the expression of another gene. GeneWeaver allows multi-species comparison of gene sets using statistical and combinatorial tools. This data resource allows users to find genetic variants that regulate differentially expressed transcripts and place them in the context of other studies of striatal gene expression and function in addiction-related behavior.

Changes in adaptation to time horizons across development

When making decisions, the amount of time remaining matters. When time horizons are long, exploring unknown options can inform later decisions, but when time horizons are short, exploiting known options should be prioritized. While adults and adolescents adapt their exploration in this way, it is unclear when such adaptation emerges and how individuals behave when time horizons are ambiguous, as in many real-life situations. We examined these questions by having 5- to 6-year-olds (N = 43), 11- to 12-year-olds (N = 40), and adult college students (N = 49) in the United States complete a Simplified Horizons Task under short, long, and ambiguous time horizons. Adaptation to time horizons increased with age: older children and adults explored more when horizons were long than when short, and while some younger children adapted to time horizons, younger children overall did not show strong evidence of adapting. Under ambiguous horizons, older children and adults preferred to exploit over explore, while younger children did not show this preference. Thus, adaptation to time horizons is evident by ages 11-12 and may begin to emerge around 5-6 years, and children decrease their tendencies to explore under short and ambiguous time horizons with development. This developmental shift may lead to less learning but more adaptive decision making. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Electrophysiological Markers of Aberrant Cue-Specific Exploration in Hazardous Drinkers

Background

Hazardous drinking is associated with maladaptive alcohol-related decision-making. Existing studies have often focused on how participants learn to exploit familiar cues based on prior reinforcement, but little is known about the mechanisms that drive hazardous drinkers to explore novel alcohol cues when their value is not known.

Methods

We investigated exploration of novel alcohol and non-alcohol cues in hazardous drinkers (N = 27) and control participants (N = 26) during electroencephalography (EEG). A normative computational model with two free parameters was fit to estimate participants' weighting of the future value of exploration and immediate value of exploitation.

Results

Hazardous drinkers demonstrated increased exploration of novel alcohol cues, and conversely, increased probability of exploiting familiar alternatives instead of exploring novel non-alcohol cues. The motivation to explore novel alcohol stimuli in hazardous drinkers was driven by an elevated relative future valuation of uncertain alcohol cues. P3a predicted more exploratory decision policies driven by an enhanced relative future valuation of novel alcohol cues. P3b did not predict choice behavior, but computational parameter estimates suggested that hazardous drinkers with enhanced P3b to alcohol cues were likely to learn to exploit their immediate expected value.

Conclusions

Hazardous drinkers did not display atypical choice behavior, different P3a/P3b amplitudes, or computational estimates to novel non-alcohol cues-diverging from previous studies in addiction showing atypical generalized explore-exploit decisions with non-drug-related cues. These findings reveal that cue-specific neural computations may drive aberrant alcohol-related decision-making in hazardous drinkers-highlighting the importance of drug-relevant cues in studies of decision-making in addiction.

Neurons in human pre-supplementary motor area encode key computations for value-based choice

Adaptive behaviour in real-world environments requires that choices integrate several variables, including the novelty of the options under consideration, their expected value and uncertainty in value estimation. Here, to probe how integration over decision variables occurs during decision-making, we recorded neurons from the human pre-supplementary motor area (preSMA), ventromedial prefrontal cortex and dorsal anterior cingulate. Unlike the other areas, preSMA neurons not only represented separate pre-decision variables for each choice option but also encoded an integrated utility signal for each choice option and, subsequently, the decision itself. Post-decision encoding of variables for the chosen option was more widely distributed and especially prominent in the ventromedial prefrontal cortex. Our findings position the human preSMA as central to the implementation of value-based decisions.

Gene expression genetics of the striatum of Diversity Outbred mice

Brain transcriptional variation is a heritable trait that mediates complex behaviors, including addiction. Expression quantitative trait locus (eQTL) mapping reveals genomic regions harboring genetic variants that influence transcript abundance. In this study, we profiled transcript abundance in the striatum of 386 Diversity Outbred (J:DO) mice of both sexes using RNA-Seq. All mice were characterized using a behavioral battery of widely-used exploratory and risk-taking assays prior to transcriptional profiling. We performed eQTL mapping, incorporated the results into a browser-based eQTL viewer, and deposited co-expression network members in GeneWeaver. The eQTL viewer allows researchers to query specific genes to obtain allelic effect plots, analyze SNP associations, assess gene expression correlations, and apply mediation analysis to evaluate whether the regulatory variant is acting through the expression of another gene. GeneWeaver allows multi-species comparison of gene sets using statistical and combinatorial tools. This data resource allows users to find genetic variants that regulate differentially expressed transcripts and place them in the context of other studies of striatal gene expression and function in addiction-related behavior.

Evidence for dopaminergic involvement in endogenous modulation of pain relief

Relief of ongoing pain is a potent motivator of behavior, directing actions to escape from or reduce potentially harmful stimuli. Whereas endogenous modulation of pain events is well characterized, relatively little is known about the modulation of pain relief and its corresponding neurochemical basis. Here, we studied pain modulation during a probabilistic relief-seeking task (a 'wheel of fortune' gambling task), in which people actively or passively received reduction of a tonic thermal pain stimulus. We found that relief perception was enhanced by active decisions and unpredictability, and greater in high novelty-seeking trait individuals, consistent with a model in which relief is tuned by its informational content. We then probed the roles of dopaminergic and opioidergic signaling, both of which are implicated in relief processing, by embedding the task in a double-blinded cross-over design with administration of the dopamine precursor levodopa and the opioid receptor antagonist naltrexone. We found that levodopa enhanced each of these information-specific aspects of relief modulation but no significant effects of the opioidergic manipulation. These results show that dopaminergic signaling has a key role in modulating the perception of pain relief to optimize motivation and behavior.

Irrelevant Threats Linger and Affect Behavior in High Anxiety

Threat-related information attracts attention and disrupts ongoing behavior, and particularly so for more anxious individuals. Yet, it is unknown how and to what extent threat-related information leave lingering influences on behavior (e.g., by impeding ongoing learning processes). Here, human male and female participants (N = 47) performed probabilistic reinforcement learning tasks where irrelevant distracting faces (neutral, happy, or fearful) were presented together with relevant monetary feedback. Behavioral modeling was combined with fMRI data (N = 27) to explore the neurocomputational bases of learning relevant and irrelevant information. In two separate studies, individuals with high trait anxiety showed increased avoidance of objects previously paired with the combination of neutral monetary feedback and fearful faces (but not neutral or happy faces). Behavioral modeling revealed that high anxiety increased the integration of fearful faces during feedback learning, and fMRI results (regarded as provisional, because of a relatively small sample size) further showed that variance in the prediction error signal, uniquely accounted for by fearful faces, correlated more strongly with activity in the right DLPFC for more anxious individuals. Behavioral and neuronal dissociations indicated that the threat-related distractors did not simply disrupt learning processes. By showing that irrelevant threats exert long-lasting influences on behavior, our results extend previous research that separately showed that anxiety increases learning from aversive feedbacks and distractibility by threat-related information. Our behavioral results, combined with the proposed neurocomputational mechanism, may help explain how increased exposure to irrelevant affective information contributes to the acquisition of maladaptive behaviors in more anxious individuals.SIGNIFICANCE STATEMENT In modern-day society, people are increasingly exposed to various types of irrelevant information (e.g., intruding social media announcements). Yet, the neurocomputational mechanisms influenced by irrelevant information during learning, and their interactions with increasingly distracted personality types are largely unknown. Using a reinforcement learning task, where relevant feedback is presented together with irrelevant distractors (emotional faces), we reveal an interaction between irrelevant threat-related information (fearful faces) and interindividual anxiety levels. fMRI shows provisional evidence for an interaction between anxiety levels and the coupling between activity in the DLPFC and learning signals specifically elicited by fearful faces. Our study reveals how irrelevant threat-related information may become entrenched in the anxious psyche and contribute to long-lasting abnormal behaviors.

A Curiosity-Based Autonomous Navigation Algorithm for Maze Robot

The external reward plays an important role in the reinforcement learning process, and the quality of its design determines the final effect of the algorithm. However, in several real-world scenarios, rewards extrinsic to the agent are extremely sparse. This is particularly evident in mobile robot navigation. To solve this problem, this paper proposes a curiosity-based autonomous navigation algorithm that consists of a reinforcement learning framework and curiosity system. The curiosity system consists of three parts: prediction network, associative memory network, and curiosity rewards. The prediction network predicts the next state. An associative memory network was used to represent the world. Based on the associative memory network, an inference algorithm and distance calibration algorithm were designed. Curiosity rewards were combined with extrinsic rewards as complementary inputs to the Q-learning algorithm. The simulation results show that the algorithm helps the agent reduce repeated exploration of the environment during autonomous navigation. The algorithm also exhibits a better convergence effect.

Real-World Exploration Increases Across Adolescence and Relates to Affect, Risk Taking, and Social Connectivity

Cross-species research suggests that exploratory behaviors increase during adolescence and relate to the social, affective, and risky behaviors characteristic of this developmental stage. However, how these typical adolescent behaviors manifest and relate in real-world settings remains unclear. Using geolocation tracking to quantify exploration—variability in daily movement patterns—over a 3-month period in 58 adolescents and adults (ages 13–27) in New York City, we investigated whether daily exploration varied with age and whether exploration related to social connectivity, risk taking, and momentary positive affect. In our cross-sectional sample, we found an association between daily exploration and age, with individuals near the transition to legal adulthood exhibiting the highest exploration levels. Days of higher exploration were associated with greater positive affect irrespective of age. Higher mean exploration was associated with greater social connectivity in all participants but was linked to higher risk taking selectively among adolescents. Our results highlight the interplay of exploration and socioemotional behaviors across development and suggest that societal norms may modulate their expression in naturalistic contexts.

Novelty, uncertainty, and the looming horizon

Novelty and uncertainty are powerful drivers of exploration that are often conflated. In this issue of Neuron, Cockburn and colleagues dissociate the two and report a key interaction: close to task termination, novel options appear much more attractive relative to uncertain options.

Development and Initial Testing of a Personalized, Adaptive, and Socially Focused Web Tool to Support Physical Activity Among Women in Midlife: Multidisciplinary and User-Centered Design Approach

Background

Women in midlife are vulnerable to developing cardiovascular disease, particularly those who have conditions such as hypertension. Physical activity (PA) can reduce risk, but efforts to promote PA in this population have been only modestly effective. More attention to social influences on PA behavior may be useful, particularly social support and social comparison processes. Activating these processes with digital tools can provide easy access that is flexible to the needs of women in midlife.

Objective

This paper describes the user-centered design processes of developing and conducting initial evaluation of a personalized and adaptive web application, tailored to the social needs of women in midlife. The goal was to gather feedback from the population of interest, before and during the design process.

Methods

This study was conducted in 4 stages. The first and second authors (DA and AFL) developed technical specifications, informed by their experience with the population of interest. We collected feedback on potential content for the web application with women in midlife using both interviews (5/10, 50%; mean age 47.4, SD 6.66 years; mean BMI 35.3, SD 9.55 kg/m²) and surveys (5/10, 50%; mean age 51, SD 6.60 years; mean BMI 32.7, SD 8.39 kg/m²). We used their feedback to inform support messages and peer profiles (ie, sources of social comparison information). Nine members of the behavioral science team and 3 testers unfamiliar with the web application completed internal testing. We conducted naturalistic functionality testing with a different group of women in midlife (n=5; mean age 50, SD 6.26 years; mean BMI 30.1, SD 5.83 kg/m²), who used the web application as intended for 4 days and provided feedback at the end of this period.

Results

Iterative storyboard development resulted in programming specifications for a prototype of the web application. We used content feedback to select and refine the support messages and peer profiles to be added. The following 2 rounds of internal testing identified bugs and other problems regarding the web application’s functioning and full data collection procedure. Problems were addressed or logged for future consideration. Naturalistic functionality testing revealed minimal further problems; findings showed preliminary acceptability of the web application and suggested that women may select different social content across days.

Conclusions

A multidisciplinary and user-centered design approach led to a personalized and adaptive web application, tailored to the social needs of women in midlife. Findings from testing with this population demonstrated the feasibility and acceptability of the new application and supported further development toward its use in daily life. We describe several potential uses of the web application and next steps for its development. We also discuss the lessons learned and offer recommendations for future collaborations between behavioral and computer scientists to develop similar tools.

Novelty and uncertainty regulate the balance between exploration and exploitation through distinct mechanisms in the human brain

Both novelty and uncertainty are potent features guiding exploration; however, they are often experimentally conflated, and an understanding of how they interact to regulate the balance between exploration and exploitation has proved elusive. Using a task designed to decouple the influence of novelty and uncertainty, we identify separable mechanisms through which exploration is directed. We show that uncertainty-directed exploration is sensitive to the prospective benefit offered by new information, whereas novelty-directed exploration is maintained regardless of its potential advantage. Using a computational framework in conjunction with fMRI, we show that uncertainty-directed choice is rooted in an adaptive bias indexing the prospective utility of exploration. In contrast, novelty persistently promotes exploration by optimistically inflating reward expectations while simultaneously dampening uncertainty signals. Our results identify separable neural substrates charged with balancing the explore/exploit trade-off to foster a manageable decomposition of an otherwise intractable problem.

An insula hierarchical network architecture for active interoceptive inference

In the brain, the insular cortex receives a vast amount of interoceptive information, ascending through deep brain structures, from multiple visceral organs. The unique hierarchical and modular architecture of the insula suggests specialization for processing interoceptive afferents. Yet, the biological significance of the insula's neuroanatomical architecture, in relation to deep brain structures, remains obscure. In this opinion piece, we propose the Insula Hierarchical Modular Adaptive Interoception Control (IMAC) model to suggest that insula modules (granular, dysgranular and agranular), forming parallel networks with the prefrontal cortex and striatum, are specialized to form higher order interoceptive representations. These interoceptive representations are recruited in a context-dependent manner to support habitual, model-based and exploratory control of visceral organs and physiological processes. We discuss how insula interoceptive representations may give rise to conscious feelings that best explain lower order deep brain interoceptive representations, and how the insula may serve to defend the body and mind against pathological depression.

The neurocomputational bases of explore-exploit decision-making

Flexible decision-making requires animals to forego immediate rewards (exploitation) and try novel choice options (exploration) to discover if they are preferable to familiar alternatives. Using the same task and a partially observable Markov decision process (POMDP) model to quantify the value of choices, we first determined that the computational basis for managing explore-exploit tradeoffs is conserved across monkeys and humans. We then used fMRI to identify where in the human brain the immediate value of exploitative choices and relative uncertainty about the value of exploratory choices were encoded. Consistent with prior neurophysiological evidence in monkeys, we observed divergent encoding of reward value and uncertainty in prefrontal and parietal regions, including frontopolar cortex, and parallel encoding of these computations in motivational regions including the amygdala, ventral striatum, and orbitofrontal cortex. These results clarify the interplay between prefrontal and motivational circuits that supports adaptive explore-exploit decisions in humans and nonhuman primates.

Novelty preferences depend on goals

People are sometimes drawn to novel items, but other times prefer familiar ones. In the present research we show, though, that both children’s and adults’ preferences for novel versus familiar items depend on their goals. Across four experiments, we showed 4- to 7-year-olds (total N = 498) and adults (total N = 659) pairs of artifacts where one was familiar and the other was novel (e.g., a four-legged chair and ten-legged chair). In Experiment 1, children wanted to have familiar artifacts, but to learn about novel ones. Experiment 2 replicated this pattern using a simpler procedure, and found the same pattern in adults. In Experiment 3, 4- to 6-year-olds and adults more strongly preferred familiar items when choosing which they would rather have than when choosing which they would rather try using. Finally, Experiment 4 replicated adults’ preferences to have familiar items and learn about novel ones with an additional set of items. Together these findings show that preferences for novelty depend on people’s goals. We suggest these effects arise because children and adults are motivated both by the promise of information and the desire for safe options in high commitment decisions that entail risk.

Contributions of expected learning progress and perceptual novelty to curiosity-driven exploration

Exploration is curiosity-driven when it relies on the intrinsic motivation to know rather than on extrinsic rewards. Recent evidence shows that artificial agents perform better on a variety of tasks when their learning is curiosity-driven, and humans often engage in curiosity-driven learning when sampling information from the environment. However, which mechanisms underlie curiosity is still unclear. Here, we let participants freely explore different unknown environments that contained learnable sequences of events with varying degrees of noise and volatility. A hierarchical reinforcement learning model captured how participants were learning in these different kinds of unknown environments, and it also tracked the errors they expected to make and the learning opportunities they were planning to seek. With this computational approach, we show that participants' exploratory behavior is guided by learning progress and perceptual novelty. Moreover, we demonstrate an overall tendency of participants to avoid extreme forms of uncertainty. These findings elucidate the cognitive mechanisms that underlie curiosity-driven exploration of unknown environments. Implications of this novel way of quantifying curiosity within a reinforcement learning framework are discussed.

A bias toward the unknown: individual and environmental factors influencing exploratory behavior

With limited resources, exploring new opportunities is crucial for survival. Exploring novel options, however, comes at the cost of uncertainty. Therefore, there is a trade‐off between exploiting options with a known beneficial outcome and exploring novel options with a potentially higher gain. Computational models have suggested that novelty may promote exploratory behavior by inducing a so‐called novelty bonus through reward‐related processes. So far, few studies have provided behavioral evidence for such a novelty bonus. In this study, we aimed to investigate whether spatial novelty can stimulate exploratory behavior (Experiment 1), and whether age, novelty‐seeking, and reduced action radius or social interactions due to COVID‐19 restrictions influenced the exploration–exploitation trade‐off (Experiment 2). In both experiments, we employed a novel paradigm in which participants made binary decisions between food items, while on rare trials, a surprise option was presented. Results from Experiment 1 are in line with a novelty bonus, with spatial novelty promoting exploratory behavior. In Experiment 2, we found that exploratory behavior declined with age, high novelty seekers made more exploratory choices than low novelty seekers, and participants with a smaller action radius made fewer exploratory choices. These findings are consistent with previous findings in animals and predictions from computational models.

Extrinsic rewards, intrinsic rewards, and non-optimal behavior

The optimality of behavior in experimental settings is usually determined with respect to an extrinsic reward defined by the experimenters. However, actions that do not lead to reward are ubiquitous in many species and in many experimental paradigms. Modern research on decision processes commonly treat non-optimal behaviors as noise, often excluding from analysis animals that do not reach behavioral performance criteria. However, non-optimal behaviors can be a window on important brain processes. Here we explore the evidence that non-optimal behaviors are the consequence of intrinsically motivated actions, related to drives that are different from that of obtaining extrinsic reward. One way of operationally characterizing these drives is by postulating intrinsic rewards associated with them. Behaviors that are apparently non-optimal can be interpreted as the consequence of optimal decisions whose goal is to optimize a combination of intrinsic and extrinsic rewards. We review intrinsic rewards that have been discussed in the literature, and suggest ways of testing their existence and role in shaping animal behavior.

Differential coding of goals and actions in ventral and dorsal corticostriatal circuits during goal-directed behavior

Goal-directed behavior requires identifying objects in the environment that can satisfy internal needs and executing actions to obtain those objects. The current study examines ventral and dorsal corticostriatal circuits that support complementary aspects of goal-directed behavior. We analyze activity from the amygdala, ventral striatum, orbitofrontal cortex, and lateral prefrontal cortex (LPFC) while monkeys perform a three-armed bandit task. Information about chosen stimuli and their value is primarily encoded in the amygdala, ventral striatum, and orbitofrontal cortex, while the spatial information is primarily encoded in the LPFC. Before the options are presented, information about the to-be-chosen stimulus is represented in the amygdala, ventral striatum, and orbitofrontal cortex; at the time of choice, the information is passed to the LPFC to direct a saccade. Thus, learned value information specifying behavioral goals is maintained throughout the ventral corticostriatal circuit, and it is routed through the dorsal circuit at the time actions are selected.

Reinforcement-learning in fronto-striatal circuits

We review the current state of knowledge on the computational and neural mechanisms of reinforcement-learning with a particular focus on fronto-striatal circuits. We divide the literature in this area into five broad research themes: the target of the learning-whether it be learning about the value of stimuli or about the value of actions; the nature and complexity of the algorithm used to drive the learning and inference process; how learned values get converted into choices and associated actions; the nature of state representations, and of other cognitive machinery that support the implementation of various reinforcement-learning operations. An emerging fifth area focuses on how the brain allocates or arbitrates control over different reinforcement-learning sub-systems or "experts". We will outline what is known about the role of the prefrontal cortex and striatum in implementing each of these functions. We then conclude by arguing that it will be necessary to build bridges from algorithmic level descriptions of computational reinforcement-learning to implementational level models to better understand how reinforcement-learning emerges from multiple distributed neural networks in the brain.

The past, present, and future of selection history

The last ten years of attention research have witnessed a revolution, replacing a theoretical dichotomy (top-down vs. bottom-up control) with a trichotomy (biased by current goals, physical salience, and selection history). This third new mechanism of attentional control, selection history, is multifaceted. Some aspects of selection history must be learned over time whereas others reflect much more transient influences. A variety of different learning experiences can shape the attention system, including reward, aversive outcomes, past experience searching for a target, target‒non-target relations, and more. In this review, we provide an overview of the historical forces that led to the proposal of selection history as a distinct mechanism of attentional control. We then propose a formal definition of selection history, with concrete criteria, and identify different components of experience-driven attention that fit within this definition. The bulk of the review is devoted to exploring how these different components relate to one another. We conclude by proposing an integrative account of selection history centered on underlying themes that emerge from our review.

To learn or to gain: neural signatures of exploration in human decision-making

Individuals not only take actions to obtain immediate rewards but also to gain more information to guide future choices. An ideal exploration-exploitation balance is crucial for maximizing reward over the long run. However, the neural signatures of exploration in humans remain unclear. Using quantitative meta-analyses of functional magnetic resonance imaging experiments on exploratory behaviors, we sought to identify the concordant activity pertaining to exploration over a range of experiments. The results revealed that exploration activates concordant brain activity associated with risk (e.g., dorsal medial prefrontal cortex and anterior insula), cognitive control (e.g., dorsolateral prefrontal cortex and inferior frontal gyrus), and motor processing (e.g., premotor cortex). These stereotaxic maps of exploration may indicate that exploration is highly linked to risk processing, but is also specifically associated with regions involved in executive control processes. Although this explanation should be treated as exploratory, these findings support theories positing an important role for the prefrontal-insular-motor cortical network in exploration.

The interplay between domain-general and domain-specific mechanisms during the time-course of verbal associative learning: An event-related potential study

Humans continuously learn new information. Here, we examined the temporal brain dynamics of explicit verbal associative learning between unfamiliar items. In the first experiment, 25 adults learned object-pseudoword associations during a 5-day training program allowing us to track the N400 dynamics across learning blocks within and across days. Successful learning was accompanied by an initial frontal N400 that decreased in amplitude across blocks during the first day and shifted to parietal sites during the last training day. In Experiment 2, we replicated our findings with 38 new participants randomly assigned to a consistent learning or an inconsistent learning group. The N400 amplitude modulations that we found, both within and between learning sessions, are taken to reflect the emergence of novel lexical traces even when learning concerns items for which no semantic information is provided. The shift in N400 topography suggests that different N400 neural generators may contribute to specific word learning steps through a balance between domain-general and language-specific mechanisms.

Vision as oculomotor reward: cognitive contributions to the dynamic control of saccadic eye movements

Humans and other primates are equipped with a foveated visual system. As a consequence, we reorient our fovea to objects and targets in the visual field that are conspicuous or that we consider relevant or worth looking at. These reorientations are achieved by means of saccadic eye movements. Where we saccade to depends on various low-level factors such as a targets' luminance but also crucially on high-level factors like the expected reward or a targets' relevance for perception and subsequent behavior. Here, we review recent findings how the control of saccadic eye movements is influenced by higher-level cognitive processes. We first describe the pathways by which cognitive contributions can influence the neural oculomotor circuit. Second, we summarize what saccade parameters reveal about cognitive mechanisms, particularly saccade latencies, saccade kinematics and changes in saccade gain. Finally, we review findings on what renders a saccade target valuable, as reflected in oculomotor behavior. We emphasize that foveal vision of the target after the saccade can constitute an internal reward for the visual system and that this is reflected in oculomotor dynamics that serve to quickly and accurately provide detailed foveal vision of relevant targets in the visual field.

Distinct motivations to seek out information in healthy individuals and problem gamblers

As massive amounts of information are becoming available to people, understanding the mechanisms underlying information-seeking is more pertinent today than ever. In this study, we investigate the underlying motivations to seek out information in healthy and addicted individuals. We developed a novel decision-making task and a novel computational model which allows dissociating the relative contribution of two motivating factors to seek out information: a desire for novelty and a general desire for knowledge. To investigate whether/how the motivations to seek out information vary between healthy and addicted individuals, in addition to healthy controls we included a sample of individuals with gambling disorder-a form of addiction without the confound of substance consumption and characterized by compulsive gambling. Our results indicate that healthy subjects and problem gamblers adopt distinct information-seeking "modes". Healthy information-seeking behavior was mostly motivated by a desire for novelty. Problem gamblers, on the contrary, displayed reduced novelty-seeking and an increased desire for accumulating knowledge compared to healthy controls. Our findings not only shed new light on the motivations driving healthy and addicted individuals to seek out information, but they also have important implications for the treatment and diagnosis of behavioral addiction.

Balancing exploration and exploitation with information and randomization

Explore-exploit decisions require us to trade off the benefits of exploring unknown options to learn more about them, with exploiting known options, for immediate reward. Such decisions are ubiquitous in nature, but from a computational perspective, they are notoriously hard. There is therefore much interest in how humans and animals make these decisions and recently there has been an explosion of research in this area. Here we provide a biased and incomplete snapshot of this field focusing on the major finding that many organisms use two distinct strategies to solve the explore-exploit dilemma: a bias for information ('directed exploration') and the randomization of choice ('random exploration'). We review evidence for the existence of these strategies, their computational properties, their neural implementations, as well as how directed and random exploration vary over the lifespan. We conclude by highlighting open questions in this field that are ripe to both explore and exploit.

Individual associations of adolescent alcohol use disorder versus cannabis use disorder symptoms in neural prediction error signaling and the response to novelty

Two of the most commonly used illegal substances by adolescents are alcohol and cannabis. Alcohol use disorder (AUD) and cannabis use disorder (CUD) are associated with poorer decision-making in adolescents. In adolescents, level of AUD symptomatology has been negatively associated with striatal reward responsivity. However, little work has explored the relationship with striatal reward prediction error (RPE) representation and the extent to which any augmentation of RPE by novel stimuli is impacted. One-hundred fifty-one adolescents participated in the Novelty Task while undergoing functional magnetic resonance imaging (fMRI). In this task, participants learn to choose novel or non-novel stimuli to gain monetary reward. Level of AUD symptomatology was negatively associated with both optimal decision-making and BOLD response modulation by RPE within striatum and regions of prefrontal cortex. The neural alterations in RPE representation were particularly pronounced when participants were exploring novel stimuli. Level of CUD symptomatology moderated the relationship between novelty propensity and RPE representation within inferior parietal lobule and dorsomedial prefrontal cortex. These data expand on an emerging literature investigating individual associations of AUD symptomatology levels versus CUD symptomatology levels and RPE representation during reinforcement processing and provide insight on the role of neuro-computational processes underlying reinforcement learning/decision-making in adolescents.

Novelty preferences and cocaine-associated cues influence regions associated with the salience network in juvenile female rats

Preferences for novel environments (novelty-seeking) is a risk factor for addiction, with little known about its underlying circuitry. Exposure to drug cues facilitates addiction maintenance, leading us to hypothesize that exposure to a novel environment activates a shared neural circuitry. Stimulation of the D1 receptor in the prelimbic cortex increases responsivity to drug-associated environments. Here, we use D1 receptor overexpression in the prelimbic cortex to probe brain responses to novelty-preferences (in a free-choice paradigm) and cocaine-associated odors following place conditioning. These same cocaine-conditioned odors were used to study neural circuitry with Blood Oxygen Level Dependent (BOLD) activity. D1 overexpressing females had deactivated BOLD signals related to novelty-preferences within the insula cortex and amygdala and activation in the frontal cortex and dopamine cell bodies. BOLD responses to cocaine cues were also sensitive to D1. Control females demonstrated a place preference for cocaine environments with no significant BOLD response, while D1 overexpressing females demonstrated a place aversion and weak BOLD responses to cocaine-conditioned odor cues within the insula cortex. For comparison, we provide data from an earlier study with juvenile males overexpressing D1 that show a strong preference for cocaine and elevated BOLD responses. The results support the use of a pharmacological manipulation (e.g., D1 overexpression) to probe the neural circuitry downstream from the prelimbic cortex.

Human complex exploration strategies are enriched by noradrenaline-modulated heuristics

An exploration-exploitation trade-off, the arbitration between sampling a lesser-known against a known rich option, is thought to be solved using computationally demanding exploration algorithms. Given known limitations in human cognitive resources, we hypothesised the presence of additional cheaper strategies. We examined for such heuristics in choice behaviour where we show this involves a value-free random exploration, that ignores all prior knowledge, and a novelty exploration that targets novel options alone. In a double-blind, placebo-controlled drug study, assessing contributions of dopamine (400 mg amisulpride) and noradrenaline (40 mg propranolol), we show that value-free random exploration is attenuated under the influence of propranolol, but not under amisulpride. Our findings demonstrate that humans deploy distinct computationally cheap exploration strategies and that value-free random exploration is under noradrenergic control.

Hierarchical models of pain: Inference, information-seeking, and adaptive control

Computational models of pain consider how the brain processes nociceptive information and allow mapping neural circuits and networks to cognition and behaviour. To date, they have generally have assumed two largely independent processes: perceptual inference, typically modelled as an approximate Bayesian process, and action control, typically modelled as a reinforcement learning process. However, inference and control are intertwined in complex ways, challenging the clarity of this distinction. Here, we consider how they may comprise a parallel hierarchical architecture that combines inference, information-seeking, and adaptive value-based control. This sheds light on the complex neural architecture of the pain system, and takes us closer to understanding from where pain 'arises' in the brain.

The effect of feedback novelty on neural correlates of feedback processing

It has been suggested that stimulus novelty itself can be rewarding and recent evidence suggests that novelty processing and reward processing share common neural mechanisms. For feedback processing, this can be beneficial as well as detrimental: If novelty lends a rewarding characteristic to a stimulus, then this should particularly decrease the impact of negative feedback. The present study investigated whether such an effect of feedback novelty on feedback processing is reflected in electrophysiological markers of reinforcement learning (feedback-related negativity, FRN) and feedback processing (feedback-P300) in a simple decision-making task. In this task, participants had to chose between two stimuli in a learning trial followed by a novel or a familiar feedback stimulus. Learning from feedback allowed them to optimize their payoff in a later test trial. As expected, we found that the FRN effect, i.e. the difference between the FRN amplitudes after negative and positive feedback, was reduced for novel compared to familiar feedback stimuli. In addition, the amplitude of the feedback-P300 was decreased by feedback novelty, both for the anterior P3a and the posterior P3b. Together, these results indicate that feedback novelty can affect feedback processing as reflected by feedback-related brain activity.

Unravelling the effects of methylphenidate on the dopaminergic and noradrenergic functional circuits

Functional magnetic resonance imaging (fMRI) can be combined with drugs to investigate the system-level functional responses in the brain to such challenges. However, most psychoactive agents act on multiple neurotransmitters, limiting the ability of fMRI to identify functional effects related to actions on discrete pharmacological targets. We recently introduced a multimodal approach, REACT (Receptor-Enriched Analysis of functional Connectivity by Targets), which offers the opportunity to disentangle effects of drugs on different neurotransmitters and clarify the biological mechanisms driving clinical efficacy and side effects of a compound. Here, we focus on methylphenidate (MPH), which binds to the dopamine transporter (DAT) and the norepinephrine transporter (NET), to unravel its effects on dopaminergic and noradrenergic functional circuits in the healthy brain at rest. We then explored the relationship between these target-enriched resting state functional connectivity (FC) maps and inter-individual variability in behavioural responses to a reinforcement-learning task encompassing a novelty manipulation to disentangle the molecular systems underlying specific cognitive/behavioural effects. Our main analysis showed a significant MPH-induced FC increase in sensorimotor areas in the functional circuit associated with DAT. In our exploratory analysis, we found that MPH-induced regional variations in the DAT and NET-enriched FC maps were significantly correlated with some of the inter-individual differences on key behavioural responses associated with the reinforcement-learning task. Our findings show that main MPH-related FC changes at rest can be understood through the distribution of DAT in the brain. Furthermore, they suggest that when compounds have mixed pharmacological profiles, REACT may be able to capture regional functional effects that are underpinned by the same cognitive mechanism but are related to engagement of distinct molecular targets.

Amygdala and ventral tegmental area differentially interact with hippocampus and cortical medial temporal lobe during rest in humans

Neuromodulatory regions that detect salience, such as amygdala and ventral tegmental area (VTA), have distinct effects on memory. Yet, questions remain about how these modulatory regions target subregions across the hippocampus and medial temporal lobe (MTL) cortex. Here, we sought to characterize how VTA and amygdala subregions (i.e., basolateral amygdala and central-medial amygdala) interact with hippocampus head, body, and tail, as well as cortical MTL areas of perirhinal cortex and parahippocampal cortex in a task-free state. To quantify these interactions, we used high-resolution resting state fMRI and characterized pair-wise, partial correlations across regions-of-interest. We found that basolateral amygdala showed greater functional coupling with hippocampus head, hippocampus tail, and perirhinal cortex when compared to either VTA or central-medial amygdala. Furthermore, the VTA showed greater functional coupling with hippocampus tail when compared to central-medial amygdala. There were no significant differences in functional coupling with hippocampus body and parahippocampal cortex. These results support a framework by which neuromodulatory regions do not indiscriminately influence all MTL subregions equally, but rather bias information processing to discrete MTL targets. These findings provide a more specified model of the intrinsic properties of systems underlying MTL neuromodulation. This emphasizes the need to consider heterogeneity both across and within neuromodulatory systems to better understand affective memory.

Novelty Manipulations, Memory Performance, and Predictive Coding: the Role of Unexpectedness

Novelty is central to the study of memory, but the wide range of experimental manipulations aimed to reveal its effects on learning produced inconsistent results. The novelty/encoding hypothesis suggests that novel information undergoes enhanced encoding and thus leads to benefits in memory, especially in recognition performance; however, recent studies cast doubts on this assumption. On the other hand, data from animal studies provided evidence on the robust effects of novelty manipulations on the neurophysiological correlates of memory processes. Conceptualizations and operationalizations of novelty are remarkably variable and were categorized into different subtypes, such as stimulus, context, associative or spatial novelty. Here, we summarize previous findings about the effects of novelty on memory and suggest that predictive coding theories provide a framework that could shed light on the differential influence of novelty manipulations on memory performance. In line with predictive coding theories, we emphasize the role of unexpectedness as a crucial property mediating the behavioral and neural effects of novelty manipulations.

Primate Orbitofrontal Cortex Codes Information Relevant for Managing Explore-Exploit Tradeoffs

Reinforcement learning (RL) refers to the behavioral process of learning to obtain reward and avoid punishment. An important component of RL is managing explore-exploit tradeoffs, which refers to the problem of choosing between exploiting options with known values and exploring unfamiliar options. We examined correlates of this tradeoff, as well as other RL related variables, in orbitofrontal cortex (OFC) while three male monkeys performed a three-armed bandit learning task. During the task, novel choice options periodically replaced familiar options. The values of the novel options were unknown, and the monkeys had to explore them to see if they were better than other currently available options. The identity of the chosen stimulus and the reward outcome were strongly encoded in the responses of single OFC neurons. These two variables define the states and state transitions in our model that are relevant to decision-making. The chosen value of the option and the relative value of exploring that option were encoded at intermediate levels. We also found that OFC value coding was stimulus specific, as opposed to coding value independent of the identity of the option. The location of the option and the value of the current environment were encoded at low levels. Therefore, we found encoding of the variables relevant to learning and managing explore-exploit tradeoffs in OFC. These results are consistent with findings in the ventral striatum and amygdala and show that this monosynaptically connected network plays an important role in learning based on the immediate and future consequences of choices.SIGNIFICANCE STATEMENT Orbitofrontal cortex (OFC) has been implicated in representing the expected values of choices. Here we extend these results and show that OFC also encodes information relevant to managing explore-exploit tradeoffs. Specifically, OFC encodes an exploration bonus, which characterizes the relative value of exploring novel choice options. OFC also strongly encodes the identity of the chosen stimulus, and reward outcomes, which are necessary for computing the value of novel and familiar options.

The role of expecting feedback during decision-making under risk

Sometimes choice is followed by outcome feedback and other times it is not. It remains unknown whether humans prefer gambling when they expect feedback to be revealed. Regarding this question, decision-making theories make alternative predictions. Some theories have proposed that choice is influenced by whether one expects to be disappointed in the future. Given that feedback is sometimes disappointing, these theories predict increased aversion towards gambling when feedback is expected compared to when feedback is not expected. The opposite effect is predicted by theories of curiosity, which postulate reduction of uncertainty as an important behavioural drive. Given that feedback reduces uncertainty, these theories predict that gambling will be favoured when feedback is expected. To examine whether expecting feedback influences gambling behaviour, we recorded functional neuroimaging data while participants performed a novel decision-making task requiring to chose between a sure option and a gamble. Crucially, participants expected to receive feedback in some trials but not in other trials. Consistent with theories of curiosity, we found that expecting feedback increased gambling propensity. At the neural level, at option presentation the increased value of gambling during feedback was reflected in activity in the ventral striatum. This suggests that, together with its established role in signalling reward, the ventral striatum also processes a form of epistemic value. Our study demonstrates that gambling becomes more attractive when feedback is expected and suggests that striatal activity could signal the value of feedback information.