Neurons in the Primate Medial Basal Forebrain Signal Combined Information about Reward Uncertainty, Value, and Punishment Anticipation

Neurons of Macaque Frontal Eye Field Signal Reward-Related Surprise

The frontal eye field (FEF) plays a well-established role in the control of visual attention. The strength of an FEF neuron's response to a visual stimulus presented in its receptive field is enhanced if the stimulus captures spatial attention by virtue of its salience. A stimulus can be rendered salient by cognitive factors as well as by physical attributes. These include surprise. The aim of the present experiment was to determine whether surprise-induced salience would result in enhanced visual-response strength in the FEF. Toward this end, we monitored neuronal activity in two male monkeys while presenting first a visual cue predicting with high probability that the reward delivered at the end of the trial would be good or bad (large or small) and then a visual cue announcing the size of the impending reward with certainty. The second cue usually confirmed but occasionally violated the expectation set up by the first cue. Neurons responded more strongly to the second cue when it violated than when it confirmed expectation. The increase in the firing rate was accompanied by a decrease in spike-count correlation as expected from capture of attention. Although both good surprise and bad surprise induced enhanced firing, the effects appeared to arise from distinct mechanisms as indicated by the fact that the bad-surprise signal appeared at a longer latency than the good-surprise signal and by the fact that the strength of the two signals varied independently across neurons.

Dorsal raphe neurons integrate the values of reward amount, delay, and uncertainty in multi-attribute decision-making

The dorsal raphe nucleus (DRN) is implicated in psychiatric disorders that feature impaired sensitivity to reward amount, impulsivity when facing reward delays, and risk-seeking when confronting reward uncertainty. However, it has been unclear whether and how DRN neurons signal reward amount, reward delay, and reward uncertainty during multi-attribute value-based decision-making, where subjects consider these attributes to make a choice. We recorded DRN neurons as monkeys chose between offers whose attributes, namely expected reward amount, reward delay, and reward uncertainty, varied independently. Many DRN neurons signaled offer attributes, and this population tended to integrate the attributes in a manner that reflected monkeys' preferences for amount, delay, and uncertainty. After decision-making, in response to post-decision feedback, these same neurons signaled signed reward prediction errors, suggesting a broader role in tracking value across task epochs and behavioral contexts. Our data illustrate how the DRN participates in value computations, guiding theories about the role of the DRN in decision-making and psychiatric disease.

Modulation of the p75NTR during Adolescent Alcohol Exposure Prevents Cholinergic Neuronal Atrophy and Associated Acetylcholine Activity and Behavioral Dysfunction

Binge alcohol consumption during adolescence can produce lasting deficits in learning and memory while also increasing the susceptibility to substance use disorders. The adolescent intermittent ethanol (AIE) rodent model mimics human adolescent binge drinking and has identified the nucleus basalis magnocellularis (NbM) as a key site of pathology. The NbM is a critical regulator of prefrontal cortical (PFC) cholinergic function and attention. The cholinergic phenotype is controlled pro/mature neurotrophin receptor activation. We sought to determine if p75NTR activity contributes to the loss of cholinergic phenotype in AIE by using a p75NTR modulator (LM11A-31) to inhibit prodegenerative signaling during ethanol exposure. Male and female rats underwent 5 g/kg ethanol (AIE) or water (CON) exposure following 2-day-on 2-day-off cycles from postnatal day 25-57. A subset of these groups also received a protective dose of LM11A-31 (50 mg/kg) during adolescence. Rats were trained on a sustained attention task (SAT) and behaviorally relevant acetylcholine (ACh) activity was recorded in the PFC with a fluorescent indicator (AChGRAB 3.0). AIE produced learning deficits on the SAT, which were spared with LM11A-31. In addition, PFC ACh activity was blunted by AIE, which LM11A-31 corrected. Investigation of NbM ChAT+ and TrkA+ neuronal expression found that AIE led to a reduction of ChAT+TrkA+ neurons, which again LM11A-31 protected. Taken together, these findings demonstrate the p75NTR activity during AIE treatment is a key regulator of cholinergic degeneration.

Basal Forebrain Cholinergic Neurons Have Specific Characteristics during the Perinatal Period

Cholinergic neurons of the basal forebrain represent the main source of cholinergic innervation of large parts of the neocortex and are involved in adults in the modulation of attention, memory, and arousal. During the first postnatal days, they play a crucial role in the development of cortical neurons and cortical cytoarchitecture. However, their characteristics, during this period have not been studied. To understand how they can fulfill this role, we investigated the morphological and electrophysiological maturation of cholinergic neurons of the substantia innominata-nucleus basalis of Meynert (SI/NBM) complex in the perinatal period in mice. We show that cholinergic neurons, whether or not they express gamma-aminobutyric acid (GABA) as a cotransmitter, are already functional at Embryonic Day 18. Until the end of the first postnatal week, they constitute a single population of neurons with a well developed dendritic tree, a spontaneous activity including bursting periods, and a short-latency response to depolarizations (early-firing). They are excited by both their GABAergic and glutamatergic afferents. During the second postnatal week, a second, less excitable, neuronal population emerges, with a longer delay response to depolarizations (late-firing), together with the hyperpolarizing action of GABAA receptor-mediated currents. This classification into early-firing (40%) and late-firing (60%) neurons is again independent of the coexpression of GABAergic markers. These results strongly suggest that during the first postnatal week, the specific properties of developing SI/NBM cholinergic neurons allow them to spontaneously release acetylcholine (ACh), or ACh and GABA, into the developing cortex.

Modulation of the p75NTR during adolescent alcohol exposure prevents cholinergic neuronal atrophy and associated acetylcholine activity and behavioral dysfunction

Binge alcohol consumption during adolescence produces lasting deficits in learning and memory, while also increasing the susceptibility to substance use disorders. The adolescent intermittent ethanol (AIE) rodent model mimics human adolescent binge drinking and has identified the Nucleus Basalis Magnocellularis (NbM) as a key site of pathology. The NbM is a critical regulator of prefrontal cortical (PFC) cholinergic function and attention. The cholinergic phenotype is controlled pro/mature neurotrophin receptor activation. We sought to determine if p75NTR activity contributes to the loss of cholinergic phenotype in AIE by using a p75NTR modulator (LM11A-31) to inhibit prodegenerative signaling during ethanol exposure. Male and female rats underwent 5g/kg ethanol (AIE) or water (CON) exposure following 2-day-on 2-day-off cycles from PND 25-57. A subset of these groups also received a protective dose of LM11A-31 (50mg/kg) during adolescence. Rats were trained on a sustained attention task (SAT) while recording activity with a fluorescent acetylcholine indicator (AChGRAB 3.0). AIE produced learning deficits on the SAT, which were spared with LM11A-31. In addition, mPFC ACh activity was blunted by AIE, which LM11A-31 corrected. Investigation of NbM ChAT+ and TrkA+ neuronal expression found that AIE led to a reduction of ChAT+TrkA+ neurons, which again LM11A-31 protected. Taken together these findings demonstrate the p75NTR activity during AIE treatment is a key regulator of cholinergic degeneration.

An opponent striatal circuit for distributional reinforcement learning

Machine learning research has achieved large performance gains on a wide range of tasks by expanding the learning target from mean rewards to entire probability distributions of rewards - an approach known as distributional reinforcement learning (RL)1. The mesolimbic dopamine system is thought to underlie RL in the mammalian brain by updating a representation of mean value in the striatum2,3, but little is known about whether, where, and how neurons in this circuit encode information about higher-order moments of reward distributions4. To fill this gap, we used high-density probes (Neuropixels) to acutely record striatal activity from well-trained, water-restricted mice performing a classical conditioning task in which reward mean, reward variance, and stimulus identity were independently manipulated. In contrast to traditional RL accounts, we found robust evidence for abstract encoding of variance in the striatum. Remarkably, chronic ablation of dopamine inputs disorganized these distributional representations in the striatum without interfering with mean value coding. Two-photon calcium imaging and optogenetics revealed that the two major classes of striatal medium spiny neurons - D1 and D2 MSNs - contributed to this code by preferentially encoding the right and left tails of the reward distribution, respectively. We synthesize these findings into a new model of the striatum and mesolimbic dopamine that harnesses the opponency between D1 and D2 MSNs5-15 to reap the computational benefits of distributional RL.

Studying the neural representations of uncertainty

The study of the brain's representations of uncertainty is a central topic in neuroscience. Unlike most quantities of which the neural representation is studied, uncertainty is a property of an observer's beliefs about the world, which poses specific methodological challenges. We analyze how the literature on the neural representations of uncertainty addresses those challenges and distinguish between 'code-driven' and 'correlational' approaches. Code-driven approaches make assumptions about the neural code for representing world states and the associated uncertainty. By contrast, correlational approaches search for relationships between uncertainty and neural activity without constraints on the neural representation of the world state that this uncertainty accompanies. To compare these two approaches, we apply several criteria for neural representations: sensitivity, specificity, invariance and functionality. Our analysis reveals that the two approaches lead to different but complementary findings, shaping new research questions and guiding future experiments.

Cholinergic Reinforcement Signaling Is Impaired by Amyloidosis Prior to Its Synaptic Loss

Alzheimer's disease (AD) is associated with amyloidosis and dysfunction of the cholinergic system, which is crucial for learning and memory. However, the nature of acetylcholine signaling within regions of cholinergic-dependent plasticity and how it changes with experience is poorly understood, much less the impact of amyloidosis on this signaling. Therefore, we optically measure the release profile of acetylcholine to unexpected, predicted, and predictive events in visual cortex (VC)-a site of known cholinergic-dependent plasticity-in a preclinical mouse model of AD that develops amyloidosis. We find that acetylcholine exhibits reinforcement signaling qualities, reporting behaviorally relevant outcomes and displaying release profiles to predictive and predicted events that change as a consequence of experience. We identify three stages of amyloidosis occurring before the degeneration of cholinergic synapses within VC and observe that cholinergic responses in amyloid-bearing mice become impaired over these stages, diverging progressively from age- and sex-matched littermate controls. In particular, amyloidosis degrades the signaling of unexpected rewards and punishments, and attenuates the experience-dependent (1) increase of cholinergic responses to outcome predictive visual cues, and (2) decrease of cholinergic responses to predicted outcomes. Hyperactive spontaneous acetylcholine release occurring transiently at the onset of impaired cholinergic signaling is also observed, further implicating disrupted cholinergic activity as an early functional biomarker in AD. Our findings suggest that acetylcholine acts as a reinforcement signal that is impaired by amyloidosis before pathologic degeneration of the cholinergic system, providing a deeper understanding of the effects of amyloidosis on acetylcholine signaling and informing future interventions for AD.SIGNIFICANCE STATEMENT The cholinergic system is especially vulnerable to the neurotoxic effects of amyloidosis, a hallmark of Alzheimer's disease (AD). Though amyloid-induced cholinergic synaptic loss is thought in part to account for learning and memory impairments in AD, little is known regarding how amyloid impacts signaling of the cholinergic system before its anatomic degeneration. Optical measurement of acetylcholine (ACh) release in a mouse model of AD that develops amyloidosis reveals that ACh signals reinforcement and outcome prediction that is disrupted by amyloidosis before cholinergic degeneration. These observations have important scientific and clinical implications: they implicate ACh signaling as an early functional biomarker, provide a deeper understanding of the action of acetylcholine, and inform on when and how intervention may best ameliorate cognitive decline in AD.

Rewarding-unrewarding prediction signals under a bivalent context in the primate lateral hypothalamus

Animals can expect rewards under equivocal situations. The lateral hypothalamus (LH) is thought to process motivational information by producing valence signals of reward and punishment. Despite rich studies using rodents and non-human primates, these signals have been assessed separately in appetitive and aversive contexts; therefore, it remains unclear what information the LH encodes in equivocal situations. To address this issue, macaque monkeys were conditioned under a bivalent context in which reward and punishment were probabilistically delivered, in addition to appetitive and aversive contexts. The monkeys increased approaching behavior similarly in the bivalent and appetitive contexts as the reward probability increased. They increased avoiding behavior under the bivalent and aversive contexts as the punishment probability increased, but the mean frequency was lower under the bivalent context than under the aversive context. The population activity correlated with these mean behaviors. Moreover, the LH produced fine prediction signals of reward expectation, uncertainty, and predictability consistently in the bivalent and appetitive contexts by recruiting context-independent and context-dependent subpopulations of neurons, while it less produced punishment signals in the aversive and bivalent contexts. Further, neural ensembles encoded context information and "rewarding-unrewarding" and "reward-punishment" valence. These signals may motivate individuals robustly in equivocal environments.

Enhanced endogenous oxytocin signaling in the brain modulates neural responses to social misalignment and promotes conformity in humans: A multi-locus genetic profile approach

The neuropeptide oxytocin (OT) is known to promote social conformity. However, the specific neurocognitive mechanisms underlying OT-induced conformity remain unclear. We aimed to address this gap by examining how genetic variation in the oxytocin receptor gene (OXTR) is linked with behavioral conformity and its underlying neural systems. Specifically, we utilized the genotype-tissue expression database (GTEx) to create a novel multi-locus genetic profile score (MPS) that reflects the level of OXTR expression in the human brain. A total of 194 participants (Neuroimaging N = 50, Behavioral N = 144) performed a novel conformity task in which they viewed a series of word pairs depicting various moral values and virtues widely recognized in the United States. In each trial, participants indicated the relative importance of these words and subsequently learned about the majority opinion. Participants later rated the same word pairs a second time. Changes in participants' ratings between the first and second sessions were measured and analyzed with respect to social feedback, blood oxygen level-dependent (BOLD) functional magnetic resonance imaging (fMRI) signals, and OXTR MPS. We found that participants adjusted their ratings in accordance with the majority opinions. Social misalignment between self and others activated brain areas such as the striatum and the posterior medial frontal cortex (pMFC). However, unlike most findings from previous studies, activation in the pMFC during the inconsistent social feedback negatively, rather than positively, predicted behavioral conformity. Notably, those with higher OXTR MPS had reduced pMFC activation in the face of social misalignment, which led to greater conformity. Our findings suggest that OT may promote conformity by dampening the conflict-related signals in the pMFC. They also show that OXTR MPS may be useful for studying the effect of genes on highly complex human social traits, such as conformity.

Seeking motivation and reward: roles of dopamine, hippocampus and supramammillo-septal pathway

Reinforcement learning and goal-seeking behavior are thought to be mediated by midbrain dopamine neurons. However, little is known about neural substrates of curiosity and exploratory behavior, which occur in the absence of clear goal or reward. This is despite behavioral scientists having long suggested that curiosity and exploratory behaviors are regulated by an innate drive. We refer to such behavior as information-seeking behavior and propose 1) key neural substrates and 2) the concept of environment prediction error as a framework to understand information-seeking processes. The cognitive aspect of information-seeking behavior, including the perception of salience and uncertainty, involves, in part, the pathways from the posterior hypothalamic supramammillary region to the hippocampal formation. The vigor of such behavior is modulated by the following: supramammillary glutamatergic neurons; their projections to medial septal glutamatergic neurons; and the projections of medial septal glutamatergic neurons to ventral tegmental dopaminergic neurons. Phasic responses of dopaminergic neurons are characterized as signaling potentially important stimuli rather than rewards. This paper describes how novel stimuli and uncertainty trigger seeking motivation and how these neural substrates modulate information-seeking behavior.

Different types of uncertainty distinguished by monkey prefrontal neurons

To adapt one's behavior, in a timely manner, to an environment that changes in many different aspects, one must be sensitive to uncertainty about each aspect of the environment. Although the medial prefrontal cortex has been implicated in the representation and reduction of a variety of uncertainties, it is unknown whether different types of uncertainty are distinguished by distinct neuronal populations. To investigate how the prefrontal cortex distinguishes between different types of uncertainty, we recorded neuronal activities from the medial and lateral prefrontal cortices of monkeys performing a visual feedback-based action-learning task in which uncertainty of coming feedback and that of context change varied asynchronously. We found that the activities of two groups of prefrontal cells represented the two different types of uncertainty. These results suggest that different types of uncertainty are represented by distinct neural populations in the prefrontal cortex.

Interactions between ventrolateral prefrontal and anterior cingulate cortex during learning and behavioural change

Hypotheses and beliefs guide credit assignment - the process of determining which previous events or actions caused an outcome. Adaptive hypothesis formation and testing are crucial in uncertain and changing environments in which associations and meanings are volatile. Despite primates' abilities to form and test hypotheses, establishing what is causally responsible for the occurrence of particular outcomes remains a fundamental challenge for credit assignment and learning. Hypotheses about what surprises are due to stochasticity inherent in an environment as opposed to real, systematic changes are necessary for identifying the environment's predictive features, but are often hard to test. We review evidence that two highly interconnected frontal cortical regions, anterior cingulate cortex and ventrolateral prefrontal area 47/12o, provide a biological substrate for linking two crucial components of hypothesis-formation and testing: the control of information seeking and credit assignment. Neuroimaging, targeted disruptions, and neurophysiological studies link an anterior cingulate - 47/12o circuit to generation of exploratory behaviour, non-instrumental information seeking, and interpretation of subsequent feedback in the service of credit assignment. Our observations support the idea that information seeking and credit assignment are linked at the level of neural circuits and explain why this circuit is important for ensuring behaviour is flexible and adaptive.

A primate temporal cortex-zona incerta pathway for novelty seeking

Primates interact with the world by exploring visual objects; they seek opportunities to view novel objects even when these have no extrinsic reward value. How the brain controls this novelty seeking is unknown. Here we show that novelty seeking in monkeys is regulated by the zona incerta (ZI). As monkeys made eye movements to familiar objects to trigger an opportunity to view novel objects, many ZI neurons were preferentially activated by predictions of novel objects before the gaze shift. Low-intensity ZI stimulation facilitated gaze shifts, whereas ZI inactivation reduced novelty seeking. ZI-dependent novelty seeking was not regulated by neurons in the lateral habenula or by many dopamine neurons in the substantia nigra, traditionally associated with reward seeking. But the anterior ventral medial temporal cortex, an area important for object vision and memory, was a prominent source of novelty predictions. These data uncover a functional pathway in the primate brain that regulates novelty seeking.

A prefrontal network integrates preferences for advance information about uncertain rewards and punishments

Humans and animals can be strongly motivated to seek information to resolve uncertainty about rewards and punishments. In particular, despite its clinical and societal relevance, very little is known about information seeking about punishments. We show that attitudes toward information about punishments and rewards are distinct and separable at both behavioral and neuronal levels. We demonstrate the existence of prefrontal neuronal populations that anticipate opportunities to gain information in a relatively valence-specific manner, separately anticipating information about either punishments or rewards. These neurons are located in anatomically interconnected subregions of anterior cingulate cortex (ACC) and ventrolateral prefrontal cortex (vlPFC) in area 12o/47. Unlike ACC, vlPFC also contains a population of neurons that integrate attitudes toward both reward and punishment information, to encode the overall preference for information in a bivalent manner. This cortical network is well suited to mediate information seeking by integrating the desire to resolve uncertainty about multiple, distinct motivational outcomes.

Contribution of the basal forebrain to corticocortical network interactions

Basal forebrain (BF) cholinergic neurons provide the cerebral cortex with acetylcholine. Despite the long-established involvement of these cells in sensory processing, attention, and memory, the mechanisms by which cholinergic signaling regulates cognitive processes remain elusive. In this study, we recorded spiking and local field potential data simultaneously from several locations in the BF, and sites in the orbitofrontal and visual cortex in transgenic ChAT-Cre rats performing a visual discrimination task. We observed distinct differences in the fine spatial distributions of gamma coherence values between specific basalo-cortical and cortico-cortical sites that shifted across task phases. Additionally, cholinergic firing induced spatial changes in cortical gamma power, and optogenetic activation of BF increased coherence between specific cortico-cortical sites, suggesting that the cholinergic system contributes to selective modulation of cortico-cortical circuits. Furthermore, the results suggest that cells in specific BF locations are dynamically recruited across behavioral epochs to coordinate interregional cortical processes underlying cognition.

Common coding of expected value and value uncertainty memories in the prefrontal cortex and basal ganglia output

Recent evidence implicates both basal ganglia and ventrolateral prefrontal cortex (vlPFC) in encoding value memories. However, comparative roles of cortical and basal nodes in value memory are not well understood. Here, single-unit recordings in vlPFC and substantia nigra reticulata (SNr), within macaque monkeys, revealed a larger value signal in SNr that was nevertheless correlated with and had a comparable onset to the vlPFC value signal. The value signal was maintained for many objects (>90) many weeks after reward learning and was resistant to extinction in both regions and to repetition suppression in vlPFC. Both regions showed comparable granularity in encoding expected value and value uncertainty, which was paralleled by enhanced gaze bias during free viewing. The value signal dynamics in SNr could be predicted by combining responses of vlPFC neurons according to their value preferences consistent with a scheme in which cortical neurons reached SNr via direct and indirect pathways.

Excitatory-inhibitory tone shapes decision strategies in a hierarchical neural network model of multi-attribute choice

We are constantly faced with decisions between alternatives defined by multiple attributes, necessitating an evaluation and integration of different information sources. Time-varying signals in multiple brain areas are implicated in decision-making; but we lack a rigorous biophysical description of how basic circuit properties, such as excitatory-inhibitory (E/I) tone and cascading nonlinearities, shape attribute processing and choice behavior. Furthermore, how such properties govern choice performance under varying levels of environmental uncertainty is unknown. We investigated two-attribute, two-alternative decision-making in a dynamical, cascading nonlinear neural network with three layers: an input layer encoding choice alternative attribute values; an intermediate layer of modules processing separate attributes; and a final layer producing the decision. Depending on intermediate layer E/I tone, the network displays distinct regimes characterized by linear (I), convex (II) or concave (III) choice indifference curves. In regimes I and II, each option's attribute information is additively integrated. In regime III, time-varying nonlinear operations amplify the separation between offer distributions by selectively attending to the attribute with the larger differences in input values. At low environmental uncertainty, a linear combination most consistently selects higher valued alternatives. However, at high environmental uncertainty, regime III is more likely than a linear operation to select alternatives with higher value. Furthermore, there are conditions where readout from the intermediate layer could be experimentally indistinguishable from the final layer. Finally, these principles are used to examine multi-attribute decisions in systems with reduced inhibitory tone, leading to predictions of different choice patterns and overall performance between those with restrictions on inhibitory tone and neurotypicals.

Social processing by the primate medial frontal cortex

The primate medial frontal cortex is comprised of several brain regions that are consistently implicated in regulating complex social behaviors. The medial frontal cortex is also critically involved in many non-social behaviors, such as those involved in reward, affective, and decision-making processes, broadly implicating the fundamental role of the medial frontal cortex in internally guided cognition. An essential question therefore is what unique contributions, if any, does the medial frontal cortex make to social behaviors? In this chapter, we outline several neural algorithms necessary for mediating adaptive social interactions and discuss selected evidence from behavioral neurophysiology experiments supporting the role of the medial frontal cortex in implementing these algorithms. By doing so, we primarily focus on research in nonhuman primates and examine several key attributes of the medial frontal cortex. Specifically, we review neuronal substrates in the medial frontal cortex uniquely suitable for enabling social monitoring, observational and vicarious learning, as well as predicting the behaviors of social partners. Moreover, by utilizing the three levels of organization in information processing systems proposed by Marr (1982) and recently adapted by Lockwood, Apps, and Chang (2020) for social information processing, we survey selected social functions of the medial frontal cortex through the lens of socially relevant algorithms and implementations. Overall, this chapter provides a broad overview of the behavioral neurophysiology literature endorsing the importance of socially relevant neural algorithms implemented by the primate medial frontal cortex for regulating social interactions.

Reward uncertainty asymmetrically affects information transmission within the monkey fronto-parietal network

A central hypothesis in research on executive function is that controlled information processing is costly and is allocated according to the behavioral benefits it brings. However, while computational theories predict that the benefits of new information depend on prior uncertainty, the cellular effects of uncertainty on the executive network are incompletely understood. Using simultaneous recordings in monkeys, we describe several mechanisms by which the fronto-parietal network reacts to uncertainty. We show that the variance of expected rewards, independently of the value of the rewards, was encoded in single neuron and population spiking activity and local field potential (LFP) oscillations, and, importantly, asymmetrically affected fronto-parietal information transmission (measured through the coherence between spikes and LFPs). Higher uncertainty selectively enhanced information transmission from the parietal to the frontal lobe and suppressed it in the opposite direction, consistent with Bayesian principles that prioritize sensory information according to a decision maker’s prior uncertainty.

Bahareh Taghizadeh and Nicholas Foley et al. show that individual neuronal responses, population spiking activity, and local field potential oscillations encode the variance of expected rewards independent of their value. They also demonstrate that reward uncertainty asymmetrically affects neuronal transmission within the monkey fronto-parietal network.

How Outcome Uncertainty Mediates Attention, Learning, and Decision-Making

Animals and humans evolved sophisticated nervous systems that endowed them with the ability to form internal-models or beliefs and make predictions about the future to survive and flourish in a world in which future outcomes are often uncertain. Crucial to this capacity is the ability to adjust behavioral and learning policies in response to the level of uncertainty. Until recently, the neuronal mechanisms that could underlie such uncertainty-guided control have been largely unknown. In this review, I discuss newly discovered neuronal circuits in primates that represent uncertainty about future rewards and propose how they guide information-seeking, attention, decision-making, and learning to help us survive in an uncertain world. Lastly, I discuss the possible relevance of these findings to learning in artificial systems.

Tuning movement for sensing in an uncertain world

While animals track or search for targets, sensory organs make small unexplained movements on top of the primary task-related motions. While multiple theories for these movements exist-in that they support infotaxis, gain adaptation, spectral whitening, and high-pass filtering-predicted trajectories show poor fit to measured trajectories. We propose a new theory for these movements called energy-constrained proportional betting, where the probability of moving to a location is proportional to an expectation of how informative it will be balanced against the movement's predicted energetic cost. Trajectories generated in this way show good agreement with measured trajectories of fish tracking an object using electrosense, a mammal and an insect localizing an odor source, and a moth tracking a flower using vision. Our theory unifies the metabolic cost of motion with information theory. It predicts sense organ movements in animals and can prescribe sensor motion for robots to enhance performance.

Manganese-Enhanced MRI in Patients with Multiple Sclerosis

BACKGROUND AND PURPOSE

Cellular uptake of the manganese ion, when administered as a contrast agent for MR imaging, can noninvasively highlight cellular activity and disease processes in both animals and humans. The purpose of this study was to explore the enhancement profile of manganese in patients with multiple sclerosis.

MATERIALS AND METHODS

Mangafodipir is a manganese chelate that was clinically approved for MR imaging of liver lesions. We present a case series of 6 adults with multiple sclerosis who were scanned at baseline with gadolinium, then injected with mangafodipir, and followed at variable time points thereafter.

RESULTS

Fourteen new lesions formed during or shortly before the study, of which 10 demonstrated manganese enhancement of varying intensity, timing, and spatial pattern. One gadolinium-enhancing extra-axial mass, presumably a meningioma, also demonstrated enhancement with manganese. Most interesting, manganese enhancement was detected in lesions that formed in the days after mangafodipir injection, and this enhancement persisted for several weeks, consistent with contrast coming from intracellular uptake of manganese. Some lesions demonstrated a diffuse pattern of manganese enhancement in an area larger than that of both gadolinium enhancement and T2-FLAIR signal abnormality.

CONCLUSIONS

This work demonstrates the first use of a manganese-based contrast agent to enhance MS lesions on MR imaging. Multiple sclerosis lesions were enhanced with a temporal and spatial profile distinct from that of gadolinium. Further experiments are necessary to uncover the mechanism of manganese contrast enhancement as well as cell-specific uptake.

Human decisions about when to act originate within a basal forebrain-nigral circuit

Decision-making studies often focus on brain mechanisms for selecting between goals and actions; however, another important, and often neglected, aspect of decision-making in humans concerns whether, at any given point in time, it is worth making any action at all. We showed that a considerable portion of the variance in when voluntary actions are emitted can be explained by a simple model that that takes into account key features of the current environment. By using ultrahigh-field MRI we identified a multilayered circuit in the human brain originating far beyond the medial frontal areas typically linked to human voluntary action starting in the basal forebrain and brain stem, converging in the dopaminergic midbrain, and only then projecting to striatum and cortex.

Decisions about when to act are critical for survival in humans as in animals, but how a desire is translated into the decision that an action is worth taking at any particular point in time is incompletely understood. Here we show that a simple model developed to explain when animals decide it is worth taking an action also explains a significant portion of the variance in timing observed when humans take voluntary actions. The model focuses on the current environment’s potential for reward, the timing of the individual’s own recent actions, and the outcomes of those actions. We show, by using ultrahigh-field MRI scanning, that in addition to anterior cingulate cortex within medial frontal cortex, a group of subcortical structures including striatum, substantia nigra, basal forebrain (BF), pedunculopontine nucleus (PPN), and habenula (HB) encode trial-by-trial variation in action time. Further analysis of the activity patterns found in each area together with psychophysiological interaction analysis and structural equation modeling suggested a model in which BF integrates contextual information that will influence the decision about when to act and communicates this information, in parallel with PPN and HB influences, to nigrostriatal circuits. It is then in the nigrostriatal circuit that action initiation per se begins.

A Basal Forebrain-Cingulate Circuit in Macaques Decides It Is Time to Act

The medial frontal cortex has been linked to voluntary action, but an explanation of why decisions to act emerge at particular points in time has been lacking. We show that, in macaques, decisions about whether and when to act are predicted by a set of features defining the animal’s current and past context; for example, respectively, cues indicating the current average rate of reward and recent previous voluntary action decisions. We show that activity in two brain areas—the anterior cingulate cortex and basal forebrain—tracks these contextual factors and mediates their effects on behavior in distinct ways. We use focused transcranial ultrasound to selectively and effectively stimulate deep in the brain, even as deep as the basal forebrain, and demonstrate that alteration of activity in the two areas changes decisions about when to act.

•

Likelihood and timing of voluntary action in macaques can be partially predicted

•

Recent experience and present context influence when voluntary action occurs

•

A basal forebrain-cingulate circuit mediated effects of these factors on behavior

•

Stimulation of this circuit by ultrasound changed decisions about when to act

Functional flexibility in cortical circuits

Cortical networks receive a highly variable stream of inputs from internal and external influences, and must flexibly adapt their operations on a short timescale. Recent work has highlighted this state-dependent functional flexibility of cortical circuits and provided initial insights into underlying circuit-level mechanisms. Transitions from quiescent to aroused or task-engaged behavioral states are associated with common motifs of network activity, including changes in correlations and enhanced sensory encoding. Evidence points to a key role for selective activation of specific GABAergic interneuron populations in mediating mode-switching in cortical networks. Finally, inhibitory interneurons may function as a critical target for convergent state-dependent neuromodulatory sculpting of cortical circuits.

Parietal neurons encode information sampling based on decision uncertainty

In natural behavior animals actively gather information that is relevant for learning or actions, but the mechanisms of active sampling are rarely investigated. We tested parietal neurons involved in oculomotor control in a task in which monkeys made saccades to gather visual information before reporting a decision based on the information. We show that the neurons encode, before the saccade, the information gains (reduction in decision uncertainty) that the saccade was expected to bring, correlating with the monkeys’ efficiency in processing the information in the post-saccadic fixation. Informational sensitivity is independent of the neurons’ reward sensitivity, which is unreliable across task contexts, inconsistent with the view that the cells encode economic utility. Instead, we suggest that parietal cells are involved in implementing active sampling policies, showing uncertainty-dependent boosts of neural gain that facilitate the selection of relevant cues and the efficient use of the information delivered by these cues.

Diverse motives for human curiosity

Curiosity-our desire to know-is a fundamental drive in human behaviour, but its mechanisms are poorly understood. A classical question concerns the curiosity motives. What drives individuals to become curious about some but not other sources of information?¹ Here we show that curiosity about probabilistic events depends on multiple aspects of the distribution of these events. Participants (n = 257) performed a task in which they could demand advance information about only one of two randomly selected monetary prizes that contributed to their income. Individuals differed markedly in the extent to which they requested information as a function of the ex ante uncertainty or ex ante value of an individual prize. This heterogeneity was not captured by theoretical models describing curiosity as a desire to learn about the total rewards of a situation^2,3. Instead, it could be explained by an extended model that allowed for attribute-specific anticipatory utility-the savouring of individual components of the eventual reward-and postulates that this utility increased nonlinearly with the certainty of receiving the reward. Parameter values fitting individual choices were consistent for information about gains or losses, suggesting that attribute-specific anticipatory utility captures fundamental heterogeneity in the determinants of curiosity.

Encoding prediction signals during appetitive and aversive Pavlovian conditioning in the primate lateral hypothalamus

The lateral hypothalamus (LH), which plays a role in homeostatic functions such as appetite regulation, is also linked to arousal and motivational behavior. However, little is known about how these components are encoded in the LH. Thus cynomolgus monkeys were conditioned with two distinct contexts, i.e., an appetitive context with available rewards and an aversive context with predicted air puffs. Different LH neuron groups encoded different degrees of expectation, predictability, and risks of rewards in a specific manner. A nearly equal number of one-third of the recorded LH neurons showed a positive or negative correlation between their response to visual conditioned stimuli (CS) that predicted the probabilistic delivery of rewards (0%, 50%, and 100%) and the associative values. For another one-third of recorded neurons, a nearly equal number showed a positive or negative correlation between their responses to rewards [appetitive unconditioned stimulus (US)] and reward predictability. Some neurons exhibited their highest or lowest trace-period responses in the 50% reward trials. These response modulations were represented independently and overlaid on a consistent excitatory or inhibitory response across the conditioning events. LH neurons also showed consistent responses in the aversive context. However, the responses to aversive conditioning events depending on the air puff value and predictability were less common. The multifaceted modulation of consistent activity related to outcome predictions may reflect motivational and arousal signals. Furthermore, it may underlie the role the LH plays in the integration and relay of signals to cortices for adaptive and goal-directed physiological and behavioral responses to environmental changes.

NEW & NOTEWORTHY The lateral hypothalamus (LH) is implicated in motivational and arousal behavior; however, the detailed information carried by single LH neurons remains unclear. We demonstrate that primate LH neurons encode multiple combinations of signals concerning different degrees of expectation, appreciation, and uncertainty of rewards in consistent responses across conditioning events and between different contexts. This multifaceted modulation of activity may underlie the role of the LH as a critical node integrating motivational signals with arousal signals.

Novelty, Salience, and Surprise Timing Are Signaled by Neurons in the Basal Forebrain

The basal forebrain (BF) is a principal source of modulation of the neocortex [1-6] and is thought to regulate cognitive functions such as attention, motivation, and learning by broadcasting information about salience [2, 3, 5, 7-19]. However, events can be salient for multiple reasons-such as novelty, surprise, or reward prediction errors [20-24]-and to date, precisely which salience-related information the BF broadcasts is unclear. Here, we report that the primate BF contains at least two types of neurons that often process salient events in distinct manners: one with phasic burst responses to cues predicting salient events and one with ramping activity anticipating such events. Bursting neurons respond to cues that convey predictions about the magnitude, probability, and timing of primary reinforcements. They also burst to the reinforcement itself, particularly when it is unexpected. However, they do not have a selective response to reinforcement omission (the unexpected absence of an event). Thus, bursting neurons do not convey value-prediction errors but do signal surprise associated with external events. Indeed, they are not limited to processing primary reinforcement: they discriminate fully expected novel visual objects from familiar objects and respond to object-sequence violations. In contrast, ramping neurons predict the timing of many salient, novel, and surprising events. Their ramping activity is highly sensitive to the subjects' confidence in event timing and on average encodes the subjects' surprise after unexpected events occur. These data suggest that the primate BF contains mechanisms to anticipate the timing of a diverse set of important external events (via ramping activity) and to rapidly deploy cognitive resources when these events occur (via short latency bursting).

Specific Basal Forebrain-Cortical Cholinergic Circuits Coordinate Cognitive Operations

Based on recent molecular genetics, as well as functional and quantitative anatomical studies, the basal forebrain (BF) cholinergic projections, once viewed as a diffuse system, are emerging as being remarkably specific in connectivity. Acetylcholine (ACh) can rapidly and selectively modulate activity of specific circuits and ACh release can be coordinated in multiple areas that are related to particular aspects of cognitive processing. This review discusses how a combination of multiple new approaches with more established techniques are being used to finally reveal how cholinergic neurons, together with other BF neurons, provide temporal structure for behavior, contribute to local cortical state regulation, and coordinate activity between different functionally related cortical circuits. ACh selectively modulates dynamics for encoding and attention within individual cortical circuits, allows for important transitions during sleep, and shapes the fidelity of sensory processing by changing the correlation structure of neural firing. The importance of this system for integrated and fluid behavioral function is underscored by its disease-modifying role; the demise of BF cholinergic neurons has long been established in Alzheimer's disease and recent studies have revealed the involvement of the cholinergic system in modulation of anxiety-related circuits. Therefore, the BF cholinergic system plays a pivotal role in modulating the dynamics of the brain during sleep and behavior, as foretold by the intricacies of its anatomical map.

Using preferred fluids and different reward schedules to motivate rhesus macaques (Macaca mulatta) in cognitive tasks

Rhesus macaques (Macaca mulatta) used in behavioural neuroscience are often required to complete cognitively complex tasks, for which a high level of motivation is essential. To induce motivation, researchers may implement fluid-restriction protocols, whereby freely available water is limited, such that fluid can be used as a reward in the laboratory. A variety of different rewards and schedules are used, but there exists a lack of data assessing their effectiveness. In this study, we aimed to quantify fluid preference in rhesus macaques and to use these preferences to compare the motivational quality of different reward schedules: the monkey's previous reward (i.e. the fluid used to reward them in past studies), their new preferred reward, a variable schedule of previous and preferred reward, and a choice between the previous and preferred rewards. We found that it may be possible to reduce the level of restriction if an adequately motivating preferred reward is identified, but that this is dependent on the animal. Each monkey responded differently to both the fluid-preference assessments and to the different reward schedules. As such, monkeys should not be subject to 'blanket' protocols but should be assessed individually to maintain adequate scientific data collection at the least severe level of fluid restriction.

Bridging the gap between rodents and humans: The role of non-human primates in oxytocin research

Oxytocin (OT), a neuropeptide that acts in the brain as a neuromodulator, has been long known to shape maternal physiology and behavior in mammals, however its role in regulating social cognition and behavior in primates has come to the forefront only in the recent decade. Many of the current perspectives on the role of OT in modulating social behavior emerged first from studies in rodents, where invasive techniques with a high degree of precision have permitted the mechanistic dissection of OT-related behaviors, as well as their underlying neural circuits in exquisite detail. In parallel, behavioral and imaging studies in humans have suggested that brain OT may similarly influence human social behavior and neural activity. These studies in rodents and humans have spurred interest in the therapeutic potential of targeting the OT system to remedy deficits in social cognition and behavior that are present across numerous psychiatric disorders. Yet there remains a tremendous gap in our mechanistic understanding of the influence of brain OT on social neural circuitry between rodents and man. In fact, very little is known regarding the neural mechanisms by which exogenous or endogenous OT influences human social cognition, limiting its therapeutic potential. Here we discuss how non-human primates (NHPs) are uniquely positioned to now bridge the gaps in knowledge provided by the precise circuit-level approaches widely used in rodent models and the behavioral, imaging, and clinical studies in humans. This review provides a perspective on what has been achieved, and what can be expected from exploring the role of OT in shaping social behaviors in NHPs in the coming years.

The Basal Forebrain Regulates Global Resting-State fMRI Fluctuations

Patterns of spontaneous brain activity, typically measured in humans at rest with fMRI, are used routinely to assess the brain's functional organization. The mechanisms that generate and coordinate the underlying neural fluctuations are largely unknown. Here we investigate the hypothesis that the nucleus basalis of Meynert (NBM), the principal source of widespread cholinergic and GABAergic projections to the cortex, contributes critically to such activity. We reversibly inactivated two distinct sites of the NBM in macaques while measuring fMRI activity across the brain. We found that inactivation led to strong, regionalized suppression of shared or "global" signal components of cortical fluctuations ipsilateral to the injection. At the same time, the commonly studied resting-state networks retained their spatial structure under this suppression. The results indicate that the NBM contributes selectively to the global component of functional connectivity but plays little if any role in the specific correlations that define resting-state networks.

Anterior cingulate is a source of valence-specific information about value and uncertainty

Anterior cingulate cortex (ACC) is thought to control a wide range of reward, punishment, and uncertainty-related behaviors. However, how it does so is unclear. Here, in a Pavlovian procedure in which monkeys displayed a diverse repertoire of reward-related, punishment-related, and uncertainty-related behaviors, we show that many ACC-neurons represent expected value and uncertainty in a valence-specific manner, signaling value or uncertainty predictions about either rewards or punishments. Other ACC-neurons signal prediction information about rewards and punishments by displaying excitation to both (rather than excitation to one and inhibition to the other). This diversity in valence representations may support the role of ACC in many behavioral states that are either enhanced by reward and punishment (e.g., vigilance) or specific to either reward or punishment (e.g., approach and avoidance). Also, this first demonstration of punishment-uncertainty signals in the brain suggests that ACC could be a target for the treatment of uncertainty-related disorders of mood.

Rewards or punishments elicit diverse behavioral responses; however, the neural circuits underlying such flexibility are unclear. Here Monosov shows that this diversity could be supported by neurons in the anterior cingulate that represent expected value and uncertainty in a valence-specific manner.

Multiple Mechanisms for Processing Reward Uncertainty in the Primate Basal Forebrain

The ability to use information about the uncertainty of future outcomes is critical for adaptive behavior in an uncertain world. We show that the basal forebrain (BF) contains at least two distinct neural-coding strategies to support this capacity. The dorsal-lateral BF, including the ventral pallidum (VP), contains reward-sensitive neurons, some of which are selectively suppressed by uncertain-reward predictions (U(-)). In contrast, the medial BF (mBF) contains reward-sensitive neurons, some of which are selectively enhanced (U(+)) by uncertain-reward predictions. In a two-alternative choice-task, U(-) neurons were selectively suppressed while monkeys chose uncertain options over certain options. During the same choice-epoch, U(+) neurons signaled the subjective reward value of the choice options. Additionally, after the choice was reported, U(+) neurons signaled reward uncertainty until the choice outcome. We suggest that uncertainty-related suppression of VP may participate in the mediation of uncertainty-seeking actions, whereas uncertainty-related enhancement of the mBF may direct cognitive resources to monitor and learn from uncertain-outcomes.

SIGNIFICANCE STATEMENT

To survive in an uncertain world, we must approach uncertainty and learn from it. Here we provide evidence for two mostly distinct mechanisms for processing uncertainty about rewards within different subregions of the primate basal forebrain (BF). We found that uncertainty suppressed the representation of certain (or safe) reward values by some neurons in the dorsal-lateral BF, in regions occupied by the ventral pallidum. This uncertainty-related suppression was evident as monkeys made risky choices. We also found that uncertainty-enhanced the activity of many medial BF neurons, most prominently after the monkeys' choices were completed (as they awaited uncertain outcomes). Based on these findings, we propose that different subregions of the BF could support action and learning under uncertainty in distinct but complimentary manners.

The menagerie of the basal forebrain: how many (neural) species are there, what do they look like, how do they behave and who talks to whom?

The diverse cell-types of the basal forebrain control sleep-wake states, cortical activity and reward processing. Large, slow-firing, cholinergic neurons suppress cortical delta activity and promote cortical plasticity in response to reinforcers. Large, fast-firing, cortically-projecting GABAergic neurons promote wakefulness and fast cortical activity. In particular, parvalbumin/GABAergic neurons promote neocortical gamma band activity. Conversely, excitation of slower-firing somatostatin/GABAergic neurons promotes sleep through inhibition of cortically-projecting neurons. Activation of glutamatergic neurons promotes wakefulness, likely by exciting other cortically-projecting neurons. Similarly, cholinergic neurons indirectly promote wakefulness by excitation of wake-promoting, cortically-projecting GABAergic neurons and/or inhibition of sleep-promoting somatostatin/GABAergic neurons. Both glia and neurons increase the levels of adenosine during prolonged wakefulness. Adenosine presynaptically inhibits glutamatergic inputs to wake-promoting cholinergic and GABAergic/parvalbumin neurons, promoting sleep.

Neurons in the primate dorsal striatum signal the uncertainty of object-reward associations

To learn, obtain reward and survive, humans and other animals must monitor, approach and act on objects that are associated with variable or unknown rewards. However, the neuronal mechanisms that mediate behaviours aimed at uncertain objects are poorly understood. Here we demonstrate that a set of neurons in an internal-capsule bordering regions of the primate dorsal striatum, within the putamen and caudate nucleus, signal the uncertainty of object–reward associations. Their uncertainty responses depend on the presence of objects associated with reward uncertainty and evolve rapidly as monkeys learn novel object–reward associations. Therefore, beyond its established role in mediating actions aimed at known or certain rewards, the dorsal striatum also participates in behaviours aimed at reward-uncertain objects.

The dorsal striatum (DS) is a brain region that is thought to aim actions at certain or known rewards. Here, the authors show that an internal-capsule bordering region of the primate DS signals the uncertainty of object-reward associations, suggesting a novel role for the DS in behavior under uncertainty.

The modulation of savouring by prediction error and its effects on choice

When people anticipate uncertain future outcomes, they often prefer to know their fate in advance. Inspired by an idea in behavioral economics that the anticipation of rewards is itself attractive, we hypothesized that this preference of advance information arises because reward prediction errors carried by such information can boost the level of anticipation. We designed new empirical behavioral studies to test this proposal, and confirmed that subjects preferred advance reward information more strongly when they had to wait for rewards for a longer time. We formulated our proposal in a reinforcement-learning model, and we showed that our model could account for a wide range of existing neuronal and behavioral data, without appealing to ambiguous notions such as an explicit value for information. We suggest that such boosted anticipation significantly drives risk-seeking behaviors, most pertinently in gambling.

DOI:http://dx.doi.org/10.7554/eLife.13747.001

People, pigeons and monkeys often want to know in advance whether they will receive a reward in the future. This behaviour is irrational when individuals pay for costly information that makes no difference to an eventual outcome. One explanation is that individuals seek information because anticipating reward has hedonic value (it produces a feeling of pleasure). Consistent with this, pigeons are more likely to seek information when they have to wait longer for the potential reward. However, existing models cannot account for why this anticipation of rewards leads to irrational information-seeking.

In many situations, animals are uncertain about what is going to happen. Providing new information can produce a “prediction error” that indexes a discrepancy between what is expected and what actually happens. Iigaya et al. have now developed a mathematical model of information-seeking in which anticipation is boosted by this prediction error.

The model accounts for a wide range of previously unexplained data from monkeys and pigeons. It also successfully explains the behaviour of a group of human volunteers from whom Iigaya et al. elicited informational and actual decisions concerning uncertain and delayed rewards. The longer that the participants had to wait for possible rewards, the more avidly they wanted to find out about them. Further research is now needed to investigate the neural underpinnings of anticipation and its boosting by prediction errors.

DOI:http://dx.doi.org/10.7554/eLife.13747.002