Value dynamics affect choice preparation during decision-making

Ventral frontostriatal circuitry mediates the computation of reinforcement from symbolic gains and losses

Reinforcement learning (RL), particularly in primates, is often driven by symbolic outcomes. However, it is usually studied with primary reinforcers. To examine the neural mechanisms underlying learning from symbolic outcomes, we trained monkeys on a task in which they learned to choose options that led to gains of tokens and avoid choosing options that led to losses of tokens. We then recorded simultaneously from the orbitofrontal cortex (OFC), ventral striatum (VS), amygdala (AMY), and mediodorsal thalamus (MDt). We found that the OFC played a dominant role in coding token outcomes and token prediction errors. The other areas contributed complementary functions, with the VS coding appetitive outcomes and the AMY coding the salience of outcomes. The MDt coded actions and relayed information about tokens between the OFC and VS. Thus, the OFC leads the processing of symbolic RL in the ventral frontostriatal circuitry.

The representation of decision variables in orbitofrontal cortex is longitudinally stable

The computation and comparison of subjective values underlying economic choices rely on the orbitofrontal cortex (OFC). In this area, distinct groups of neurons encode the value of individual options, the binary choice outcome, and the chosen value. These variables capture both the choice input and the choice output, suggesting that the cell groups found in the OFC constitute the building blocks of a decision circuit. Here, we show that this neural circuit is longitudinally stable. Using two-photon calcium imaging, we record from the OFC of mice engaged in a juice-choice task. Imaging of individual cells continues for up to 40 weeks. For each cell and each session pair, we compare activity profiles using cosine similarity, and we assess whether the neuron encodes the same variable in both sessions. We find a high degree of stability and a modest representational drift. Quantitative estimates indicate that this drift would not randomize the circuit within the animal's lifetime.

Neuronal Activity in the Gustatory Cortex during Economic Choice

An economic choice entails computing and comparing the values of individual offers. Offer values are represented in the orbitofrontal cortex (OFC)-an area that participates in value comparison-but it is unknown where offer values are computed in the first place. One possibility is that this computation takes place in OFC. Alternatively, offer values might be computed upstream of OFC. For choices between edible goods, a primary candidate is the gustatory region of the anterior insula (gustatory cortex, GC). Here we recorded from the GC of male rhesus monkeys choosing between different juice types. As a population, neurons in GC represented the flavor, the quantity, and the subjective value of the juice chosen by the animal. These variables were represented by distinct groups of cells and with different time courses. Specifically, chosen value signals emerged shortly after offer presentation, while neurons encoding the chosen juice and the chosen quantity peaked after juice delivery. Surprisingly, neurons in GC did not represent individual offer values in a systematic way. In a computational sense, the variables encoded in GC follow the process of value comparison. Thus our results argue against the hypothesis that offer values are computed in GC. At the same time, signals representing the subjective value of the expected reward indicate that responses in GC are not purely sensory. Thus neuronal responses in GC appear consummatory in nature.

Ventral frontostriatal circuitry mediates the computation of reinforcement from symbolic gains and losses

Reinforcement learning (RL), particularly in primates, is often driven by symbolic outcomes. However, it is usually studied with primary reinforcers. To examine the neural mechanisms underlying learning from symbolic outcomes, we trained monkeys on a task in which they learned to choose options that led to gains of tokens and avoid choosing options that led to losses of tokens. We then recorded simultaneously from the orbitofrontal cortex (OFC), ventral striatum (VS), amygdala (AMY), and the mediodorsal thalamus (MDt). We found that the OFC played a dominant role in coding token outcomes and token prediction errors. The other areas contributed complementary functions with the VS coding appetitive outcomes and the AMY coding the salience of outcomes. The MDt coded actions and relayed information about tokens between the OFC and VS. Thus, OFC leads the process of symbolic reinforcement learning in the ventral frontostriatal circuitry.

Reward Circuit Local Field Potential Modulations Precede Risk Taking

Risk taking behavior is a symptom of multiple neuropsychiatric disorders and often lacks effective treatments. Reward circuitry regions including the amygdala, orbitofrontal cortex, insula, and anterior cingulate have been implicated in risk-taking by neuroimaging studies. Electrophysiological activity associated with risk taking in these regions is not well understood in humans. Further characterizing the neural signalling that underlies risk-taking may provide therapeutic insight into disorders associated with risk-taking. Eleven patients with pharmacoresistant epilepsy who underwent stereotactic electroencephalography with electrodes in the amygdala, orbitofrontal cortex, insula, and/or anterior cingulate participated. Patients participated in a gambling task where they wagered on a visible playing card being higher than a hidden card, betting $5 or $20 on this outcome, while local field potentials were recorded from implanted electrodes. We used cluster-based permutation testing to identify reward prediction error signals by comparing oscillatory power following unexpected and expected rewards. We also used cluster-based permutation testing to compare power preceding high and low bets in high-risk (<50% chance of winning) trials and two-way ANOVA with bet and risk level to identify signals associated with risky, risk averse, and optimized decisions. We used linear mixed effects models to evaluate the relationship between reward prediction error and risky decision signals across trials, and a linear regression model for associations between risky decision signal power and Barratt Impulsiveness Scale scores for each patient. Reward prediction error signals were identified in the amygdala (p=0.0066), anterior cingulate (p=0.0092), and orbitofrontal cortex (p=6.0E-4, p=4.0E-4). Risky decisions were predicted by increased oscillatory power in high-gamma frequency range during card presentation in the orbitofrontal cortex (p=0.0022), and by increased power following bet cue presentation across the theta-to-beta range in the orbitofrontal cortex ( p =0.0022), high-gamma in the anterior cingulate ( p =0.0004), and high-gamma in the insula ( p =0.0014). Risk averse decisions were predicted by decreased orbitofrontal cortex gamma power ( p =2.0E-4). Optimized decisions that maximized earnings were preceded by decreases within the theta to beta range in orbitofrontal cortex ( p =2.0E-4), broad frequencies in amygdala ( p =2.0E-4), and theta to low-gamma in insula ( p =4.0E-4). Insula risky decision power was associated with orbitofrontal cortex high-gamma reward prediction error signal ( p =0.0048) and with patient impulsivity ( p =0.00478). Our findings identify and help characterize reward circuitry activity predictive of risk-taking in humans. These findings may serve as potential biomarkers to inform the development of novel treatment strategies such as closed loop neuromodulation for disorders of risk taking.

The Representation of Decision Variables in Orbitofrontal Cortex is Longitudinally Stable

The computation and comparison of subjective values underlying economic choices rely on the orbitofrontal cortex (OFC). In this area, distinct groups of neurons encode the value of individual options, the binary choice outcome, and the chosen value. These variables capture both the input and the output of the choice process, suggesting that the cell groups found in OFC constitute the building blocks of a decision circuit. Here we show that this neural circuit is longitudinally stable. Using two-photon calcium imaging, we recorded from mice choosing between different juice flavors. Recordings of individual cells continued for up to 20 weeks. For each cell and each pair of sessions, we compared the activity profiles using cosine similarity, and we assessed whether the cell encoded the same variable in both sessions. These analyses revealed a high degree of stability and a modest representational drift. A quantitative estimate indicated this drift would not randomize the circuit within the animal's lifetime.

Neural circuit models for evidence accumulation through choice-selective sequences

Decision making is traditionally thought to be mediated by populations of neurons whose firing rates persistently accumulate evidence across time. However, recent decision-making experiments in rodents have observed neurons across the brain that fire sequentially as a function of spatial position or time, rather than persistently, with the subset of neurons in the sequence depending on the animal's choice. We develop two new candidate circuit models, in which evidence is encoded either in the relative firing rates of two competing chains of neurons or in the network location of a stereotyped pattern ("bump") of neural activity. Encoded evidence is then faithfully transferred between neuronal populations representing different positions or times. Neural recordings from four different brain regions during a decision-making task showed that, during the evidence accumulation period, different brain regions displayed tuning curves consistent with different candidate models for evidence accumulation. This work provides mechanistic models and potential neural substrates for how graded-value information may be precisely accumulated within and transferred between neural populations, a set of computations fundamental to many cognitive operations.

Single basolateral amygdala neurons in macaques exhibit distinct connectional motifs with frontal cortex

Basolateral amygdala (BLA) projects widely across the macaque frontal cortex, and amygdalo-frontal projections are critical for appropriate emotional responding and decision making. While it is appreciated that single BLA neurons branch and project to multiple areas in frontal cortex, the organization and frequency of this branching has yet to be fully characterized. Here, we determined the projection patterns of more than 3,000 macaque BLA neurons. We found that one-third of BLA neurons had two or more distinct projection targets in frontal cortex and subcortical structures. The patterns of single BLA neuron projections to multiple areas were organized into repeating motifs that targeted distinct sets of areas in medial and ventral frontal cortex, indicative of separable BLA networks. Our findings begin to reveal the rich structure of single-neuron connections in the non-human primate brain, providing a neuroanatomical basis for the role of BLA in coordinating brain-wide responses to valent stimuli.