Primate anterior insular cortex represents economic decision variables proposed by prospect theory

Computational Mechanisms Underlying Motivation to Earn Symbolic Reinforcers

Reinforcement learning is a theoretical framework that describes how agents learn to select options that maximize rewards and minimize punishments over time. We often make choices, however, to obtain symbolic reinforcers (e.g., money, points) that are later exchanged for primary reinforcers (e.g., food, drink). Although symbolic reinforcers are ubiquitous in our daily lives, widely used in laboratory tasks because they can be motivating, mechanisms by which they become motivating are less understood. In the present study, we examined how monkeys learn to make choices that maximize fluid rewards through reinforcement with tokens. The question addressed here is how the value of a state, which is a function of multiple task features (e.g., the current number of accumulated tokens, choice options, task epoch, trials since the last delivery of primary reinforcer, etc.), drives value and affects motivation. We constructed a Markov decision process model that computes the value of task states given task features to then correlate with the motivational state of the animal. Fixation times, choice reaction times, and abort frequency were all significantly related to values of task states during the tokens task (n = 5 monkeys, three males and two females). Furthermore, the model makes predictions for how neural responses could change on a moment-by-moment basis relative to changes in the state value. Together, this task and model allow us to capture learning and behavior related to symbolic reinforcement.

Ventral frontostriatal circuitry mediates the computation of reinforcement from symbolic gains and losses

Reinforcement learning (RL), particularly in primates, is often driven by symbolic outcomes. However, it is usually studied with primary reinforcers. To examine the neural mechanisms underlying learning from symbolic outcomes, we trained monkeys on a task in which they learned to choose options that led to gains of tokens and avoid choosing options that led to losses of tokens. We then recorded simultaneously from the orbitofrontal cortex (OFC), ventral striatum (VS), amygdala (AMY), and the mediodorsal thalamus (MDt). We found that the OFC played a dominant role in coding token outcomes and token prediction errors. The other areas contributed complementary functions with the VS coding appetitive outcomes and the AMY coding the salience of outcomes. The MDt coded actions and relayed information about tokens between the OFC and VS. Thus, OFC leads the process of symbolic reinforcement learning in the ventral frontostriatal circuitry.

Intrinsic functional clustering of the macaque insular cortex

The functional organization of the primate insula has been studied using a variety of techniques focussing on regional differences in either architecture, connectivity, or function. These complementary methods offered insights into the complex organization of the insula and proposed distinct parcellation schemes at varying levels of detail and complexity. The advent of imaging techniques that allow non-invasive assessment of structural and functional connectivity, has popularized data-driven connectivity-based parcellation methods to investigate the organization of the human insula. Yet, it remains unclear if the subdivisions derived from these data-driven clustering methods reflect meaningful descriptions of the functional specialization of the insula. In this study, we employed hierarchical clustering to examine the cluster parcellations of the macaque insula. As our aim was exploratory, we examined parcellations consisting of two up to ten clusters. Three different cluster validation methods (fingerprinting, silhouette, elbow) converged on a four-cluster solution as the most optimal representation of our data. Examining functional response properties of these clusters, in addition to their brain-wide functional connectivity suggested a functional specialization related to processing gustatory, somato-motor, vestibular and social visual cues. However, a more detailed functional differentiation aligning with previous functional investigations of insula subfields became evident at higher cluster numbers beyond the proposed optimal four clusters. Overall, our findings demonstrate that resting-state-based hierarchical clustering can provide a meaningful description of the insula's functional organization at some level of detail. Nonetheless, cluster parcellations derived from this method are best combined with data obtained through other modalities, to provide a more comprehensive and detailed account of the insula's complex functional organization.

A longitudinal study of perceived social position and health-related quality of life

BACKGROUND

A large body of evidence suggests that there is a social gradient in the association between perceived social position and various health outcomes. Yet only a fraction of this research uses longitudinal data, and these studies usually rely on two data points in time, consider a single health outcome measure, overlook non-linear effects of perceived social position, and come almost exclusively from the Western welfare democracies.

METHODS

Using data for 1921 individuals from three waves (2008, 2013, 2018) of the Polish Panel Survey (POLPAN), we fit between- and within-individuals hybrid-effects models with cluster-robust standard errors to investigate the association between one's perceived social position (self-placement on a socioeconomic hierarchy scale varying 1 to 10) and subsequent health-related quality of life (HRQoL) measured using the Nottingham Health Profile (NHP) and its six components.

RESULTS

We find that the association between perceived social position and health-related quality of life is larger when estimated between individuals than within individuals, yet in fixed- and hybrid-effects models perceived social position remains significantly and negatively linked with both the aggregated NHP measure as well as with its components such as emotional reaction, physical abilities, sleep, and social isolation. We also identify that starting to perceive oneself at the lower end of the social hierarchy is associated with a deteriorating health-related quality of life but a change at the top of the perceived social hierarchy is not linked with an improvement in NHP scores.

CONCLUSIONS

We provide new evidence on the significant and non-linear links between perceived social position and health-related quality of life and highlight possible pathways linking these two aspects of individuals' lives.

Asymmetric coding of reward prediction errors in human insula and dorsomedial prefrontal cortex

The signed value and unsigned salience of reward prediction errors (RPEs) are critical to understanding reinforcement learning (RL) and cognitive control. Dorsomedial prefrontal cortex (dMPFC) and insula (INS) are key regions for integrating reward and surprise information, but conflicting evidence for both signed and unsigned activity has led to multiple proposals for the nature of RPE representations in these brain areas. Recently developed RL models allow neurons to respond differently to positive and negative RPEs. Here, we use intracranially recorded high frequency activity (HFA) to test whether this flexible asymmetric coding strategy captures RPE coding diversity in human INS and dMPFC. At the region level, we found a bias towards positive RPEs in both areas which paralleled behavioral adaptation. At the local level, we found spatially interleaved neural populations responding to unsigned RPE salience and valence-specific positive and negative RPEs. Furthermore, directional connectivity estimates revealed a leading role of INS in communicating positive and unsigned RPEs to dMPFC. These findings support asymmetric coding across distinct but intermingled neural populations as a core principle of RPE processing and inform theories of the role of dMPFC and INS in RL and cognitive control.

A view-based decision mechanism for rewards in the primate amygdala

Primates make decisions visually by shifting their view from one object to the next, comparing values between objects, and choosing the best reward, even before acting. Here, we show that when monkeys make value-guided choices, amygdala neurons encode their decisions in an abstract, purely internal representation defined by the monkey's current view but not by specific object or reward properties. Across amygdala subdivisions, recorded activity patterns evolved gradually from an object-specific value code to a transient, object-independent code in which currently viewed and last-viewed objects competed to reflect the emerging view-based choice. Using neural-network modeling, we identified a sequence of computations by which amygdala neurons implemented view-based decision making and eventually recovered the chosen object's identity when the monkeys acted on their choice. These findings reveal a neural mechanism in the amygdala that derives object choices from abstract, view-based computations, suggesting an efficient solution for decision problems with many objects.

Electrophysiological population dynamics reveal context dependencies during decision making in human frontal cortex

Evidence from monkeys and humans suggests that the orbitofrontal cortex (OFC) encodes the subjective value of options under consideration during choice. Data from non-human primates suggests that these value signals are context-dependent, representing subjective value in a way influenced by the decision makers' recent experience. Using electrodes distributed throughout cortical and subcortical structures, human epilepsy patients performed an auction task where they repeatedly reported the subjective values they placed on snack food items. High-gamma activity in many cortical and subcortical sites including the OFC positively correlated with subjective value. Other OFC sites showed signals contextually modulated by the subjective value of previously offered goods-a context dependency predicted by theory but not previously observed in humans. These results suggest that value and value-context signals are simultaneously present but separately represented in human frontal cortical activity.

Computational mechanisms underlying motivation to earn symbolic reinforcers

Reinforcement learning (RL) is a theoretical framework that describes how agents learn to select options that maximize rewards and minimize punishments over time. We often make choices, however, to obtain symbolic reinforcers (e.g. money, points) that can later be exchanged for primary reinforcers (e.g. food, drink). Although symbolic reinforcers are motivating, little is understood about the neural or computational mechanisms underlying the motivation to earn them. In the present study, we examined how monkeys learn to make choices that maximize fluid rewards through reinforcement with tokens. The question addressed here is how the value of a state, which is a function of multiple task features (e.g. current number of accumulated tokens, choice options, task epoch, trials since last delivery of primary reinforcer, etc.), drives value and affects motivation. We constructed a Markov decision process model that computes the value of task states given task features to capture the motivational state of the animal. Fixation times, choice reaction times, and abort frequency were all significantly related to values of task states during the tokens task (n=5 monkeys). Furthermore, the model makes predictions for how neural responses could change on a moment-by-moment basis relative to changes in state value. Together, this task and model allow us to capture learning and behavior related to symbolic reinforcement.

Functional characterization of macaque insula using task-based and resting-state fMRI

Neurophysiological investigations over the past decades have demonstrated the involvement of the primate insula in a wide array of sensory, cognitive, affective and regulatory functions, yet the complex functional organization of the insula remains unclear. Here we examined to what extent non-invasive task-based and resting-state fMRI provides support for functional specialization and integration of sensory and motor information in the macaque insula. Task-based fMRI experiments suggested a functional specialization related to processing of ingestive/taste/distaste information in anterior insula, grasping-related sensorimotor responses in middle insula and vestibular information in posterior insula. Visual stimuli depicting social information involving conspecific`s lip-smacking gestures yielded responses in middle and anterior portions of dorsal and ventral insula, overlapping partially with the sensorimotor and ingestive/taste/distaste fields. Functional specialization/integration of the insula was further corroborated by seed-based whole brain resting-state analyses, showing distinct functional connectivity gradients across the anterio-posterior extent of both dorsal and ventral insula. Posterior insula showed functional correlations in particular with vestibular/optic flow network regions, mid-dorsal insula with vestibular/optic flow as well as parieto-frontal regions of the sensorimotor grasping network, mid-ventral insula with social/affiliative network regions in temporal, cingulate and prefrontal cortices and anterior insula with taste and mouth motor networks including premotor and frontal opercular regions.

Pathways to the persistence of drug use despite its adverse consequences

The persistence of drug taking despite its adverse consequences plays a central role in the presentation, diagnosis, and impacts of addiction. Eventual recognition and appraisal of these adverse consequences is central to decisions to reduce or cease use. However, the most appropriate ways of conceptualizing persistence in the face of adverse consequences remain unclear. Here we review evidence that there are at least three pathways to persistent use despite the negative consequences of that use. A cognitive pathway for recognition of adverse consequences, a motivational pathway for valuation of these consequences, and a behavioral pathway for responding to these adverse consequences. These pathways are dynamic, not linear, with multiple possible trajectories between them, and each is sufficient to produce persistence. We describe these pathways, their characteristics, brain cellular and circuit substrates, and we highlight their relevance to different pathways to self- and treatment-guided behavior change.

Systematic comparison of risky choices in humans and monkeys

The past decades have seen tremendous progress in fundamental studies on economic choice in humans. However, elucidation of the underlying neuronal processes requires invasive neurophysiological studies that are met with difficulties in humans. Monkeys as evolutionary closest relatives offer a solution. The animals display sophisticated and well-controllable behavior that allows to implement key constructs of proven economic choice theories. However, the similarity of economic choice between the two species has never been systematically investigated. We investigated compliance with the independence axiom (IA) of expected utility theory as one of the most demanding choice tests and compared IA violations between humans and monkeys. Using generalized linear modeling and cumulative prospect theory (CPT), we found that humans and monkeys made comparable risky choices, although their subjective values (utilities) differed. These results suggest similar fundamental choice mechanism across these primate species and encourage to study their underlying neurophysiological mechanisms.

A neuronal prospect theory model in the brain reward circuitry

Prospect theory, arguably the most prominent theory of choice, is an obvious candidate for neural valuation models. How the activity of individual neurons, a possible computational unit, obeys prospect theory remains unknown. Here, we show, with theoretical accuracy equivalent to that of human neuroimaging studies, that single-neuron activity in four core reward-related cortical and subcortical regions represents the subjective valuation of risky gambles in monkeys. The activity of individual neurons in monkeys passively viewing a lottery reflects the desirability of probabilistic rewards parameterized as a multiplicative combination of utility and probability weighting functions, as in the prospect theory framework. The diverse patterns of valuation signals were not localized but distributed throughout most parts of the reward circuitry. A network model aggregating these signals reconstructed the risk preferences and subjective probability weighting revealed by the animals’ choices. Thus, distributed neural coding explains the computation of subjective valuations under risk.

It is unclear how the activity of individual neurons conform to prospect theory. Here, the authors demonstrate that the activity of single neurons in various reward-related regions in the monkey brain can be described as encoding a multiplicative combination of utility and probability weighting, and that this subjective valuation process is achieved via a distributed coding scheme.