Neural coding of computational factors affecting decision making

Misperceiving Momentum: Computational Mechanisms of Biased Striatal Reward Prediction Errors in Bipolar Disorder

Background

Dysregulated reward processing and mood instability are core features of bipolar disorder that have largely been considered separately, with contradictory findings. We sought to test a mechanistic account that emphasizes an excessive tendency in bipolar disorder to enter recursive cycles in which reward perception is biased by signals that the environment may be changing for the better or worse.

Methods

Participants completed a probabilistic reward task with functional magnetic resonance imaging. Using an influential computational model, we ascertained whether participants with bipolar disorder (n = 21) showed greater striatal tracking of momentum-biased reward prediction errors (RPEs) than matched control participants (n = 21). We conducted psychophysiological interaction analyses to quantify the degree to which each group modulated functional connectivity between the ventral striatum and left anterior insula in response to fluctuations in momentum.

Results

In participants with bipolar disorder, but not control participants, the momentum-biased RPE model accounted for significant additional variance in striatal activity beyond a standard model of veridical RPEs. Compared with control participants, participants with bipolar disorder exhibited lower insular-striatal functional connectivity modulated by momentum-biased RPEs, an effect that was more pronounced as a function of current manic symptoms.

Conclusions

Consistent with existing theory, we found evidence that bipolar disorder is associated with a tendency for momentum to excessively bias striatal tracking of RPEs. We identified impaired insular-striatal connectivity as a possible locus for this propensity. We argue that computational psychiatric approaches that examine momentary shifts in reward and mood dynamics have strong potential for yielding new mechanistic insights and intervention targets.

Art Value Creation and Destruction

I present a theory of creative and destructive value state referring to abstract art. Value is a probabilistic state held as a mixture of its expectation and information forces that coexist in a give-and-take relationship. Expectations are driven by the disclosure of novel information about the value state of various events of desire. Each bit of accumulated information contributes to the improvement of perception up to a threshold level, beyond which begin conscious states. The desire to disclose a value state triggers a triadic system of evaluation which uses concepts, observables and approaches. While the triadic valuation mechanisms can be used to assess various commodities, the scope of this work is limited to the case of artworks, in particular abstract paintings. I assume that art value is basically mediated by the interplay between these value state mechanisms of creation and destruction. Expectations in artwork develop attraction by challenging its contemplator to evaluate (predict) its meaning. Once the relevant information, corresponding to its creative expectations, is acquired (and conditioned), emotional states of indifference, disinterest and desensitization develop.

Prediction errors drive dynamic changes in neural patterns that guide behavior

Learning describes the process by which our internal expectation models of the world are updated by surprising outcomes (prediction errors [PEs]) to improve predictions of future events. However, the mechanisms through which error signals dynamically influence existing neural representations are unknown. Here, we use functional magnetic resonance imaging (fMRI) in humans solving a two-step Markov decision task to investigate changes in neural activation patterns following PEs. Using a dynamic multivariate pattern analysis, we can show that PE-related fMRI responses in error-coding regions predict trial-by-trial changes in multivariate neural patterns in the orbitofrontal cortex, the precuneus, and the ventromedial prefrontal cortex (vmPFC). Importantly, the dynamics of these pattern changes in the vmPFC also predicted upcoming changes in choice strategies and thus highlight the importance of these pattern changes for behavior.

The neural basis of effort valuation: A meta-analysis of functional magnetic resonance imaging studies

Choosing how much effort to expend is critical for everyday decisions. While several neuroimaging studies have examined effort-based decision-making, results have been highly heterogeneous, leaving unclear which brain regions process effort-related costs and integrate them with rewards. We conducted two meta-analyses of functional magnetic resonance imaging data to examine consistent neural correlates of effort demands (23 studies, 15 maps, 549 participants) and net value (15 studies, 11 maps, 428 participants). The pre-supplementary motor area (pre-SMA) scaled positively with pure effort demand, whereas the ventromedial prefrontal cortex (vmPFC) showed the opposite effect. Moreover, regions that have been previously implicated in value integration in other cost domains, such as the vmPFC and ventral striatum, were consistently involved in signaling net value. The opposite response patterns of the pre-SMA and vmPFC imply that they are differentially involved in the representation of effort costs and value integration. These findings provide conclusive evidence that the vmPFC is a central node for net value computation and reveal potential brain targets to treat motivation-related disorders.

Using pharmacological manipulations to study the role of dopamine in human reward functioning: A review of studies in healthy adults

Dopamine (DA) plays a key role in reward processing and is implicated in psychological disorders such as depression, substance use, and schizophrenia. The role of DA in reward processing is an area of highly active research. One approach to this question is drug challenge studies with drugs known to alter DA function. These studies provide good experimental control and can be performed in parallel in laboratory animals and humans. This review aimed to summarize results of studies using pharmacological manipulations of DA in healthy adults. 'Reward' is a complex process, so we separated 'phases' of reward, including anticipation, evaluation of cost and benefits of upcoming reward, execution of actions to obtain reward, pleasure in response to receiving a reward, and reward learning. Results indicated that i) DAergic drugs have different effects on different phases of reward; ii) the relationship between DA and reward functioning appears unlikely to be linear; iii) our ability to detect the effects of DAergic drugs varies depending on whether subjective, behavioral, imaging measures are used.

Dopaminergic modulation of the exploration/exploitation trade-off in human decision-making

Involvement of dopamine in regulating exploration during decision-making has long been hypothesized, but direct causal evidence in humans is still lacking. Here, we use a combination of computational modeling, pharmacological intervention and functional magnetic resonance imaging to address this issue. Thirty-one healthy male participants performed a restless four-armed bandit task in a within-subjects design under three drug conditions: 150 mg of the dopamine precursor L-dopa, 2 mg of the D2 receptor antagonist haloperidol, and placebo. Choices were best explained by an extension of an established Bayesian learning model accounting for perseveration, directed exploration and random exploration. Modeling revealed attenuated directed exploration under L-dopa, while neural signatures of exploration, exploitation and prediction error were unaffected. Instead, L-dopa attenuated neural representations of overall uncertainty in insula and dorsal anterior cingulate cortex. Our results highlight the computational role of these regions in exploration and suggest that dopamine modulates how this circuit tracks accumulating uncertainty during decision-making.

Neural mechanisms for learning self and other ownership

Sense of ownership is a ubiquitous and fundamental aspect of human cognition. Here we used model-based functional magnetic resonance imaging and a novel minimal ownership paradigm to probe the behavioural and neural mechanisms underpinning ownership acquisition for ourselves, friends and strangers. We find a self-ownership bias at multiple levels of behaviour from initial preferences to reaction times and computational learning rates. Ventromedial prefrontal cortex (vmPFC) and anterior cingulate sulcus (ACCs) responded more to self vs. stranger associations, but despite a pervasive neural bias to track self-ownership, no brain area tracked self-ownership exclusively. However, ACC gyrus (ACCg) specifically coded ownership prediction errors for strangers and ownership associative strength for friends and strangers but not for self. Core neural mechanisms for associative learning are biased to learn in reference to self but also engaged when learning in reference to others. In contrast, ACC gyrus exhibits specialization for learning about others.

Separate neural representations of prediction error valence and surprise: Evidence from an fMRI meta-analysis

Learning occurs when an outcome differs from expectations, generating a reward prediction error signal (RPE). The RPE signal has been hypothesized to simultaneously embody the valence of an outcome (better or worse than expected) and its surprise (how far from expectations). Nonetheless, growing evidence suggests that separate representations of the two RPE components exist in the human brain. Meta-analyses provide an opportunity to test this hypothesis and directly probe the extent to which the valence and surprise of the error signal are encoded in separate or overlapping networks. We carried out several meta-analyses on a large set of fMRI studies investigating the neural basis of RPE, locked at decision outcome. We identified two valence learning systems by pooling studies searching for differential neural activity in response to categorical positive-versus-negative outcomes. The first valence network (negative > positive) involved areas regulating alertness and switching behaviours such as the midcingulate cortex, the thalamus and the dorsolateral prefrontal cortex whereas the second valence network (positive > negative) encompassed regions of the human reward circuitry such as the ventral striatum and the ventromedial prefrontal cortex. We also found evidence of a largely distinct surprise-encoding network including the anterior cingulate cortex, anterior insula and dorsal striatum. Together with recent animal and electrophysiological evidence this meta-analysis points to a sequential and distributed encoding of different components of the RPE signal, with potentially distinct functional roles.

Impact of juvenile chronic stress on adult cortico-accumbal function: Implications for cognition and addiction

Repeated exposure to stress during childhood is associated with increased risk for neuropsychiatric illness, substance use disorders and other behavioral problems in adulthood. However, it is not clear how chronic childhood stress can lead to emergence of such a wide range of symptoms and disorders in later life. One possible explanation lies in stress-induced disruption to the development of specific brain regions associated with executive function and reward processing, deficits in which are common to the disorders promoted by childhood stress. Evidence of aberrations in prefrontal cortex and nucleus accumbens function following repeated exposure of juvenile (pre- and adolescent) organisms to a variety of different stressors would account not only for the similarity in symptoms across the wide range of childhood stress-associated mental illnesses, but also their persistence into adulthood in the absence of further stress. Therefore, the goal of this review is to evaluate the current knowledge regarding disruption to executive function and reward processing in adult animals or humans exposed to chronic stress over the juvenile period, and the underlying neurobiology, with particular emphasis on the prefrontal cortex and nucleus accumbens. First, the role of these brain regions in mediating executive function and reward processing is highlighted. Second, the neurobehavioral development of these systems is discussed to illustrate how juvenile stress may exert long-lasting effects on prefrontal cortex-accumbal activity and related behavioral functions. Finally, a critical review of current animal and human findings is presented, which strongly supports the supposition that exposure to chronic stress (particularly social aggression and isolation in animal studies) in the juvenile period produces impairments in executive function in adulthood, especially in working memory and inhibitory control. Chronic juvenile stress also results in aberrations to reward processing and seeking, with increased sensitivity to drugs of abuse particularly noted in animal models, which is in line with greater incidence of substance use disorders seen in clinical studies. These consequences are potentially mediated by monoamine and glutamatergic dysfunction in the prefrontal cortex and nucleus accumbens, providing translatable therapeutic targets. However, the predominant use of male subjects and social-based stressors in preclinical studies points to a clear need for determining how both sex differences and stressor heterogeneity may differentially contribute to stress-induced changes to substrates mediating executive function and reward processing, before the impact of chronic juvenile stress in promoting adult psychopathology can be fully understood.

Brain correlates of the intrinsic subjective cost of effort in sedentary volunteers

One key aspect of motivation is the ability of agents to overcome excessive weighting of intrinsic subjective costs. This contribution aims to analyze the subjective cost of effort and assess its neural correlates in sedentary volunteers. We recruited a sample of 57 subjects who underwent a decision-making task using a prospective, moderate, and sustained physical effort as devaluating factor. Effort discounting followed a hyperbolic function, and individual discounting constants correlated with an indicator of sedentary lifestyle (global physical activity questionnaire; R=-0.302, P=0.033). A subsample of 24 sedentary volunteers received a functional magnetic resonance imaging scan while performing a similar effort-discounting task. BOLD signal of a cluster located in the dorsomedial prefrontal cortex correlated with the subjective value of the pair of options under consideration (Z>2.3, P<0.05; cluster corrected for multiple comparisons for the whole brain). Furthermore, effort-related discounting of reward correlated with the signal of a cluster in the ventrolateral prefrontal cortex (Z>2.3, P<0.05; small volume cluster corrected for a region of interest including the ventral prefrontal cortex and striatum). This study offers empirical data about the intrinsic subjective cost of effort and its neural correlates in sedentary individuals.

Applying principles from the game theory to acute stroke care: Learning from the prisoner's dilemma, stag-hunt, and other strategies

BACKGROUND

Acute stroke care represents a challenge for decision makers. Decisions based on erroneous assessments may generate false expectations of patients and their family members, and potentially inappropriate medical advice. Game theory is the analysis of interactions between individuals to study how conflict and cooperation affect our decisions.

AIMS

We reviewed principles of game theory that could be applied to medical decisions under uncertainty.

SUMMARY

Medical decisions in acute stroke care are usually made under constrains: short period of time, with imperfect clinical information, limit understanding about patients and families' values and beliefs. Game theory brings some strategies to help us manage complex medical situations under uncertainty. For example, it offers a different perspective by encouraging the consideration of different alternatives through the understanding of patients' preferences and the careful evaluation of cognitive distortions when applying 'real-world' data. The stag-hunt game teaches us the importance of trust to strength cooperation for a successful patient-physician interaction that is beyond a good or poor clinical outcome.

CONCLUSIONS

The application of game theory to stroke care may improve our understanding of complex medical situations and help clinicians make practical decisions under uncertainty.

Auditory hedonic phenotypes in dementia: A behavioural and neuroanatomical analysis

Patients with dementia may exhibit abnormally altered liking for environmental sounds and music but such altered auditory hedonic responses have not been studied systematically. Here we addressed this issue in a cohort of 73 patients representing major canonical dementia syndromes (behavioural variant frontotemporal dementia (bvFTD), semantic dementia (SD), progressive nonfluent aphasia (PNFA) amnestic Alzheimer's disease (AD)) using a semi-structured caregiver behavioural questionnaire and voxel-based morphometry (VBM) of patients' brain MR images. Behavioural responses signalling abnormal aversion to environmental sounds, aversion to music or heightened pleasure in music (‘musicophilia’) occurred in around half of the cohort but showed clear syndromic and genetic segregation, occurring in most patients with bvFTD but infrequently in PNFA and more commonly in association with MAPT than C9orf72 mutations. Aversion to sounds was the exclusive auditory phenotype in AD whereas more complex phenotypes including musicophilia were common in bvFTD and SD. Auditory hedonic alterations correlated with grey matter loss in a common, distributed, right-lateralised network including antero-mesial temporal lobe, insula, anterior cingulate and nucleus accumbens. Our findings suggest that abnormalities of auditory hedonic processing are a significant issue in common dementias. Sounds may constitute a novel probe of brain mechanisms for emotional salience coding that are targeted by neurodegenerative disease.

The role of stop-signal probability and expectation in proactive inhibition

The subjective belief of what will happen plays an important role across many cognitive domains, including response inhibition. However, tasks that study inhibition do not distinguish between the processing of objective contextual cues indicating stop-signal probability and the subjective expectation that a stop-signal will or will not occur. Here we investigated the effects of stop-signal probability and the expectation of a stop-signal on proactive inhibition. Twenty participants performed a modified stop-signal anticipation task while being scanned with functional magnetic resonance imaging. At the beginning of each trial, the stop-signal probability was indicated by a cue (0% or > 0%), and participants had to indicate whether they expected a stop-signal to occur (yes/no/don't know). Participants slowed down responding on trials with a > 0% stop-signal probability, but this proactive response slowing was even greater when they expected a stop-signal to occur. Analyses were performed in brain regions previously associated with proactive inhibition. Activation in the striatum, supplementary motor area and left dorsal premotor cortex during the cue period was increased when participants expected a stop-signal to occur. In contrast, activation in the right inferior frontal gyrus and right inferior parietal cortex activity during the stimulus-response period was related to the processing of contextual cues signalling objective stop-signal probability, regardless of expectation. These data show that proactive inhibition depends on both the processing of objective contextual task information and the subjective expectation of stop-signals.

Past makes future: role of pFC in prediction

The pFC enables the essential human capacities for predicting future events and preadapting to them. These capacities rest on both the structure and dynamics of the human pFC. Structurally, pFC, together with posterior association cortex, is at the highest hierarchical level of cortical organization, harboring neural networks that represent complex goal-directed actions. Dynamically, pFC is at the highest level of the perception-action cycle, the circular processing loop through the cortex that interfaces the organism with the environment in the pursuit of goals. In its predictive and preadaptive roles, pFC supports cognitive functions that are critical for the temporal organization of future behavior, including planning, attentional set, working memory, decision-making, and error monitoring. These functions have a common future perspective and are dynamically intertwined in goal-directed action. They all utilize the same neural infrastructure: a vast array of widely distributed, overlapping, and interactive cortical networks of personal memory and semantic knowledge, named cognits, which are formed by synaptic reinforcement in learning and memory acquisition. From this cortex-wide reservoir of memory and knowledge, pFC generates purposeful, goal-directed actions that are preadapted to predicted future events.

Frontal theta as a mechanism for cognitive control

Recent advancements in cognitive neuroscience have afforded a description of neural responses in terms of latent algorithmic operations. However, the adoption of this approach to human scalp electroencephalography (EEG) has been more limited, despite the ability of this methodology to quantify canonical neuronal processes. Here, we provide evidence that theta band activities over the midfrontal cortex appear to reflect a common computation used for realizing the need for cognitive control. Moreover, by virtue of inherent properties of field oscillations, these theta band processes may be used to communicate this need and subsequently implement such control across disparate brain regions. Thus, frontal theta is a compelling candidate mechanism by which emergent processes, such as 'cognitive control', may be biophysically realized.

A common currency for the computation of motivational values in the human striatum

Reward comparison in the brain is thought to be achieved through the use of a 'common currency', implying that reward value representations are computed on a unique scale in the same brain regions regardless of the reward type. Although such a mechanism has been identified in the ventro-medial prefrontal cortex and ventral striatum in the context of decision-making, it is less clear whether it similarly applies to non-choice situations. To answer this question, we scanned 38 participants with fMRI while they were presented with single cues predicting either monetary or erotic rewards, without the need to make a decision. The ventral striatum was the main brain structure to respond to both cues while showing increasing activity with increasing expected reward intensity. Most importantly, the relative response of the striatum to monetary vs erotic cues was correlated with the relative motivational value of these rewards as inferred from reaction times. Similar correlations were observed in a fronto-parietal network known to be involved in attentional focus and motor readiness. Together, our results suggest that striatal reward value signals not only obey to a common currency mechanism in the absence of choice but may also serve as an input to adjust motivated behaviour accordingly.

Neural dynamics of reward probability coding: a Magnetoencephalographic study in humans

Prediction of future rewards and discrepancy between actual and expected outcomes (prediction error) are crucial signals for adaptive behavior. In humans, a number of fMRI studies demonstrated that reward probability modulates these two signals in a large brain network. Yet, the spatio-temporal dynamics underlying the neural coding of reward probability remains unknown. Here, using magnetoencephalography, we investigated the neural dynamics of prediction and reward prediction error computations while subjects learned to associate cues of slot machines with monetary rewards with different probabilities. We showed that event-related magnetic fields (ERFs) arising from the visual cortex coded the expected reward value 155 ms after the cue, demonstrating that reward value signals emerge early in the visual stream. Moreover, a prediction error was reflected in ERF peaking 300 ms after the rewarded outcome and showing decreasing amplitude with higher reward probability. This prediction error signal was generated in a network including the anterior and posterior cingulate cortex. These findings pinpoint the spatio-temporal characteristics underlying reward probability coding. Together, our results provide insights into the neural dynamics underlying the ability to learn probabilistic stimuli-reward contingencies.

Neural substrates underlying effort computation in schizophrenia

The lack of initiative, drive or effort in patients with schizophrenia is linked to marked functional impairments. However, our assessment of effort and motivation is crude, relying on clinical rating scales based largely on patient recall. In order to better understand the neurobiology of effort in schizophrenia, we need more rigorous measurements of this construct. In the behavioural neuroscience literature, decades of work has been carried out developing various paradigms to examine the neural underpinnings of an animal's willingness to expend effort for a reward. Here, we shall review this literature on the nature of paradigms used in rodents to assess effort, as well as those used in humans. Next, the neurobiology of these effort-based decisions will be discussed. We shall then review what is known about effort in schizophrenia, and what might be inferred from experiments done in other human populations. Lastly, we shall discuss future directions of research that may assist in shedding light on the neurobiology of effort cost computations in schizophrenia.