Prediction error for free monetary reward in the human prefrontal cortex

The Role of the Cerebellum in Learning to Predict Reward: Evidence from Cerebellar Ataxia

Recent findings in animals have challenged the traditional view of the cerebellum solely as the site of motor control, suggesting that the cerebellum may also be important for learning to predict reward from trial-and-error feedback. Yet, evidence for the role of the cerebellum in reward learning in humans is lacking. Moreover, open questions remain about which specific aspects of reward learning the cerebellum may contribute to. Here we address this gap through an investigation of multiple forms of reward learning in individuals with cerebellum dysfunction, represented by cerebellar ataxia cases. Nineteen participants with cerebellar ataxia and 57 age- and sex-matched healthy controls completed two separate tasks that required learning about reward contingencies from trial-and-error. To probe the selectivity of reward learning processes, the tasks differed in their underlying structure: while one task measured incremental reward learning ability alone, the other allowed participants to use an alternative learning strategy based on episodic memory alongside incremental reward learning. We found that individuals with cerebellar ataxia were profoundly impaired at reward learning from trial-and-error feedback on both tasks, but retained the ability to learn to predict reward based on episodic memory. These findings provide evidence from humans for a specific and necessary role for the cerebellum in incremental learning of reward associations based on reinforcement. More broadly, the findings suggest that alongside its role in motor learning, the cerebellum likely operates in concert with the basal ganglia to support reinforcement learning from reward.

Neural substrates of reward anticipation and outcome in schizophrenia: a meta-analysis of fMRI findings in the monetary incentive delay task

Dysfunction of the mesocorticolimbic dopaminergic reward system is a core feature of schizophrenia (SZ), yet its precise contributions to different stages of reward processing and their relevance to disease symptomology are not fully understood. We performed a coordinate-based meta-analysis, using the monetary incentive delay task, to identify which brain regions are implicated in different reward phases in functional magnetic resonance imaging in SZ. A total of 17 studies (368 SZ and 428 controls) were included in the reward anticipation, and 10 studies (229 SZ and 281 controls) were included in the reward outcome. Our meta-analysis revealed that during anticipation, patients showed hypoactivation in the striatum, anterior cingulate cortex, median cingulate cortex (MCC), amygdala, precentral gyrus, and superior temporal gyrus compared with controls. Striatum hypoactivation was negatively associated with negative symptoms and positively associated with the proportion of second-generation antipsychotic users (percentage of SGA users). During outcome, patients displayed hyperactivation in the striatum, insula, amygdala, hippocampus, parahippocampal gyrus, cerebellum, postcentral gyrus, and MCC, and hypoactivation in the dorsolateral prefrontal cortex (DLPFC) and medial prefrontal cortex (mPFC). Hypoactivity of mPFC during outcome was negatively associated with positive symptoms. Moderator analysis showed that the percentage of SGA users was a significant moderator of the association between symptom severity and brain activity in both the anticipation and outcome stages. Our findings identified the neural substrates for different reward phases in SZ and may help explain the neuropathological mechanisms underlying reward processing deficits in the disorder.

Neural correlates of risky decision making in Parkinson’s disease patients with impulse control disorders

Some patients with Parkinson's disease (PD) experience impulse control disorders (ICDs), characterized by deficient voluntary control over impulses, drives, or temptations regarding excessive hedonic behavior. The present study aimed to better understand the neural basis of impulsive, risky decision making in PD patients with ICDs by disentangling potential dysfunctions in decision and outcome mechanisms. We collected fMRI data from 20 patients with ICDs and 28 without ICDs performing an information gathering task. Patients viewed sequences of bead colors drawn from hidden urns and were instructed to infer the majority bead color in each urn. With each new bead, they could choose to either seek more evidence by drawing another bead (draw choice) or make an urn-inference (urn choice followed by feedback). We manipulated risk via the probability of bead color splits (80/20 vs. 60/40) and potential loss following an incorrect inference ($10 vs. $0). Patients also completed the Barratt Impulsiveness Scale (BIS) to assess impulsivity. Patients with ICDs showed greater urn choice-specific activation in the right middle frontal gyrus, overlapping the dorsal premotor cortex. Across all patients, fewer draw choices (i.e., more impulsivity) were associated with greater activation during both decision making and outcome processing in a variety of frontal and parietal areas, cerebellum, and bilateral striatum. Our findings demonstrate that ICDs in PD are associated with differences in neural processing of risk-related information and outcomes, implicating both reward and sensorimotor dopaminergic pathways.

Abnormal white matter structure in hoarding disorder

Although preliminary neuroimaging research suggests that patients with hoarding disorder (HD) show widespread abnormal task-related activity in the brain, there has been no research on alterations in the white matter tracts in these patients. The aim of this study was to investigate the characteristics of the major white matter tracts in patients with HD. Tract-based spatial statistics were used to search for white matter tract abnormalities throughout the brain in 25 patients with HD and 36 healthy controls. Post hoc analysis of regions of interest was performed to detect correlations with clinical features. Compared with the controls, patients with HD showed decreased fractional anisotropy and increased radial diffusivity in anatomically widespread white matter tracts. Post hoc analysis of regions of interest revealed a significant negative correlation between the severity of hoarding symptoms and fractional anisotropy in the left anterior limb of the internal capsule and a positive correlation between the severity of these symptoms and radial diffusivity in the right anterior thalamic radiation. Patients with HD showed a broad range of alterations in the frontal white matter tracts, including the frontothalamic circuit, frontoparietal network, and frontolimbic pathway. The findings of this study indicate associations between frontal white matter abnormalities related to the severity of hoarding symptoms in HD and the cortical regions involved in cognitive dysfunction. The insights provided would be useful for understanding the neurobiological basis of HD.

Reward signals in the cerebellum: origins, targets, and functional implications

The cerebellum has long been proposed to play a role in cognitive function, although this has remained controversial. This idea has received renewed support with the recent discovery that signals associated with reward can be observed in the cerebellar circuitry, particularly in goal-directed learning tasks involving an interplay between the cerebellar cortex, basal ganglia, and cerebral cortex. Remarkably, a wide range of reward contingencies-including reward expectation, delivery, size, and omission-can be encoded by specific circuit elements in a manner that reflects the microzonal organization of the cerebellar cortex. The facts that reward signals have been observed in both the mossy fiber and climbing fiber input pathways to the cerebellar cortex and that their convergence may trigger plasticity in Purkinje cells suggest that these interactions may be crucial for the role of the cerebellar cortex in learned behavior. These findings strengthen the emerging consensus that the cerebellum plays a pivotal role in shaping cognitive processing and suggest that the cerebellum may combine both supervised learning and reinforcement learning to optimize goal-directed action. We make specific predictions about how cerebellar circuits can work in concert with the basal ganglia to guide different stages of learning.

Schizophrenia Patients Show Largely Similar Salience Signaling Compared to Healthy Controls in an Observational Task Environment

Recent evidence suggests that the aberrant signaling of salience is associated with psychotic illness. Salience, however, can take many forms in task environments. For example, salience may refer to any of the following: (1) the valence of an outcome, (2) outcomes that are unexpected, called reward prediction errors (PEs), or (3) cues associated with uncertain outcomes. Here, we measure brain responses to different forms of salience in the context of a passive PE-signaling task, testing whether patients with schizophrenia (SZ) showed aberrant signaling of particular types of salience. We acquired event-related MRI data from 29 SZ patients and 23 controls during the performance of a passive outcome prediction task. Across groups, we found that the anterior insula and posterior parietal cortices were activated to multiple different types of salience, including PE magnitude and heightened levels of uncertainty. However, BOLD activation to salient events was not significantly different between patients and controls in many regions, including the insula, posterior parietal cortices, and default mode network nodes. Such results suggest that deficiencies in salience processing in SZ may not result from an impaired ability to signal salience per se, but instead the ability to use such signals to guide future actions. Notably, no between-group differences were observed in BOLD signal changes associated with PE-signaling in the striatum. However, positive symptom severity was found to significantly correlate with the magnitudes of salience contrasts in default mode network nodes. Our results suggest that, in an observational environment, SZ patients may show an intact ability to activate striatal and cortical regions to rewarding and non-rewarding salient events. Furthermore, reduced deactivation of a hypothesized default mode network node for SZ participants with high levels of positive symptoms, following salient events, point to abnormalities in interactions of the salience network with other brain networks, and their potential importance to positive symptoms.

The basal ganglia and the cerebellum in human emotion

The basal ganglia (BG) and the cerebellum historically have been relegated to a functional role in producing or modulating motor output. Recent research, however, has emphasized the importance of these subcortical structures in multiple functional domains, including affective processes such as emotion recognition, subjective feeling elicitation and reward valuation. The pathways through the thalamus that connect the BG and cerebellum directly to each other and with extensive regions of the cortex provide a structural basis for their combined influence on limbic function. By regulating cortical oscillations to guide learning and strengthening rewarded behaviors or thought patterns to achieve a desired goal state, these regions can shape the way an individual processes emotional stimuli. This review will discuss the basic structure and function of the BG and cerebellum and propose an updated view of their functional role in human affective processing.

Maladaptive Avoidance Learning in the Orbitofrontal Cortex in Adolescents With Major Depression

BACKGROUND

Understanding the mechanisms in the brain's incentive network that give rise to symptoms of major depressive disorder (MDD) during adolescence provides new perspectives to address MDD in early stages of development. This functional magnetic resonance imaging study determines whether instrumental vigor and brain responses to appetitive and aversive monetary incentives are altered in adolescent MDD and associated with symptom severity.

METHODS

Adolescents with moderate to severe MDD (n = 30, mean age [SD] = 16.1 [1.4] years) and healthy control subjects (n = 33, mean age = 16.2 [1.9] years) matched for age, sex, and IQ performed a monetary incentive delay task. During outcome presentation, prediction error signals were used to study the response and coupling of the incentive network during learning of cue-outcome associations. A computational reinforcement model was used to assess adaptation of response vigor. Brain responses and effective connectivity to model-derived prediction errors were assessed and related to depression severity and anhedonia levels.

RESULTS

Participants with MDD behaved according to a more simplistic learning model and exhibited slower learning. Effective connectivity analysis of functional magnetic resonance imaging data revealed that impaired loss error processing in the orbitofrontal cortex was associated with aberrant gain control. Anhedonia scores correlated with loss-related error signals in the posterior insula and habenula.

CONCLUSIONS

Adolescent MDD is selectively related to impaired processing of error signals during loss, but not reward, in the orbitofrontal cortex. Aberrant evaluation of loss outcomes might reflect an early mechanism of how negative bias and helplessness manifest in the brain. This approach sheds light on pathomechanisms in MDD and may improve early diagnosis and treatment selection.

Matched neurofeedback during fMRI differentially activates reward-related circuits in active and sham groups

BACKGROUND AND PURPOSE

Functional MRI neurofeedback (fMRI-nf) leverages the brain's ability to self-regulate its own activity. However, self-regulation processes engaged during fMRI-nf are incompletely understood. Here, we used matched feedback in an fMRI-nf experimental protocol to investigate whether brain processes recognize true neurofeedback signals.

METHODS

We implemented an existing fMRI-nf protocol to train lateralized motor activity using a finger-tap task in conjunction with real-time feedback. Twelve healthy, right-handed, adult participants were assigned into age- and sex-matched active and sham study groups. Matched participant pairs received the same visual feedback, based on brain activity of the participant from the active group. We compared group-averaged activation maps before, during, and after neurofeedback, and analyzed changes in lateralized motor activity due to neurofeedback.

RESULTS

Active and sham groups demonstrated different brain activation to the same feedback during neurofeedback. In particular, there was higher activation in visual cortex, secondary somatosensory cortex, and right inferior frontal gyrus in the active group compared to the sham group. Conversely, sham participants demonstrated higher activation in anterior cingulate cortex, left frontal pole, and posterior superior temporal gyrus. Despite differing brain activations during neurofeedback, neither group demonstrated significant improvement in lateralized motor activity from pre to postfeedback scan in the same session. We also observed no significant difference between pre and postfeedback activation maps, suggesting that no significant finger-tap related functional reorganization had occurred.

CONCLUSIONS

These findings suggest that fMRI neurofeedback paradigms that monitor or incorporate activity from regions reported here would provide enhanced efficacy for research investigation and clinical intervention.

Cerebellar Dysfunction in Autism Spectrum Disorders: Deriving Mechanistic Insights from an Internal Model Framework

Autism spectrum disorders (ASD) are highly prevalent neurodevelopmental disorders; however, the neurobiological mechanisms underlying disordered behavior in ASD remain poorly understood. Notably, individuals with ASD have demonstrated difficulties generating implicitly derived behavioral predictions and adaptations. Although many brain regions are involved in these processes, the cerebellum contributes an outsized role to these behavioral functions. Consistent with this prominent role, cerebellar dysfunction has been increasingly implicated in ASD. In this review, we will utilize the foundational, theoretical contributions of the late neuroscientist Masao Ito to establish an internal model framework for the cerebellar contribution to ASD-relevant behavioral predictions and adaptations. Additionally, we will also explore and then apply his key experimental contributions towards an improved, mechanistic understanding of the contribution of cerebellar dysfunction to ASD.

Basal ganglia lateralization in different types of reward

Reward processing is a fundamental human activity. The basal ganglia are recognized for their role in reward processes; however, specific roles of the different nuclei (e.g., nucleus accumbens, caudate, putamen and globus pallidus) remain unclear. Using quantitative meta-analyses we assessed whole-brain and basal ganglia specific contributions to money, erotic, and food reward processing. We analyzed data from 190 fMRI studies which reported stereotaxic coordinates of whole-brain, within-group results from healthy adult participants. Results showed concordance in overlapping and distinct cortical and sub-cortical brain regions as a function of reward type. Common to all reward types was concordance in basal ganglia nuclei, with distinct differences in hemispheric dominance and spatial extent in response to the different reward types. Food reward processing favored the right hemisphere; erotic rewards favored the right lateral globus pallidus and left caudate body. Money rewards engaged the basal ganglia bilaterally including its most anterior part, nucleus accumbens. We conclude by proposing a model of common reward processing in the basal ganglia and separate models for money, erotic, and food rewards.

Entorhinal and ventromedial prefrontal cortices abstract and generalize the structure of reinforcement learning problems

Knowledge of the structure of a problem, such as relationships between stimuli, enables rapid learning and flexible inference. Humans and other animals can abstract this structural knowledge and generalize it to solve new problems. For example, in spatial reasoning, shortest-path inferences are immediate in new environments. Spatial structural transfer is mediated by cells in entorhinal and (in humans) medial prefrontal cortices, which maintain their co-activation structure across different environments and behavioral states. Here, using fMRI, we show that entorhinal and ventromedial prefrontal cortex (vmPFC) representations perform a much broader role in generalizing the structure of problems. We introduce a task-remapping paradigm, where subjects solve multiple reinforcement learning (RL) problems differing in structural or sensory properties. We show that, as with space, entorhinal representations are preserved across different RL problems only if task structure is preserved. In vmPFC and ventral striatum, representations of prediction error also depend on task structure.

Enhanced neural responses in specific phases of reward processing in individuals with Internet gaming disorder

BACKGROUND AND AIMS

Internet gaming disorder (IGD) has become a global health problem. The self-regulation model noted that a shift to reward system, whether due to overwhelming reward-seeking or impaired control, can lead to self-regulation failures, e.g., addiction. The present study focused on the reward processing of IGD, aiming to provide insights into the etiology of IGD. Reward processing includes three phases: reward anticipation, outcome monitoring and choice evaluation. However, it is not clear which phases of reward processing are different between individuals with IGD and healthy controls (HC).

METHODS

To address this issue, the present study asked 27 individuals with IGD and 26 HC to complete a roulette task during a functional MRI scan.

RESULTS

Compared with HC, individuals with IGD preferred to take risks in pursuit of high rewards behaviorally and showed exaggerated brain activity in the striatum (nucleus accumbens and caudate) during the reward anticipation and outcome monitoring but not during the choice evaluation.

DISCUSSION

These results reveal that the oversensitivity of the reward system to potential and positive rewards in college students with IGD drives them to approach risky options more frequently although they are able to assess the risk values of options and the correctness of decisions properly as HC do.

CONCLUSIONS

These findings provide partial support for the application of the self-regulation model to the IGD population. Moreover, this study enriches this model from the perspective of three phases of reward processing and provides specific targets for future research regarding effective treatment of IGD.

Trauma Affects Prospective Relationships Between Reward-Related Ventral Striatal and Amygdala Activation and 1-Year Future Hypo/Mania Trajectories

BACKGROUND

Trauma exposure is associated with a more severe, persistent course of affective and anxiety symptoms. Markers of reward neural circuitry function, specifically activation to reward prediction error (RPE), are impacted by trauma and predict the future course of affective symptoms. This study's purpose was to determine how lifetime trauma exposure influences relationships between reward neural circuitry function and the course of future affective and anxiety symptoms in a naturalistic, transdiagnostic observational context.

METHODS

A total of 59 young adults aged 18-25 (48 female and 11 male participants, mean ± SD = 21.5 ± 2.0 years) experiencing psychological distress completed the study. Participants were evaluated at baseline, 6, and 12 months. At baseline, the participants reported lifetime trauma events and completed a monetary reward functional magnetic resonance imaging task. Affective and anxiety symptoms were reported at each visit, and trajectories were calculated using MPlus. Neural activation during RPE and other phases of reward processing were determined using SPM8. Trauma and reward neural activation were entered as predictors of symptom trajectories.

RESULTS

Trauma exposure moderated prospective relationships between left ventral striatum (β = -1.29, p = .02) and right amygdala (β = 0.58, p = .04) activation to RPE and future hypo/mania severity trajectory: the interaction between greater trauma and greater left ventral striatum activation to RPE was associated with a shallower increase in hypo/mania severity, whereas the interaction between greater trauma and greater right amygdala activation to RPE was associated with increasing hypo/mania severity.

CONCLUSIONS

Trauma exposure affects prospective relationships between markers of reward circuitry function and affective symptom trajectories. Evaluating trauma exposure is thus crucial in naturalistic and treatment studies aiming to identify neural predictors of future affective symptom course.

Neural Mechanisms of Social and Nonsocial Reward Prediction Errors in Adolescents with Autism Spectrum Disorder

Autism spectrum disorder (ASD) is characterized by impaired predictive abilities; however, the neural mechanisms subsuming reward prediction errors in ASD are poorly understood. In the current study, we investigated neural responses during social and nonsocial reward prediction errors in 22 adolescents with ASD (ages 12-17) and 20 typically developing control adolescents (ages 12-18). Participants performed a reward prediction error task using both social (i.e., faces) and nonsocial (i.e., objects) rewards during a functional magnetic resonance imaging scan. Reward prediction errors were defined in two ways: (a) the signed prediction error, the difference between the experienced and expected reward; and (b) the thresholded unsigned prediction error, the difference between expected and unexpected outcomes regardless of magnitude. During social reward prediction errors, the ASD group demonstrated the following differences relative to the TD group: (a) signed prediction error: decreased activation in the right precentral gyrus and increased activation in the right frontal pole; and (b) thresholded unsigned prediction error: increased activation in the right anterior cingulate gyrus and bilateral precentral gyrus. Groups did not differ in brain activation during nonsocial reward prediction errors. Within the ASD group, exploratory analyses revealed that reaction times and social-communication impairments were related to precentral gyrus activation during social prediction errors. These findings elucidate the neural mechanisms of social reward prediction errors in ASD and suggest that ASD is characterized by greater neural atypicalities during social, relative to nonsocial, reward prediction errors in ASD. Autism Res 2020, 13: 715-728. © 2020 International Society for Autism Research, Wiley Periodicals, Inc. LAY SUMMARY: We used brain imaging to evaluate differences in brain activation in adolescents with autism while they performed tasks that involved learning about social and nonsocial information. We found no differences in brain responses during the nonsocial condition, but differences during the social condition of the learning task. This study provides evidence that autism may involve different patterns of brain activation when learning about social information.

Encoding of eye movements explains reward-related activity in cerebellar simple spikes

The cerebellum exhibits both motor and reward-related signals. However, it remains unclear whether reward is processed independently from the motor command or might reflect the motor consequences of the reward drive. To test how reward-related signals interact with sensorimotor processing in the cerebellum, we recorded Purkinje cell simple spike activity in the cerebellar floccular complex while monkeys were engaged in smooth pursuit eye movement tasks. The color of the target signaled the size of the reward the monkeys would receive at the end of the target motion. When the tracking task presented a single target, both pursuit and neural activity were only slightly modulated by the reward size. The reward modulations in single cells were rarely large enough to be detected. These modulations were only significant in the population analysis when we averaged across many neurons. In two-target tasks where the monkey learned to select based on the size of the reward outcome, both behavior and neural activity adapted rapidly. In both the single- and two-target tasks, the size of the reward-related modulation matched the size of the effect of reward on behavior. Thus, unlike cortical activity in eye movement structures, the reward-related signals could not be dissociated from the motor command. These results suggest that reward information is integrated with the eye movement command upstream of the Purkinje cells in the floccular complex. Thus reward-related modulations of the simple spikes are akin to modulations found in motor behavior and not to the central processing of the reward value.NEW & NOTEWORTHY Disentangling sensorimotor and reward signals is only possible if these signals do not completely overlap. We recorded activity in the floccular complex of the cerebellum while monkeys performed tasks designed to separate representations of reward from those of movement. Activity modulation by reward could be accounted for by the coding of eye movement parameters, suggesting that reward information is already integrated into motor commands upstream of the floccular complex.

Anhedonia Reduction and the Association Between Left Ventral Striatal Reward Response and 6-Month Improvement in Life Satisfaction Among Young Adults

IMPORTANCE

Anhedonia is a symptom of multiple psychiatric conditions in young adults that is associated with poorer mental health and psychosocial function and abnormal ventral striatum reward processing. Aberrant function of neural reward circuitry is well documented in anhedonia and other psychiatric disorders. Longitudinal studies to identify potential biomarkers associated with a reduction in anhedonia are necessary for the development of novel treatment targets.

OBJECTIVE

To identify neural reward-processing factors associated with improved psychiatric symptoms and psychosocial function in a naturalistic, observational context.

DESIGN, SETTING, AND PARTICIPANTS

A longitudinal cohort follow-up study was conducted from March 1, 2014, to June 5, 2018, at the University of Pittsburgh Medical Center after baseline functional magnetic resonance imaging in 52 participants between the ages of 18 and 25 years who were experiencing psychological distress.

MAIN OUTCOMES AND MEASURES

Participants were evaluated at baseline and 6 months. At baseline, participants underwent functional magnetic resonance imaging during a card-guessing monetary reward task. Participants completed measures of affective symptoms and psychosocial function at each visit. Neural activation during reward prediction error (RPE), a measure of reward learning, was determined using Statistical Parametric Mapping software. Neural reward regions with significant RPE activation were entered as regions associated with future symptoms in multiple linear regression models.

RESULTS

A total of 52 young adults (42 women and 10 men; mean [SD] age, 21.4 [2.2] years) completed the study. Greater RPE activation in the left ventral striatum was associated with a decrease in anhedonia symptoms during a 6-month period (β = -6.152; 95% CI, -11.870 to -0.433; P = .04). The decrease in anhedonia between baseline and 6 months mediated the association between left ventral striatum activation to RPE and improvement in life satisfaction between baseline and 6 months (total [c path] association: β = 0.245; P = .01; direct [c' path] association: β = 0.133; P = .16; and indirect [ab path] association: 95% CI, 0.026-0.262). Results were not associated with psychotropic medication use.

CONCLUSIONS AND RELEVANCE

Greater left ventral striatum responsiveness to RPE may serve as a biomarker or potential target for novel treatments to improve the severity of anhedonia, overall mental health, and psychosocial function.

The basal ganglia and the cerebellum: nodes in an integrated network

The basal ganglia and the cerebellum are considered to be distinct subcortical systems that perform unique functional operations. The outputs of the basal ganglia and the cerebellum influence many of the same cortical areas but do so by projecting to distinct thalamic nuclei. As a consequence, the two subcortical systems were thought to be independent and to communicate only at the level of the cerebral cortex. Here, we review recent data showing that the basal ganglia and the cerebellum are interconnected at the subcortical level. The subthalamic nucleus in the basal ganglia is the source of a dense disynaptic projection to the cerebellar cortex. Similarly, the dentate nucleus in the cerebellum is the source of a dense disynaptic projection to the striatum. These observations lead to a new functional perspective that the basal ganglia, the cerebellum and the cerebral cortex form an integrated network. This network is topographically organized so that the motor, cognitive and affective territories of each node in the network are interconnected. This perspective explains how synaptic modifications or abnormal activity at one node can have network-wide effects. A future challenge is to define how the unique learning mechanisms at each network node interact to improve performance.

Neural Mechanisms of Reward Prediction Error in Autism Spectrum Disorder

Few studies have explored neural mechanisms of reward learning in ASD despite evidence of behavioral impairments of predictive abilities in ASD. To investigate the neural correlates of reward prediction errors in ASD, 16 adults with ASD and 14 typically developing controls performed a prediction error task during fMRI scanning. Results revealed greater activation in the ASD group in the left paracingulate gyrus during signed prediction errors and the left insula and right frontal pole during thresholded unsigned prediction errors. Findings support atypical neural processing of reward prediction errors in ASD in frontostriatal regions critical for prediction coding and reward learning. Results provide a neural basis for impairments in reward learning that may contribute to traits common in ASD (e.g., intolerance of unpredictability).

Neural substrates of self- and external-preoccupation: A voxel-based morphometry study

INTRODUCTION

Self- and external-preoccupation have been linked to psychopathological states. The neural substrates underlying self- and external-preoccupation remain unclear. In the present study, we aim to provide insight into the information-processing mechanisms associated with self- and external-preoccupation at the structural level.

METHODS

To investigate the neural substrates of self- and external-preoccupation, we acquired high-resolution T1-weighted structural images and Preoccupation Scale scores from 1,122 young subjects. Associations between regional gray matter volume (rGMV) and Preoccupation Scale subscores for self- and external-preoccupation were estimated using voxel-based morphometry.

RESULTS

Significant positive associations between self-preoccupation and rGMV were observed in widespread brain areas such as the bilateral precuneus and posterior cingulate gyri, structures known to be associated with self-triggered self-reference during rest. Significant negative associations between external-preoccupation and rGMV were observed only in the bilateral cerebellum, regions known to be associated with behavioral addiction, sustained attention, and reward system.

CONCLUSION

Our results reveal distinct neural substrates for self- and external-preoccupation at the structural level.

Neural evidence for automatic value-modulated approach behaviour

Reward learning has the ability to bias both attention and behaviour. The current study presents behavioural and neural evidence that irrelevant responses evoked by previously reward-associated stimuli are more robustly represented in the motor system using a combined go/no-go and flankers task. Following a colour-reward association training, participants were instructed to respond to a central target only in a response-relevant context, while ignoring flankers that appeared either in a high-value or low-value colour. The motor cortex and cerebellum exhibited reduced activation to low-value flankers in a response-irrelevant context, consistent with goal-directed response suppression. However, these same regions exhibited similar activation to high-value flankers regardless of their response relevance, indicating less effective suppression, and the resulting interaction in motor cortex activation was strongly predicted by the influence of the flankers on behaviour. These findings suggest that associative reward learning produces a general approach bias, which is particularly evident when it conflicts with task goals, extending the principle of value-driven attention to stimulus-evoked responses in the motor system.

Acupuncture – Self-Appraisal and the Reward System

Acupuncture is an ancient therapy with a variety of different explanatory models. A cascade of physiological effects has been reported, both in the peripheral and the central nervous system, following the insertion of a needle or light tapping of the skin. Clinical trials testing the specific claims of acupuncture have generally tried to focus on testing the efficacy of applying specific techniques and/or specified points. However, different conditions may respond differently to different modes of stimulation.

Recently, it was demonstrated that both superficial and deep needling (with de qi/Hibiki) resulted in amelioration of patellofemoral pain and unpleasantness. The pleasurable aspect of the acupuncture experience has largely been ignored as it has been considered secondary to its pain alleviating effects. This aspect of acupuncture treatment is likely to be related to activation of self-appraisal and the reward system.

When a patient seeks a therapist there are expectations of a specific effect. These expectations are partly based on self-relevant phenomena and self-referentia introspection and constitute the preference. Also, when asked about the effect of the treatment, processes that orientate pre-attentive anticipatory or mnemonic information and processes that mediate self-reflection and recollection are integrated together with sensory detection to enable a decision about the patient's perception of the effect of acupuncture treatment. These ‘self-appraisal’ processes are dependent on two integrated networks: a ventral medial prefrontal cortex paralimbic limbic ‘affective’ pathway and a dorsal medial prefrontal cortex cortical hippocampal ‘cognitive’ pathway.

The limbic structures are implicated in the reward system and play a key role in most diseases and illness responses including chronic pain and depression, regulating mood and neuromodulatory responses (eg sensory, autonomic, and endocrine). The pleasurable and neuromodulatory aspects of acupuncture as well as ‘placebo needling’ may partly be explained by the activation or deactivation of limbic structures including the hippocampus, amygdala, and their connections with the hypothalamus.

In patients with patellofemoral pain, the effects of superficial and deep needling remained for six months. These long term pain-alleviating effects have been attributed to activation of pain inhibiting systems in cortical and subcortical pathways. When considering long term effects the cortical cerebellar system needs to be taken into account. The cortical cerebellar system is probably central to the development of neural models that learn and eventually stimulate routinely executed (eg motor skills) and long term (eg pain alleviation) cognitive processes. These higher order cognitive processes are initially mediated in prefrontal cortical loci but later shift control iteratively to internal cerebellar representations of these processes. Possibly part of the long term healing effects of acupuncture may be attributed to changes in the cerebellar system thereby sparing processing load in cortical and subcortical areas.

As cortical and subcortical structures are activated and/or de-activated following stimulation of receptors in the skin, disregarding site, ‘placebo or sham needling’ does not exist and conclusions drawn on the basis that it is an inert control are invalid.

‘Self’ may be seen as a shifting illusion, ceaselessly constructed and deconstructed, and the effect of acupuncture may reflect its status (as well as that of the therapist).

Orbitofrontal overactivation in reward processing in borderline personality disorder: the role of non-suicidal self-injury

Borderline Personality Disorder (BPD) is a disabling and difficult-to-treat mental disease. One of its core features is a significant difficulty in affect regulation, which is often accompanied by Non-Suicidal Self-Injury (NSSI). It is suggested that this type of behavior elicits positive emotions and mitigates emotional distress, and therefore can ultimately be reinforced and promoted. In spite of the high prevalence of NSSI behaviors (also in non-BPD samples), their role in modulating reward-related processes has not yet been investigated in BPD patients. In the present study, this lack of research was addressed. A large sample of BPD patients (N = 40), divided into two groups depending on the presence of NSSI, and a group of matched healthy controls underwent functional Magnetic Resonance Imaging (fMRI) while performing a gambling task. Patients who committed NSSI acts exhibited enhanced activation of the orbitofrontal cortex following an unexpected reward, when compared with controls and BPD patients with no NSSI behavior. In addition, the NSSI group showed diminished functional connectivity between the left orbitofrontal cortex and the right parahippocampal gyrus. These findings might suggest impaired ability to update reward associations of potential choices when both BPD and NSSI are present. We propose that the presence of NSSI involves alterations in the reward system independently of BPD, and thus can be considered as a possible phenotype for reward-related alterations.

Differences in BOLD responses in brain reward network reflect the tendency to assimilate a surprising flavor stimulus to an expected stimulus

External information can modify the subjective value of a tasted stimulus, but little is known about neural mechanisms underlying these behavioral modifications. This study used flavored drinks to produce variable degrees of discrepancy between expected and received flavor. During a learning session, 43 healthy young men learned 4 symbol-flavor associations. In a separate session, associations were presented again during an fMRI scan, but half of the trials introduced discrepancy with previously learned associations. Liking ratings of drinks were collected and were analyzed using a linear model to define the degree to which discrepant symbols affected liking ratings of the subjects during the fMRI session. Based on these results, a GLM analysis of fMRI data was conducted to determine neural correlates of observed behavior. Groups of subjects were composed based on their behavior in response to discrepant symbols, and comparison of brain activity between groups showed that activation in the PCC and the caudate nucleus was more potent in those subjects in which liking was not affected by discrepant symbols. These activations were not found in subjects who assimilated unexpected flavors to flavors preceeded by discrepant symbols. Instead, these subjects showed differences in the activity in the parietal operculum. The activity of reward network appears to be related to assimilation of received flavor to expected flavor in response to symbol-flavor discrepancy.

The role of the cerebellum for feedback processing and behavioral switching in a reversal-learning task

Previous studies have reported cerebellar activations during error and reward processing. The present study investigated if the cerebellum differentially processes feedback depending on changes in response strategy during reversal learning, as is conceivable given its internal models for movement and thought. Negative relative to positive feedback in an fMRI-based reversal learning task was hypothesized to be associated with increased cerebellar activations. Moreover, increased activations were expected for negative feedback followed by a change in response strategy compared to negative feedback not followed by such a change, and for first positive feedback after compared to final negative feedback before a change, due to updating of internal models. As predicted, activation in lobules VI and VIIa/Crus I was increased for negative relative to positive feedback, and for final negative feedback before a change in response strategy relative to negative feedback not associated with a change. Moreover, activation was increased for first positive feedback after relative to final negative feedback before a change. These findings are consistent with updating of cerebellar internal models to accommodate new behavioral strategies. Recruitment of posterior regions in reversal learning is in line with the cerebellar functional topography, with posterior regions involved in complex motor and cognitive functions.

Role of Frontostriatal Connectivity in Adolescents With Excessive Smartphone Use

As smartphone use has grown rapidly over recent decade, it has been a growing interest in the potential negative impact of excessive smartphone use. In this study, we aim to identify altered brain connectivity associated with excessive smartphone use, and to investigate correlations between withdrawal symptoms, cortisol concentrations, and frontostriatal connectivity. We focused on investigating functional connectivity in frontostriatal regions, including the orbitofrontal cortex (OFC), midcingulate cortex (MCC), and nucleus accumbens (NAcc), which is related to reward processing and cognitive control. We analyzed data from 38 adolescents with excessive smartphone use (SP) and 42 healthy controls (HC). In the SP group compared with HC, we observed lower functional connectivity between the right OFC and NAcc, and between the left OFC and MCC. Moreover, functional connectivity between the MCC and NAcc was greater in SP compared with HC. Subsequently, we examined the relationship between Internet use withdrawal symptoms, cortisol concentrations, and functional connectivity between the OFC and NAcc in SP and HC. We observed that more severe withdrawal symptoms were associated with higher cortisol concentrations in adolescents with excessive smartphone use. The most interesting finding was that we observed a negative correlation between OFC connectivity with the NAcc and both withdrawal symptoms and cortisol concentrations. The functional connectivity between the OFC and NAcc, and between the OFC and MCC are related to cognitive control of emotional stimuli including reward. The current study suggests that adolescents with SP had reduced functional connectivity in these regions related to cognitive control. Furthermore, Internet use withdrawal symptoms appear to elicit cortisol secretion, and this psychophysiological change may affect frontostriatal connectivity. Our findings provide important clues to understanding the effects of excessive use of smartphones on brain functional connectivity in adolescence.

Learning about Expectation Violation from Prediction Error Paradigms - A Meta-Analysis on Brain Processes Following a Prediction Error

Modifying patients' expectations by exposing them to expectation violation situations (thus maximizing the difference between the expected and the actual situational outcome) is proposed to be a crucial mechanism for therapeutic success for a variety of different mental disorders. However, clinical observations suggest that patients often maintain their expectations regardless of experiences contradicting their expectations. It remains unclear which information processing mechanisms lead to modification or persistence of patients' expectations. Insight in the processing could be provided by Neuroimaging studies investigating prediction error (PE, i.e., neuronal reactions to non-expected stimuli). Two methods are often used to investigate the PE: (1) paradigms, in which participants passively observe PEs ("passive" paradigms) and (2) paradigms, which encourage a behavioral adaptation following a PE ("active" paradigms). These paradigms are similar to the methods used to induce expectation violations in clinical settings: (1) the confrontation with an expectation violation situation and (2) an enhanced confrontation in which the patient actively challenges his expectation. We used this similarity to gain insight in the different neuronal processing of the two PE paradigms. We performed a meta-analysis contrasting neuronal activity of PE paradigms encouraging a behavioral adaptation following a PE and paradigms enforcing passiveness following a PE. We found more neuronal activity in the striatum, the insula and the fusiform gyrus in studies encouraging behavioral adaptation following a PE. Due to the involvement of reward assessment and avoidance learning associated with the striatum and the insula we propose that the deliberate execution of action alternatives following a PE is associated with the integration of new information into previously existing expectations, therefore leading to an expectation change. While further research is needed to directly assess expectations of participants, this study provides new insights into the information processing mechanisms following an expectation violation.

Effects of loss aversion on neural responses to loss outcomes: An event-related potential study

Loss aversion is the tendency to prefer avoiding losses over acquiring gains of the same amount. To shed light on the spatio-temporal processes underlying loss aversion, we analysed the associations between individual loss aversion and electrophysiological responses to loss and gain outcomes in a monetary gamble task. Electroencephalographic feedback-related negativity (FRN) was computed in 29 healthy participants as the difference in electrical potentials between losses and gains. Loss aversion was evaluated using non-linear parametric fitting of choices in a separate gamble task. Loss aversion correlated positively with FRN amplitude (233-263ms) at electrodes covering the lower face. Feedback related potentials were modelled by five equivalent source dipoles. From these dipoles, stronger activity in a source located in the orbitofrontal cortex was associated with loss aversion. The results suggest that loss aversion implemented during risky decision making is related to a valuation process in the orbitofrontal cortex, which manifests during learning choice outcomes.

Nicotine increases anterior insula activation to expected and unexpected outcomes among nonsmokers

RATIONALE

Tobacco has a higher rate of dependence than other drugs of abuse. However, the psychopharmacological effects of nicotine are incongruent with the tenacity of tobacco addiction since nicotine does not produce robust euphoria in humans or self-administration in rodents. A potential explanation is that nicotine amplifies the salience of other stimuli that have some incentive value, which could influence the initiation and persistence of smoking. However, the neural mechanisms of this process are unknown.

OBJECTIVES

One way that nicotine may amplify the salience of other stimuli is by enhancing reward prediction errors. We hypothesized that nicotine would enhance the neural response to unexpected (relative to expected) rewards compared to placebo.

METHODS

Twenty-three nonsmokers underwent two fMRI scans, following nicotine (1 mg) or placebo administration, while performing an outcome expectation task. In the task, a pair of cues was associated with either a subsequent reward (the image of a $100 bill) or a nonreward (the image of a blurry rectangle). On 20% of trials, the cue was followed by an unexpected outcome.

RESULTS

Although nicotine did not affect the magnitude of prediction errors relative to placebo, nicotine did increase BOLD activation in the anterior insula/inferior frontal gyrus and decrease activation in the caudate across all outcome types (including both rewards and nonrewards).

CONCLUSIONS

The insula and caudate could play a role in the initial effects of nicotine in nonsmokers, and these changes in baseline may be the mechanism that underlies how nicotine amplifies the salience of nondrug stimuli.

Processing of Positive and Negative Feedback in Patients with Cerebellar Lesions

It is well accepted that the cerebellum plays a crucial role in the prediction of the sensory consequences of movements. Recent findings of altered error processing in patients with selective cerebellar lesions led to the hypothesis that feedback processing and feedback-based learning might be affected by cerebellar damage as well. Thus, the present study investigated learning from and processing of positive and negative feedback in 12 patients with selective cerebellar lesions and healthy control subjects. Participants performed a monetary feedback learning task. The processing of positive and negative feedback was assessed by means of event-related potentials (ERPs) during the learning task and during a separate task in which the frequencies of positive and negative feedback were balanced. Patients did not show a general learning deficit compared to controls. Relative to the control group, however, patients with cerebellar lesions showed significantly higher ERP difference wave amplitudes (rewards-losses) in a time window between 250 and 450 ms after feedback presentation, possibly indicating impaired outcome prediction. The analysis of the original waveforms suggested that patients and controls primarily differed in their pattern of feedback-related negativity and P300 amplitudes. Our results add to recent findings on altered performance monitoring associated with cerebellar damage and demonstrate, for the first time, alterations of feedback processing in patients with cerebellar damage. Unaffected learning performance appears to suggest that chronic cerebellar lesions can be compensated in behaviour.

Ventral Striatum Functional Connectivity during Rewards and Losses and Symptomatology in Depressed Patients

BACKGROUND

The ventral striatum (VS) and striatal network supports goal motivated behavior. Identifying how depressed patients differ in their striatal network during the processing of emotionally salient events is a step towards uncovering biomarkers for diagnosis and treatment.

METHODS

38 depressed and 30 healthy adults completed a task that examined brain activation to the anticipation and receipt of monetary rewards and losses. Data were collected using a 3T Siemens Trio scanner. Functional connectivity differences were examined with seeds in the Left or Right VS. FC estimates were regressed on specific symptoms.

RESULTS

Depressed patients displayed higher functional connectivity between the VS and midline cortical areas during loss versus reward trials. Anhedonia and depressed mood were associated to fairly similar striatal circuits but suicidality was associated to a unique VS-midline structures coupling, while depression severity was linked to higher VS to caudate and precuneus connectivity during loss versus reward trials.

CONCLUSIONS

Depression is characterized by excessive VS coupling to cognitive control and associative networks during losses versus rewards. High VS to midline cortical structures coupling may index suicidality.

Firing Variability of Frontal Pole Neurons during a Cued Strategy Task

In previous reports, we described neuronal activity in the polar (PFp), dorsolateral (PFdl), and orbital (PFo) PFC as monkeys performed a cued strategy task with two spatial goals. On each trial, a cue instructed one of two strategies: Stay with the previous goal or shift to the alternative. A delay period followed each cue, and feedback followed each choice, also at a delay. Our initial analysis showed that the mean firing rate of a population of PFp cells encoded the goal chosen on a trial, but only near the time of feedback, not earlier in the trial. In contrast, PFdl cells encoded goals and strategies during the cue and delay periods, and PFo cells encoded strategies in those task periods. Both areas also signaled goals near feedback time. Here we analyzed trial-to-trial variability of neuronal firing, as measured by the Fano factor (FF): the ratio of variance to the mean. Goal-selective PFp neurons had two properties: (1) a lower FF from the beginning of the trial compared with PFp cells that did not encode goals and (2) a weak but significant inverse correlation between FF throughout a trial and the degree of goal selectivity at feedback time. Cells in PFdl and PFo showed neither of these properties. Our findings indicate that goal-selective PFp neurons were engaged in the task throughout a trial, although they only encoded goals near feedback time. Their lower FF could improve the ability of other cortical areas to decode its selected-goal signal.

Reduced Neural Recruitment for Bayesian Adjustment of Inhibitory Control in Methamphetamine Dependence

Delineating the processes that contribute to the progression and maintenance of substance dependence is critical to understanding and preventing addiction. Several previous studies have shown inhibitory control deficits in individuals with stimulant use disorder. We used a Bayesian computational approach to examine potential neural deficiencies in the dynamic predictive processing underlying inhibitory function among recently abstinent methamphetamine-dependent individuals (MDIs), a population at high risk of relapse. Sixty-two MDIs were recruited from a 28-day inpatient treatment program at the San Diego Veterans Affairs Medical Center and compared with 34 healthy control subjects. They completed a stop-signal task during functional magnetic resonance imaging. A Bayesian ideal observer model was used to predict individuals' trial-to-trial probabilistic expectations of inhibitory response, P(stop), to identify group differences specific to Bayesian expectation and prediction error computation. Relative to control subjects, MDIs were more likely to make stop errors on difficult trials and had attenuated slowing following stop errors. MDIs further exhibited reduced sensitivity as measured by the neural tracking of a Bayesian measure of surprise (unsigned prediction error), which was evident across all trials in the left posterior caudate and orbitofrontal cortex (Brodmann area 11), and selectively on stop error trials in the right thalamus and inferior parietal lobule. MDIs are less sensitive to surprising task events, both across trials and upon making commission errors, which may help explain why these individuals may not engage in switching strategy when the environment changes, leading to adverse consequences.

Associations between polygenic risk for schizophrenia and brain function during probabilistic learning in healthy individuals

A substantial proportion of schizophrenia liability can be explained by additive genetic factors. Risk profile scores (RPS) directly index risk using a summated total of common risk variants weighted by their effect. Previous studies suggest that schizophrenia RPS predict alterations to neural networks that support working memory and verbal fluency. In this study, we apply schizophrenia RPS to fMRI data to elucidate the effects of polygenic risk on functional brain networks during a probabilistic‐learning neuroimaging paradigm. The neural networks recruited during this paradigm have previously been shown to be altered to unmedicated schizophrenia patients and relatives of schizophrenia patients, which may reflect genetic susceptibility. We created schizophrenia RPS using summary data from the Psychiatric Genetic Consortium (Schizophrenia Working Group) for 83 healthy individuals and explore associations between schizophrenia RPS and blood oxygen level dependency (BOLD) during periods of choice behavior (switch–stay) and reflection upon choice outcome (reward–punishment). We show that schizophrenia RPS is associated with alterations in the frontal pole (P_{WHOLE‐BRAIN‐CORRECTED} = 0.048) and the ventral striatum (P_{ROI‐CORRECTED} = 0.036), during choice behavior, but not choice outcome. We suggest that the common risk variants that increase susceptibility to schizophrenia can be associated with alterations in the neural circuitry that support the processing of changing reward contingencies. Hum Brain Mapp 37:491–500, 2016. © 2015 Wiley Periodicals, Inc.

Development of the uncinate fasciculus: Implications for theory and developmental disorders

The uncinate fasciculus (UF) is a long-range white matter tract that connects limbic regions in the temporal lobe to the frontal lobe. The UF is one of the latest developing tracts, and continues maturing into the third decade of life. As such, individual differences in the maturational profile of the UF may serve to explain differences in behavior. Indeed, atypical macrostructure and microstructure of the UF have been reported in numerous studies of individuals with developmental and psychiatric disorders such as social deprivation and maltreatment, autism spectrum disorders, conduct disorder, risk taking, and substance abuse. The present review evaluates what we currently know about the UF's developmental trajectory and reviews the literature relating UF abnormalities to specific disorders. Additionally, we take a dimensional approach and critically examine symptoms and behavioral impairments that have been demonstrated to cluster with UF aberrations, in an effort to relate these impairments to our speculations regarding the functionality of the UF. We suggest that developmental disorders with core problems relating to memory retrieval, reward and valuation computation, and impulsive decision making may be linked to aberrations in uncinate microstructure.

Automatic and controlled processing in the corticocerebellar system

During learning, performance changes often involve a transition from controlled processing in which performance is flexible and responsive to ongoing error feedback, but effortful and slow, to a state in which processing becomes swift and automatic. In this state, performance is unencumbered by the requirement to process feedback, but its insensitivity to feedback reduces its flexibility. Many properties of automatic processing are similar to those that one would expect of forward models, and many have suggested that these may be instantiated in cerebellar circuitry. Since hierarchically organized frontal lobe areas can both send and receive commands, I discuss the possibility that they can act both as controllers and controlled objects and that their behaviors can be independently modeled by forward models in cerebellar circuits. Since areas of the prefrontal cortex contribute to this hierarchically organized system and send outputs to the cerebellar cortex, I suggest that the cerebellum is likely to contribute to the automation of cognitive skills, and to the formation of habitual behavior which is resistant to error feedback. An important prerequisite to these ideas is that cerebellar circuitry should have access to higher order error feedback that signals the success or failure of cognitive processing. I have discussed the pathways through which such feedback could arrive via the inferior olive and the dopamine system. Cerebellar outputs inhibit both the inferior olive and the dopamine system. It is possible that learned representations in the cerebellum use this as a mechanism to suppress the processing of feedback in other parts of the nervous system. Thus, cerebellar processes that control automatic performance may be completed without triggering the engagement of controlled processes by prefrontal mechanisms.

Reward-based spatial learning in unmedicated adults with obsessive-compulsive disorder

OBJECTIVE

The authors assessed the functioning of mesolimbic and striatal areas involved in reward-based spatial learning in unmedicated adults with obsessive-compulsive disorder (OCD).

METHOD

Functional MRI blood-oxygen-level-dependent response was compared in 33 unmedicated adults with OCD and 33 healthy, age-matched comparison subjects during a reward-based learning task that required learning to use extramaze cues to navigate a virtual eight-arm radial maze to find hidden rewards. The groups were compared in their patterns of brain activation associated with reward-based spatial learning versus a control condition in which rewards were unexpected because they were allotted pseudorandomly to experimentally prevent learning.

RESULTS

Both groups learned to navigate the maze to find hidden rewards, but group differences in neural activity during navigation and reward processing were detected in mesolimbic and striatal areas. During navigation, the OCD group, unlike the healthy comparison group, exhibited activation in the left posterior hippocampus. Unlike healthy subjects, participants in the OCD group did not show activation in the left ventral putamen and amygdala when anticipating rewards or in the left hippocampus, amygdala, and ventral putamen when receiving unexpected rewards (control condition). Signal in these regions decreased relative to baseline during unexpected reward receipt among those in the OCD group, and the degree of activation was inversely associated with doubt/checking symptoms.

CONCLUSIONS

Participants in the OCD group displayed abnormal recruitment of mesolimbic and ventral striatal circuitry during reward-based spatial learning. Whereas healthy comparison subjects exhibited activation in this circuitry in response to the violation of reward expectations, unmedicated OCD participants did not and instead over-relied on the posterior hippocampus during learning. Thus, dopaminergic innervation of reward circuitry may be altered, and future study of anterior/posterior hippocampal dysfunction in OCD is warranted.

Toward sophisticated basal ganglia neuromodulation: Review on basal ganglia deep brain stimulation

This review presents state-of-the-art knowledge about the roles of the basal ganglia (BG) in action-selection, cognition, and motivation, and how this knowledge has been used to improve deep brain stimulation (DBS) treatment of neurological and psychiatric disorders. Such pathological conditions include Parkinson's disease, Huntington's disease, Tourette syndrome, depression, and obsessive-compulsive disorder. The first section presents evidence supporting current hypotheses of how the cortico-BG circuitry works to select motor and emotional actions, and how defects in this circuitry can cause symptoms of the BG diseases. Emphasis is given to the role of striatal dopamine on motor performance, motivated behaviors and learning of procedural memories. Next, the use of cutting-edge electrochemical techniques in animal and human studies of BG functioning under normal and disease conditions is discussed. Finally, functional neuroimaging studies are reviewed; these works have shown the relationship between cortico-BG structures activated during DBS and improvement of disease symptoms.

Smoking automaticity and tolerance moderate brain activation during explore-exploit behavior

The adaptive trade-off between exploration and exploitation is a key component in models of reinforcement learning. Over the past decade, these models have been applied to the study of reward-seeking behavior. Drugs of addiction induce reward-seeking behavior and modify its underlying neurophysiological processes. These neurophysiological changes may underlie a behavioral shift from a flexible, exploratory mode to a focused, exploitative mode, which precedes the development of inflexible, habitual drug use. The goal of this study was to investigate the relationship between explore/exploit behavior and drug addiction by examining the neural correlates of this behavior in cigarette smokers. Participants (n=22) with a range of smoking behaviors completed a smoking dependence motives questionnaire and played a 6-armed bandit task while undergoing functional magnetic resonance imaging (fMRI). Exploratory behavior produced greater activation in the bilateral superior parietal and bilateral frontal cortices than exploitative behavior. Exploitative behavior produced greater activation in the bilateral superior and middle temporal gyri than exploratory behavior. fMRI data and orthogonalized smoking dependence motive scores were entered into multiple linear regression analyses. After controlling for nicotine tolerance, smoking automaticity positively correlated with activation in the same bilateral parietal regions preferentially activated by exploratory choices. These preliminary results link smoking dependence motives to variation in the neural processes that mediate exploratory decision making.

Processing of action- but not stimulus-related prediction errors differs between active and observational feedback learning

Learning of stimulus-response-outcome associations is driven by outcome prediction errors (PEs). Previous studies have shown larger PE-dependent activity in the striatum for learning from own as compared to observed actions and the following outcomes despite comparable learning rates. We hypothesised that this finding relates primarily to a stronger integration of action and outcome information in active learners. Using functional magnetic resonance imaging, we investigated brain activations related to action-dependent PEs, reflecting the deviation between action values and obtained outcomes, and action-independent PEs, reflecting the deviation between subjective values of response-preceding cues and obtained outcomes. To this end, 16 active and 15 observational learners engaged in a probabilistic learning card-guessing paradigm. On each trial, active learners saw one out of five cues and pressed either a left or right response button to receive feedback (monetary win or loss). Each observational learner observed exactly those cues, responses and outcomes of one active learner. Learning performance was assessed in active test trials without feedback and did not differ between groups. For both types of PEs, activations were found in the globus pallidus, putamen, cerebellum, and insula in active learners. However, only for action-dependent PEs, activations in these structures and the anterior cingulate were increased in active relative to observational learners. Thus, PE-related activity in the reward system is not generally enhanced in active relative to observational learning but only for action-dependent PEs. For the cerebellum, additional activations were found across groups for cue-related uncertainty, thereby emphasising the cerebellum's role in stimulus-outcome learning.

Dopamine, Salience, and Response Set Shifting in Prefrontal Cortex

Dopamine is implicated in multiple functions, including motor execution, action learning for hedonically salient outcomes, maintenance, and switching of behavioral response set. Here, we used a novel within-subject psychopharmacological and combined functional neuroimaging paradigm, investigating the interaction between hedonic salience, dopamine, and response set shifting, distinct from effects on action learning or motor execution. We asked whether behavioral performance in response set shifting depends on the hedonic salience of reversal cues, by presenting these as null (neutral) or salient (monetary loss) outcomes. We observed marked effects of reversal cue salience on set-switching, with more efficient reversals following salient loss outcomes. L-Dopa degraded this discrimination, leading to inappropriate perseveration. Generic activation in thalamus, insula, and striatum preceded response set switches, with an opposite pattern in ventromedial prefrontal cortex (vmPFC). However, the behavioral effect of hedonic salience was reflected in differential vmPFC deactivation following salient relative to null reversal cues. l-Dopa reversed this pattern in vmPFC, suggesting that its behavioral effects are due to disruption of the stability and switching of firing patterns in prefrontal cortex. Our findings provide a potential neurobiological explanation for paradoxical phenomena, including maintenance of behavioral set despite negative outcomes, seen in impulse control disorders in Parkinson's disease.

With you or against you: social orientation dependent learning signals guide actions made for others

In social environments, it is crucial that decision-makers take account of the impact of their actions not only for oneself, but also on other social agents. Previous work has identified neural signals in the striatum encoding value-based prediction errors for outcomes to oneself; also, recent work suggests that neural activity in prefrontal cortex may similarly encode value-based prediction errors related to outcomes to others. However, prior work also indicates that social valuations are not isomorphic, with social value orientations of decision-makers ranging on a cooperative to competitive continuum; this variation has not been examined within social learning environments. Here, we combine a computational model of learning with functional neuroimaging to examine how individual differences in orientation impact neural mechanisms underlying 'other-value' learning. Across four experimental conditions, reinforcement learning signals for other-value were identified in medial prefrontal cortex, and were distinct from self-value learning signals identified in striatum. Critically, the magnitude and direction of the other-value learning signal depended strongly on an individual's cooperative or competitive orientation toward others. These data indicate that social decisions are guided by a social orientation-dependent learning system that is computationally similar but anatomically distinct from self-value learning. The sensitivity of the medial prefrontal learning signal to social preferences suggests a mechanism linking such preferences to biases in social actions and highlights the importance of incorporating heterogeneous social predispositions in neurocomputational models of social behavior.

Predictability influences whether outcomes are processed in terms of original or relative values

Previous studies have provided insights into the representations of original and relative values and the influence of predictability on decision making. However, whether the predictability of outcomes can influence the processing manner of outcomes (i.e. whether the outcomes are processed in terms of original or relative values) is still unknown. To investigate this issue, we had participants perform a monetary decision task which resulted in two outcomes with the same relative values but different original values in either a predictable or unpredictable condition, while recording event-related potentials (ERP). ERP results showed that the outcome processing in the unpredictable condition elicited more positive deflections in the time window of 300-500ms (P300) than did those in the predictable condition. More importantly, the outcome with high original value elicited a greater P300 component than did that with low original value in the unpredictable condition even though these two outcomes had the same relative values, while in the predictable condition no significant difference was observed between ERPs elicited by the two outcomes even though their original values were different. These results suggest that the outcomes might be processed in terms of relative values in the predictable condition but original values in the unpredictable condition.

An fMRI study of reward circuitry in patients with minimal or extensive history of major depression

Functional abnormalities in regions associated with reward processing are apparent in people with depression, but the extent to which disease burden impacts on the processing of reward is unknown. This research examined the neural correlates of reward processing in patients with major depressive disorder and varying degrees of past illness burden. Twenty-nine depressed patients and twenty-five healthy subjects with no lifetime history of psychiatric illness completed the study. Subsets of fourteen patients were presenting for first lifetime treatment of a depressive episode, and fifteen patients had at least three treated episodes of depression. We used functional magnetic resonance imaging to study blood oxygen level-dependent signals during the performance of a contingency reversal reward paradigm. The results identified group differences in the response to punishers bilaterally in the orbitofrontal and medial prefrontal regions. In addition, areas such as the nucleus accumbens, anterior cingulate and ventral prefrontal cortices were activated greatest by controls during reward processing, less by patients early in the course of illness and least by patients with highly recurrent illness-suggesting that these areas are sensitive to the impact of disease burden and repeated episodes of depression. Reward processing in people with depression may be associated with diminished signaling of incentive salience, a reduction in the formation of reward-related associations and heightened sensitivities for negatively valenced stimuli, all of which could contribute to symptoms of depression.

Electrophysiological indicators of surprise and entropy in dynamic task-switching environments

This event-related brain potential (ERP) study aimed at bridging two hitherto widely separated domains of cognitive neuroscience. Specifically, we combined the analysis of cognitive control in a cued task-switching paradigm with the fundamental question of how uncertainty is encoded in the brain. Two functional models of P3 amplitude variation in cued task-switching paradigms were put to an empirical test: (1) According to the P3b surprise hypothesis, parietal P3b waveforms are related to surprise over switch cues. (2) According to the P3a entropy hypothesis, frontal P3a waveforms are associated with entropy over switch outcomes. In order to examine these hypotheses, we measured the EEG while sixteen healthy young participants performed cued task-switching paradigms closely modeled to the Wisconsin Card Sorting Test (WCST). We applied a factorial design, with number of tasks (two vs. three viable tasks), cue explicitness (task cuing vs. transition cuing), and cue contingency (prospectively-signaled cuing vs. feedback-based cuing) as independent variables. The ERP results replicated the commonly reported P3b effect associated with task switches, and further showed that P3a amplitudes were related to the entropy of switch outcomes, thereby supporting both hypotheses. Based on these ERP data, we suggest that surprise over task switches, and entropy over switch outcomes, constitute dissociable functional correlates of P3b and P3a ERP components in task-switching paradigms, respectively. Finally, a theoretical integration of the findings is proposed within the framework of Sokolov's (1966) entropy model of the orienting response (OR).