A simple solution for model comparison in bold imaging: the special case of reward prediction error and reward outcomes

Independent and interacting value systems for reward and information in the human brain

Theories of prefrontal cortex (PFC) as optimizing reward value have been widely deployed to explain its activity in a diverse range of contexts, with substantial empirical support in neuroeconomics and decision neuroscience. Similar neural circuits, however, have also been associated with information processing. By using computational modeling, model-based functional magnetic resonance imaging analysis, and a novel experimental paradigm, we aim at establishing whether a dedicated and independent value system for information exists in the human PFC. We identify two regions in the human PFC that independently encode reward and information. Our results provide empirical evidence for PFC as an optimizer of independent information and reward signals during decision-making under realistic scenarios, with potential implications for the interpretation of PFC activity in both healthy and clinical populations.

Value estimation and latent-state update-related neural activity during fear conditioning predict posttraumatic stress disorder symptom severity

Learning theories of posttraumatic stress disorder (PTSD) purport that fear-learning processes, such as those that support fear acquisition and extinction, are impaired. Computational models designed to capture specific processes involved in fear learning have primarily assessed model-free, or trial-and-error, reinforcement learning (RL). Although previous studies indicated that aspects of model-free RL are disrupted among individuals with PTSD, research has yet to identify whether model-based RL, which is inferential and contextually driven, is impaired. Given empirical evidence of aberrant contextual modulation of fear in PTSD, the present study sought to identify whether model-based RL processes are altered during fear conditioning among women with interpersonal violence (IPV)-related PTSD (n = 85) using computational modeling. Model-free, hybrid, and model-based RL models were applied to skin conductance responses (SCR) collected during fear acquisition and extinction, and the model-based RL model was found to provide the best fit to the SCR data. Parameters from the model-based RL model were carried forward to neuroimaging analyses (voxel-wise and independent component analysis). Results revealed that reduced activity within visual processing regions during model-based updating uniquely predicted higher PTSD symptoms. Additionally, after controlling for model-based updating, greater value estimation encoding within the left frontoparietal network during fear acquisition and reduced value estimation encoding within the dorsomedial prefrontal cortex during fear extinction predicted greater PTSD symptoms. Results provide evidence of disrupted RL processes in women with assault-related PTSD, which may contribute to impaired fear and safety learning, and, furthermore, may relate to treatment response (e.g., poorer response to exposure therapy).

Towards Automaticity in Reinforcement Learning: A Model-Based Functional Magnetic Resonance Imaging Study

Introduction

Previous studies showed that over the course of learning many neurons in the medial prefrontal cortex adapt their firing rate towards the options with highest predicted value reward but it was showed that during later learning trials the brain switches to a more automatic processing mode governed by the basal ganglia. Based on this evidence, we hypothesized that during the early learning trials the predicted values of chosen options will be coded by a goal directed system in the medial frontal cortex but during the late trials the predicted values will be coded by the habitual learning system in the dorsal striatum.

Methods

In this study, using a 3 Tesla functional magnetic resonance imaging scanner (fMRI), blood oxygen level dependent signal (BOLD) data was collected whilst participants (N=12) performed a reinforcement learning task. The task consisted of instrumental conditioning trials wherein each trial a participant choose one of the two available options in order to win or avoid losing money. In addition to that, depending on the experimental condition, participants received either monetary reward (gain money), monetary penalty (lose money) or neural outcome.

Results

Using model-based analysis for functional magnetic resonance imaging (fMRI) event related designs; region of interest (ROI) analysis was performed to nucleus accumbens, medial frontal cortex, caudate nucleus, putamen and globus pallidus internal and external segments. In order to compare the difference in brain activity for early (goal directed) versus late learning (habitual, automatic) trials, separate ROI analyses were performed for each anatomical sub-region. For the reward condition, we found significant activity in the medial frontal cortex (p<0.05) only for early learning trials but activity is shifted to bilateral putamen (p<0.05) during later trials. However, for the loss condition no significant activity was found for early trials except globus pallidus internal segment showed a significant activity (p<0.05) for later trials.

Conclusion

We found that during reinforcement learning activation in the brain shifted from the medial frontal regions to dorsal regions of the striatum. These findings suggest that there are two separable (early goal directed and late habitual) learning systems in the brain.

Differential Roles of the Salience Network During Prediction Error Encoding and Facial Emotion Processing Among Female Adolescent Assault Victims

BACKGROUND

Early-life assaultive violence exposure is a potent risk factor for posttraumatic stress disorder (PTSD) and other mood and anxiety disorders. Neurocircuitry models posit that increased risk is mediated by heightened emotion processing in a salience network including the dorsal anterior cingulate cortex, anterior insula, and amygdala. However, the processes of reinforcement learning (RL) also engage the salience network and are implicated in responses to early-life trauma and PTSD. To define their relative roles in response to early-life trauma and PTSD symptoms, the current study compared engagement of the salience network during emotion processing and RL as a function of early-life assault exposure.

METHODS

Adolescent girls (n = 30 girls who had previously been physically or sexually assaulted; n = 30 healthy girls for comparison) 11 to 17 years of age completed two types of tasks during functional magnetic resonance imaging: a facial emotion processing task and an RL task using either social or nonsocial stimuli. Independent component analysis was used to identify a salience network and characterize its engagement in response to emotion processing and prediction error encoding during the RL tasks.

RESULTS

Assault was related to greater reactivity of the salience network during emotion processing. By contrast, we found lesser encoding of negative prediction errors in the salience network, particularly during the social RL task, in girls who had been assaulted. The dysfunction of salience network activity during emotion processing and prediction error encoding was not associated with PTSD symptoms.

CONCLUSIONS

These results suggest that hyper- versus hypoactivity of the salience network among trauma-exposed youths depends on the cognitive-affective domain.

Functional hemispheric asymmetries during the planning and manual control of virtual avatar movements

Both hemispheres contribute to motor control beyond the innervation of the contralateral alpha motoneurons. The left hemisphere has been associated with higher-order aspects of motor control like sequencing and temporal processing, the right hemisphere with the transformation of visual information to guide movements in space. In the visuomotor context, empirical evidence regarding the latter has been limited though the right hemisphere’s specialization for visuospatial processing is well-documented in perceptual tasks. This study operationalized temporal and spatial processing demands during visuomotor processing and investigated hemispheric asymmetries in neural activation during the unimanual control of a visual cursor by grip force. Functional asymmetries were investigated separately for visuomotor planning and online control during functional magnetic resonance imaging in 19 young, healthy, right-handed participants. The expected cursor movement was coded with different visual trajectories. During planning when spatial processing demands predominated, activity was right-lateralized in a hand-independent manner in the inferior temporal lobe, occipito-parietal border, and ventral premotor cortex. When temporal processing demands overweighed spatial demands, BOLD responses during planning were left-lateralized in the temporo-parietal junction. During online control of the cursor, right lateralization was not observed. Instead, left lateralization occurred in the intraparietal sulcus. Our results identify movement phase and spatiotemporal demands as important determinants of dynamic hemispheric asymmetries during visuomotor processing. We suggest that, within a bilateral visuomotor network, the right hemisphere exhibits a processing preference for planning global spatial movement features whereas the left hemisphere preferentially times local features of visual movement trajectories and adjusts movement online.

Multicollinearity in Regression Analyses Conducted in Epidemiologic Studies

The adverse impact of ignoring multicollinearity on findings and data interpretation in regression analysis is very well documented in the statistical literature. The failure to identify and report multicollinearity could result in misleading interpretations of the results. A review of epidemiological literature in PubMed from January 2004 to December 2013, illustrated the need for a greater attention to identifying and minimizing the effect of multicollinearity in analysis of data from epidemiologic studies. We used simulated datasets and real life data from the Cameron County Hispanic Cohort to demonstrate the adverse effects of multicollinearity in the regression analysis and encourage researchers to consider the diagnostic for multicollinearity as one of the steps in regression analysis.

Reinforcement learning models and their neural correlates: An activation likelihood estimation meta-analysis

Reinforcement learning describes motivated behavior in terms of two abstract signals. The representation of discrepancies between expected and actual rewards/punishments-prediction error-is thought to update the expected value of actions and predictive stimuli. Electrophysiological and lesion studies have suggested that mesostriatal prediction error signals control behavior through synaptic modification of cortico-striato-thalamic networks. Signals in the ventromedial prefrontal and orbitofrontal cortex are implicated in representing expected value. To obtain unbiased maps of these representations in the human brain, we performed a meta-analysis of functional magnetic resonance imaging studies that had employed algorithmic reinforcement learning models across a variety of experimental paradigms. We found that the ventral striatum (medial and lateral) and midbrain/thalamus represented reward prediction errors, consistent with animal studies. Prediction error signals were also seen in the frontal operculum/insula, particularly for social rewards. In Pavlovian studies, striatal prediction error signals extended into the amygdala, whereas instrumental tasks engaged the caudate. Prediction error maps were sensitive to the model-fitting procedure (fixed or individually estimated) and to the extent of spatial smoothing. A correlate of expected value was found in a posterior region of the ventromedial prefrontal cortex, caudal and medial to the orbitofrontal regions identified in animal studies. These findings highlight a reproducible motif of reinforcement learning in the cortico-striatal loops and identify methodological dimensions that may influence the reproducibility of activation patterns across studies.

Goal or gold: overlapping reward processes in soccer players upon scoring and winning money

Social rewards are important incentives for human behavior. This is especially true in team sports such as the most popular one worldwide: soccer. We investigated reward processing upon scoring a soccer goal in a standard two-versus-one situation and in comparison to winning in a monetary incentive task. The results show a strong overlap in brain activity between the two conditions in established reward regions of the mesolimbic dopaminergic system, including the ventral striatum and ventromedial pre-frontal cortex. The three main components of reward-associated learning i.e. reward probability (RP), reward reception (RR) and reward prediction errors (RPE) showed highly similar activation in both con-texts, with only the RR and RPE components displaying overlapping reward activity. Passing and shooting behavior did not correlate with individual egoism scores, but we observe a positive correlation be-tween egoism and activity in the left middle frontal gyrus upon scoring after a pass versus a direct shot. Our findings suggest that rewards in the context of soccer and monetary incentives are based on similar neural processes.