Reward-based contextual learning supported by anterior cingulate cortex

Unraveling the influence of trial-based motivational changes on performance monitoring stages in a flanker task

Performance monitoring (PM) is a vital component of adaptive behavior and known to be influenced by motivation. We examined effects of potential gain (PG) and loss avoidance (LA) on neural correlates of PM at different processing stages, using a task with trial-based changes in these motivational contexts. Findings suggest more attention is allocated to the PG context, with higher amplitudes for respective correlates of stimulus and feedback processing. The PG context favored rapid responses, while the LA context emphasized accurate responses. Lower response thresholds in the PG context after correct responses derived from a drift-diffusion model also indicate a more approach-oriented response style in the PG context. This cognitive shift is mirrored in neural correlates: negative feedback in the PG context elicited a higher feedback-related negativity (FRN) and higher theta power, whereas positive feedback in the LA context elicited higher P3a and P3b amplitudes, as well as higher theta power. There was no effect of motivational context on response-locked brain activity. Given the similar frequency of negative feedback in both contexts, the elevated FRN and theta power in PG trials cannot be attributed to variations in reward prediction error. The observed variations in the FRN indicate that the effect of outcome valence is modulated by motivational salience.

Abstraction of Reward Context Facilitates Relative Reward Coding in Neural Populations of the Macaque Anterior Cingulate Cortex

The anterior cingulate cortex (ACC) is believed to be involved in many cognitive processes, including linking goals to actions and tracking decision-relevant contextual information. ACC neurons robustly encode expected outcomes, but how this relates to putative functions of ACC remains unknown. Here, we approach this question from the perspective of population codes by analyzing neural spiking data in the ventral and dorsal banks of the ACC in two male monkeys trained to perform a stimulus-motor mapping task to earn rewards or avoid losses. We found that neural populations favor a low dimensional representational geometry that emphasizes the valence of potential outcomes while also facilitating the independent, abstract representation of multiple task-relevant variables. Valence encoding persisted throughout the trial, and realized outcomes were primarily encoded in a relative sense, such that cue valence acted as a context for outcome encoding. This suggests that the population coding we observe could be a mechanism that allows feedback to be interpreted in a context-dependent manner. Together, our results point to a prominent role for ACC in context setting and relative interpretation of outcomes, facilitated by abstract, or untangled, representations of task variables.SIGNIFICANCE STATEMENT The ability to interpret events in light of the current context is a critical facet of higher-order cognition. The ACC is suggested to be important for tracking contextual information, whereas alternate views hold that its function is more related to the motor system and linking goals to appropriate actions. We evaluated these possibilities by analyzing geometric properties of neural population activity in monkey ACC when contexts were determined by the valence of potential outcomes and found that this information was represented as a dominant, abstract concept. Ensuing outcomes were then coded relative to these contexts, suggesting an important role for these representations in context-dependent evaluation. Such mechanisms may be critical for the abstract reasoning and generalization characteristic of biological intelligence.

Connectome-based prediction of functional impairment in experimental stroke models

Experimental rat models of stroke and hemorrhage are important tools to investigate cerebrovascular disease pathophysiology mechanisms, yet how significant patterns of functional impairment induced in various models of stroke are related to changes in connectivity at the level of neuronal populations and mesoscopic parcellations of rat brains remain unresolved. To address this gap in knowledge, we employed two middle cerebral artery occlusion models and one intracerebral hemorrhage model with variant extent and location of neuronal dysfunction. Motor and spatial memory function was assessed and the level of hippocampal activation via Fos immunohistochemistry. Contribution of connectivity change to functional impairment was analyzed for connection similarities, graph distances and spatial distances as well as the importance of regions in terms of network architecture based on the neuroVIISAS rat connectome. We found that functional impairment correlated with not only the extent but also the locations of the injury among the models. In addition, via coactivation analysis in dynamic rat brain models, we found that lesioned regions led to stronger coactivations with motor function and spatial learning regions than with other unaffected regions of the connectome. Dynamic modeling with the weighted bilateral connectome detected changes in signal propagation in the remote hippocampus in all 3 stroke types, predicting the extent of hippocampal hypoactivation and impairment in spatial learning and memory function. Our study provides a comprehensive analytical framework in predictive identification of remote regions not directly altered by stroke events and their functional implication.

Reinforcing effect of tramadol in the rat

Tramadol is one of the most commonly prescribed analgesic opioids in various pharmacopeias. Tramadol has been linked to abuse in recent clinical investigations. However, the behavioral effects and neural substrates of the drug have not been well characterized in preclinical studies. As a result, the present study investigated the effects of tramadol on behavioral sensitizations in rats. Its impacts on cellular and molecular alterations in the brain were also investigated. In conditioned place preference (CPP) paradigm, tramadol induced behavioral as well as motor sensitizations. These effects were dramatically reduced by intraperitoneal administration of naltrexone, an opioid receptor antagonist. Tramadol caused changes in several molecular markers (pERK1/2, Δ-FosB, PKCγ, PKMζ GAD67) in the anterior cingulate cortex, which could indicate an increase in excitation within this structure. Tramadol is demonstrated in the present study to be a reinforcing drug in rats, as it increased both behavioral and motor sensitizations. Tramadol's effects are most likely due to the high levels of excitation it causes in the brain, which is mostly caused by the activation of opioid receptors.

Task-level value affects trial-level reward processing

Despite disagreement about how anterior cingulate cortex (ACC) supports decision making, a recent hypothesis suggests that activity in this region is best understood in the context of a task or series of tasks. One important task-level variable is average reward because it is both a known driver of effortful behaviour and an important determiner of the tasks in which we choose to engage. Here we asked how average task value affects reward-related ACC activity. To answer this question, we measured a reward-related signal said to be generated in ACC called the reward positivity (RewP) while participants gambled in three tasks of differing average value. The RewP was reduced in the high-value task, an effect that was not explainable by either reward magnitude or outcome expectancy. This result suggests that ACC does not evaluate outcomes and cues in isolation, but in the context of the value of the current task.

Baseline brain and behavioral factors distinguish adolescent substance initiators and non-initiators at follow-up

Background

Earlier substance use (SU) initiation is associated with greater risk for the development of SU disorders (SUDs), while delays in SU initiation are associated with a diminished risk for SUDs. Thus, identifying brain and behavioral factors that are markers of enhanced risk for earlier SU has major public health import. Heightened reward-sensitivity and risk-taking are two factors that confer risk for earlier SU.

Materials and methods

We characterized neural and behavioral factors associated with reward-sensitivity and risk-taking in substance-naïve adolescents (N = 70; 11.1-14.0 years), examining whether these factors differed as a function of subsequent SU initiation at 18- and 36-months follow-up. Adolescents completed a reward-related decision-making task while undergoing functional MRI. Measures of reward sensitivity (Behavioral Inhibition System-Behavioral Approach System; BIS-BAS), impulsive decision-making (delay discounting task), and SUD risk [Drug Use Screening Inventory, Revised (DUSI-R)] were collected. These metrics were compared for youth who did [Substance Initiators (SI); n = 27] and did not [Substance Non-initiators (SN); n = 43] initiate SU at follow-up.

Results

While SI and SN youth showed similar task-based risk-taking behavior, SI youth showed more variable patterns of activation in left insular cortex during high-risk selections, and left anterior cingulate cortex in response to rewarded outcomes. Groups displayed similar discounting behavior. SI participants scored higher on the DUSI-R and the BAS sub-scale.

Conclusion

Activation patterns in the insula and anterior cingulate cortex may serve as a biomarker for earlier SU initiation. Importantly, these brain regions are implicated in the development and experience of SUDs, suggesting differences in these regions prior to substance exposure.

Abnormal brain activity in nonsuicidal self-injury: a coordinate-based activation likelihood meta-analysis of functional neuroimaging studies

Background

The high prevalence of nonsuicidal self-injury (NSSI) in youths demonstrates a substantial population-level burden on society. NSSI is often associated with emotional and social skill deficits. To date, several studies have aimed to identify the underlying neural mechanism of those deficits in NSSI by using functional magnetic resonance imaging (fMRI). However, their conclusions display poor consistency.

Objective

We aimed to conduct a meta-analysis using activation likelihood estimation (ALE) for fMRI data based on emotional and cognitive tasks to clarify the underlying neural processing deficits of NSSI.

Methods

We searched for MRI studies of NSSI in the PubMed, Cochrane, and Embase databases. We identified significant foci for the included studies and conducted two ALE meta-analyses as follows: (i) activation for the NSSI contrast healthy control group and (ii) deactivation for the NSSI contrast healthy controls. Considering the diverse sex composition of study participants and possible bias from one large sample study, we conducted sensitivity analyses for the meta-analysis.

Results

Nine studies comprising 359 participants were included, and the results demonstrated substantial activation in NSSI patients compared with healthy controls in two clusters located in the right medial frontal gyrus extending to the rostral anterior cingulate and the left inferior frontal gyrus extending to the insula.

Conclusions

The results suggest that individuals with NSSI show brain activity alterations that underpin their core symptoms, including poor emotional regulation and reward processing deficits. Our findings provide new insights into the neural mechanism of NSSI, which may serve as functional biomarkers for developing effective diagnosis and therapeutic interventions for these patients.

The Paradoxical Effect Hypothesis of Abused Drugs in a Rat Model of Chronic Morphine Administration

A growing body of studies has recently shown that abused drugs could simultaneously induce the paradoxical effect in reward and aversion to influence drug addiction. However, whether morphine induces reward and aversion, and which neural substrates are involved in morphine’s reward and aversion remains unclear. The present study first examined which doses of morphine can simultaneously produce reward in conditioned place preference (CPP) and aversion in conditioned taste aversion (CTA) in rats. Furthermore, the aversive dose of morphine was determined. Moreover, using the aversive dose of 10 mg/kg morphine tested plasma corticosterone (CORT) levels and examined which neural substrates were involved in the aversive morphine-induced CTA on conditioning, extinction, and reinstatement. Further, we analyzed c-Fos and p-ERK expression to demonstrate the paradoxical effect—reward and aversion and nonhomeostasis or disturbance by morphine-induced CTA. The results showed that a dose of more than 20 mg/kg morphine simultaneously induced reward in CPP and aversion in CTA. A dose of 10 mg/kg morphine only induced the aversive CTA, and it produced higher plasma CORT levels in conditioning and reacquisition but not extinction. High plasma CORT secretions by 10 mg/kg morphine-induced CTA most likely resulted from stress-related aversion but were not a rewarding property of morphine. For assessments of c-Fos and p-ERK expression, the cingulate cortex 1 (Cg1), prelimbic cortex (PrL), infralimbic cortex (IL), basolateral amygdala (BLA), nucleus accumbens (NAc), and dentate gyrus (DG) were involved in the morphine-induced CTA, and resulted from the aversive effect of morphine on conditioning and reinstatement. The c-Fos data showed fewer neural substrates (e.g., PrL, IL, and LH) on extinction to be hyperactive. In the context of previous drug addiction data, the evidence suggests that morphine injections may induce hyperactivity in many neural substrates, which mediate reward and/or aversion due to disturbance and nonhomeostasis in the brain. The results support the paradoxical effect hypothesis of abused drugs. Insight from the findings could be used in the clinical treatment of drug addiction.

Neural Dynamics of the Combined Discounting of Delay and Probability During the Evaluation of a Delayed Risky Reward

Delay discounting and probability discounting are two important processes, but in daily life there are many more situations that involve delayed risky outcomes. Although neuroscience research has extensively investigated delay and probability discounting in isolation, little research has explored the neural correlates of the combined discounting of delay and probability. Using the event-related brain potentials (ERPs) technique, we designed a novel paradigm to investigate neural processes related to the combined discounting of delay and probability during the evaluation of a delayed risky reward. ERP results suggested distinct temporal dynamics for delay and probability processing during combined discounting. Both the early frontal P200 and the N2 reflected only probability, not delay, while the parietal P300 was sensitive to both probability and delay. Furthermore, the late positive potential (LPP) was sensitive to probability, but insensitive to delay. These results suggest that probability has a prolonged modulatory effect on reward evaluation in the information processing stream. These findings contribute to an understanding of the neural processes underlying the combined discounting of delay and probability. The limitation of this study is to only consider four delay and probability combinations. Future studies can explore the combined discounting of more probability and delay combinations to further test the robustness of the conclusion.

Supplementation with a putative calorie restriction mimetic micronutrient blend increases glutathione concentrations and improves neuroenergetics in brain of healthy middle-aged men and women

BACKGROUND

Caloric restriction (CR) without micronutrient deficiency has been shown to increase both lifespan and healthspan. In animals, CR has been demonstrated to increase glutathione (GSH), a neuroprotective antioxidant, in the brain and preserve brain mitochondrial function by altering neuroenergetics. In humans it has been associated with improvements in mood states and cognitive function. However, most CR studies have employed a 30-60% reduction in calories which is likely too stringent for most people to adhere to long-term. Thus, there is an unmet need for nutritional supplements which can mimic the biological effects of CR, without the need for calorie limitations.

AIM

The purpose of the present randomized, placebo-controlled clinical trial was to use Proton (¹H) Magnetic Resonance Spectroscopic (MRS) measurements to determine non-invasively whether a blend of micronutrients, a putative CR mimetic, positively modulates metabolites related to neuroprotection and neuroenergetics in the brain.

METHODS

Healthy middle-aged men and women (N = 63 [33 women]; age: 40-60 years) were randomized in a double-blind manner to 6 weeks supplementation with either the putative CR mimetic or placebo. At baseline and 6 weeks, subjects underwent MRS at 3 T to investigate changes in brain chemistry, including the neurometabolites: GSH, Glutamate (Glu), Glutamine (Gln) and N-Acetylaspartate (NAA).

RESULTS

GSH, a marker of antioxidant and cellular redox status, increased in the brain of participants in the supplement group. The supplement group also showed an increase in the Glu/Gln ratio, a marker of excitatory neurotransmission and bioenergetics. A trend for an increase in NAA/H₂O, a marker of neuronal integrity, was observed in females in the supplement group.

CONCLUSIONS

The present study reveals that 6-weeks daily supplementation with a micronutrient blend elicits positive changes in brain neurochemistry. This is the first study to demonstrate that a putative CR mimetic increases brain GSH concentrations and improves neuroprotection and neuroenergetics in the brain of healthy humans. This study was registered at www.clinicaltrials.gov as NCT02439983.

Individual differences of conflict monitoring and feedback processing during reinforcement learning in a mock forensic context

This study investigated individual differences of conflict monitoring (N2 component), feedback processing (feedback negativity component), and reinforcement learning in a discrimination learning task using a mock (fictitious) forensic scenario to set participants in a semantic task context. We investigated individual differences of anxiety-related, impulsivity-related traits and reasoning ability during trial-and-error learning of mock suspect and nonsuspect faces. Thereby, we asked how the differential investment of cognitive-motivational processes facilitates learning in a mock forensic context. As learning can be studied by means of time-on-task effects (i.e., variations of cognitive processes across task blocks), we investigated the differential investment of cognitive-motivational processes block-wise in N = 100 participants. By performing structural equation modeling, we demonstrate that conflict monitoring decreased across task blocks, whereas the percentage of correct responses increased across task blocks. Individuals with higher reasoning scores and higher impulsivity-related traits relied rather on feedback processing (i.e., external indicators) during reinforcement learning. Individuals with higher anxiety-related traits intensified their conflict monitoring throughout the task to learn successfully. Observation by relevant others intensified conflict monitoring more than nonobservation. Our data highlight that individual differences and social context modulate the intensity of information processing in a discrimination learning task using a mock forensic task scenario. We discuss our data with regard to recent cognitive-motivational approaches and in terms of reinforcement learning.

Electrophysiological measures reveal the role of anterior cingulate cortex in learning from unreliable feedback

Although a growing number of studies have investigated the neural mechanisms of reinforcement learning, it remains unclear how the brain responds to feedback that is unreliable. A recent theory proposes that the reward positivity (RewP) component of the event-related brain potential (ERP) and frontal midline theta (FMT) power reflect separate feedback-related processing functions of anterior cingulate cortex (ACC). In the present study, the electroencephalogram (EEG) was recorded from participants as they engaged in a time estimation task in which feedback reliability was manipulated across conditions. After each response, they received a cue that indicated that the following feedback stimulus was 100%, 75%, or 50% reliable. The results showed that participants' time estimates adjusted linearly according to the feedback reliability. Moreover, presentation of the cue indicating 100% reliability elicited a larger RewP-like ERP component than the other cues did, and feedback presentation elicited a RewP of approximately equal amplitude for all of the three reliability conditions. By contrast, FMT power elicited by negative feedback decreased linearly from the 100% condition to 75% and 50% condition, and only FMT power predicted behavioral adjustments on the following trials. In addition, an analysis of Beta power and cross-frequency coupling (CFC) of Beta power with FMT phase suggested that Beta-FMT communication modulated motor areas for the purpose of adjusting behavior. We interpreted these findings in terms of the hierarchical reinforcement learning account of ACC, in which the RewP and FMT are proposed to reflect reward processing and control functions of ACC, respectively.

Passively learned spatial navigation cues evoke reinforcement learning reward signals

Since the suggestion by Tolman (1948) that both rodents and humans create cognitive maps during navigation, the specifics of how navigators learn about their environment has been mired in debate. One facet of this debate is whether or not the creation of cognitive maps - also known as allocentric navigation - involves reinforcement learning. Here, we demonstrate a role for reinforcement learning during allocentric navigation using event-related brain potentials (ERPs). In the present experiment, participants navigated in a virtual environment that allowed the use of three different navigation strategies (allocentric, egocentric-response, & egocentric-cue), in which their goal was to locate and remember a hidden platform. Following the navigation phase of the experiment, participants were shown "cue images" representative of the three navigation strategies. Specifically, we examined whether or not these passively learned strategy images elicited a reward positivity - an ERP component associated with reinforcement learning and the anterior cingulate cortex. We found that when allocentric navigators were shown previously learned cues predicting the goal location a reward positivity was elicited. The present findings demonstrate that allocentric navigational cues carry long-term value after navigation and lend support to the claim that reinforcement learning plays a role in the acquisition of allocentric navigation and thus the generation of cognitive maps.

Neural Dynamics of Processing Probability Weight and Monetary Magnitude in the Evaluation of a Risky Reward

Risky decision-making involves risky reward valuation, choice, and feedback processes. However, the temporal dynamics of risky reward processing are not well understood. Using event-related brain potential, we investigated the neural correlates of probability weight and money magnitude in the evaluation of a risky reward. In this study, each risky choice consisted of two risky options, which were presented serially to separate decision-making and option evaluation processes. The early P200 component reflected the process of probability weight, not money magnitude. The medial frontal negativity (MFN) reflected both probability weight and money magnitude processes. The late positive potential (LPP) only reflected the process of probability weight. These results demonstrate distinct temporal dynamics for probability weight and money magnitude processes when evaluating a risky outcome, providing a better understanding of the possible mechanism underlying risky reward processing.

Subgoal- and Goal-related Reward Prediction Errors in Medial Prefrontal Cortex

A longstanding view of the organization of human and animal behavior holds that behavior is hierarchically organized—in other words, directed toward achieving superordinate goals through the achievement of subordinate goals or subgoals. However, most research in neuroscience has focused on tasks without hierarchical structure. In past work, we have shown that negative reward prediction error (RPE) signals in medial prefrontal cortex (mPFC) can be linked not only to superordinate goals but also to subgoals. This suggests that mPFC tracks impediments in the progression toward subgoals. Using fMRI of human participants engaged in a hierarchical navigation task, here we found that mPFC also processes positive prediction errors at the level of subgoals, indicating that this brain region is sensitive to advances in subgoal completion. However, when subgoal RPEs were elicited alongside with goal-related RPEs, mPFC responses reflected only the goal-related RPEs. These findings suggest that information from different levels of hierarchy is processed selectively, depending on the task context.

Computational Models of Anterior Cingulate Cortex: At the Crossroads between Prediction and Effort

In the last two decades the anterior cingulate cortex (ACC) has become one of the most investigated areas of the brain. Extensive neuroimaging evidence suggests countless functions for this region, ranging from conflict and error coding, to social cognition, pain and effortful control. In response to this burgeoning amount of data, a proliferation of computational models has tried to characterize the neurocognitive architecture of ACC. Early seminal models provided a computational explanation for a relatively circumscribed set of empirical findings, mainly accounting for EEG and fMRI evidence. More recent models have focused on ACC's contribution to effortful control. In parallel to these developments, several proposals attempted to explain within a single computational framework a wider variety of empirical findings that span different cognitive processes and experimental modalities. Here we critically evaluate these modeling attempts, highlighting the continued need to reconcile the array of disparate ACC observations within a coherent, unifying framework.