The Dopamine Prediction Error: Contributions to Associative Models of Reward Learning

Electrical brain activations in preadolescents during a probabilistic reward-learning task reflect cognitive processes and behavioral strategy

Both adults and children learn through feedback which environmental events and choices are associated with higher probability of reward, an ability thought to be supported by the development of fronto-striatal reward circuits. Recent developmental studies have applied computational models of reward learning to investigate such learning in children. However, tasks and measures effective for assaying the cascade of reward-learning neural processes in children have been limited. Using a child-version of a probabilistic reward-learning task while recording event-related-potential (ERP) measures of electrical brain activity, this study examined key processes of reward learning in preadolescents (8-12 years old; n=30), namely: (1) reward-feedback sensitivity, as measured by the early-latency, reward-related, frontal ERP positivity, (2) rapid attentional shifting of processing toward favored visual stimuli, as measured by the N2pc component, and (3) longer-latency attention-related responses to reward feedback as a function of behavioral strategies (i.e., Win-Stay-Lose-Shift), as measured by the central-parietal P300. Consistent with our prior work in adults, the behavioral findings indicate preadolescents can learn stimulus-reward outcome associations, but at varying levels of performance. Neurally, poor preadolescent learners (those with slower learning rates) showed greater reward-related positivity amplitudes relative to good learners, suggesting greater reward-feedback sensitivity. We also found attention shifting towards to-be-chosen stimuli, as evidenced by the N2pc, but not to more highly rewarded stimuli as we have observed in adults. Lastly, we found the behavioral learning strategy (i.e., Win-Stay-Lose-Shift) reflected by the feedback-elicited parietal P300. These findings provide novel insights into the key neural processes underlying reinforcement learning in preadolescents.

Dopamine projections to the basolateral amygdala drive the encoding of identity-specific reward memories

To make adaptive decisions, we build an internal model of the associative relationships in an environment and use it to make predictions and inferences about specific available outcomes. Detailed, identity-specific cue-reward memories are a core feature of such cognitive maps. Here we used fiber photometry, cell-type and pathway-specific optogenetic manipulation, Pavlovian cue-reward conditioning and decision-making tests in male and female rats, to reveal that ventral tegmental area dopamine (VTADA) projections to the basolateral amygdala (BLA) drive the encoding of identity-specific cue-reward memories. Dopamine is released in the BLA during cue-reward pairing; VTADA→BLA activity is necessary and sufficient to link the identifying features of a reward to a predictive cue but does not assign general incentive properties to the cue or mediate reinforcement. These data reveal a dopaminergic pathway for the learning that supports adaptive decision-making and help explain how VTADA neurons achieve their emerging multifaceted role in learning.

Asymmetric coding of reward prediction errors in human insula and dorsomedial prefrontal cortex

The signed value and unsigned salience of reward prediction errors (RPEs) are critical to understanding reinforcement learning (RL) and cognitive control. Dorsomedial prefrontal cortex (dMPFC) and insula (INS) are key regions for integrating reward and surprise information, but conflicting evidence for both signed and unsigned activity has led to multiple proposals for the nature of RPE representations in these brain areas. Recently developed RL models allow neurons to respond differently to positive and negative RPEs. Here, we use intracranially recorded high frequency activity (HFA) to test whether this flexible asymmetric coding strategy captures RPE coding diversity in human INS and dMPFC. At the region level, we found a bias towards positive RPEs in both areas which paralleled behavioral adaptation. At the local level, we found spatially interleaved neural populations responding to unsigned RPE salience and valence-specific positive and negative RPEs. Furthermore, directional connectivity estimates revealed a leading role of INS in communicating positive and unsigned RPEs to dMPFC. These findings support asymmetric coding across distinct but intermingled neural populations as a core principle of RPE processing and inform theories of the role of dMPFC and INS in RL and cognitive control.

Dopaminergic D2-like receptor stimulation affects attention on contextual information and modulates BOLD activation of extinction-related brain areas

Contextual information is essential for learning and memory processes and plays a crucial role during the recall of extinction memory, and in the renewal effect, which is the context-dependent recovery of an extinguished response. The dopaminergic system is known to be involved in regulating attentional processes by shifting attention to novel and salient contextual cues. Higher dopamine levels are associated with a better recall of previously learned stimulus-outcome associations and enhanced encoding, as well as retrieval of contextual information which promotes renewal. In this fMRI study, we aimed to investigate the impact of processing contextual information and the influence of dopaminergic D2-like receptor activation on attention to contextual information during a predictive learning task as well as upon extinction learning, memory performance, and activity of extinction-related brain areas. A single oral dose of 1.25 mg bromocriptine or an identical-looking placebo was administered to the participants. We modified a predictive learning task that in previous studies reliably evoked a renewal effect, by increasing the complexity of contextual information. We analysed fixations and dwell on contextual cues by use of eye-tracking and correlated these with behavioural performance and BOLD activation of extinction-related brain areas. Our results indicate that the group with dopaminergic D2-like receptor stimulation had higher attention to task-relevant contextual information and greater/lower BOLD activation of brain regions associated with cognitive control during extinction learning and recall. Moreover, renewal responses were almost completely absent. Since this behavioural effect was observed for both treatment groups, we assume that this was due to the complexity of the altered task design.

Dynamics of Lateral Habenula-Ventral Tegmental Area Microcircuit on Pain-Related Cognitive Dysfunctions

Chronic pain is a health problem that affects the ability to work and perform other activities, and it generally worsens over time. Understanding the complex pain interaction with brain circuits could help predict which patients are at risk of developing central dysfunctions. Increasing evidence from preclinical and clinical studies suggests that aberrant activity of the lateral habenula (LHb) is associated with depressive symptoms characterized by excessive negative focus, leading to high-level cognitive dysfunctions. The primary output region of the LHb is the ventral tegmental area (VTA), through a bidirectional connection. Recently, there has been growing interest in the complex interactions between the LHb and VTA, particularly regarding their crucial roles in behavior regulation and their potential involvement in the pathological impact of chronic pain on cognitive functions. In this review, we briefly discuss the structural and functional roles of the LHb-VTA microcircuit and their impact on cognition and mood disorders in order to support future studies addressing brain plasticity during chronic pain conditions.

Role of Dopamine Neurons in Familiarity

Dopamine neurons signal the salience of environmental stimuli, influencing learning and motivation. However, research has not yet identified whether dopamine neurons also modulate the salience of memory content. Dopamine neuron activity in the ventral tegmental area (VTA) increases in response to novel objects and diminishes as objects become familiar through repeated presentations. We proposed that the declined rate of dopamine neuron activity during familiarization affects the salience of a familiar object's memory. This, in turn, influences the degree to which an animal distinguishes between familiar and novel objects in a subsequent novel object recognition (NOR) test. As such, a single familiarization session may not sufficiently reduce dopamine activity, allowing the memory of a familiar object to maintain its salience and potentially attenuating NOR. In contrast, multiple familiarization sessions could lead to more pronounced dopamine activity suppression, strengthening NOR. Our data in mice reveals that, compared to a single session, multiple sessions result in decreased VTA dopamine neuron activation, as indicated by c-Fos measurements, and enhanced novelty discrimination. Critically, when VTA dopamine neurons are chemogenetically inhibited during a single familiarization session, NOR improves, mirroring the effects of multiple familiarization sessions. In summary, our findings highlight the pivotal function of dopamine neurons in familiarity and suggest a role in modulating the salience of memory content.

Enhancing reinforcement learning models by including direct and indirect pathways improves performance on striatal dependent tasks

A major advance in understanding learning behavior stems from experiments showing that reward learning requires dopamine inputs to striatal neurons and arises from synaptic plasticity of cortico-striatal synapses. Numerous reinforcement learning models mimic this dopamine-dependent synaptic plasticity by using the reward prediction error, which resembles dopamine neuron firing, to learn the best action in response to a set of cues. Though these models can explain many facets of behavior, reproducing some types of goal-directed behavior, such as renewal and reversal, require additional model components. Here we present a reinforcement learning model, TD2Q, which better corresponds to the basal ganglia with two Q matrices, one representing direct pathway neurons (G) and another representing indirect pathway neurons (N). Unlike previous two-Q architectures, a novel and critical aspect of TD2Q is to update the G and N matrices utilizing the temporal difference reward prediction error. A best action is selected for N and G using a softmax with a reward-dependent adaptive exploration parameter, and then differences are resolved using a second selection step applied to the two action probabilities. The model is tested on a range of multi-step tasks including extinction, renewal, discrimination; switching reward probability learning; and sequence learning. Simulations show that TD2Q produces behaviors similar to rodents in choice and sequence learning tasks, and that use of the temporal difference reward prediction error is required to learn multi-step tasks. Blocking the update rule on the N matrix blocks discrimination learning, as observed experimentally. Performance in the sequence learning task is dramatically improved with two matrices. These results suggest that including additional aspects of basal ganglia physiology can improve the performance of reinforcement learning models, better reproduce animal behaviors, and provide insight as to the role of direct- and indirect-pathway striatal neurons.

Dopamine in the dorsal bed nucleus of stria terminalis signals Pavlovian sign-tracking and reward violations

Midbrain and striatal dopamine signals have been extremely well characterized over the past several decades, yet novel dopamine signals and functions in reward learning and motivation continue to emerge. A similar characterization of real-time sub-second dopamine signals in areas outside of the striatum has been limited. Recent advances in fluorescent sensor technology and fiber photometry permit the measurement of dopamine binding correlates, which can divulge basic functions of dopamine signaling in non-striatal dopamine terminal regions, like the dorsal bed nucleus of the stria terminalis (dBNST). Here, we record GRABDA signals in the dBNST during a Pavlovian lever autoshaping task. We observe greater Pavlovian cue-evoked dBNST GRABDA signals in sign-tracking (ST) compared to goal-tracking/intermediate (GT/INT) rats and the magnitude of cue-evoked dBNST GRABDA signals decreases immediately following reinforcer-specific satiety. When we deliver unexpected rewards or omit expected rewards, we find that dBNST dopamine signals encode bidirectional reward prediction errors in GT/INT rats, but only positive prediction errors in ST rats. Since sign- and goal-tracking approach strategies are associated with distinct drug relapse vulnerabilities, we examined the effects of experimenter-administered fentanyl on dBNST dopamine associative encoding. Systemic fentanyl injections do not disrupt cue discrimination but generally potentiate dBNST dopamine signals. These results reveal multiple dBNST dopamine correlates of learning and motivation that depend on the Pavlovian approach strategy employed.

Choice between food and cocaine reinforcers under fixed and variable schedules in female and male rhesus monkeys

Illicit drugs like cocaine may be uncertain in terms of the time and effort required to obtain them. Behavior maintained by variable schedules resembles excessive drug-taking compared with fixed schedules. However, no prior research has examined fixed versus variable schedules in drug versus nondrug choice. The present study evaluated cocaine versus food choice under fixed- (FR) and variable-ratio (VR) schedules. The simpler food versus food and cocaine versus cocaine arrangements also were included. Adult female (n = 6) and male (n = 7) rhesus monkeys chose between cocaine (0.01-0.18 mg/kg/injection) and food (4 pellets/delivery), food and food (4 pellets/delivery), or cocaine and cocaine (0.018-0.03 mg/kg/injection) under FR and VR 100 and 200 schedules. In cocaine versus food choice, cocaine's potency to maintain choice was greatest when available under a VR 100 or 200 schedule and food under an FR schedule and was lowest when cocaine was available under an FR 200 schedule and food was available under a VR 200 schedule. In food versus food choice, males chose food associated with a VR schedule more than food associated with an FR schedule. In cocaine versus cocaine choice, females and males chose cocaine associated with a VR schedule more than cocaine associated with an FR schedule, particularly under VR 200. These findings suggest that uncertainty in terms of time and effort required to obtain cocaine, or perhaps the occasional low-cost access that results from VR schedules, results in greater allocation of behavior toward drug reinforcers at the expense of more certain, nondrug alternatives. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Reward-Mediated, Model-Free Reinforcement-Learning Mechanisms in Pavlovian and Instrumental Tasks Are Related

Model-free and model-based computations are argued to distinctly update action values that guide decision-making processes. It is not known, however, if these model-free and model-based reinforcement learning mechanisms recruited in operationally based instrumental tasks parallel those engaged by pavlovian-based behavioral procedures. Recently, computational work has suggested that individual differences in the attribution of incentive salience to reward predictive cues, that is, sign- and goal-tracking behaviors, are also governed by variations in model-free and model-based value representations that guide behavior. Moreover, it is not appreciated if these systems that are characterized computationally using model-free and model-based algorithms are conserved across tasks for individual animals. In the current study, we used a within-subject design to assess sign-tracking and goal-tracking behaviors using a pavlovian conditioned approach task and then characterized behavior using an instrumental multistage decision-making (MSDM) task in male rats. We hypothesized that both pavlovian and instrumental learning processes may be driven by common reinforcement-learning mechanisms. Our data confirm that sign-tracking behavior was associated with greater reward-mediated, model-free reinforcement learning and that it was also linked to model-free reinforcement learning in the MSDM task. Computational analyses revealed that pavlovian model-free updating was correlated with model-free reinforcement learning in the MSDM task. These data provide key insights into the computational mechanisms mediating associative learning that could have important implications for normal and abnormal states.SIGNIFICANCE STATEMENT Model-free and model-based computations that guide instrumental decision-making processes may also be recruited in pavlovian-based behavioral procedures. Here, we used a within-subject design to test the hypothesis that both pavlovian and instrumental learning processes were driven by common reinforcement-learning mechanisms. Sign-tracking and goal-tracking behaviors were assessed in rats using a pavlovian conditioned approach task, and then instrumental behavior was characterized using an MSDM task. We report that sign-tracking behavior was associated with greater model-free, but not model-based, learning in the MSDM task. These data suggest that pavlovian and instrumental behaviors may be driven by conserved reinforcement-learning mechanisms.

Reinforcement learning deficits exhibited by postnatal PCP-treated rats enable deep neural network classification

The ability to appropriately update the value of a given action is a critical component of flexible decision making. Several psychiatric disorders, including schizophrenia, are associated with impairments in flexible decision making that can be evaluated using the probabilistic reversal learning (PRL) task. The PRL task has been reverse-translated for use in rodents. Disrupting glutamate neurotransmission during early postnatal neurodevelopment in rodents has induced behavioral, cognitive, and neuropathophysiological abnormalities relevant to schizophrenia. Here, we tested the hypothesis that using the NMDA receptor antagonist phencyclidine (PCP) to disrupt postnatal glutamatergic transmission in rats would lead to impaired decision making in the PRL. Consistent with this hypothesis, compared to controls the postnatal PCP-treated rats completed fewer reversals and exhibited disruptions in reward and punishment sensitivity (i.e., win-stay and lose-shift responding, respectively). Moreover, computational analysis of behavior revealed that postnatal PCP-treatment resulted in a pronounced impairment in the learning rate throughout PRL testing. Finally, a deep neural network (DNN) trained on the rodent behavior could accurately predict the treatment group of subjects. These data demonstrate that disrupting early postnatal glutamatergic neurotransmission impairs flexible decision making and provides evidence that DNNs can be trained on behavioral datasets to accurately predict the treatment group of new subjects, highlighting the potential for DNNs to aid in the diagnosis of schizophrenia.

An Adaptive Motivation Approach to Understanding the 'How' and 'Why' of Wellbeing

A new model provides insight into the 'how' and 'why' of wellbeing to better understand the 'what'. Informed by evolutionary psychology and neuroscience, it proposes that systems for adaptive motivation underpin experiential and reflective wellbeing. The model proposes that the brain learns to predict situations, and errors arise between the predictions and experience. These prediction errors drive emotional experience, learning, motivation, decision-making, and the formation of wellbeing-relevant memories. The model differentiates four layers of wellbeing: objective, experiential, reflective, and narrative, which relate to the model in different ways. Constituents of wellbeing, human motives, and specific emotions integrate into the model. A simple computational implementation of the model reproduced several established wellbeing phenomena, including: the greater frequency of pleasant to unpleasant emotions, the stronger emotional salience of unpleasant emotions, hedonic adaptation to changes in circumstances, heritable influences on wellbeing, and affective forecasting errors. It highlights the importance of individual differences, and implies that high wellbeing will correlate with the experience of infrequent, routine, and predictable avoidance cues and frequent, varied, and novel approach cues. The model suggests that wellbeing arises directly from a system for adaptive motivation. This system functions like a mental dashboard that calls attention to situational changes and motivates the kinds of behaviours that gave humans a relative advantage in their ancestral environment. The model offers a set of fundamental principles and processes that may underlie diverse conceptualisations of wellbeing.

Utility of 'substance use disorder' as a heuristic for understanding overeating and obesity

Rates of obesity and obesity-associated diseases have increased dramatically in countries with developed economies. Substance use disorders (SUDs) are characterized by the persistent use of the substance despite negative consequences. It has been hypothesized that overconsumption of palatable energy dense food can elicit SUD-like maladaptive behaviors that contribute to persistent caloric intake beyond homeostatic need even in the face of negative consequences. Palatable food and drugs of abuse act on many of the same motivation-related circuits in the brain, and can induce, at least superficially, similar molecular, cellular, and physiological adaptations on these circuits. As such, applying knowledge about the neurobiological mechanisms of SUDs may serve as useful heuristic to better understand the persistent overconsumption of palatable food that contributes to obesity. However, many important differences exist between the actions of drugs of abuse and palatable food in the brain. This warrants caution when attributing weight gain and obesity to the manifestation of a putative SUD-related behavioral disorder. Here, we describe similarities and differences between compulsive drug use in SUDs and overconsumption in obesity and consider the merit of the concept of "food addiction".

Neural signals implicated in the processing of appetitive and aversive events in social and non-social contexts

In 2014, we participated in a special issue of Frontiers examining the neural processing of appetitive and aversive events. Specifically, we reviewed brain areas that contribute to the encoding of prediction errors and value versus salience, attention and motivation. Further, we described how we disambiguated these cognitive processes and their neural substrates by using paradigms that incorporate both appetitive and aversive stimuli. We described a circuit in which the orbitofrontal cortex (OFC) signals expected value and the basolateral amygdala (BLA) encodes the salience and valence of both appetitive and aversive events. This information is integrated by the nucleus accumbens (NAc) and dopaminergic (DA) signaling in order to generate prediction and prediction error signals, which guide decision-making and learning via the dorsal striatum (DS). Lastly, the anterior cingulate cortex (ACC) is monitoring actions and outcomes, and signals the need to engage attentional control in order to optimize behavioral output. Here, we expand upon this framework, and review our recent work in which within-task manipulations of both appetitive and aversive stimuli allow us to uncover the neural processes that contribute to the detection of outcomes delivered to a conspecific and behaviors in social contexts. Specifically, we discuss the involvement of single-unit firing in the ACC and DA signals in the NAc during the processing of appetitive and aversive events in both social and non-social contexts.

Dopamine D2Rs coordinate cue-evoked changes in striatal acetylcholine levels

In the striatum, acetylcholine (ACh) neuron activity is modulated co-incident with dopamine (DA) release in response to unpredicted rewards and reward predicting cues and both neuromodulators are thought to regulate each other. While this co-regulation has been studied using stimulation studies, the existence of this mutual regulation in vivo during natural behavior is still largely unexplored. One long-standing controversy has been whether striatal DA is responsible for the induction of the cholinergic pause or whether D2R modulate a pause that is induced by other mechanisms. Here, we used genetically encoded sensors in combination with pharmacological and genetic inactivation of D2Rs from cholinergic interneurons (CINs) to simultaneously measure ACh and DA levels after CIN D2R inactivation in mice. We found that CIN D2Rs are not necessary for the initiation of cue induced decrease in ACh levels. Rather, they prolong the duration of the decrease and inhibit ACh rebound levels. Notably, the change in task evoked ACh levels is not associated with altered DA levels. Moreover, D2R inactivation strongly decreased the temporal correlation between DA and ACh signals not only at cue presentation but also during the intertrial interval pointing to a general mechanism by which D2Rs coordinate both signals. At the behavioral level D2R antagonism increased the latency to lever press, which was not observed in CIN-selective D2R knock out mice. Press latency correlated with the cue evoked decrease in ACh levels and artificial inhibition of CINs revealed that longer inhibition shortens the latency to press compared to shorter inhibition. This supports a role of the ACh signal and it's regulation by D2Rs in the motivation to initiate actions.

Distributed Neural Systems Support Flexible Attention Updating during Category Learning

To accurately categorize items, humans learn to selectively attend to stimulus dimensions that are most relevant to the task. Models of category learning describe the interconnected cognitive processes that contribute to attentional tuning as labeled stimuli are progressively observed. The Adaptive Attention Representation Model (AARM), for example, provides an account whereby categorization decisions are based on the perceptual similarity of a new stimulus to stored exemplars, and dimension-wise attention is updated on every trial in the direction of a feedback-based error gradient. As such, attention modulation as described by AARM requires interactions among orienting, visual perception, memory retrieval, prediction error, and goal maintenance to facilitate learning across trials. The current study explored the neural bases of attention mechanisms using quantitative predictions from AARM to analyze behavioral and fMRI data collected while participants learned novel categories. Generalized linear model analyses revealed patterns of BOLD activation in the parietal cortex (orienting), visual cortex (perception), medial temporal lobe (memory retrieval), basal ganglia (prediction error), and pFC (goal maintenance) that covaried with the magnitude of model-predicted attentional tuning. Results are consistent with AARM's specification of attention modulation as a dynamic property of distributed cognitive systems.

Do psychedelics change beliefs?

Renewed interest in psychedelics has reignited the debate about whether and how they change human beliefs. In both the clinical and social-cognitive domains, psychedelic consumption may be accompanied by profound, and sometimes lasting, belief changes. We review these changes and their possible underlying mechanisms. Rather than inducing de novo beliefs, we argue psychedelics may instead change the impact of affect and of others' suggestions on how beliefs are imputed. Critically, we find that baseline beliefs (in the possible effects of psychedelics, for example) might color the acute effects of psychedelics as well as longer-term changes. If we are to harness the apparent potential of psychedelics in the clinic and for human flourishing more generally, these possibilities must be addressed empirically.

Phasic Dopamine Changes and Hebbian Mechanisms during Probabilistic Reversal Learning in Striatal Circuits: A Computational Study

Cognitive flexibility is essential to modify our behavior in a non-stationary environment and is often explored by reversal learning tasks. The basal ganglia (BG) dopaminergic system, under a top-down control of the pre-frontal cortex, is known to be involved in flexible action selection through reinforcement learning. However, how adaptive dopamine changes regulate this process and learning mechanisms for training the striatal synapses remain open questions. The current study uses a neurocomputational model of the BG, based on dopamine-dependent direct (Go) and indirect (NoGo) pathways, to investigate reinforcement learning in a probabilistic environment through a task that associates different stimuli to different actions. Here, we investigated: the efficacy of several versions of the Hebb rule, based on covariance between pre- and post-synaptic neurons, as well as the required control in phasic dopamine changes crucial to achieving a proper reversal learning. Furthermore, an original mechanism for modulating the phasic dopamine changes is proposed, assuming that the expected reward probability is coded by the activity of the winner Go neuron before a reward/punishment takes place. Simulations show that this original formulation for an automatic phasic dopamine control allows the achievement of a good flexible reversal even in difficult conditions. The current outcomes may contribute to understanding the mechanisms for active control of dopamine changes during flexible behavior. In perspective, it may be applied in neuropsychiatric or neurological disorders, such as Parkinson’s or schizophrenia, in which reinforcement learning is impaired.

The prediction-error hypothesis of schizophrenia: new data point to circuit-specific changes in dopamine activity

Schizophrenia is a severe psychiatric disorder affecting 21 million people worldwide. People with schizophrenia suffer from symptoms including psychosis and delusions, apathy, anhedonia, and cognitive deficits. Strikingly, schizophrenia is characterised by a learning paradox involving difficulties learning from rewarding events, whilst simultaneously 'overlearning' about irrelevant or neutral information. While dysfunction in dopaminergic signalling has long been linked to the pathophysiology of schizophrenia, a cohesive framework that accounts for this learning paradox remains elusive. Recently, there has been an explosion of new research investigating how dopamine contributes to reinforcement learning, which illustrates that midbrain dopamine contributes in complex ways to reinforcement learning, not previously envisioned. This new data brings new possibilities for how dopamine signalling contributes to the symptomatology of schizophrenia. Building on recent work, we present a new neural framework for how we might envision specific dopamine circuits contributing to this learning paradox in schizophrenia in the context of models of reinforcement learning. Further, we discuss avenues of preclinical research with the use of cutting-edge neuroscience techniques where aspects of this model may be tested. Ultimately, it is hoped that this review will spur to action more research utilising specific reinforcement learning paradigms in preclinical models of schizophrenia, to reconcile seemingly disparate symptomatology and develop more efficient therapeutics.

Reinforcement-learning in fronto-striatal circuits

We review the current state of knowledge on the computational and neural mechanisms of reinforcement-learning with a particular focus on fronto-striatal circuits. We divide the literature in this area into five broad research themes: the target of the learning-whether it be learning about the value of stimuli or about the value of actions; the nature and complexity of the algorithm used to drive the learning and inference process; how learned values get converted into choices and associated actions; the nature of state representations, and of other cognitive machinery that support the implementation of various reinforcement-learning operations. An emerging fifth area focuses on how the brain allocates or arbitrates control over different reinforcement-learning sub-systems or "experts". We will outline what is known about the role of the prefrontal cortex and striatum in implementing each of these functions. We then conclude by arguing that it will be necessary to build bridges from algorithmic level descriptions of computational reinforcement-learning to implementational level models to better understand how reinforcement-learning emerges from multiple distributed neural networks in the brain.

Brainstem Mechanisms of Pain Modulation: A within-Subjects 7T fMRI Study of Placebo Analgesic and Nocebo Hyperalgesic Responses

Pain perception can be powerfully influenced by an individual's expectations and beliefs. Although the cortical circuitry responsible for pain modulation has been thoroughly investigated, the brainstem pathways involved in the modulatory phenomena of placebo analgesia and nocebo hyperalgesia remain to be directly addressed. This study used ultra-high-field 7 tesla functional MRI (fMRI) to accurately resolve differences in brainstem circuitry present during the generation of placebo analgesia and nocebo hyperalgesia in healthy human participants (N = 25, 12 male). Over 2 successive days, through blinded application of altered thermal stimuli, participants were deceptively conditioned to believe that two inert creams labeled lidocaine (placebo) and capsaicin (nocebo) were acting to modulate their pain relative to a third Vaseline (control) cream. In a subsequent test phase, fMRI image sets were collected while participants were given identical noxious stimuli to all three cream sites. Pain intensity ratings were collected and placebo and nocebo responses determined. Brainstem-specific fMRI analysis revealed altered activity in key pain modulatory nuclei, including a disparate recruitment of the periaqueductal gray (PAG)-rostral ventromedial medulla (RVM) pathway when both greater placebo and nocebo effects were observed. Additionally, we found that placebo and nocebo responses differentially activated the parabrachial nucleus but overlapped in engagement of the substantia nigra and locus coeruleus. These data reveal that placebo and nocebo effects are generated through differential engagement of the PAG-RVM pathway, which in concert with other brainstem sites likely influences the experience of pain by modulating activity at the level of the dorsal horn.SIGNIFICANCE STATEMENT Understanding endogenous pain modulatory mechanisms would support development of effective clinical treatment strategies for both acute and chronic pain. Specific brainstem nuclei have long been known to play a central role in nociceptive modulation; however, because of the small size and complex organization of the nuclei, previous neuroimaging efforts have been limited in directly identifying how these subcortical networks interact during the development of antinociceptive and pro-nociceptive effects. We used ultra-high-field fMRI to resolve brainstem structures and measure signal change during placebo analgesia and nocebo hyperalgesia. We define overlapping and disparate brainstem circuitry responsible for altering pain perception. These findings extend our understanding of the detailed organization and function of discrete brainstem nuclei involved in pain processing and modulation.

Uncertainty and predictiveness modulate attention in human predictive learning

[Correction Notice: An Erratum for this article was reported online in Journal of Experimental Psychology: General on Jan 14 2021 (see record 2021-07705-001). In the article, formatting for UK Research Councils funding was omitted. The author note and copyright line now reflect the standard acknowledgment of and formatting for the funding received for this article. All versions of this article have been corrected.] Attention determines which cues receive processing and are learned about. Learning, however, leads to attentional biases. In the study of animal learning, in some circumstances, cues that have been previously predictive of their consequences are subsequently learned about more than are nonpredictive cues, suggesting that they receive more attention. In other circumstances, cues that have previously led to uncertain consequences are learned about more than are predictive cues. In human learning, there is a clear role for predictiveness, but a role for uncertainty has been less clear. Here, in a human learning task, we show that cues that led to uncertain outcomes were subsequently learned about more than were cues that were previously predictive of their outcomes. This effect occurred when there were few uncertain cues. When the number of uncertain cues was increased, attention switched to predictive cues. This pattern of results was found for cues (1) that were uncertain because they led to 2 different outcomes equally often in a nonpredictable manner and (2) that were used in a nonlinear discrimination and were not predictive individually but were predictive in combination with other cues. This suggests that both the opposing predictiveness and uncertainty effects were determined by the relationship between individual cues and outcomes rather than the predictive strength of combined cues. These results demonstrate that learning affects attention; however, the precise nature of the effect on attention depends on the level of task complexity, which reflects a potential switch between exploration and exploitation of cues. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

The Role of Dopamine in Associative Learning in Drosophila: An Updated Unified Model

Learning to associate a positive or negative experience with an unrelated cue after the presentation of a reward or a punishment defines associative learning. The ability to form associative memories has been reported in animal species as complex as humans and as simple as insects and sea slugs. Associative memory has even been reported in tardigrades [1], species that diverged from other animal phyla 500 million years ago. Understanding the mechanisms of memory formation is a fundamental goal of neuroscience research. In this article, we work on resolving the current contradictions between different Drosophila associative memory circuit models and propose an updated version of the circuit model that predicts known memory behaviors that current models do not. Finally, we propose a model for how dopamine may function as a reward prediction error signal in Drosophila, a dopamine function that is well-established in mammals but not in insects [2, 3].

Does openness/intellect predict sensitivity to the reward value of information?

A recent theory proposes that the personality trait openness/intellect is underpinned by differential sensitivity to the reward value of information. This theory draws on evidence that midbrain dopamine neurons respond to unpredicted information gain, mirroring their responses to unpredicted primary rewards. Using a choice task modelled on this seminal work (Experiment 1, N = 139, 69% female), we examined the relation between openness/intellect and willingness to pay for non-instrumental information (i.e., information with no secondary utility). We also assessed whether any such relation was moderated by the dopamine D2 receptor antagonist sulpiride (Experiment 2, N = 164, 100% male). Unexpectedly, most measures of openness/intellect were unrelated to costly information preference in both experiments, and some predicted a decreased willingness to incur a cost for information. In Experiment 2, this cost-dependent association between openness/intellect and information valuation appeared in the placebo condition but not under sulpiride. In addition, participants were more willing to pay for moderately costly information under sulpiride compared to placebo, consistent with a dopaminergic basis to information valuation. Potential refinements to the information valuation theory of openness/intellect are discussed in the light of these and other emerging findings.

The Role of the Rodent Lateral Orbitofrontal Cortex in Simple Pavlovian Cue-Outcome Learning Depends on Training Experience

The orbitofrontal cortex (OFC) is a critical structure in the flexible control of value-based behaviors. OFC dysfunction is typically only detected when task or environmental contingencies change, against a backdrop of apparently intact initial acquisition and behavior. While intact acquisition following OFC lesions in simple Pavlovian cue-outcome conditioning is often predicted by models of OFC function, this predicted null effect has not been thoroughly investigated. Here, we test the effects of lesions and temporary muscimol inactivation of the rodent lateral OFC on the acquisition of a simple single cue-outcome relationship. Surprisingly, pretraining lesions significantly enhanced acquisition after overtraining, whereas post-training lesions and inactivation significantly impaired acquisition. This impaired acquisition to the cue reflects a disruption of behavioral control and not learning since the cue could also act as an effective blocking stimulus in an associative blocking procedure. These findings suggest that even simple cue-outcome representations acquired in the absence of OFC function are impoverished. Therefore, while OFC function is often associated with flexible behavioral control in complex environments, it is also involved in very simple Pavlovian acquisition where complex cue-outcome relationships are irrelevant to task performance.

Basolateral Amygdala to Nucleus Accumbens Communication Differentially Mediates Devaluation Sensitivity of Sign- and Goal-Tracking Rats

Rats rely on communication between the basolateral amygdala (BLA) and nucleus accumbens (NAc) to express lever directed approach in a Pavlovian lever autoshaping (PLA) task that distinguishes sign- and goal-tracking rats. During PLA, sign-tracking rats preferentially approach an insertable lever cue, while goal-tracking rats approach a foodcup where rewards are delivered. While sign-tracking rats inflexibly respond to cues even after the associated reward is devalued, goal-tracking rats flexibly reduce responding to cues during outcome devaluation. Here, we sought to determine whether BLA-NAc communication, which is necessary for sign, but not goal-tracking, drives a rigid appetitive approach of sign-tracking rats that are insensitive to manipulations of outcome value. Using a contralateral chemogenetic inactivation design, we injected contralateral BLA and NAc core with inhibitory DREADD (hm4Di-mCherry) or control (mCherry) constructs. To determine sign- and goal-tracking groups, we trained rats in five PLA sessions in which brief lever insertion predicts food pellet delivery. We sated rats on training pellets (devalued condition) or chow (valued condition) before systemic clozapine injections (0.1 mg/kg) to inactivate BLA and contralateral NAc during two outcome devaluation probe tests, in which we measured lever and foodcup approach. Contralateral BLA-NAc chemogenetic inactivation promoted a flexible lever approach in sign-tracking rats but disrupted the flexible foodcup approach in goal-tracking rats. Consistent with a prior BLA-NAc disconnection lesion study, we find contralateral chemogenetic inactivation of BLA and NAc core reduces lever, but not the foodcup approach in PLA. Together these findings suggest rigid appetitive associative encoding in BLA-NAc of sign-tracking rats hinders the expression of flexible behavior when outcome value changes.

The basal ganglia corticostriatal loops and conditional learning

Brief maneuvering of the literature as to the various roles attributed to the basal ganglia corticostriatal circuits in a variety of cognitive processes such as working memory, selective attention, and category learning has inspired us to investigate the interplay of the two major basal ganglia open-recurrent loops, namely, visual and executive loops specifically the possible involvement of their overlap in conditional learning. We propose that the interaction of the visual and executive loops reflected through their cortical overlap in the dorsolateral prefrontal cortex (DL-PFC), lateral orbitofrontal cortex (LO-PFC), and presupplementary motor area (SMA) plays an instrumental role preliminary first in forming associations between a series of correct responses following similar stimuli and then in shifting, abstracting, and generalizing conditioned responses. The premotor and supplementary motor areas have been shown essential to producing a sequence of movements while the SMA is engaged in monitoring complex movements. In light of the recent studies, we will suggest that the interaction of visual and executive loops could strengthen or weaken learned associations following different reward values. Furthermore, we speculate that the overlap of the visual and executive loops can account for the switching between the associative vs. rule-based category learning systems.

Reward-predictive representations generalize across tasks in reinforcement learning

In computer science, reinforcement learning is a powerful framework with which artificial agents can learn to maximize their performance for any given Markov decision process (MDP). Advances over the last decade, in combination with deep neural networks, have enjoyed performance advantages over humans in many difficult task settings. However, such frameworks perform far less favorably when evaluated in their ability to generalize or transfer representations across different tasks. Existing algorithms that facilitate transfer typically are limited to cases in which the transition function or the optimal policy is portable to new contexts, but achieving "deep transfer" characteristic of human behavior has been elusive. Such transfer typically requires discovery of abstractions that permit analogical reuse of previously learned representations to superficially distinct tasks. Here, we demonstrate that abstractions that minimize error in predictions of reward outcomes generalize across tasks with different transition and reward functions. Such reward-predictive representations compress the state space of a task into a lower dimensional representation by combining states that are equivalent in terms of both the transition and reward functions. Because only state equivalences are considered, the resulting state representation is not tied to the transition and reward functions themselves and thus generalizes across tasks with different reward and transition functions. These results contrast with those using abstractions that myopically maximize reward in any given MDP and motivate further experiments in humans and animals to investigate if neural and cognitive systems involved in state representation perform abstractions that facilitate such equivalence relations.

Unpredictability as a modulator of drug self-administration: Relevance for substance-use disorders

Drug self-administration has been regarded as a gold-standard preclinical model of addiction and substance-use disorder (SUD). However, investigators are becoming increasingly aware, that certain aspects of addiction or SUDs experienced by humans are not accurately captured in our preclinical self-administration models. The current review will focus on two such aspects of current preclinical drug self-administration models: 1) Predictable vs. unpredictable drug access in terms of the time and effort put into obtaining drugs (i.e., response requirement) and drug quality (i.e., amount) and 2) rich vs. lean access to drugs. Some behavioral and neurobiological mechanisms that could contribute to excessive allocation of behavior toward drug-seeking and drug-taking at the expense of engaging in nondrug-related activities are discussed, and some directions for future research are identified. Based on the experiments reviewed, lean and unpredictable drug access could worsen drug-seeking and drug-taking behavior in individuals with SUDs. Once more fully explored, this area of research will help determine whether and how unpredictable and lean cost requirements affect drug self-administration in preclinical laboratory studies with nonhuman subjects and will help determine whether incorporating these conditions in current self-administration models will increase their predictive validity.

Dynamic resource allocation during reinforcement learning accounts for ramping and phasic dopamine activity

For an animal to learn about its environment with limited motor and cognitive resources, it should focus its resources on potentially important stimuli. However, too narrow focus is disadvantageous for adaptation to environmental changes. Midbrain dopamine neurons are excited by potentially important stimuli, such as reward-predicting or novel stimuli, and allocate resources to these stimuli by modulating how an animal approaches, exploits, explores, and attends. The current study examined the theoretical possibility that dopamine activity reflects the dynamic allocation of resources for learning. Dopamine activity may transition between two patterns: (1) phasic responses to cues and rewards, and (2) ramping activity arising as the agent approaches the reward. Phasic excitation has been explained by prediction errors generated by experimentally inserted cues. However, when and why dopamine activity transitions between the two patterns remain unknown. By parsimoniously modifying a standard temporal difference (TD) learning model to accommodate a mixed presentation of both experimental and environmental stimuli, we simulated dopamine transitions and compared them with experimental data from four different studies. The results suggested that dopamine transitions from ramping to phasic patterns as the agent focuses its resources on a small number of reward-predicting stimuli, thus leading to task dimensionality reduction. The opposite occurs when the agent re-distributes its resources to adapt to environmental changes, resulting in task dimensionality expansion. This research elucidates the role of dopamine in a broader context, providing a potential explanation for the diverse repertoire of dopamine activity that cannot be explained solely by prediction error.

Effect of Expectation on Pain Processing: A Psychophysics and Functional MRI Analysis

Pain is a complex phenomenon that is highly modifiable by expectation. Whilst the intensity of incoming noxious information plays a key role in the intensity of perceived pain, this intensity can be profoundly shaped by an individual’s expectations. Modern brain imaging investigations have begun to detail the brain regions responsible for placebo and nocebo related changes in pain, but less is known about the neural basis of stimulus-expectancy changes in pain processing. In this functional magnetic resonance imaging study, we administered two separate protocols of the same noxious thermal stimuli to 24 healthy subjects. However, different expectations were elicited by different explanations to subjects prior to each protocol. During one protocol, pain intensities were matched to expectation and in the other protocol they were not. Pain intensity was measured continuously via a manually operated computerized visual analogue scale. When individuals expected the stimulus intensity to remain constant, but in reality it was surreptitiously increased or decreased, pain intensity ratings were significantly lower than when expectation and pain intensities were matched. When the stimulus intensities did not match expectations, various areas in the brain such as the amygdala, anterior cingulate cortex (ACC), dorsolateral prefrontal cortex (dlPFC), and the midbrain periaqueductal gray matter (PAG) displayed significantly different patterns of activity compared to instances when stimulus intensity and pain expectations were matched. These results show that stimulus-expectancy manipulation of pain intensity alters activity in both higher brain and brainstem centers which are known to modulate pain under various conditions.

A roadmap to integrate astrocytes into Systems Neuroscience

Systems neuroscience is still mainly a neuronal field, despite the plethora of evidence supporting the fact that astrocytes modulate local neural circuits, networks, and complex behaviors. In this article, we sought to identify which types of studies are necessary to establish whether astrocytes, beyond their well-documented homeostatic and metabolic functions, perform computations implementing mathematical algorithms that sub-serve coding and higher-brain functions. First, we reviewed Systems-like studies that include astrocytes in order to identify computational operations that these cells may perform, using Ca2+ transients as their encoding language. The analysis suggests that astrocytes may carry out canonical computations in a time scale of subseconds to seconds in sensory processing, neuromodulation, brain state, memory formation, fear, and complex homeostatic reflexes. Next, we propose a list of actions to gain insight into the outstanding question of which variables are encoded by such computations. The application of statistical analyses based on machine learning, such as dimensionality reduction and decoding in the context of complex behaviors, combined with connectomics of astrocyte-neuronal circuits, is, in our view, fundamental undertakings. We also discuss technical and analytical approaches to study neuronal and astrocytic populations simultaneously, and the inclusion of astrocytes in advanced modeling of neural circuits, as well as in theories currently under exploration such as predictive coding and energy-efficient coding. Clarifying the relationship between astrocytic Ca2+ and brain coding may represent a leap forward toward novel approaches in the study of astrocytes in health and disease.

Characterization of glutamatergic VTA neural population responses to aversive and rewarding conditioning in freely-moving mice

The Ventral Tegmental Area (VTA) is a midbrain structure known to integrate aversive and rewarding stimuli, but little is known about the role of VTA glutamatergic (VGluT2) neurons in these functions. Direct activation of VGluT2 soma evokes rewarding behaviors, while activation of their downstream projections evokes aversive behaviors. To facilitate our understanding of these conflicting properties, we recorded calcium signals from VTA^VGluT2+ neurons using fiber photometry in VGluT2-cre mice to investigate how this population was recruited by aversive and rewarding stimulation, both during unconditioned and conditioned protocols. Our results revealed that, as a population, VTA^VGluT2+ neurons responded similarly to unconditioned-aversive and unconditioned-rewarding stimulation. During aversive and rewarding conditioning, the CS-evoked responses gradually increased across trials whilst the US-evoked response remained stable. Retrieval 24 h after conditioning, during which mice received only CS presentation, resulted in VTA^VGluT2+ neurons strongly responding to CS presentation and to the expected-US but only for aversive conditioning. To help understand these differences based on VTA^VGluT2+ neuronal networks, the inputs and outputs of VTA^VGluT2+ neurons were investigated using Cholera Toxin B (CTB) and rabies virus. Based on our results, we propose that the divergent VTA^VGluT2+ neuronal responses to aversion and reward conditioning may be partly due to the existence of VTA^VGluT2+ subpopulations that are characterized by their connectivity.

Pattern of dopamine signaling during aversive events predicts active avoidance learning

Learning to avoid aversive outcomes is an adaptive strategy to limit one's future exposure to stressful events. However, there is considerable variance in active avoidance learning across a population. The mesolimbic dopamine system contributes to behaviors elicited by aversive stimuli, although it is unclear if the heterogeneity in active avoidance learning is explained by differences in dopamine transmission. Furthermore, it is not known how dopamine signals evolve throughout active avoidance learning. To address these questions, we performed voltammetry recordings of dopamine release in the ventral medial striatum throughout training on inescapable footshock and signaled active avoidance tasks. This approach revealed differences in the pattern of dopamine signaling during the aversive cue and the safety period that corresponded to subsequent task performance. Dopamine transmission throughout the footshock bout did not predict performance but rather was modulated by the prior stress exposure. Additionally, we demonstrate that dopamine encodes a safety prediction error signal, which illustrates that ventral medial striatal dopamine release conveys a learning signal during both appetitive and aversive conditions.

Holistic Reinforcement Learning: The Role of Structure and Attention

Compact representations of the environment allow humans to behave efficiently in a complex world. Reinforcement learning models capture many behavioral and neural effects but do not explain recent findings showing that structure in the environment influences learning. In parallel, Bayesian cognitive models predict how humans learn structured knowledge but do not have a clear neurobiological implementation. We propose an integration of these two model classes in which structured knowledge learned via approximate Bayesian inference acts as a source of selective attention. In turn, selective attention biases reinforcement learning towards relevant dimensions of the environment. An understanding of structure learning will help to resolve the fundamental challenge in decision science: explaining why people make the decisions they do.

Ventral Tegmental Dopamine Neurons Participate in Reward Identity Predictions

Dopamine (DA) neurons in the ventral tegmental area (VTA) and substantia nigra (SNc) encode reward prediction errors (RPEs) and are proposed to mediate error-driven learning. However, the learning strategy engaged by DA-RPEs remains controversial. RPEs might imbue predictive cues with pure value, independently of representations of their associated outcome. Alternatively, RPEs might promote learning about the sensory features (the identity) of the rewarding outcome. Here, we show that, although both VTA and SNc DA neuron activation reinforces instrumental responding, only VTA DA neuron activation during consumption of expected sucrose reward restores error-driven learning and promotes formation of a new cue→sucrose association. Critically, expression of VTA DA-dependent Pavlovian associations is abolished following sucrose devaluation, a signature of identity-based learning. These findings reveal that activation of VTA- or SNc-DA neurons engages largely dissociable learning processes with VTA-DA neurons capable of participating in outcome-specific predictive learning, and the role of SNc-DA neurons appears limited to reinforcement of instrumental responses.

Acute tramadol enhances brain activity associated with reward anticipation in the nucleus accumbens

BACKGROUND

Tramadol is an analgesic with monoamine reuptake inhibition and μ-opioid receptor activation. Although tramadol has been widely used for treatment of various pain conditions, there is controversy over the risk of abuse potential. We examined the effects of tramadol on the reward system in humans using functional magnetic resonance imaging (fMRI) to assess the potential of tramadol for drug abuse or dependence.

METHODS

A randomized, double-blind, placebo-controlled, crossover study was conducted for 19 healthy adults under tramadol or placebo. In association with subjective mood questionnaires, monetary incentive delay (MID) task was performed to assess the neural response to reward anticipation during fMRI. Subjective mood measures and blood oxygenation level-dependent (BOLD) signal during gain and loss anticipation were compared between tramadol and placebo.

RESULTS

Tramadol significantly reduced anxiety (Z = - 2.513, p = 0.012) and enhanced vigor (Z = - 2.725, p = 0.006) compared with placebo. By Mood Rating Scale, tramadol provoked contented (Z = - 2.316, p = 0.021), relaxed (Z = - 2.236, p = 0.025), and amicable feelings (Z = - 2.015, p = 0.044) as well as increased alertness (Z = - 1.972, p = 0.049) and contentedness domains (Z = - 2.174, p = 0.030) compared with placebo. Several brain regions including nucleus accumbens (NAc) were activated during gain anticipation in the MID task under both tramadol and placebo. Tramadol increased the %BOLD signal change in NAc at +¥500 cue significantly more than the placebo (Z = - 2.295, p = 0.022).

CONCLUSION

Tramadol enhances the reward system and thereby may have abuse potential or precipitate drug abuse in human.

Cannabinoid disruption of learning mechanisms involved in reward processing

The increasing use of cannabis, its derivatives, and synthetic cannabinoids for medicinal and recreational purposes has led to burgeoning interest in understanding the addictive potential of this class of molecules. It is estimated that ∼10% of marijuana users will eventually show signs of dependence on the drug, and the diagnosis of cannabis use disorder (CUD) is increasing in the United States. The molecule that sustains the use of cannabis is Δ⁹-tetrahydrocannabinol (Δ⁹-THC), and our knowledge of its effects, and those of other cannabinoids on brain function has expanded rapidly in the past two decades. Additionally, the identification of endogenous cannabinoid (endocannabinoid) systems in brain and their roles in physiology and behavior, demonstrate extensive involvement of these lipid signaling molecules in regulating CNS function. Here, we examine roles for endogenous cannabinoids in shaping synaptic activity in cortical and subcortical brain circuits, and we discuss mechanisms in which exogenous cannabinoids, such as Δ⁹-THC, interact with endocannabinoid systems to disrupt neuronal network oscillations. We then explore how perturbation of the interaction of this activity within brain reward circuits may lead to impaired learning. Finally, we propose that disruption of cellular plasticity mechanisms by exogenous cannabinoids in cortical and subcortical circuits may explain the difficulty in establishing viable cannabinoid self-administration models in animals.

Model-based fMRI reveals dissimilarity processes underlying base rate neglect

Extensive evidence suggests that people use base rate information inconsistently in decision making. A classic example is the inverse base rate effect (IBRE), whereby participants classify ambiguous stimuli sharing features of both common and rare categories as members of the rare category. Computational models of the IBRE have posited that it arises either from associative similarity-based mechanisms or from dissimilarity-based processes that may depend on higher-level inference. Here we develop a hybrid model, which posits that similarity- and dissimilarity-based evidence both contribute to the IBRE, and test it using functional magnetic resonance imaging data collected from human subjects completing an IBRE task. Consistent with our model, multivoxel pattern analysis reveals that activation patterns on ambiguous test trials contain information consistent with dissimilarity-based processing. Further, trial-by-trial activation in left rostrolateral prefrontal cortex tracks model-based predictions for dissimilarity-based processing, consistent with theories positing a role for higher-level symbolic processing in the IBRE.

Paradoxical Decision-Making: A Framework for Understanding Cognition in Parkinson's Disease

People with Parkinson's disease (PD) show impaired decision-making when sensory and memory information must be combined. This recently identified impairment results from an inability to accumulate the proper amount of information needed to make a decision and appears to be independent of dopamine tone and reinforcement learning mechanisms. Although considerable work focuses on PD and decisions involving risk and reward, in this Opinion article we propose that the emerging findings in perceptual decision-making highlight the multisystem nature of PD, and that unraveling the neuronal circuits underlying perceptual decision-making impairment may help in understanding other cognitive impairments in people with PD. We also discuss how a decision-making framework may be extended to gain insights into mechanisms of motor impairments in PD.

Stress Elevates Frontal Midline Theta in Feedback-based Category Learning of Exceptions

Adapting behavior based on category knowledge is a fundamental cognitive function, which can be achieved via different learning strategies relying on different systems in the brain. Whereas the learning of typical category members has been linked to implicit, prototype abstraction learning, which relies predominantly on prefrontal areas, the learning of exceptions is associated with explicit, exemplar-based learning, which has been linked to the hippocampus. Stress is known to foster implicit learning strategies at the expense of explicit learning. Procedural, prefrontal learning and cognitive control processes are reflected in frontal midline theta (4–8 Hz) oscillations during feedback processing. In the current study, we examined the effect of acute stress on feedback-based category learning of typical category members and exceptions and the oscillatory correlates of feedback processing in the EEG. A computational modeling procedure was applied to estimate the use of abstraction and exemplar strategies during category learning. We tested healthy, male participants who underwent either the socially evaluated cold pressor test or a nonstressful control procedure before they learned to categorize typical members and exceptions based on feedback. The groups did not differ significantly in their categorization accuracy or use of categorization strategies. In the EEG, however, stressed participants revealed elevated theta power specifically during the learning of exceptions, whereas the theta power during the learning of typical members did not differ between the groups. Elevated frontal theta power may reflect an increased involvement of medial prefrontal areas in the learning of exceptions under stress.

Illusory Motion Reproduced by Deep Neural Networks Trained for Prediction

The cerebral cortex predicts visual motion to adapt human behavior to surrounding objects moving in real time. Although the underlying mechanisms are still unknown, predictive coding is one of the leading theories. Predictive coding assumes that the brain's internal models (which are acquired through learning) predict the visual world at all times and that errors between the prediction and the actual sensory input further refine the internal models. In the past year, deep neural networks based on predictive coding were reported for a video prediction machine called PredNet. If the theory substantially reproduces the visual information processing of the cerebral cortex, then PredNet can be expected to represent the human visual perception of motion. In this study, PredNet was trained with natural scene videos of the self-motion of the viewer, and the motion prediction ability of the obtained computer model was verified using unlearned videos. We found that the computer model accurately predicted the magnitude and direction of motion of a rotating propeller in unlearned videos. Surprisingly, it also represented the rotational motion for illusion images that were not moving physically, much like human visual perception. While the trained network accurately reproduced the direction of illusory rotation, it did not detect motion components in negative control pictures wherein people do not perceive illusory motion. This research supports the exciting idea that the mechanism assumed by the predictive coding theory is one of basis of motion illusion generation. Using sensory illusions as indicators of human perception, deep neural networks are expected to contribute significantly to the development of brain research.

Dopaminergic dysfunction in neurodevelopmental disorders: recent advances and synergistic technologies to aid basic research

Neurodevelopmental disorders (NDDs) represent a diverse group of syndromes characterized by abnormal development of the central nervous system and whose symptomatology includes cognitive, emotional, sensory, and motor impairments. The identification of causative genetic defects has allowed for creation of transgenic NDD mouse models that have revealed pathophysiological mechanisms of disease phenotypes in a neural circuit- and cell type-specific manner. Mouse models of several syndromes, including Rett syndrome, Fragile X syndrome, Angelman syndrome, Neurofibromatosis type 1, etc., exhibit abnormalities in the structure and function of dopaminergic circuitry, which regulates motivation, motor behavior, sociability, attention, and executive function. Recent advances in technologies for functional circuit mapping, including tissue clearing, viral vector-based tracing methods, and optical readouts of neural activity, have refined our knowledge of dopaminergic circuits in unperturbed states, yet these tools have not been widely applied to NDD research. Here, we will review recent findings exploring dopaminergic function in NDD models and discuss the promise of new tools to probe NDD pathophysiology in these circuits.

Associability-modulated loss learning is increased in posttraumatic stress disorder

Disproportionate reactions to unexpected stimuli in the environment are a cardinal symptom of posttraumatic stress disorder (PTSD). Here, we test whether these heightened responses are associated with disruptions in distinct components of reinforcement learning. Specifically, using functional neuroimaging, a loss-learning task, and a computational model-based approach, we assessed the mechanistic hypothesis that overreactions to stimuli in PTSD arise from anomalous gating of attention during learning (i.e., associability). Behavioral choices of combat-deployed veterans with and without PTSD were fit to a reinforcement learning model, generating trial-by-trial prediction errors (signaling unexpected outcomes) and associability values (signaling attention allocation to the unexpected outcomes). Neural substrates of associability value and behavioral parameter estimates of associability updating, but not prediction error, increased with PTSD during loss learning. Moreover, the interaction of PTSD severity with neural markers of associability value predicted behavioral choices. These results indicate that increased attention-based learning may underlie aspects of PTSD and suggest potential neuromechanistic treatment targets.

Differential Noradrenergic Modulation of Monetary Reward and Visual Erotic Stimulus Processing

We recently investigated the effects of the noradrenergic antidepressant reboxetine and the antipsychotic amisulpride compared to placebo on neural correlates of primary reinforcers by visual erotic stimulation in healthy subjects. Whereas, amisulpride left subjective sexual functions and corresponding neural activations unimpaired, attenuated neural activations were observed under reboxetine within the nucleus accumbens (Nacc) along with diminished behavioral sexual functioning. However, a global dampening of the reward system under reboxetine seemed not intuitive considering the complementary role of the noradrenergic to the dopamine system in reward-related learning mediated by prediction error processing. We therefore investigated the sample of 17 healthy males in a mean age of 23.8 years again by functional magnetic resonance imaging (fMRI), to explore the noradrenergic effects on neural reward prediction error signaling. Participants took reboxetine (4 mg/d), amisulpride (200 mg/d), and placebo each for 7 days within a randomized, double-blind, within-subject cross-over design. During fMRI, we used an established monetary incentive task to assess neural reward expectation and prediction error signals within the bilateral Nacc using an independent anatomical mask for a region of interest (ROI) analysis. Activations within the same ROI were also assessed for the erotic picture paradigm. We confirmed our previous results from the whole brain analysis for the selected ROI by significant (p < 0.05 FWE-corrected) attenuated activations within the Nacc during visual sexual stimulation under reboxetine compared to placebo. However, activations in the Nacc concerning prediction error processing and monetary reward expectation were unimpaired under reboxetine compared to placebo, along with unimpaired reaction times in the reward task. For both tasks, neural activations and behavioral processing were not altered by amisulpride compared to placebo. The observed attenuated neural activations within the Nacc during visual erotic stimulation along with unimpaired neural prediction error and monetary reward expectation processing provide evidence for a differential modulation of the neural reward system by the noradrenergic agent reboxetine depending on the presence of primary reinforcers such as erotic stimuli in contrast to secondary such as monetary rewards.

Drosophila: An Emergent Model for Delineating Interactions between the Circadian Clock and Drugs of Abuse

Endogenous circadian oscillators orchestrate rhythms at the cellular, physiological, and behavioral levels across species to coordinate activity, for example, sleep/wake cycles, metabolism, and learning and memory, with predictable environmental cycles. The 21st century has seen a dramatic rise in the incidence of circadian and sleep disorders with globalization, technological advances, and the use of personal electronics. The circadian clock modulates alcohol- and drug-induced behaviors with circadian misalignment contributing to increased substance use and abuse. Invertebrate models, such as Drosophila melanogaster, have proven invaluable for the identification of genetic and molecular mechanisms underlying highly conserved processes including the circadian clock, drug tolerance, and reward systems. In this review, we highlight the contributions of Drosophila as a model system for understanding the bidirectional interactions between the circadian system and the drugs of abuse, alcohol and cocaine, and illustrate the highly conserved nature of these interactions between Drosophila and mammalian systems. Research in Drosophila provides mechanistic insights into the corresponding behaviors in higher organisms and can be used as a guide for targeted inquiries in mammals.