Modeling dopaminergic and other processes involved in learning from reward prediction error: contributions from an individual differences perspective

Individual differences in naturalistic learning link negative emotionality to the development of anxiety

Organisms learn from prediction errors (PEs) to predict the future. Laboratory studies using small financial outcomes find that humans use PEs to update expectations and link individual differences in PE-based learning to internalizing disorders. Because of the low-stakes outcomes in most tasks, it is unclear whether PE learning emerges in naturalistic, high-stakes contexts and whether individual differences in PE learning predict psychopathology risk. Using experience sampling to assess 625 college students' expected exam grades, we found evidence of PE-based learning and a general tendency to discount negative PEs, an "optimism bias." However, individuals with elevated negative emotionality, a personality trait linked to the development of anxiety disorders, displayed a global pessimism and learning differences that impeded accurate expectations and predicted future anxiety symptoms. A sensitivity to PEs combined with an aversion to negative PEs may result in a pessimistic and inaccurate model of the world, leading to anxiety.

Dissociable feedback valence effects on frontal midline theta during reward gain versus threat avoidance learning

While frontal midline theta (FMθ) has been associated with threat processing, with cognitive control in the context of anxiety, and with reinforcement learning, most reinforcement learning studies on FMθ have used reward rather than threat-related stimuli as reinforcer. Accordingly, the role of FMθ in threat-related reinforcement learning is largely unknown. Here, n = 23 human participants underwent one reward-, and one punishment-, based reversal learning task, which differed only with regard to the kind of reinforcers that feedback was tied to (i.e., monetary gain vs. loud noise burst, respectively). In addition to single-trial EEG, we assessed single-trial feedback expectations based on both a reinforcement learning computational model and trial-by-trial subjective feedback expectation ratings. While participants' performance and feedback expectations were comparable between the reward and punishment tasks, FMθ was more reliably amplified to negative vs. positive feedback in the reward vs. punishment task. Regressions with feedback valence, computationally derived, and self-reported expectations as predictors and FMθ as criterion further revealed that trial-by-trial variations in FMθ specifically relate to reward-related feedback-valence and not to threat-related feedback or to violated expectations/prediction errors. These findings suggest that FMθ as measured in reinforcement learning tasks may be less sensitive to the processing of events with direct relevance for fear and anxiety.

Personality Neuroscience: An Emerging Field with Bright Prospects

Personality neuroscience is the study of persistent psychological individual differences, typically in the general population, using neuroscientific methods. It has the potential to shed light on the neurobiological mechanisms underlying individual differences and their manifestation in ongoing behavior and experience. The field was inaugurated many decades ago, yet has only really gained momentum in the last two, as suitable technologies have become widely available. Personality neuroscience employs a broad range of methods, including molecular genetics, pharmacological assays or manipulations, electroencephalography, and various neuroimaging modalities, such as magnetic resonance imaging and positron emission tomography. Although exciting progress is being made in this young field, much remains unknown. In this brief review, we discuss discoveries that have been made, methodological challenges and advances, and important questions that remain to be answered. We also discuss best practices for personality neuroscience research and promising future directions for the field.

Modulation of Dopamine for Adaptive Learning: A Neurocomputational Model

There have been many proposals that learning rates in the brain are adaptive, in the sense that they increase or decrease depending on environmental conditions. The majority of these models are abstract and make no attempt to describe the neural circuitry that implements the proposed computations. This article describes a biologically detailed computational model that overcomes this shortcoming. Specifically, we propose a neural circuit that implements adaptive learning rates by modulating the gain on the dopamine response to reward prediction errors, and we model activity within this circuit at the level of spiking neurons. The model generates a dopamine signal that depends on the size of the tonically active dopamine neuron population and the phasic spike rate. The model was tested successfully against results from two single-neuron recording studies and a fast-scan cyclic voltammetry study. We conclude by discussing the general applicability of the model to dopamine mediated tasks that transcend the experimental phenomena it was initially designed to address.

Decoding personality trait measures from resting EEG: An exploratory report

Can personality be predicted from oscillatory patterns produced by the brain at rest? To date, relatively few studies using electroencephalography (EEG) have yielded consistent relations between personality trait measures and spectral power. Thus, new exploratory research may help develop targeted hypotheses about how neural processes associated with EEG activity may relate to personality differences. We used multivariate pattern analysis to decode personality scores (i.e., Big Five traits) from resting EEG frequency power spectra. Up to 8 minutes of EEG data was recorded per participant prior to completing an unrelated task (N = 168, M_age = 23.51, 57% female) and, in a subset of participants, after task completion (N = 96, M_age = 23.22, 52% female). In each recording, participants alternated between open and closed eyes. Linear support vector regression with 10-fold cross validation was performed using the power from 62 scalp electrodes within 1 Hz frequency bins from 1 to 30 Hz. One Big Five trait, agreeableness, could be decoded from EEG power ranging from 8 to 19 Hz, and this was consistent across all four recording periods. Neuroticism was decodable using data within the 3-6 Hz range, albeit less consistently. Posterior alpha power negatively correlated with agreeableness, whereas parietal beta power positively correlated with agreeableness. We suggest methods to draw from our results and develop targeted future hypotheses, such as linking to individual alpha frequency and incorporating self-reported emotional states. Our open dataset can be harnessed to reproduce results or investigate new research questions concerning the biological basis of personality.

Extraversion and reward-processing: Consolidating evidence from an electroencephalographic index of reward-prediction-error

Trait extraversion has been theorized to emerge from functioning of the dopaminergic reward system. Recent evidence for this view shows that extraversion modulates the scalp-recorded Reward Positivity, a putative marker of dopaminergic signaling of reward-prediction-error. We attempt to replicate this association amid several improvements on previous studies in this area, including an adequately-powered sample (N = 100) and thorough examination of convergent-divergent validity. Participants completed a passive associative learning task presenting rewards and non-rewards that were either predictable or unexpected. Frequentist and Bayesian analyses confirmed that the scalp recorded Reward Positivity (i.e., the Feedback-Related-Negativity contrasting unpredicted rewards and unpredicted non-rewards) was significantly associated with three measures of extraversion and unrelated to other basic traits from the Big Five personality model. Narrower sub-traits of extraversion showed similar, though weaker associations with the Reward Positivity. These findings consolidate previous evidence linking extraversion with a putative marker of dopaminergic reward-processing.

The Virtual Personalities Neural Network Model: Neurobiological Underpinnings

The Virtual Personalities Model is a motive-based neural network model that provides both a psychological model and a computational implementation that explicates the dynamics and often large within-person variability in behavior that arises over time. At the same time the same model can produce -- across many virtual personalities - between subject variability in behavior that when factor analyzed yields familiar personality structure (e.g., the Big-5). First, we describe our personality model and its implementation as a neural network model. Second, we focus on detailing the neurobiological underpinnings of this model. Third, we examine the learning mechanisms, and their biological substrates, as ways that the model gets "wired up", discussing Pavlovian and instrumental conditioning, Pavlovian to instrumental transfer (PIT), and habits. Finally, we describe the dynamics of how initial differences in propensities (e.g., dopamine functioning), wiring differences due to experience, and other factors could operate together to develop and change personality over time, and how this might be empirically examined. Thus, our goal is to contribute to the rising chorus of voices seeking a more precise neurobiologically-based science of the complex dynamics underlying personality.

'Proactive' use of cue-context congruence for building reinforcement learning's reward function

BACKGROUND

Reinforcement learning is a fundamental form of learning that may be formalized using the Bellman equation. Accordingly an agent determines the state value as the sum of immediate reward and of the discounted value of future states. Thus the value of state is determined by agent related attributes (action set, policy, discount factor) and the agent's knowledge of the environment embodied by the reward function and hidden environmental factors given by the transition probability. The central objective of reinforcement learning is to solve these two functions outside the agent's control either using, or not using a model.

RESULTS

In the present paper, using the proactive model of reinforcement learning we offer insight on how the brain creates simplified representations of the environment, and how these representations are organized to support the identification of relevant stimuli and action. Furthermore, we identify neurobiological correlates of our model by suggesting that the reward and policy functions, attributes of the Bellman equitation, are built by the orbitofrontal cortex (OFC) and the anterior cingulate cortex (ACC), respectively.

CONCLUSIONS

Based on this we propose that the OFC assesses cue-context congruence to activate the most context frame. Furthermore given the bidirectional neuroanatomical link between the OFC and model-free structures, we suggest that model-based input is incorporated into the reward prediction error (RPE) signal, and conversely RPE signal may be used to update the reward-related information of context frames and the policy underlying action selection in the OFC and ACC, respectively. Furthermore clinical implications for cognitive behavioral interventions are discussed.

The effects of methylphenidate on cerebral responses to conflict anticipation and unsigned prediction error in a stop-signal task

To adapt flexibly to a rapidly changing environment, humans must anticipate conflict and respond to surprising, unexpected events. To this end, the brain estimates upcoming conflict on the basis of prior experience and computes unsigned prediction error (UPE). Although much work implicates catecholamines in cognitive control, little is known about how pharmacological manipulation of catecholamines affects the neural processes underlying conflict anticipation and UPE computation. We addressed this issue by imaging 24 healthy young adults who received a 45 mg oral dose of methylphenidate (MPH) and 62 matched controls who did not receive MPH prior to performing the stop-signal task. We used a Bayesian Dynamic Belief Model to make trial-by-trial estimates of conflict and UPE during task performance. Replicating previous research, the control group showed anticipation-related activation in the presupplementary motor area and deactivation in the ventromedial prefrontal cortex and parahippocampal gyrus, as well as UPE-related activations in the dorsal anterior cingulate, insula, and inferior parietal lobule. In group comparison, MPH increased anticipation activity in the bilateral caudate head and decreased UPE activity in each of the aforementioned regions. These findings highlight distinct effects of catecholamines on the neural mechanisms underlying conflict anticipation and UPE, signals critical to learning and adaptive behavior.