Subjective and model-estimated reward prediction: association with the feedback-related negativity (FRN) and reward prediction error in a reinforcement learning task

Dissociable feedback valence effects on frontal midline theta during reward gain versus threat avoidance learning

While frontal midline theta (FMθ) has been associated with threat processing, with cognitive control in the context of anxiety, and with reinforcement learning, most reinforcement learning studies on FMθ have used reward rather than threat-related stimuli as reinforcer. Accordingly, the role of FMθ in threat-related reinforcement learning is largely unknown. Here, n = 23 human participants underwent one reward-, and one punishment-, based reversal learning task, which differed only with regard to the kind of reinforcers that feedback was tied to (i.e., monetary gain vs. loud noise burst, respectively). In addition to single-trial EEG, we assessed single-trial feedback expectations based on both a reinforcement learning computational model and trial-by-trial subjective feedback expectation ratings. While participants' performance and feedback expectations were comparable between the reward and punishment tasks, FMθ was more reliably amplified to negative vs. positive feedback in the reward vs. punishment task. Regressions with feedback valence, computationally derived, and self-reported expectations as predictors and FMθ as criterion further revealed that trial-by-trial variations in FMθ specifically relate to reward-related feedback-valence and not to threat-related feedback or to violated expectations/prediction errors. These findings suggest that FMθ as measured in reinforcement learning tasks may be less sensitive to the processing of events with direct relevance for fear and anxiety.

Single-trial modeling separates multiple overlapping prediction errors during reward processing in human EEG

Learning signals during reinforcement learning and cognitive control rely on valenced reward prediction errors (RPEs) and non-valenced salience prediction errors (PEs) driven by surprise magnitude. A core debate in reward learning focuses on whether valenced and non-valenced PEs can be isolated in the human electroencephalogram (EEG). We combine behavioral modeling and single-trial EEG regression to disentangle sequential PEs in an interval timing task dissociating outcome valence, magnitude, and probability. Multiple regression across temporal, spatial, and frequency dimensions characterized a spatio-tempo-spectral cascade from early valenced RPE value to non-valenced RPE magnitude, followed by outcome probability indexed by a late frontal positivity. Separating negative and positive outcomes revealed the valenced RPE value effect is an artifact of overlap between two non-valenced RPE magnitude responses: frontal theta feedback-related negativity on losses and posterior delta reward positivity on wins. These results reconcile longstanding debates on the sequence of components representing reward and salience PEs in the human EEG.

Increased Anxiety is Associated with Better Learning from Negative Feedback

Anxiety is one of the most prevalent mental health problems; it is known to impede cognitive functioning. It is believed to alter preferences for feedback-based learning in anxious and non-anxious learners. Thus, the present study measured feedback processing in adults ( N = 30) with and without anxiety symptoms using a probabilistic learning task. Event-related potential (ERP) measures were used to assess how the bias for either positive or negative feedback learning is reflected by the feedback-related negativity component (FRN), an ERP extracted from the electroencephalogram. Anxious individuals, identified by means of the Penn State Worry Questionnaire, showed a diminished FRN and increased accuracy after negative compared to positive feedback. Non-anxious individuals exhibited the reversed pattern with better learning from positive feedback, highlighting their preference for positive feedback. Our ERP results imply that impairments with feedback-based learning in anxious individuals are due to alterations in the mesolimbic dopaminergic system. Our finding that anxious individuals seem to favor negative as opposed to positive feedback has important implications for teacher–student feedback communication.

Revisiting the importance of model fitting for model-based fMRI: It does matter in computational psychiatry

Computational modeling has been applied for data analysis in psychology, neuroscience, and psychiatry. One of its important uses is to infer the latent variables underlying behavior by which researchers can evaluate corresponding neural, physiological, or behavioral measures. This feature is especially crucial for computational psychiatry, in which altered computational processes underlying mental disorders are of interest. For instance, several studies employing model-based fMRI-a method for identifying brain regions correlated with latent variables-have shown that patients with mental disorders (e.g., depression) exhibit diminished neural responses to reward prediction errors (RPEs), which are the differences between experienced and predicted rewards. Such model-based analysis has the drawback that the parameter estimates and inference of latent variables are not necessarily correct-rather, they usually contain some errors. A previous study theoretically and empirically showed that the error in model-fitting does not necessarily cause a serious error in model-based fMRI. However, the study did not deal with certain situations relevant to psychiatry, such as group comparisons between patients and healthy controls. We developed a theoretical framework to explore such situations. We demonstrate that the parameter-misspecification can critically affect the results of group comparison. We demonstrate that even if the RPE response in patients is completely intact, a spurious difference to healthy controls is observable. Such a situation occurs when the ground-truth learning rate differs between groups but a common learning rate is used, as per previous studies. Furthermore, even if the parameters are appropriately fitted to individual participants, spurious group differences in RPE responses are observable when the model lacks a component that differs between groups. These results highlight the importance of appropriate model-fitting and the need for caution when interpreting the results of model-based fMRI.

Visual Feedback Modulates Aftereffects and Electrophysiological Markers of Prism Adaptation

Prism adaptation (PA) is both a model for visuomotor learning and a promising treatment for visuospatial neglect after stroke. The task involves reaching for targets while prism glasses horizontally displace the visual field. Adaptation is hypothesized to occur through two processes: strategic recalibration, a rapid self-correction of pointing errors; and spatial realignment, a more gradual adjustment of visuomotor reference frames that produce prism aftereffects (i.e., reaching errors upon glasses removal in the direction opposite to the visual shift). While aftereffects can ameliorate neglect, not all patients respond to PA, and the neural mechanisms underlying successful adaptation are unclear. We investigated the feedback-related negativity (FRN) and the P300 event-related potential (ERP) components as candidate markers of strategic recalibration and spatial realignment, respectively. Healthy young adults wore prism glasses and performed memory-guided reaching toward vertical-line targets. ERPs were recorded in response to three different between-subject error feedback conditions at screen-touch: view of hand and target (Experiment 1), view of hand only (Experiment 2), or view of lines to mark target and hand position (view of hand occluded; Experiment 3). Conditions involving a direct view of the hand-produced stronger aftereffects than indirect hand feedback, and also evoked a P300 that decreased in amplitude as adaptation proceeded. Conversely, the FRN was only seen in conditions involving target feedback, even when aftereffects were smaller. Since conditions producing stronger aftereffects were associated with a phase-sensitive P300, this component may index a “context-updating” realignment process critical for strong aftereffects, whereas the FRN may reflect an error monitoring process related to strategic recalibration.

A tradeoff relationship between internal monitoring and external feedback during the dynamic process of reinforcement learning

Effective behavior monitoring, including internal monitoring/error detection and external monitoring/feedback, is very pivotal for reinforcement learning. However, less attention has been paid to internal monitoring and the dynamic learning performance in reinforcement learning, and there is still a heated debate on which kind of external feedback is relied on in the reinforcement learning. In order to address these questions, an adaption probabilistic selection task was used to examine the effect of the internal monitoring, external feedback and the relationship between them for approach learners and avoidance learners during dynamic learning process of reinforcement learning and behavior adaption. Error-related negativity (ERN), feedback-related negativity (FRN) and feedback-related P300 are three ERPs components, which can be used as the indexes of internal monitoring, external feedback and behavior adaption. For our results, the ERN effect of avoidance learners become large in block 3, which is earlier than approach learners (block 4). This phenomenon suggests that avoidance learners learned faster than approach learners. In addition, the FRN amplitude of avoidance learners in block 4 was significantly smaller than the other three blocks. The aforementioned results demonstrated a tradeoff relationship between the ERN and FRN effects.

Multiple Dissociations Between Comorbid Depression and Anxiety on Reward and Punishment Processing: Evidence From Computationally Informed EEG

In this report, we provide the first evidence that mood and anxiety dimensions are associated with unique aspects of EEG responses to reward and punishment, respectively. We reanalyzed data from our prior publication of a categorical depiction of depression to address more sophisticated dimensional hypotheses. Highly symptomatic depressed individuals (N = 46) completed a probabilistic learning task with concurrent EEG. Measures of anxiety and depression symptomatology were significantly correlated with each other; however, only anxiety predicted better avoidance learning due to a tighter coupling of negative prediction error signaling with punishment-specific EEG features. In contrast, depression predicted a smaller reward-related EEG feature, but this did not affect prediction error coupling or the ability to learn from reward. We suggest that this reward-related alteration reflects motivational or hedonic aspects of reward and not a diminishment in the ability to represent the information content of reinforcements. These findings compel further research into the domain-specific neural systems underlying dimensional aspects of psychiatric disease.

Choice predicts the feedback negativity

Choosing the appropriate response given the circumstance is integral to all aspects of human behavior. One way of elucidating the mechanisms of choice is to relate behavior to neural correlates. Electrophysiological evidence implicates the ERP feedback-negativity (FN) and the P300 as promising neural correlates of reward processing, an integral component of learning. However, prior research has not adequately addressed how the development of a preference to select one option over another (choice preference) relates to the FN and the P300. We assessed whether variation in choice preference predicted the FN and P300 amplitude within subjects. We used a discrete-trials two-alternative choice procedure, where the reinforcer rate for each option was dependently scheduled by a concurrent variable interval. The reinforcer ratio for selecting each option was varied between sessions. Choice was quantified using both the generalized matching law sensitivity and the log odds of staying on the same versus switching to the other alternative (stay preference). The relationship between stay preference, FN, and P300 amplitudes was assessed using the innovative application of hierarchical Bayesian linear regression. The results demonstrate that stay preference was controlled by the reinforcer ratios and credibly predicted the FN amplitude but not P300 amplitude. The findings are consistent with the view that reinforcers may guide behavior by what they signal about future reinforcement, with the FN related to such a process.

Frontal cortical effects on feedback processing and reinforcement learning: Relation of EEG asymmetry with the feedback-related negativity and behavior

Reinforcement learning refers to the acquisition of approach or avoidance action tendencies through repeated reward/nonreward feedback. Although much research on reinforcement learning has focused on the striatum, the prefrontal cortex likely modulates this process. Given prior research demonstrating a consistent pattern of lateralized frontal cortical activity in affective responses and approach/avoidance tendencies in the EEG literature, we aimed to elucidate the role of frontal EEG asymmetry in reinforcement learning. Thirty-two participants completed a probabilistic selection task in which they learned to select some targets and avoid others though correct/incorrect feedback. EEG indices of frontal cortical asymmetry were computed from alpha power recorded at baseline and during task completion. We also examined the feedback-related negativity ERP component to assess feedback processing associated with activity in the dorsal anterior cingulate cortex. Results revealed that greater right-lateralized frontal cortical activity during learning was associated with better avoidance learning, but neither left- nor right-sided asymmetry reliably related to approach learning. Results also suggested that left frontal activity may relate to reinforcement feedback processing, as indicated by the feedback-related negativity (FRN). These findings offer preliminary evidence regarding the role of frontal cortical activity in reinforcement learning while integrating classic and contemporary research on lateralized frontal cortical functions.

Valence-separated representation of reward prediction error in feedback-related negativity and positivity

Feedback-related negativity (FRN) is an event-related brain potential (ERP) component elicited by errors and negative outcomes. Previous studies proposed that FRN reflects the activity of a general error-processing system that incorporates reward prediction error (RPE). However, other studies reported inconsistent results on this issue - namely, that FRN only reflects the valence of feedback and that the magnitude of RPE is reflected by the other ERP component called P300. The present study focused on the relationship between the FRN amplitude and RPE. ERPs were recorded during a reversal learning task performed by the participants, and a computational model was used to estimate trial-by-trial RPEs, which we correlated with the ERPs. The results indicated that FRN and P300 reflected the magnitude of RPE in negative outcomes and positive outcomes, respectively. In addition, the correlation between RPE and the P300 amplitude was stronger than the correlation between RPE and the FRN amplitude. These differences in the correlation between ERP and RPE components may explain the inconsistent results reported by previous studies; the asymmetry in the correlations might make it difficult to detect the effect of the RPE magnitude on the FRN and makes it appear that the FRN only reflects the valence of feedback.

Frontal midline theta reflects anxiety and cognitive control: meta-analytic evidence

Evidence from imaging and anatomical studies suggests that the midcingulate cortex (MCC) is a dynamic hub lying at the interface of affect and cognition. In particular, this neural system appears to integrate information about conflict and punishment in order to optimize behavior in the face of action-outcome uncertainty. In a series of meta-analyses, we show how recent human electrophysiological research provides compelling evidence that frontal-midline theta signals reflecting MCC activity are moderated by anxiety and predict adaptive behavioral adjustments. These findings underscore the importance of frontal theta activity to a broad spectrum of control operations. We argue that frontal-midline theta provides a neurophysiologically plausible mechanism for optimally adjusting behavior to uncertainty, a hallmark of situations that elicit anxiety and demand cognitive control. These observations compel a new perspective on the mechanisms guiding motivated learning and behavior and provide a framework for understanding the role of the MCC in temperament and psychopathology.

Learning from experience: event-related potential correlates of reward processing, neural adaptation, and behavioral choice

To behave adaptively, we must learn from the consequences of our actions. Studies using event-related potentials (ERPs) have been informative with respect to the question of how such learning occurs. These studies have revealed a frontocentral negativity termed the feedback-related negativity (FRN) that appears after negative feedback. According to one prominent theory, the FRN tracks the difference between the values of actual and expected outcomes, or reward prediction errors. As such, the FRN provides a tool for studying reward valuation and decision making. We begin this review by examining the neural significance of the FRN. We then examine its functional significance. To understand the cognitive processes that occur when the FRN is generated, we explore variables that influence its appearance and amplitude. Specifically, we evaluate four hypotheses: (1) the FRN encodes a quantitative reward prediction error; (2) the FRN is evoked by outcomes and by stimuli that predict outcomes; (3) the FRN and behavior change with experience; and (4) the system that produces the FRN is maximally engaged by volitional actions.

Frontal theta reflects uncertainty and unexpectedness during exploration and exploitation

In order to understand the exploitation/exploration trade-off in reinforcement learning, previous theoretical and empirical accounts have suggested that increased uncertainty may precede the decision to explore an alternative option. To date, the neural mechanisms that support the strategic application of uncertainty-driven exploration remain underspecified. In this study, electroencephalography (EEG) was used to assess trial-to-trial dynamics relevant to exploration and exploitation. Theta-band activities over middle and lateral frontal areas have previously been implicated in EEG studies of reinforcement learning and strategic control. It was hypothesized that these areas may interact during top-down strategic behavioral control involved in exploratory choices. Here, we used a dynamic reward-learning task and an associated mathematical model that predicted individual response times. This reinforcement-learning model generated value-based prediction errors and trial-by-trial estimates of exploration as a function of uncertainty. Mid-frontal theta power correlated with unsigned prediction error, although negative prediction errors had greater power overall. Trial-to-trial variations in response-locked frontal theta were linearly related to relative uncertainty and were larger in individuals who used uncertainty to guide exploration. This finding suggests that theta-band activities reflect prefrontal-directed strategic control during exploratory choices.

Decision-making based on emotional images

The emotional outcome of a choice affects subsequent decision making. While the relationship between decision making and emotion has attracted attention, studies on emotion and decision making have been independently developed. In this study, we investigated how the emotional valence of pictures, which was stochastically contingent on participants’ choices, influenced subsequent decision making. In contrast to traditional value-based decision-making studies that used money or food as a reward, the “reward value” of the decision outcome, which guided the update of value for each choice, is unknown beforehand. To estimate the reward value of emotional pictures from participants’ choice data, we used reinforcement learning models that have successfully been used in previous studies for modeling value-based decision making. Consequently, we found that the estimated reward value was asymmetric between positive and negative pictures. The negative reward value of negative pictures (relative to neutral pictures) was larger in magnitude than the positive reward value of positive pictures. This asymmetry was not observed in valence for an individual picture, which was rated by the participants regarding the emotion experienced upon viewing it. These results suggest that there may be a difference between experienced emotion and the effect of the experienced emotion on subsequent behavior. Our experimental and computational paradigm provides a novel way for quantifying how and what aspects of emotional events affect human behavior. The present study is a first step toward relating a large amount of knowledge in emotion science and in taking computational approaches to value-based decision making.