Reward, Salience, and Agency in Event-Related Potentials for Appetitive and Aversive Contexts

Distinct processing of the state prediction error signals in frontal and parietal correlates in learning the environment model

Goal-directed reinforcement learning constructs a model of how the states in the environment are connected and prospectively evaluates action values by simulating experience. State prediction error (SPE) is theorized as a crucial signal for learning the environment model. However, the underlying neural mechanisms remain unclear. Here, using electroencephalogram, we verified in a two-stage Markov task two neural correlates of SPEs: an early negative correlate transferring from frontal to central electrodes and a late positive correlate over parietal regions. Furthermore, by investigating the effects of explicit knowledge about the environment model and rewards in the environment, we found that, for the parietal correlate, rewards enhanced the representation efficiency (beta values of regression coefficient) of SPEs, whereas explicit knowledge elicited a larger SPE representation (event-related potential activity) for rare transitions. However, for the frontal and central correlates, rewards increased activities in a content-independent way and explicit knowledge enhanced activities only for common transitions. Our results suggest that the parietal correlate of SPEs is responsible for the explicit learning of state transition structure, whereas the frontal and central correlates may be involved in cognitive control. Our study provides novel evidence for distinct roles of the frontal and the parietal cortices in processing SPEs.

The detection of self-group conflicts in exercise behaviors differs with social network centrality: ERP evidence

BACKGROUND

The influence of social norms on exercise behaviors has been explored in studies over the years. However, little is known about whether an individual's role (central or peripheral) in his or her social network, which is associated with social skills, could shift his or her susceptibility to normative effects on exercise behaviors. To that end, event-related potentials (ERPs) were recorded to examine the underlying cognitive mechanism of the effects of network centrality on normative social influence.

METHODS

We manipulated network centrality by assigning participants to exercise support groups, with group members who were their nominated friends (high centrality) or nonnominated classmates (low centrality). Participants were asked to evaluate their willingness to engage in various exercises, after viewing discrepant group ratings (peer influence) or not viewing (no-influence).

RESULTS

Peer influence evoked a larger negative-going feedback-related negativity (FRN) wave, which was linked to automatic social conflict detection, and a larger positive-going P3 wave, which was linked to subsequent conformity behavioral changes. However, effects on the FRN, not the P3, were observed only in the high-centrality group.

CONCLUSION

Our results highlight the important roles of network centrality in encoding self-group exercise attitude discrepancy rather than in decision-making regarding exercise attitude adjustments. Interventions aimed at promoting exercise behaviors should be considered in a broader social environmental framework.

Neural dissociation between reward and salience prediction errors through the lens of optimistic bias

The question of how the brain represents reward prediction errors is central to reinforcement learning and adaptive, goal-directed behavior. Previous studies have revealed prediction error representations in multiple electrophysiological signatures, but it remains elusive whether these electrophysiological correlates underlying prediction errors are sensitive to valence (in a signed form) or to salience (in an unsigned form). One possible reason concerns the loose correspondence between objective probability and subjective prediction resulting from the optimistic bias, that is, the tendency to overestimate the likelihood of encountering positive future events. In the present electroencephalography (EEG) study, we approached this question by directly measuring participants' idiosyncratic, trial-to-trial prediction errors elicited by subjective and objective probabilities across two experiments. We adopted monetary gain and loss feedback in Experiment 1 and positive and negative feedback as communicated by the same zero-value feedback in Experiment 2. We provided electrophysiological evidence in time and time-frequency domains supporting both reward and salience prediction error signals. Moreover, we showed that these electrophysiological signatures were highly flexible and sensitive to an optimistic bias and various forms of salience. Our findings shed new light on multiple presentations of prediction error in the human brain, which differ in format and functional role.

Feedback-related negativity in perfectionists: An index of performance outcome evaluation

It has been suggested that maladaptive perfectionists are more prone to concern over their performance outcomes than adaptive perfectionists. Performance outcome evaluation is reflected in the amplitude of feedback-related negativity (FRN) in brain electroencephalography (EEG). Hence, the amplitude of the FRN after receiving unfavorable feedback indicating a negative performance outcome may reflect personality characteristics. In other words, EEG could be a better marker of personality characteristics than self-report measures. However, the FRN component has not yet been investigated between different types of perfectionists. In the present study, group differences in the FRN were examined between two groups of adaptive and maladaptive perfectionists and a group of non-perfectionists during a monetary gambling task. We observed a larger FRN amplitude for adaptive perfectionists than for maladaptive perfectionists. This finding is consistent with previous reports that reward prediction error is reflected in the amplitude of the FRN. This difference in FRN could be interpreted as the pessimistic outcome expectation biases in maladaptive perfectionists.

Valence precedes value in neural encoding of prediction error

Event-related potentials that follow feedback in reinforcement learning tasks have been proposed to reflect neural encoding of prediction errors. Prior research has shown that in the interval of 240-340 ms multiple different prediction error encodings appear to co-occur, including a value signal carrying signed quantitative prediction error and a valence signal merely carrying sign. The effects used to identify these two encoders, respectively a sign main effect and a sign × size interaction, do not reliably discriminate them. A full discrimination is made possible by comparing tasks in which the reinforcer available on a given trial is set to be either appetitive or aversive against tasks where a trial allows the possibility of either. This study presents a meta-analysis of reinforcement learning experiments, the majority of which presented the possibility of winning or losing money. Value and valence encodings were identified by conventional difference wave methodology but additionally by an analysis of their predicted behavior using a Bayesian analysis that incorporated nulls into the evidence for each encoder. The results suggest that a valence encoding, sensitive only to the available outcomes on the trial at hand precedes a later value encoding sensitive to the outcomes available in the wider experimental context. The implications of this for modeling computational processes of reinforcement learning in humans are discussed.

Disentangling performance-monitoring signals encoded in feedback-related EEG dynamics

The feedback-related negativity (FRN) is a well-established electrophysiological correlate of feedback-processing. However, there is still an ongoing debate whether the FRN is driven by negative or positive reward prediction errors (RPE), valence of feedback, or mere surprise. Our study disentangles independent contributions of valence, surprise, and RPE on the feedback-related neuronal signal including the FRN and P3 components using the statistical power of a sample of N = 992 healthy individuals. The participants performed a modified time-estimation task, while EEG from 64 scalp electrodes was recorded. Our results show that valence coding is present during the FRN with larger amplitudes for negative feedback. The FRN is further modulated by surprise in a valence-dependent way being more positive-going for surprising positive outcomes. The P3 was strongly driven by both global and local surprise, with larger amplitudes for unexpected feedback and local deviants. Behavioral adaptations after feedback and FRN just show small associations. Results support the theory of the FRN as a representation of a signed RPE. Additionally, our data indicates that surprising positive feedback enhances the EEG response in the time window of the P3. These results corroborate previous findings linking the P3 to the evaluation of PEs in decision making and learning tasks.

Reward prediction error in the ERP following unconditioned aversive stimuli

Reinforcement learning in humans and other animals is driven by reward prediction errors: deviations between the amount of reward or punishment initially expected and that which is obtained. Temporal difference methods of reinforcement learning generate this reward prediction error at the earliest time at which a revision in reward or punishment likelihood is signalled, for example by a conditioned stimulus. Midbrain dopamine neurons, believed to compute reward prediction errors, generate this signal in response to both conditioned and unconditioned stimuli, as predicted by temporal difference learning. Electroencephalographic recordings of human participants have suggested that a component named the feedback-related negativity (FRN) is generated when this signal is carried to the cortex. If this is so, the FRN should be expected to respond equivalently to conditioned and unconditioned stimuli. However, very few studies have attempted to measure the FRN's response to unconditioned stimuli. The present study attempted to elicit the FRN in response to a primary aversive stimulus (electric shock) using a design that varied reward prediction error while holding physical intensity constant. The FRN was strongly elicited, but earlier and more transiently than typically seen, suggesting that it may incorporate other processes than the midbrain dopamine system.