Effects of feedback delay and agency on feedback‐locked beta and theta power during reinforcement learning

Dysfunctional feedback processing in male methamphetamine abusers: Evidence from neurophysiological and computational approaches

Methamphetamine use disorder (MUD) as a major public health risk is associated with dysfunctional neural feedback processing. Although dysfunctional feedback processing in people who are substance dependent has been explored in several behavioral, computational, and electrocortical studies, this mechanism in MUDs requires to be well understood. Furthermore, the current understanding of latent components of their behavior such as learning speed and exploration-exploitation dilemma is still limited. In addition, the association between the latent cognitive components and the related neural mechanisms also needs to be explored. Therefore, in this study, the underlying neurocognitive mechanisms of feedback processing of such impairment, and age/gender-matched healthy controls are evaluated within a probabilistic learning task with rewards and punishments. Mathematical modeling results based on the Q-learning paradigm suggested that MUDs show less sensitivity in distinguishing optimal options. Additionally, it may be worth noting that MUDs exhibited a slight decrease in their ability to learn from negative feedback compared to healthy controls. Also through the lens of underlying neural mechanisms, MUDs showed lower theta power at the medial-frontal areas while responding to negative feedback. However, other EEG measures of reinforcement learning including feedback-related negativity, parietal-P300, and activity flow from the medial frontal to lateral prefrontal regions, remained intact in MUDs. On the other hand, the elimination of the linkage between value sensitivity and medial-frontal theta activity in MUDs was observed. The observed dysfunction could be due to the adverse effects of methamphetamine on the cortico-striatal dopamine circuit, which is reflected in the anterior cingulate cortex activity as the most likely region responsible for efficient behavior adjustment. These findings could help us to pave the way toward tailored therapeutic approaches.

Prediction-error-dependent processing of immediate and delayed positive feedback

Learning often involves trial-and-error, i.e. repeating behaviours that lead to desired outcomes, and adjusting behaviour when outcomes do not meet our expectations and thus lead to prediction errors (PEs). PEs have been shown to be reflected in the reward positivity (RewP), an event-related potential (ERP) component between 200 and 350 ms after performance feedback which is linked to striatal processing and assessed via electroencephalography (EEG). Here we show that this is also true for delayed feedback processing, for which a critical role of the hippocampus has been suggested. We found a general reduction of the RewP for delayed feedback, but the PE was similarly reflected in the RewP and the later P300 for immediate and delayed positive feedback, while no effect was found for negative feedback. Our results suggest that, despite processing differences between immediate and delayed feedback, positive PEs drive feedback processing and learning irrespective of delay.

Freedom of choice boosts midfrontal theta power during affective feedback processing of goal-directed actions

Sense of agency, the feeling of being in control of one's actions and their effects, is particularly relevant during goal-directed actions. During feedback learning, action effects provide information about the best course of action to reinforce positive and prevent negative outcomes. However, it is unclear whether agency experience selectively affects the processing of negative or positive feedback during the performance of goal-directed actions. As an important marker of feedback processing, we examined agency-related changes in midfrontal oscillatory activity in response to performance feedback using electroencephalography. Thirty-three participants completed a reinforcement learning task during which they received positive (monetary gain) or negative (monetary loss) feedback following item choices made either by themselves (free-choice) or by the computer (forced-choice). Independent of choice context, midfrontal theta activity was more enhanced for negative than positive feedback. In addition, free, compared to forced choices increased midfrontal theta power for both gain and loss feedback. These results indicate that freedom of choice in a motivationally salient learning task leads to a general enhancement in the processing of affective action outcomes. Our findings contribute to an understanding of the neuronal mechanisms underlying agency-related changes during action regulation and indicate midfrontal theta activity as a neurophysiological marker important for the monitoring of affective action outcomes, irrespective of feedback valence.

The modulation of pain in reward processing is reflected by increased P300 and delta oscillation

Pain elicits the desire for a reward to alleviate the unpleasant sensation. This may be a consequence of facilitated neural activities in the reward circuit. However, the temporal modulation of pain on reward processing remains unclear. We addressed this issue by recording electroencephalogram when participants received win or loss feedback in a simple gambling task. Pain treatment was conducted on 33 participants with topical capsaicin cream and on 33 participants with hand cream as a control. Results showed that pain generally increased the P300 amplitude for both types of feedback but did not affect feedback-related negativity (FRN). A significant interaction effect of treatment (painful, non-painful) and outcome (win, loss) was observed on delta oscillation as pain only enhanced the power of win feedback. In addition, the FRN and theta oscillation responded more to loss feedback, but this effect was unaffected by pain. These findings indicate that pain may enhance secondary value representation and evaluation processes of rewards, but does not influence primary distinction of reward or reward expectation. The temporal unfolding of how pain affects reward-related neural activities highlights the prominent impact of pain on high-level cognitive processes associated with reward.

Acting in Temporal Contexts: On the Behavioral and Neurophysiological Consequences of Feedback Delays

Feedback on success or failure is critical to increase rewards through behavioral adaptation or learning of dependencies from trial and error. Learning from reward feedback is thereby treated as embedded in a reinforcement learning framework. Due to temporal discounting of reward, learning in this framework is suspected to be vulnerable to feedback delay. Together, investigations of reinforcement learning in learned decision making tasks show that performance and learning impairments due to feedback delay vary as a function of task type. Performance in tasks that require implicit processing is affected by the delayed availability of feedback compared to tasks that can be accomplished with explicit processing. At the same time, the feedback related negativity, an event related potential component in the electroencephalogram that is associated with feedback processing, is affected by feedback delay similarly independent of task type. With the idea of fully implicit or explicit processing as opposite endpoints of a continuum of reciprocal shares of the implicit and explicit processing systems with feedback delay as the determinant of where on this continuum processing can be located, a common explanatory approach of both, behavioral and electrophysiological findings, is suggested.

Learning to deal with delayed outcomes: EEG oscillatory and slow potentials during the prefeedback interval

It is well established that the stimulus-preceding negativity (SPN) decreases in amplitude as a task is mastered, a phenomenon generally attributed to the reduction in anticipatory attention as feedback becomes less needed. Typically, the experiments supporting this assumption have used relatively short delays (<3 s). However, we found in a previous study that this decline in amplitude, although present during the 2.5-s prefeedback delay of a patterned key-pressing task, was absent with an 8-s delay. We reexamined this finding using a 6-s delay and found that the SPN diminished at frontal sites as participants learned a sequence of four keypress durations, but that this modulation was limited to the early half of the delay (maximum at 2 s). Decline of lateralized sensorimotor theta activity across trials was also limited to early portions of the delay. These findings suggest that processes other than anticipatory attention to feedback may be more relevant for explaining SPN diminution. Such processes could include adjustment and maintenance of action-outcome expectancies (e.g., forward models) during the prefeedback interval.

Beta oscillations following performance feedback predict subsequent recall of task-relevant information

Reward delivery in reinforcement learning tasks elicits increased beta power in the human EEG over frontal areas of the scalp but it is unclear whether these 20–30 Hz oscillations directly facilitate reward learning. We previously proposed that frontal beta is not specific to reward processing but rather reflects the role of prefrontal cortex in maintaining and transferring task-related information to other brain areas. To test this proposal, we had subjects perform a reinforcement learning task followed by a memory recall task in which subjects were asked to recall stimuli associated either with reward feedback (Reward Recall condition) or error feedback (Error Recall condition). We trained a classifier on post-feedback beta power in the Reward Recall condition to discriminate trials associated with reward feedback from those associated with error feedback and then tested the classifier on post-feedback beta power in the Error Recall condition. Crucially, the model classified error-related beta in the Error Recall condition as reward-related. The model also predicted stimulus recall from post-feedback beta power irrespective of feedback valence and task condition. These results indicate that post-feedback beta power is not specific to reward processing but rather reflects a more general task-related process.