Frontal theta links prediction errors to behavioral adaptation in reinforcement learning

How the Level of Reward Awareness Changes the Computational and Electrophysiological Signatures of Reinforcement Learning

The extent to which subjective awareness influences reward processing, and thereby affects future decisions, is currently largely unknown. In the present report, we investigated this question in a reinforcement learning framework, combining perceptual masking, computational modeling, and electroencephalographic recordings (human male and female participants). Our results indicate that degrading the visibility of the reward decreased, without completely obliterating, the ability of participants to learn from outcomes, but concurrently increased their tendency to repeat previous choices. We dissociated electrophysiological signatures evoked by the reward-based learning processes from those elicited by the reward-independent repetition of previous choices and showed that these neural activities were significantly modulated by reward visibility. Overall, this report sheds new light on the neural computations underlying reward-based learning and decision-making and highlights that awareness is beneficial for the trial-by-trial adjustment of decision-making strategies.SIGNIFICANCE STATEMENT The notion of reward is strongly associated with subjective evaluation, related to conscious processes such as "pleasure," "liking," and "wanting." Here we show that degrading reward visibility in a reinforcement learning task decreases, without completely obliterating, the ability of participants to learn from outcomes, but concurrently increases subjects' tendency to repeat previous choices. Electrophysiological recordings, in combination with computational modeling, show that neural activities were significantly modulated by reward visibility. Overall, we dissociate different neural computations underlying reward-based learning and decision-making, which highlights a beneficial role of reward awareness in adjusting decision-making strategies.

Oscillatory signatures of reward prediction errors in declarative learning

Reward prediction errors (RPEs) are crucial to learning. Whereas these mismatches between reward expectation and reward outcome are known to drive procedural learning, their role in declarative learning remains underexplored. Earlier work from our lab addressed this, and consistently found that signed reward prediction errors (SRPEs; "better-than-expected" signals) boost declarative learning. In the current EEG study, we sought to explore the neural signatures of SRPEs. Participants studied 60 Dutch-Swahili word pairs while RPE magnitudes were parametrically manipulated. Behaviorally, we replicated our previous findings that SRPEs drive declarative learning, with increased recognition for word pairs accompanied by large, positive RPEs. In the EEG data, at the start of reward feedback processing, we found an oscillatory (theta) signature consistent with unsigned reward prediction errors (URPEs; "different-than-expected" signals). Slightly later during reward feedback processing, we observed oscillatory (high-beta and high-alpha) signatures for SRPEs during reward feedback, similar to SRPE signatures during procedural learning. These findings illuminate the time course of neural oscillations in processing reward during declarative learning, providing important constraints for future theoretical work.

Feedback-related negativity reflects omission of monetary gains: Evidence from ERP gambling study

Feedback processing is an important aspect of learning. In the human brain, feedback processing is often examined by measuring an event-related potential, the feedback-related negativity component. Typically, the feedback-related negativity component is investigated by directly comparing gain with loss feedback randomized across trials; however, this method does not control for confounds associated with unexpected feedback. For this study we used a blocked design gambling task to investigate the sensitivity of feedback-related negativity to positive and negative feedback separately for gains and losses. While there appeared to be no significant feedback-related negativity in the loss domain, results revealed an enlarged feedback-related negativity during the omission of gains compared to the reception of gains. These findings further support the reward positivity hypothesis which declares that the feedback-related negativity is associated with the processing of outcomes in the context of gains as opposed to losses, irrespective of unexpectedness.

Theta oscillations promote temporal sequence learning

Many theoretical models suggest that neural oscillations play a role in learning or retrieval of temporal sequences, but the extent to which oscillations support sequence representation remains unclear. To address this question, we used scalp electroencephalography (EEG) to examine oscillatory activity over learning of different object sequences. Participants made semantic decisions on each object as they were presented in a continuous stream. For three "Consistent" sequences, the order of the objects was always fixed. Activity during Consistent sequences was compared to "Random" sequences that consisted of the same objects presented in a different order on each repetition. Over the course of learning, participants made faster semantic decisions to objects in Consistent, as compared to objects in Random sequences. Thus, participants were able to use sequence knowledge to predict upcoming items in Consistent sequences. EEG analyses revealed decreased oscillatory power in the theta (4-7 Hz) band at frontal sites following decisions about objects in Consistent sequences, as compared with objects in Random sequences. The theta power difference between Consistent and Random only emerged in the second half of the task, as participants were more effectively able to predict items in Consistent sequences. Moreover, we found increases in parieto-occipital alpha (10-13 Hz) and beta (14-28 Hz) power during the pre-response period for objects in Consistent sequences, relative to objects in Random sequences. Linear mixed effects modeling revealed that single trial theta oscillations were related to reaction time for future objects in a sequence, whereas beta and alpha oscillations were only predictive of reaction time on the current trial. These results indicate that theta and alpha/beta activity preferentially relate to future and current events, respectively. More generally our findings highlight the importance of band-specific neural oscillations in the learning of temporal order information.

Time-frequency approaches to investigating changes in feedback processing during childhood and adolescence

Processing feedback from the environment is an essential function during development to adapt behavior in advantageous ways. One measure of feedback processing, the feedback negativity (FN), is an ERP observed following the presentation of feedback. Findings detailing developmental changes in the FN have been mixed, possibly due to limitations in traditional ERP measurement methods. Recent work shows that both theta and delta frequency activity contribute to the FN; utilizing time-frequency methods to measure change in power and phase in these frequency bands may provide more accurate representation of feedback processing development in childhood and adolescence. We employ time-frequency power and intertrial phase synchrony measures, in addition to conventional time-domain ERP methods, to examine the development of feedback processing in the theta (4-7 Hz) and delta (.1-3 Hz) bands throughout adolescence. A sample of 54 female participants (8-17 years old) completed a gambling task while EEG was recorded. As expected, time-domain ERP amplitudes showed no association with age. In contrast, significant effects were observed for the time-frequency measures, with theta power decreasing with age and delta power increasing with age. For intertrial phase synchrony, delta synchrony increased with age, while age-related changes in theta synchrony differed for gains and losses. Collectively, these findings highlight the importance of considering time-frequency dynamics when exploring how the processing of feedback develops through late childhood and adolescence. In particular, the role of delta band activity and theta synchrony appear central to understanding age-related changes in the neural response to feedback.

Conflict and adaptation signals in the anterior cingulate cortex and ventral tegmental area

The integration and utilization of feedback in order to determine which decision strategy to use in different contexts is the core of executive function. The anterior cingulate cortex (ACC) is central to these processes but how feedback is made available to the ACC is unclear. To address this question, we trained rats with implants in the ACC and the ventral tegmental area (VTA), a dopaminergic brain region implicated in feedback processing, in a spatial decision reversal task with rule switching occurring approximately every 12 trials. Following a rule switch, the rats had to shift and sustain responses to the alternative side in order to obtain reward. Partial directed coherence (PDC) models of signal directionality between the ACC and VTA indicated that VTA → ACC communication (near 4 Hz) increased immediately prior to incorrect choices and during post-error decisions. This increase did not occur during correct choices. These data indicate that the VTA provides a feedback-driven, bottom-up modulating signal to the ACC which may be involved in assessing, and correcting for, decision conflict.

The Influence of Emotional State on Learning From Reward and Punishment in Borderline Personality Disorder

Despite preliminary evidence that individuals with borderline personality disorder (BPD) demonstrate deficits in learning from corrective feedback, no studies have examined the influence of emotional state on these learning deficits in BPD. This laboratory study examined the influence of negative emotions on learning among participants with BPD (n = 17), compared with clinical (past-year mood/anxiety disorder; n = 20) and healthy (n = 23) controls. Participants completed a reinforcement learning task before and after a negative emotion induction. The learning task involved presenting pairs of stimuli with probabilistic feedback in the training phase, and subsequently assessing accuracy for choosing previously rewarded stimuli or avoiding previously punished stimuli. ANOVAs and ANCOVAs revealed no significant between-group differences in overall learning accuracy. However, there was an effect of group in the ANCOVA for postemotion induction high-conflict punishment learning accuracy, with the BPD group showing greater decrements in learning accuracy than controls following the negative emotion induction.

Resting-state theta/beta EEG ratio is associated with reward- and punishment-related reversal learning

Prior research has shown that the ratio between resting-state theta (4–7 Hz)-beta (13–30 Hz) oscillations in the electroencephalogram (EEG) is associated with reward- and punishment-related feedback learning and risky decision making. However, it remains unclear whether the theta/beta EEG ratio is also an electrophysiological index for poorer behavioral adaptation when reward and punishment contingencies change over time. The aim of the present study was to investigate whether resting-state theta (4–7 Hz)-beta (13–30 Hz) EEG ratio correlated with reversal learning. A 4-min resting-state EEG was recorded and a gambling task with changing reward-punishment contingencies was administered in 128 healthy volunteers. Results showed an inverse relationship between theta/beta EEG ratio and reversal learning. Our findings replicate and extend previous findings by showing that higher midfrontal theta/beta EEG ratios are associated with poorer reversal learning and behavioral adaptive responses under changing environmental demands.

Interactions between frontal and posterior oscillatory dynamics support adjustment of stimulus processing during reinforcement learning

Reinforcement learning (RL) in humans is subserved by a network of striatal and frontal brain areas. The electrophysiological signatures of feedback evaluation are increasingly well understood, but how those signatures relate to the use of feedback to guide subsequent behavioral adjustment remains unclear. One mechanism for post-feedback behavioral optimization is the modulation of sensory processing. We used source-reconstructed MEG to test whether feedback affects the interactions between sources of oscillatory activity in the learning network and task-relevant stimulus-processing areas. Participants performed a probabilistic RL task in which they learned associations between colored faces and response buttons using trial-and-error feedback. Delta-band (2-4 Hz) and theta-band (4-8 Hz) power in multiple frontal regions were sensitive to feedback valence. Low and high beta-band power (12-20 and 20-30 Hz) in occipital, parietal, and temporal regions differentiated between color and face information. Consistent with our hypothesis, single-trial power-power correlations between frontal and posterior-sensory areas were modulated by the interaction between feedback valence and the relevant stimulus characteristic (color versus identity). These results suggest that long-range oscillatory coupling supports post-feedback updating of stimulus processing.

Non-motor Characterization of the Basal Ganglia: Evidence From Human and Non-human Primate Electrophysiology

Although the basal ganglia have been implicated in a growing list of human behaviors, they include some of the least understood nuclei in the brain. For several decades studies have employed numerous methodologies to uncover evidence pointing to the basal ganglia as a hub of both motor and non-motor function. Recently, new electrophysiological characterization of the basal ganglia in humans has become possible through direct access to these deep structures as part of routine neurosurgery. Electrophysiological approaches for identifying non-motor function have the potential to unlock a deeper understanding of pathways that may inform clinical interventions and particularly neuromodulation. Various electrophysiological modalities can also be combined to reveal functional connections between the basal ganglia and traditional structures throughout the neocortex that have been linked to non-motor behavior. Several reviews have previously summarized evidence for non-motor function in the basal ganglia stemming from behavioral, clinical, computational, imaging, and non-primate animal studies; in this review, instead we turn to electrophysiological studies of non-human primates and humans. We begin by introducing common electrophysiological methodologies for basal ganglia investigation, and then we discuss studies across numerous non-motor domains–emotion, response inhibition, conflict, decision-making, error-detection and surprise, reward processing, language, and time processing. We discuss the limitations of current approaches and highlight the current state of the information.

Added value of money on motor performance feedback: Increased left central beta-band power for rewards and fronto-central theta-band power for punishments

Monetary rewards and punishments have been shown to respectively enhance retention of motor memories and short-term motor performance, but their underlying neural bases in the context of motor control tasks remain unclear. Using electroencephalography (EEG), the present study tested the hypothesis that monetary rewards and punishments are respectively reflected in post-feedback beta-band (20-30 Hz) and theta-band (3-8 Hz) oscillatory power. While participants performed upper limb reaching movements toward visual targets using their right hand, the delivery of monetary rewards and punishments was manipulated as well as their probability (i.e., by changing target size). Compared to unrewarded and unpunished trials, monetary rewards and the successful avoidance of punishments both entailed greater beta-band power at left central electrodes overlaying contralateral motor areas. In contrast, monetary punishments and reward omissions both entailed increased theta-band power at fronto-central scalp sites. Additional analyses revealed that beta-band power was further increased when rewards were lowly probable. In light of previous work demonstrating similar beta-band modulations in basal ganglia during reward processing, the present results may reflect functional communication of reward-related information between the basal ganglia and motor cortical regions. In turn, the increase in fronto-central theta-band power after monetary punishments may reflect an emphasized cognitive need for behavioral adjustments. Globally, the present work identifies possible neural substrates for the growing behavioral evidence showing beneficial effects of monetary feedback on motor learning and performance.

Stress Elevates Frontal Midline Theta in Feedback-based Category Learning of Exceptions

Adapting behavior based on category knowledge is a fundamental cognitive function, which can be achieved via different learning strategies relying on different systems in the brain. Whereas the learning of typical category members has been linked to implicit, prototype abstraction learning, which relies predominantly on prefrontal areas, the learning of exceptions is associated with explicit, exemplar-based learning, which has been linked to the hippocampus. Stress is known to foster implicit learning strategies at the expense of explicit learning. Procedural, prefrontal learning and cognitive control processes are reflected in frontal midline theta (4–8 Hz) oscillations during feedback processing. In the current study, we examined the effect of acute stress on feedback-based category learning of typical category members and exceptions and the oscillatory correlates of feedback processing in the EEG. A computational modeling procedure was applied to estimate the use of abstraction and exemplar strategies during category learning. We tested healthy, male participants who underwent either the socially evaluated cold pressor test or a nonstressful control procedure before they learned to categorize typical members and exceptions based on feedback. The groups did not differ significantly in their categorization accuracy or use of categorization strategies. In the EEG, however, stressed participants revealed elevated theta power specifically during the learning of exceptions, whereas the theta power during the learning of typical members did not differ between the groups. Elevated frontal theta power may reflect an increased involvement of medial prefrontal areas in the learning of exceptions under stress.

Utilizing time-frequency amplitude and phase synchrony measure to assess feedback processing in a gambling task

The neurophysiological mechanisms involved in the evaluation of performance feedback have been widely studied in the ERP literature over the past twenty years, but understanding has been limited by the use of traditional time-domain amplitude analytic approaches. Gambling outcome valence has been identified as an important factor modulating event-related potential (ERP) components, most notably the feedback negativity (FN). Recent work employing time-frequency analysis has shown that processes indexed by the FN are confounded in the time-domain and can be better represented as separable feedback-related processes in the theta (3-7 Hz) and delta (0-3 Hz) frequency bands. In addition to time-frequency amplitude analysis, phase synchrony measures have begun to further our understanding of performance evaluation by revealing how feedback information is processed within and between various brain regions. The current study aimed to provide an integrative assessment of time-frequency amplitude, inter-trial phase synchrony, and inter-channel phase synchrony changes following monetary feedback in a gambling task. Results revealed that time-frequency amplitude activity explained separable loss and gain processes confounded in the time-domain. Furthermore, phase synchrony measures explained unique variance above and beyond amplitude measures and demonstrated enhanced functional integration between medial prefrontal and bilateral frontal, motor, and occipital regions for loss relative to gain feedback. These findings demonstrate the utility of assessing time-frequency amplitude, inter-trial phase synchrony, and inter-channel phase synchrony together to better elucidate the neurophysiology of feedback processing.

Increased conflict-induced slowing, but no differences in conflict-induced positive or negative prediction error learning in patients with schizophrenia

People with schizophrenia (PSZ) often fail to pursue rewarding activities despite largely intact in-the-moment hedonic experiences. Deficits in effort-based decision making in PSZ may be related to enhanced effects of cost or reduced reward, i.e., through the amplification of negative prediction errors or by dampened positive prediction errors (here, positive and negative prediction errors refer to outcomes that are better or worse than expected respectively). We administered a modified Simon task to people with schizophrenia (PSZ; N = 46) and healthy controls (N = 32). The modification included a reinforcement learning component, where positive and negative prediction errors are dampened or boosted through the use of cognitively-effortful response conflict. EEG was recorded concurrently to investigate potential differences in conflict enhanced mid-frontal theta power between PSZ and controls. We found an enhanced effect of response conflict on response time in people with schizophrenia, but no discernible difference in conflict processing as reflected by the lack of a difference in theta-power enhancement to conflict in mid-frontal regions. Using the reinforcement learning transfer phase of the modified Simon task, PSZ also showed clear deficits in selecting the most rewarding stimulus during the 'easy' (most discriminable in terms of value) stimulus contrasts. However, we failed to find a difference between patients and controls in their gain or avoidance learning bias, nor did these biases correlate with negative symptoms. Previous studies had failed to find significant conflict effects on the Simon task likely due to its modest effect size. Our results show that PSZ do indeed possess subtle impairments in response-conflict, suggesting an increase in cognitive effort required for appropriate responding. In addition, while the lack of an overt positive or negative prediction error bias (i.e., a bias towards punishment or reward learning) was unexpected, it is consistent with recent work showing intact estimation of value when the reinforcement learning system is isolated from other contributors to value learning.

Anterior Cingulate Cortex Modulation of the Ventral Tegmental Area in an Effort Task

Information gained during goal pursuit motivates adaptive behavior. The anterior cingulate cortex (ACC) supports adaptive behavior, but how ACC signals are translated into motivational signals remains unclear. Rats with implants in the ACC and ventral tegmental area (VTA), a dopaminergic brain area implicated in motivation, were trained to run laps around a rectangular track for a fixed reward, where each lap varied in physical effort (a 30-cm climbable barrier). Partial directed coherence analysis of local field potentials revealed that ACC theta (4-12 Hz) activity increased as rats entered the barrier-containing region of the maze in trials when the barrier was absent and predicted similar changes in VTA theta activity. This did not occur in effortful, barrier-present trials. These data suggest that the ACC provides a top-down modulating signal to the VTA that can influence the motivation with which to pursue a reward.

Rewarding images do not invoke the reward positivity: They inflate it

Increasing evidence suggests that the reward positivity conforms to an axiomatic reward prediction error - that is, it closely follows the rule-like encoding of surprising reinforcers. However, a major limitation in these EEG studies is the over-reliance on a single class of secondary rewards like points or money, constraining dimensionality and limiting generalizability. In the current suite of studies we address this limitation by leveraging different classes of rewards outcomes, specifically emotionally pleasant pictures. Over a series of three experiments, participants were able to choose idiosyncratically preferred pictures as rewards. During the first two experiments, participants were rewarded with either high or low points or high or low preferred pictures. The reward positivity was modulated by points, but not by pictures (regardless of preference), which instead evoked enhanced N2 amplitudes. In a third study that paired high/low points and preferred/non-preferred pictures, the point-induced reward positivity was inflated by the presence of a preferred picture. In line with past research stating the reward positivity is primarily sensitive to positive reward prediction error, this report finds that it is also influenced by a liking dimension, which possibly acts as an affective state to frame the motivational aspect of extrinsic rewards.

Behavioral preference in sequential decision-making and its association with anxiety

In daily life, people often make consecutive decisions before the ultimate goal is reached (i.e., sequential decision-making). However, this kind of decision-making has been largely overlooked in the literature. The current study investigated whether behavioral preference would change during sequential decisions, and the neural processes underlying the potential changes. For this purpose, we revised the classic balloon analogue risk task and recorded the electroencephalograph (EEG) signals associated with each step of decision-making. Independent component analysis performed on EEG data revealed that four EEG components elicited by periodic feedback in the current step predicted participants' decisions (gamble vs. no gamble) in the next step. In order of time sequence, these components were: bilateral occipital alpha rhythm, bilateral frontal theta rhythm, middle frontal theta rhythm, and bilateral sensorimotor mu rhythm. According to the information flows between these EEG oscillations, we proposed a brain model that describes the temporal dynamics of sequential decision-making. Finally, we found that the tendency to gamble (as well as the power intensity of bilateral frontal theta rhythms) was sensitive to the individual level of trait anxiety in certain steps, which may help understand the role of emotion in decision-making.

Mood congruent tuning of reward expectation in positive mood: evidence from FRN and theta modulations

Positive mood broadens attention and builds additional mental resources. However, its effect on performance monitoring and reward prediction errors remain unclear. To examine this issue, we used a standard mood induction procedure (based on guided imagery) and asked 45 participants to complete a gambling task suited to study reward prediction errors by means of the feedback-related negativity (FRN) and mid-frontal theta band power. Results showed a larger FRN for negative feedback as well as a lack of reward expectation modulation for positive feedback at the theta level with positive mood, relative to a neutral mood condition. A control analysis showed that this latter result could not be explained by the mere superposition of the event-related brain potential component on the theta oscillations. Moreover, these neurophysiological effects were evidenced in the absence of impairments at the behavioral level or increase in autonomic arousal with positive mood, suggesting that this mood state reliably altered brain mechanisms of reward prediction errors during performance monitoring. We interpret these new results as reflecting a genuine mood congruency effect, whereby reward is anticipated as the default outcome with positive mood and therefore processed as unsurprising (even when it is unlikely), while negative feedback is perceived as unexpected.

Proactive vs. reactive car driving: EEG evidence for different driving strategies of older drivers

Aging is associated with a large heterogeneity in the extent of age-related changes in sensory, motor, and cognitive functions. All these functions can influence the performance in complex tasks like car driving. The present study aims to identify potential differences in underlying cognitive processes that may explain inter-individual variability in driving performance. Younger and older participants performed a one-hour monotonous driving task in a driving simulator under varying crosswind conditions, while behavioral and electrophysiological data were recorded. Overall, younger and older drivers showed comparable driving performance (lane keeping). However, there was a large difference in driving lane variability within the older group. Dividing the older group in two subgroups with low vs. high driving lane variability revealed differences between the two groups in electrophysiological correlates of mental workload, consumption of mental resources, and activation and sustaining of attention: Older drivers with high driving lane variability showed higher frontal Alpha and Theta activity than older drivers with low driving lane variability and—with increasing crosswind—a more pronounced decrease in Beta activity. These results suggest differences in driving strategies of older and younger drivers, with the older drivers using either a rather proactive and alert driving strategy (indicated by low driving lane variability and lower Alpha and Beta activity), or a rather reactive strategy (indicated by high driving lane variability and higher Alpha activity).

Predictions through evidence accumulation over time

It has been proposed that the brain specializes in predicting future states of the environment. These predictions are probabilistic, and must be continuously updated on the basis of their mismatch with actual evidence. Although electrophysiological data disclose neural activity patterns in relation to predictive processes, little is known about how this activity supports prediction build-up through evidence accumulation. Here we addressed this gap. Participants were required to make moment-by-moment predictions about stimuli presented in sequences in which gathering evidence from previous items as they were presented was either possible or not. Two event-related potentials (ERP), a frontocentral P2 and a central P3, were sensitive to information accumulation throughout the sequence. Time-frequency (TF) analyses revealed that prediction build-up process also modulated centrally distributed theta activity, and that alpha power was suppressed in anticipation to fully predictable stimuli. Results are in agreement with the notion of predictions as probability distributions and highlight the ability of observers to extract those probabilities in a changing environment and to adjust their predictions consequently.

Transcranial Alternating Current Stimulation Modulates Risky Decision Making in a Frequency-Controlled Experiment

In this study, we investigated the effect of transcranial alternating current stimulation (tACS) on voluntary risky decision making and executive control in humans. Stimulation was delivered online at 5 Hz (θ), 10 Hz (α), 20 Hz (β), and 40 Hz (γ) on the left and right frontal area while participants performed a modified risky decision-making task. This task allowed participants to voluntarily select between risky and certain decisions associated with potential gains or losses, while simultaneously measuring the cognitive control component (voluntary switching) of decision making. The purpose of this experimental design was to test whether voluntary risky decision making and executive control can be modulated with tACS in a frequency-specific manner. Our results revealed a robust effect of a 20-Hz stimulation over the left prefrontal area that significantly increased voluntary risky decision making, which may suggest a possible link between risky decision making and reward processing, underlined by β-oscillatory activity.

Medial frontal cortex response to unexpected motivationally salient outcomes

The medial frontal cortex (MFC) plays a central role allocating resources to process salient information, in part by responding to prediction errors. While there is some recent debate, the feedback-related negativity (FRN) is thought to index a reward prediction error by signaling outcomes that are worse than expected. A recent study utilizing electric shock provided data inconsistent with these accounts and reported that the omission of both appetitive (money) and aversive outcomes (electric shocks) elicited a medial frontal negativity. These data suggest that the ERPs within this time range support a salience prediction error that responds to unexpected events regardless of valence. To compare the reward and salience prediction error models, we employed a design that delivered both appetitive (monetary) and aversive (noise burst) outcomes. Participants completed a passive S1/S2 prediction design where S1 predicted S2 with 80% accuracy and S2 predicted the outcome with 100% accuracy. We compared both earlier and later ERP responses over the medial frontal cortex to compare the salience and reward prediction hypotheses. Considering both time windows, the ERP response to S2 in the early time window was most positive when S2 signaled that an outcome was unexpectedly delivered and in the later time window, was most negative when an outcome was unexpectedly withheld, regardless of outcome valence. Thus, these results are more consistent with a salience prediction error rather than a reward prediction error.

Cingulate and cerebellar beta oscillations are engaged in the acquisition of auditory-motor sequences

Singing, music performance, and speech rely on the retrieval of complex sounds, which are generated by the corresponding actions and are organized into sequences. It is crucial in these forms of behavior that the serial organization (i.e., order) of both the actions and associated sounds be monitored and learned. To investigate the neural processes involved in the monitoring of serial order during the initial learning of sensorimotor sequences, we performed magnetoencephalographic recordings while participants explicitly learned short piano sequences under the effect of occasional alterations of auditory feedback (AAF). The main result was a prominent and selective modulation of beta (13-30 Hz) oscillations in cingulate and cerebellar regions during the processing of AAF that simulated serial order errors. Furthermore, the AAF-induced modulation of beta oscillations was associated with higher error rates, reflecting compensatory changes in sequence planning. This suggests that cingulate and cerebellar beta oscillations play a role in tracking serial order during initial sensorimotor learning and in updating the mapping of the sensorimotor representations. The findings support the notion that the modulation of beta oscillations is a candidate mechanism for the integration of sequential motor and auditory information during an early stage of skill acquisition in music performance. This has potential implications for singing and speech. Hum Brain Mapp 38:5161-5179, 2017. © 2017 Wiley Periodicals, Inc.

Name conditioning in event-related brain potentials

Four experiments are reported in which two harmonic tones (CS+ and CS-) were paired with a participant's own name (SON) and different names (DN), respectively. A third tone was not paired with any other stimulus and served as a standard (frequent stimulus) in a three-stimuli oddball paradigm. The larger posterior positivity (P3) to SON than DN, found in previous studies, was replicated in all experiments. Conditioning of the P3 response was albeit observed in two similar experiments (1 and 3), but the obtained effects were weak and not identical in the two experiments. Only Experiment 4, where the number of CS/UCS pairings and the Stimulus-Onset Asynchrony between CS and UCS were increased, showed clear CS+/CS- differences both in time and time-frequency domains. Surprisingly, differential responses to CS+ and CS- were also obtained in Experiment 2, although SON and DN in that experiment were masked and never consciously recognized as meaningful words (recognition rate 0/63 participants). The results are discussed in the context of other ERP conditioning experiments and, particularly, the studies of non-conscious effect on ERP. Several further experiments are suggested to replicate and extend the present findings and to remove the remaining methodological limitations.

Expectancy effects in feedback processing are explained primarily by time-frequency delta not theta

The roles of outcome valence and expectancy in feedback processing have been investigated as important factors modulating event-related potential (ERP) measures including the feedback negativity (FN) and P300, but results have been inconsistent. Recent work from our group has shown that processes underlying the FN and P300 are better represented as separable processes in the theta (3-7Hz) and delta (0-3Hz) ranges using time-frequency analysis. The current study evaluated the modulation of time-domain FN and P300 and time-frequency theta and delta to outcome valence and expectancy in a gambling feedback task paradigm. Results revealed that the FN was sensitive to valence but not expectancy, and that valence effects were driven by loss-sensitive theta and gain-sensitive delta. Alternatively, the P300 was sensitive to the expectedness of outcomes but only for gain trials, and these expectancy differences were explained by time-frequency delta not theta. These results add to a growing body of research showing that time-frequency measures reflect separable processes underlying time-domain components, where theta is more sensitive to primary task features and less sensitive to secondary features while delta is sensitive to primary and more complex, secondary task features.

Cortical Networks for Correcting Errors in Sensorimotor Synchronization Depend on the Direction of Asynchrony

Recent work provides clues that different cortical mechanisms may be employed when correcting for errors in sensorimotor synchronization that increase tap-tone asynchrony compared with those that decrease it. The authors tested this hypothesis by recording 64-channel electroencephalography while participants synchronized with an auditory metronome. We systematically introduced positive and negative phase-shift perturbations that were either liminal (10%) and subliminal (3%). We used a distributed source modeling approach to evaluate oscillatory activity and connectivity of discrete cortical sources. Three key findings support our hypothesis. First was a theta band response indicative of error detection and top-down control observed in frontomedial presupplementary motor area (pre-SMA) and anterior cingulate for liminal positive perturbations. Second was an increase in theta band coupling between the SMA and contralateral motor cortex exclusively for positive perturbations suggesting a top-down modulation of motor parameters. Third, when compared with other conditions, liminal positive perturbations result in an increase in postmovement beta rebound within contralateral primary motor cortex. The authors propose that frontomedial motor areas exert a top-down inhibitory influence over the primary motor cortex to effectively lengthen tap intervals in response to lengthening tap-tone asynchronies.

I Don't Want to Miss a Thing - Learning Dynamics and Effects of Feedback Type and Monetary Incentive in a Paired Associate Deterministic Learning Task

Effective functioning in a complex environment requires adjusting of behavior according to changing situational demands. To do so, organisms must learn new, more adaptive behaviors by extracting the necessary information from externally provided feedback. Not surprisingly, feedback-guided learning has been extensively studied using multiple research paradigms. The purpose of the present study was to test the newly designed Paired Associate Deterministic Learning task (PADL), in which participants were presented with either positive or negative deterministic feedback. Moreover, we manipulated the level of motivation in the learning process by comparing blocks with strictly cognitive, informative feedback to blocks where participants were additionally motivated by anticipated monetary reward or loss. Our results proved the PADL to be a useful tool not only for studying the learning process in a deterministic environment, but also, due to the varying task conditions, for assessing differences in learning patterns. Particularly, we show that the learning process itself is influenced by manipulating both the type of feedback information and the motivational significance associated with the expected monetary reward.

Slow and Fast Responses: Two Mechanisms of Trial Outcome Processing Revealed by EEG Oscillations

Cognitive control includes maintenance of task-specific processes related to attention, and non-specific regulation of motor threshold. Depending upon the nature of the behavioral tasks, these mechanisms may predispose to different kinds of errors, with either increased or decreased response time (RT) of erroneous responses relative to correct responses. Specifically, slow responses are related to attentional lapses and decision uncertainty, these conditions tending to delay RTs of both erroneous and correct responses. Here we studied if RT may be a valid approximation distinguishing trials with high and low levels of sustained attention and decision uncertainty. We analyzed response-related and feedback-related modulations in theta, alpha and beta band activity in the auditory version of the two-choice condensation task, which is highly demanding for sustained attention while involves no inhibition of prepotent responses. Depending upon response speed and accuracy, trials were divided into slow correct, slow erroneous, fast correct and fast erroneous. We found that error-related frontal midline theta (FMT) was present only on fast erroneous trials. The feedback-related FMT was equally strong on slow erroneous and fast erroneous trials. Late post-response posterior alpha suppression was stronger on erroneous slow trials. Feedback-related frontal beta was present only on slow correct trials. The data obtained cumulatively suggests that RT allows distinguishing the two types of trials, with fast trials related to higher levels of attention and low uncertainty, and slow trials related to lower levels of attention and higher uncertainty.

Inter-trial Coherence of Medial Frontal Theta Oscillations Linked to Differential Feedback Processing in Youth and Young Adults with Autism

BACKGROUND

Impairment in prediction and appreciation for choice outcomes could contribute to several core symptoms of ASD. We examined electroencephalography (EEG) oscillations in 27 youth and young adults diagnosed with autism spectrum disorder (ASD) and 22 IQ-matched neurotypical controls while they performed a chance-based reward prediction task.

METHOD

We re-analyzed our previously published ERP data (Larson et al., 2011) and examined theta band oscillations (4-8 Hz) at frontal midline sites, within a timing window that overlaps with the feedback-related negativity (FRN). We focused on event-related changes after presentation of feedback for reward (WIN) and punitive (LOSE) outcomes, both for spectral power and inter-trial phase coherence.

RESULTS

In our reward prediction task, for both groups, medial frontal theta power and phase coherence were greater following LOSE compared to WIN feedback. However, compared to controls, inter-trial coherence of medial frontal theta was significantly lower overall (across both feedback types) for individuals with ASD. Our results indicate that while individuals with ASD are sensitive to the valence of reward feedback, comparable to their neurotypical peers, they have reduced synchronization of medial frontal theta activity during feedback processing.

CONCLUSIONS

This finding are consistent with previous studies showing neural variability in ASD and suggest that the processes underlying decision-making and reinforcement learning may be atypical and less efficient in ASD.

Error-related oscillatory activity is modulated by novelty seeking in the reward condition

Behavioural research has revealed the influence of motivation conditions on cognitive task performance and demonstrated that these influences are modulated by temperament factors. Modern neuroimaging methods enable analysis of neuropsychological mechanisms through which individual differences in reinforcement sensitivity may influence cognitive functioning. In the study, fifty-six participants were scored on the Cloninger's Temperament and Character Inventory to assess punishment and reward sensitivity. Then, subjects participated in an EEG experiment using the numerical Stroop task under different motivational conditions. In one condition, they were punished for erroneous responses; in the other, they were rewarded for correct performance. We analysed event related changes in EEG spectral power to investigate the influence of temperamentally driven differences on error-related oscillatory brain activity. In agreement with previous findings, after incorrect responses an increase in frontocentral theta (3-7Hz) and a decrease in occipital alpha (10-11Hz) power were observed. Moreover, a multivariate regression analysis showed that these spectral markers were modulated by temperamental trait Novelty Seeking in the reward condition. To our knowledge, we are the first to demonstrate such a relationship between individual differences and error-related oscillatory activity. This neuronal pattern may explain why participants that score high on Novelty Seeking trait are highly motivated and strongly engaged in a task when a reward might be earned. Thus, in conclusion we emphasise that to understand an individual's response to errors, it is necessary to account simultaneously for motivational conditions as well as temperament traits.

Characterizing the roles of alpha and theta oscillations in multisensory attention

Cortical alpha oscillations (8-13Hz) appear to play a role in suppressing distractions when just one sensory modality is being attended, but do they also contribute when attention is distributed over multiple sensory modalities? For an answer, we examined cortical oscillations in human subjects who were dividing attention between auditory and visual sequences. In Experiment 1, subjects performed an oddball task with auditory, visual, or simultaneous audiovisual sequences in separate blocks, while the electroencephalogram was recorded using high-density scalp electrodes. Alpha oscillations were present continuously over posterior regions while subjects were attending to auditory sequences. This supports the idea that the brain suppresses processing of visual input in order to advantage auditory processing. During a divided-attention audiovisual condition, an oddball (a rare, unusual stimulus) occurred in either the auditory or the visual domain, requiring that attention be divided between the two modalities. Fronto-central theta band (4-7Hz) activity was strongest in this audiovisual condition, when subjects monitored auditory and visual sequences simultaneously. Theta oscillations have been associated with both attention and with short-term memory. Experiment 2 sought to distinguish these possible roles of fronto-central theta activity during multisensory divided attention. Using a modified version of the oddball task from Experiment 1, Experiment 2 showed that differences in theta power among conditions were independent of short-term memory load. Ruling out theta's association with short-term memory, we conclude that fronto-central theta activity is likely a marker of multisensory divided attention.

Single neuron activity and theta modulation in the posterior parietal cortex in a visuospatial attention task

The posterior parietal cortex (PPC) is implicated in directing and maintaining visual attention to locations in space. We hypothesized that the PPC also engages other cognitive processes in the transformation of behaviorally relevant visual inputs into appropriate actions, for example, monitoring of multiple locations, selection of responses to locations in space, and monitoring the outcome of response selections. We recorded single cells and local field potentials in the rat PPC during performance on a novel visuospatial attention (VSA) task that requires visually monitoring locations in space in order to make appropriate stimulus-guided locomotor responses. In each trial, rats attended to four locations on the floor of a maze. A randomly chosen location was briefly illuminated. Approach to the correct target location was followed by food reward. We observed that PPC activity correlated with multiple phases of the VSA task, including monitoring for stimulus onset, detection of a target, spatial location of the target, and target choice. A substantial proportion of cells with behavioral correlates were also modulated by outcome of the trial. Our analyses of local field potentials revealed strong oscillatory rhythms in the theta frequency band, and more than a third of PPC neurons were phase locked to theta oscillations. As in other brain regions, theta power correlated with running speed. Peak theta power was higher in superficial layers than deep layers providing evidence against volume conduction from the hippocampus. In addition, theta power was sensitive to the outcome of a choice. Theta power was significantly higher following incorrect choices compared with correct choices, possibly providing a prediction error signal. Our study provides evidence that the rat PPC has multiple roles in the translation of visual information into appropriate behavioral actions. © 2016 Wiley Periodicals, Inc.

MEG Signatures of a Perceived Match or Mismatch between Individual and Group Opinions

Humans often adjust their opinions to the perceived opinions of others. Neural responses to a perceived match or mismatch between individual and group opinions have been investigated previously, but some findings are inconsistent. In this study, we used magnetoencephalographic source imaging to investigate further neural responses to the perceived opinions of others. We found that group opinions mismatching with individual opinions evoked responses in the anterior and posterior medial prefrontal cortices, as well as in the temporoparietal junction and ventromedial prefrontal cortex in the 220–320 and 380–530 ms time windows. Evoked responses were accompanied by an increase in the power of theta oscillations (4–8 Hz) over a number of frontal cortical sites. Group opinions matching with individual opinions evoked an increase in amplitude of beta oscillations (13–30 Hz) in the anterior cingulate and ventral medial prefrontal cortices. Based on these results, we argue that distinct valuation and performance-monitoring neural circuits in the medial cortices of the brain may monitor compliance of individual behavior to the perceived group norms.

Genetic influences on functional connectivity associated with feedback processing and prediction error: Phase coupling of theta-band oscillations in twins

Detection and evaluation of the mismatch between the intended and actually obtained result of an action (reward prediction error) is an integral component of adaptive self-regulation of behavior. Extensive human and animal research has shown that evaluation of action outcome is supported by a distributed network of brain regions in which the anterior cingulate cortex (ACC) plays a central role, and the integration of distant brain regions into a unified feedback-processing network is enabled by long-range phase synchronization of cortical oscillations in the theta band. Neural correlates of feedback processing are associated with individual differences in normal and abnormal behavior, however, little is known about the role of genetic factors in the cerebral mechanisms of feedback processing. Here we examined genetic influences on functional cortical connectivity related to prediction error in young adult twins (age 18, n=399) using event-related EEG phase coherence analysis in a monetary gambling task. To identify prediction error-specific connectivity pattern, we compared responses to loss and gain feedback. Monetary loss produced a significant increase of theta-band synchronization between the frontal midline region and widespread areas of the scalp, particularly parietal areas, whereas gain resulted in increased synchrony primarily within the posterior regions. Genetic analyses showed significant heritability of frontoparietal theta phase synchronization (24 to 46%), suggesting that individual differences in large-scale network dynamics are under substantial genetic control. We conclude that theta-band synchronization of brain oscillations related to negative feedback reflects genetically transmitted differences in the neural mechanisms of feedback processing. To our knowledge, this is the first evidence for genetic influences on task-related functional brain connectivity assessed using direct real-time measures of neuronal synchronization.

Neural signatures of experience-based improvements in deterministic decision-making

Feedback about our choices is a crucial part of how we gather information and learn from our environment. It provides key information about decision experiences that can be used to optimize future choices. However, our understanding of the processes through which feedback translates into improved decision-making is lacking. Using neuroimaging (fMRI) and cognitive models of decision-making and learning, we examined the influence of feedback on multiple aspects of decision processes across learning. Subjects learned correct choices to a set of 50 word pairs across eight repetitions of a concurrent discrimination task. Behavioral measures were then analyzed with both a drift-diffusion model and a reinforcement learning model. Parameter values from each were then used as fMRI regressors to identify regions whose activity fluctuates with specific cognitive processes described by the models. The patterns of intersecting neural effects across models support two main inferences about the influence of feedback on decision-making. First, frontal, anterior insular, fusiform, and caudate nucleus regions behave like performance monitors, reflecting errors in performance predictions that signal the need for changes in control over decision-making. Second, temporoparietal, supplementary motor, and putamen regions behave like mnemonic storage sites, reflecting differences in learned item values that inform optimal decision choices. As information about optimal choices is accrued, these neural systems dynamically adjust, likely shifting the burden of decision processing from controlled performance monitoring to bottom-up, stimulus-driven choice selection. Collectively, the results provide a detailed perspective on the fundamental ability to use past experiences to improve future decisions.

What can time-frequency and phase coherence measures tell us about the genetic basis of P3 amplitude?

In a recent comprehensive investigation, we largely failed to identify significant genetic markers associated with P3 amplitude or to corroborate previous associations between P3 and specific single nucleotide polymorphisms (SNPs) or genes. In the present study we extended this line of investigation to examine time-frequency (TF) activity and intertrial phase coherence (ITPC) in the P3 time window, both of which are associated with P3 amplitude. Previous genome-wide research has reported associations between P3-related theta and delta activity and individual genetic variants. A large, population-based sample of 4211 subjects, comprising male and female adolescent twins and their parents, was genotyped for 527,828 single nucleotide polymorphisms (SNPs), from which over six million SNPs were accurately imputed. Heritability estimates were greater for TF energy than ITPC, whether based on biometric models or the combined influence of all measured SNPs (derived from genome-wide complex trait analysis). The magnitude of overlap in the specific SNPs associated with delta energy and ITPC and P3 amplitude was significant. A genome-wide analysis of all SNPs, accompanied by an analysis of approximately 17,600 genes, indicated a region of chromosome 2 around TEKT4 that was significantly associated with theta ITPC. Analysis of candidate SNPs and genes previously reported to be associated with P3 or related phenotypes yielded one association surviving correction for multiple tests: between theta energy and CRHR1. However, we did not obtain significant associations for SNPs implicated in previous genome-wide studies of TF measures. Identifying specific genetic variants associated with P3 amplitude remains a challenge.

Time-frequency phase-synchrony approaches with ERPs

Time-frequency signal processing approaches are well-developed, and have been widely employed for the study of the energy distribution of event-related potential (ERP) data across time and frequency. Wavelet time-frequency transform (TFT) and Cohen's class of time-frequency distributions (TFD) are the most widely used in the field. While ERP TFT approaches have been most extensively developed for amplitude measures, reflecting the magnitude of regional neuronal activity, time-frequency phase-synchrony measures have gained increased utility in recent years for the assessment of functional connectivity. Phase synchrony measures can be used to index the functional integration between regions (interregional), in addition to the consistency of activity within region (intertrial). In this paper, we focus on a particular class of time-frequency distributions belonging to Cohen's class, known as the Reduced Interference Distribution (RID) for quantifying functional connectivity, which we recently introduced (Aviyente et al., 2011). The present report first summarizes common time-frequency approaches to computing phase-synchrony with ERP data in order to highlight the similarities and differences relative to the RID. In previous work, we demonstrated differences between the RID and wavelet approaches to indexing phase-synchrony, and have applied the RID to demonstrate that RID-based time-frequency phase-synchrony measures can index increased functional connectivity between medial and lateral prefrontal regions during control processing, observed in the theta band during the error-related negativity (ERN). Because ERN amplitude measures have been associated with two other widely studied medial-frontal theta components (no-go N2; feedback negativity, FN), the application of the RID phase synchrony measure in the present report extends our previous work with ERN to include theta activity during the no-go N2 (inhibitory processing) and the feedback negativity (FN; loss feedback processing). Findings support the idea that similar medial-lateral prefrontal functional connectivity underlies the ERN, no-go N2, and FN components, and provide initial validation that the proposed RID-based time-frequency phase-synchrony measure can index this activity.

To Know or Not to Know? Theta and Delta Reflect Complementary Information about an Advanced Cue before Feedback in Decision-Making

To investigate brain activity during the reinforcement learning process in social contexts is a topic of increasing research interest. Previous studies have mainly focused on using electroencephalograms (EEGs) for feedback evaluation in reinforcement learning tasks by measuring event-related potentials. Few studies have investigated the time–frequency (TF) profiles of a cue that manifested whether a following feedback is available or not after decision-making. Moreover, it remains unclear whether the TF profiles of the cue interact with different agents to whom the feedback related. In this study we used the TF approach to test EEG oscillations of the cue stimuli in three agents (‘Self’, ‘Other’, and ‘Computer’) conditions separately. The results showed that the increased central-posterior delta power was elicited by the feedback unavailable cues more so than with the feedback available cue within 200–350 ms after the onset of the cue, but only in the self-condition. Moreover, a frontal-central theta oscillation had enhanced power when following the feedback unavailable cue as opposed to the feedback available cue across three agencies. These findings demonstrated that the cue for knowing an outcome produced reward prediction error-like signals, which were mirrored by the delta and theta oscillations during decision-making. More importantly, the present study demonstrated that the theta and delta oscillations reflected separable components of the advanced cue processing before the feedback in decision-making.