Frontal theta links prediction errors to behavioral adaptation in reinforcement learning

Frontal Oscillatory Dynamics Predict Feedback Learning and Action Adjustment

Frontal oscillatory dynamics in the theta (4–8 Hz) and beta (20–30 Hz) frequency bands have been implicated in cognitive control processes. Here we investigated the changes in coordinated activity within and between frontal brain areas during feedback-based response learning. In a time estimation task, participants learned to press a button after specific, randomly selected time intervals (300–2000 msec) using the feedback after each button press (correct, too fast, too slow). Consistent with previous findings, theta-band activity over medial frontal scalp sites (presumably reflecting medial frontal cortex activity) was stronger after negative feedback, whereas beta-band activity was stronger after positive feedback. Theta-band power predicted learning only after negative feedback, and beta-band power predicted learning after positive and negative feedback. Furthermore, negative feedback increased theta-band intersite phase synchrony (a millisecond resolution measure of functional connectivity) among right lateral prefrontal, medial frontal, and sensorimotor sites. These results demonstrate the importance of frontal theta- and beta-band oscillations and intersite communication in the realization of reinforcement learning.

Frontal theta reflects uncertainty and unexpectedness during exploration and exploitation

In order to understand the exploitation/exploration trade-off in reinforcement learning, previous theoretical and empirical accounts have suggested that increased uncertainty may precede the decision to explore an alternative option. To date, the neural mechanisms that support the strategic application of uncertainty-driven exploration remain underspecified. In this study, electroencephalography (EEG) was used to assess trial-to-trial dynamics relevant to exploration and exploitation. Theta-band activities over middle and lateral frontal areas have previously been implicated in EEG studies of reinforcement learning and strategic control. It was hypothesized that these areas may interact during top-down strategic behavioral control involved in exploratory choices. Here, we used a dynamic reward-learning task and an associated mathematical model that predicted individual response times. This reinforcement-learning model generated value-based prediction errors and trial-by-trial estimates of exploration as a function of uncertainty. Mid-frontal theta power correlated with unsigned prediction error, although negative prediction errors had greater power overall. Trial-to-trial variations in response-locked frontal theta were linearly related to relative uncertainty and were larger in individuals who used uncertainty to guide exploration. This finding suggests that theta-band activities reflect prefrontal-directed strategic control during exploratory choices.

Contextual novelty modulates the neural dynamics of reward anticipation

We investigated how rapidly the reward-predicting properties of visual cues are signaled in the human brain and the extent these reward prediction signals are contextually modifiable. In a magnetoencephalography study, we presented participants with fractal visual cues that predicted monetary rewards with different probabilities. These cues were presented in the temporal context of a preceding novel or familiar image of a natural scene. Starting at ∼100 ms after cue onset, reward probability was signaled in the event-related fields (ERFs) over temporo-occipital sensors and in the power of theta (5-8 Hz) and beta (20-30 Hz) band oscillations over frontal sensors. While theta decreased with reward probability beta power showed the opposite effect. Thus, in humans anticipatory reward responses are generated rapidly, within 100 ms after the onset of reward-predicting cues, which is similar to the timing established in non-human primates. Contextual novelty enhanced the reward anticipation responses in both ERFs and in beta oscillations starting at ∼100 ms after cue onset. This very early context effect is compatible with a physiological model that invokes the mediation of a hippocampal-VTA loop according to which novelty modulates neural response properties within the reward circuitry. We conclude that the neural processing of cues that predict future rewards is temporally highly efficient and contextually modifiable.

Theta lingua franca: a common mid-frontal substrate for action monitoring processes

We present evidence that a multitude of mid-frontal event-related potential (ERP) components partially reflect a common theta band oscillatory process. Specifically, mid-frontal ERP components in the N2 time range and error-related negativity time range are parsimoniously characterized as reflections of theta band activities. Forty participants completed three different tasks with varying stimulus-response demands. Permutation tests were used to identify the dominant time-frequency responses of stimulus- and response-locked conditions as well as the enhanced responses to novelty, conflict, punishment, and error. A dominant theta band feature was found in all conditions, and both ERP component amplitudes and theta power measures were similarly modulated by novelty, conflict, punishment, and error. The findings support the hypothesis that generic and reactive medial prefrontal cortex processes are parsimoniously reflected by theta band activities.

Subthalamic nucleus stimulation reverses mediofrontal influence over decision threshold

It takes effort and time to tame one's impulses. Although medial prefrontal cortex (mPFC) is broadly implicated in effortful control over behavior, the subthalamic nucleus (STN) is specifically thought to contribute by acting as a brake on cortico-striatal function during decision conflict, buying time until the right decision can be made. Using the drift diffusion model of decision making, we found that trial-to-trial increases in mPFC activity (EEG theta power, 4-8 Hz) were related to an increased threshold for evidence accumulation (decision threshold) as a function of conflict. Deep brain stimulation of the STN in individuals with Parkinson's disease reversed this relationship, resulting in impulsive choice. In addition, intracranial recordings of the STN area revealed increased activity (2.5-5 Hz) during these same high-conflict decisions. Activity in these slow frequency bands may reflect a neural substrate for cortico-basal ganglia communication regulating decision processes.

Human Scalp Electroencephalography Reveals that Repetition Suppression Varies with Expectation

Repetitions of a sensory event elicit lower levels of brain activity than its initial presentation (“repetition suppression,” RS). According to one view, RS depends on the biophysics of neuronal discharge, and is thus an automatic consequence of stimulus processing (“fatigue” model). Another account suggests that RS depends on the statistical structure of the environment, and occurs when repeated stimuli are less surprising than novel stimuli (“surprise reduction” model). In support of the latter view, functional magnetic resonance imaging studies have shown that RS is modulated by the local probability of repetition. However, single-cell recordings from macaque inferotemporal area (IT) have failed to replicate this finding. Here, we recorded scalp electroencephalography from human participants viewing pairs of faces that repeated (face₁–face₁) or alternated (face₁–face₂), in contexts in which repetitions were expected or unexpected. As previously described, event-related potentials in the range of 100–400 ms recorded at posterior electrode sites and at the vertex differed between repetitions and alternations. Critically, at central electrodes, we observed that the difference between repeated and alternating stimuli was attenuated when repetitions were unexpected, as predicted by the surprise reduction model. These findings demonstrate that the modulation of RS by repetition probability is observable using direct neural recording methods in human participants, and that it occurs relatively late (>300 ms) post-stimulus. Finally, we found that theta-band (4–8 Hz) spectral power over central electrodes varied with the three-way interaction between of repetition, expectation, and the rate of change of the environment, consistent with recent reports that frontal theta may be a hallmark of learning processes originating in the anterior cingulate and medial prefrontal cortex.

Dopaminergic genes predict individual differences in susceptibility to confirmation bias

The striatum is critical for the incremental learning of values associated with behavioral actions. The prefrontal cortex (PFC) represents abstract rules and explicit contingencies to support rapid behavioral adaptation in the absence of cumulative experience. Here we test two alternative models of the interaction between these systems, and individual differences thereof, when human subjects are instructed with prior information about reward contingencies that may or may not be accurate. Behaviorally, subjects are overly influenced by prior instructions, at the expense of learning true reinforcement statistics. Computational analysis found that this pattern of data is best accounted for by a confirmation bias mechanism in which prior beliefs--putatively represented in PFC--influence the learning that occurs in the striatum such that reinforcement statistics are distorted. We assessed genetic variants affecting prefrontal and striatal dopaminergic neurotransmission. A polymorphism in the COMT gene (rs4680), associated with prefrontal dopaminergic function, was predictive of the degree to which participants persisted in responding in accordance with prior instructions even as evidence against their veracity accumulated. Polymorphisms in genes associated with striatal dopamine function (DARPP-32, rs907094, and DRD2, rs6277) were predictive of learning from positive and negative outcomes. Notably, these same variants were predictive of the degree to which such learning was overly inflated or neglected when outcomes are consistent or inconsistent with prior instructions. These findings indicate dissociable neurocomputational and genetic mechanisms by which initial biases are strengthened by experience.

What is the Functional Relevance of Prefrontal Cortex Entrainment to Hippocampal Theta Rhythms?

There has been considerable interest in the importance of oscillations in the brain and in how these oscillations relate to the firing of single neurons. Recently a number of studies have shown that the spiking of individual neurons in the medial prefrontal cortex (mPFC) become entrained to the hippocampal (HPC) theta rhythm. We recently showed that theta-entrained mPFC cells lost theta-entrainment specifically on error trials even though the firing rates of these cells did not change (Hyman et al., 2010). This implied that the level of HPC theta-entrainment of mPFC units was more predictive of trial outcome than differences in firing rates and that there is more information encoded by the mPFC on working memory tasks than can be accounted for by a simple rate code. Nevertheless, the functional meaning of mPFC entrainment to HPC theta remains a mystery. It is also unclear as to whether there are any differences in the nature of the information encoded by theta-entrained and non-entrained mPFC cells. In this review we discuss mPFC entrainment to HPC theta within the context of previous results as well as provide a more detailed analysis of the Hyman et al. (2010) data set. This re-analysis revealed that theta-entrained mPFC cells selectively encoded a variety of task-relevant behaviors and stimuli while never theta-entrained mPFC cells were most strongly attuned to errors or the lack of expected rewards. In fact, these error responsive neurons were responsible for the error representations exhibited by the entire ensemble of mPFC neurons. A theta reset was also detected in the post-error period. While it is becoming increasingly evident that mPFC neurons exhibit correlates to virtually all cues and behaviors, perhaps phase-locking directs attention to the task-relevant representations required to solve a spatially based working memory task while the loss of theta-entrainment at the start of error trials may represent a shift of attention away from these representations. The subsequent theta reset following error commission, when coupled with the robust responses of never theta-entrained cells, could produce a potent error-evoked signal used to alert the rat to changes in the relationship between task-relevant cues and reward expectations.

Single-trial regression elucidates the role of prefrontal theta oscillations in response conflict

In most cognitive neuroscience experiments there are many behavioral and experimental dynamics, and many indices of brain activity, that vary from trial to trial. For example, in studies of response conflict, conflict is usually treated as a binary variable (i.e., response conflict exists or does not in any given trial), whereas some evidence and intuition suggests that conflict may vary in intensity from trial to trial. Here we demonstrate that single-trial multiple regression of time–frequency electrophysiological activity reveals neural mechanisms of cognitive control that are not apparent in cross-trial averages. We also introduce a novel extension to oscillation phase coherence and synchronization analyses, based on “weighted” phase modulation, that has advantages over standard coherence measures in terms of linking electrophysiological dynamics to trial-varying behavior and experimental variables. After replicating previous response conflict findings using trial-averaged data, we extend these findings using single-trial analytic methods to provide novel evidence for the role of medial frontal–lateral prefrontal theta-band synchronization in conflict-induced response time dynamics, including a role for lateral prefrontal theta-band activity in biasing response times according to perceptual conflict. Given that these methods shed new light on the prefrontal mechanisms of response conflict, they are also likely to be useful for investigating other neurocognitive processes.

The effect of pre- vs. post-reward attainment on EEG asymmetry in melancholic depression

Clinical investigators have long theorized about the role of reward processing and positive affect in depression. One theory posits that compared to nonmelancholic depressives, melancholic depressives experience less consummatory (i.e., post-reward), but comparably low anticipatory (prior to reward), positive affect. We tested whether frontal EEG asymmetry, a putative marker of the anticipatory reward system, is present only before an individual receives a reward or also after receiving a reward (i.e., during consummatory reward processing). We also examined whether melancholic depression, a condition characterized by a deficit in consummatory reward processing, is associated with abnormal EEG asymmetries in alpha band power. Effects in other frequency bands (delta, theta, or beta) were also explored. EEG was recorded in 34 controls, 48 nonmelancholic depressives, and 17 melancholic depressives during a slot machine task designed to elicit anticipatory and consummatory reward processing. Results indicated that, for alpha, the frontal EEG asymmetry of greater relative left activity was specific to anticipatory reward processing. During the consummatory phase, individuals with melancholic depression exhibited different posterior EEG asymmetries than individuals with nonmelancholic depression (and controls at a trend level). This second finding was largely due to melancholics exhibiting relatively lower right posterior activity and nonmelancholics exhibiting relatively lower left activity. These results suggest that a posterior asymmetry may be a marker for melancholic depression and aberrant consummatory reward processing.

Emotion blocks the path to learning under stereotype threat

Gender-based stereotypes undermine females' performance on challenging math tests, but how do they influence their ability to learn from the errors they make? Females under stereotype threat or non-threat were presented with accuracy feedback after each problem on a GRE-like math test, followed by an optional interactive tutorial that provided step-wise problem-solving instruction. Event-related potentials tracked the initial detection of the negative feedback following errors [feedback related negativity (FRN), P3a], as well as any subsequent sustained attention/arousal to that information [late positive potential (LPP)]. Learning was defined as success in applying tutorial information to correction of initial test errors on a surprise retest 24-h later. Under non-threat conditions, emotional responses to negative feedback did not curtail exploration of the tutor, and the amount of tutor exploration predicted learning success. In the stereotype threat condition, however, greater initial salience of the failure (FRN) predicted less exploration of the tutor, and sustained attention to the negative feedback (LPP) predicted poor learning from what was explored. Thus, under stereotype threat, emotional responses to negative feedback predicted both disengagement from learning and interference with learning attempts. We discuss the importance of emotion regulation in successful rebound from failure for stigmatized groups in stereotype-salient environments.

It's about Time

The purpose of this review/opinion paper is to argue that human cognitive neuroscience has focused too little attention on how the brain may use time and time-based coding schemes to represent, process, and transfer information within and across brain regions. Instead, the majority of cognitive neuroscience studies rest on the assumption of functional localization. Although the functional localization approach has brought us a long way toward a basic characterization of brain functional organization, there are methodological and theoretical limitations of this approach. Further advances in our understanding of neurocognitive function may come from examining how the brain performs computations and forms transient functional neural networks using the rich multi-dimensional information available in time. This approach rests on the assumption that information is coded precisely in time but distributed in space; therefore, measures of rapid neuroelectrophysiological dynamics may provide insights into brain function that cannot be revealed using localization-based approaches and assumptions. Space is not an irrelevant dimension for brain organization; rather, a more complete understanding of how brain dynamics lead to behavior dynamics must incorporate how the brain uses time-based coding and processing schemes.

Neurogenetics and pharmacology of learning, motivation, and cognition

Many of the individual differences in cognition, motivation, and learning-and the disruption of these processes in neurological conditions-are influenced by genetic factors. We provide an integrative synthesis across human and animal studies, focusing on a recent spate of evidence implicating a role for genes controlling dopaminergic function in frontostriatal circuitry, including COMT, DARPP-32, DAT1, DRD2, and DRD4. These genetic effects are interpreted within theoretical frameworks developed in the context of the broader cognitive and computational neuroscience literature, constrained by data from pharmacological, neuroimaging, electrophysiological, and patient studies. In this framework, genes modulate the efficacy of particular neural computations, and effects of genetic variation are revealed by assays designed to be maximally sensitive to these computations. We discuss the merits and caveats of this approach and outline a number of novel candidate genes of interest for future study.

Decision processes in human performance monitoring

The ability to detect and compensate for errors is crucial in producing effective, goal-directed behavior. Human error processing is reflected in two event-related brain potential components, the error-related negativity (Ne/ERN) and error positivity (Pe), but the functional significance of both components remains unclear. Our approach was to consider error detection as a decision process involving an evaluation of available evidence that an error has occurred against an internal criterion. This framework distinguishes two fundamental stages of error detection--accumulating evidence (input), and reaching a decision (output)--that should be differentially affected by changes in internal criterion. Predictions from this model were tested in a brightness discrimination task that required human participants to signal their errors, with incentives varied to encourage participants to adopt a high or low criterion for signaling their errors. Whereas the Ne/ERN was unaffected by this manipulation, the Pe varied consistently with criterion: A higher criterion was associated with larger Pe amplitude for signaled errors, suggesting that the Pe reflects the strength of accumulated evidence. Across participants, Pe amplitude was predictive of changes in behavioral criterion as estimated through signal detection theory analysis. Within participants, Pe amplitude could be estimated robustly with multivariate machine learning techniques and used to predict error signaling behavior both at the level of error signaling frequencies and at the level of individual signaling responses. These results suggest that the Pe, rather than the Ne/ERN, is closely related to error detection, and specifically reflects the accumulated evidence that an error has been committed.

Social stress reactivity alters reward and punishment learning

To examine how stress affects cognitive functioning, individual differences in trait vulnerability (punishment sensitivity) and state reactivity (negative affect) to social evaluative threat were examined during concurrent reinforcement learning. Lower trait-level punishment sensitivity predicted better reward learning and poorer punishment learning; the opposite pattern was found in more punishment sensitive individuals. Increasing state-level negative affect was directly related to punishment learning accuracy in highly punishment sensitive individuals, but these measures were inversely related in less sensitive individuals. Combined electrophysiological measurement, performance accuracy and computational estimations of learning parameters suggest that trait and state vulnerability to stress alter cortico-striatal functioning during reinforcement learning, possibly mediated via medio-frontal cortical systems.