Goal-directed EEG activity evoked by discriminative stimuli in reinforcement learning

Oxytocin modulates neural activity during early perceptual salience attribution

Leading hypotheses of oxytocin's (OT) role in human cognition posit that it enhances salience attribution. However, whether OT exerts its effects predominantly in social (vs non-social) contexts remains debatable, and the time-course of intranasal OT's effects' on salience attribution processing is still unknown. We used the social Salience Attribution Task modified (sSAT) in a double-blind, placebo-controlled intranasal OT (inOT) administration, between-subjects design, with 54 male participants, to test existing theories of OT's role in cognition. Namely, we aimed to test whether inOT would differently affect salience attribution processing of social stimuli (expressing fearfulness) and non-social stimuli (fruits) made relevant via monetary reinforcement, and its neural processing time-course. During electroencephalography (EEG) recording, participants made speeded responses to emotional social (fearful faces) and non-emotional non-social (fruits) stimuli - which were matched for task-relevant motivational salience through their (color-dependent) probability of monetary reinforcement. InOT affected early (rather than late, P3b and LPP) EEG components, increasing N170 amplitude (p = .041) and P2b latency (p .001; albeit not of P1), regardless of stimuli's (emotional) socialness or reinforcement probability. Fear-related socialness affected salience attribution processing EEG (p .05) across time (N170, P2b and P3b), being later modulated by reinforcement probability (LPP). Our data suggest that OT's effects on neural activity during early perception, may exist irrespective of fear-related social- or reward-contexts. This partially supports the tri-phasic model of OT (which posits OT enhances salience attribution in an early perception stage regardless of socialness), and not the social salience nor the general approach-withdrawal hypotheses of OT, for early salience processing event-related potentials.

Differential effects of intra-modal and cross-modal reward value on perception: ERP evidence

In natural environments objects comprise multiple features from the same or different sensory modalities but it is not known how perception of an object is affected by the value associations of its constituent parts. The present study compares intra- and cross-modal value-driven effects on behavioral and electrophysiological correlates of perception. Human participants first learned the reward associations of visual and auditory cues. Subsequently, they performed a visual discrimination task in the presence of previously rewarded, task-irrelevant visual or auditory cues (intra- and cross-modal cues, respectively). During the conditioning phase, when reward associations were learned and reward cues were the target of the task, high value stimuli of both modalities enhanced the electrophysiological correlates of sensory processing in posterior electrodes. During the post-conditioning phase, when reward delivery was halted and previously rewarded stimuli were task-irrelevant, cross-modal value significantly enhanced the behavioral measures of visual sensitivity, whereas intra-modal value produced only an insignificant decrement. Analysis of the simultaneously recorded event-related potentials (ERPs) of posterior electrodes revealed similar findings. We found an early (90-120 ms) suppression of ERPs evoked by high-value, intra-modal stimuli. Cross-modal stimuli led to a later value-driven modulation, with an enhancement of response positivity for high- compared to low-value stimuli starting at the N1 window (180-250 ms) and extending to the P3 (300-600 ms) responses. These results indicate that sensory processing of a compound stimulus comprising a visual target and task-irrelevant visual or auditory cues is modulated by the reward value of both sensory modalities, but such modulations rely on distinct underlying mechanisms.

Stimulus-response learning and expected reward value enhance stimulus cognitive processing: An ERP study

Reward affects our attention to stimuli, prioritizing those that lead to high-value outcomes. Recently, it has been suggested that such reward-related cognitive prioritization might be associated with the process of learning new stimulus-response (S-R) associations, because both are acquired through extended reward training, and once established, they are hard to overcome. We used event-related potentials (ERP) to analyze the contribution of S-R links to the formation of reward-related cognitive prioritization during reinforcement learning. Reward-related cognitive prioritization was measured by comparing the ERP signals for stimuli predicting high-value and low-value outcomes. In addition, we compared a strong S-R link (same stimulus, same response), with a weak S-R link condition (same stimulus, two different responses). The participants' performance was more accurate and faster when the procedure allowed for establishing strong S-R links and for high-value outcomes. Furthermore, those stimuli associated with strong S-R links showed a larger P3 amplitude at parietal sites. Value effects (larger ERP activity for those stimuli predicting a high-value outcome) were obtained at parietal and occipital sites in the P3 time window. However, value effects did not benefit from strong S-R links in either the P1 or the P3 components. These results suggest that strong S-R learning is not necessary to develop reward-related modulations of ERP activity.

Learned value and predictiveness affect gaze but not figure assignment

Many factors affect figure-ground segregation, but the contributions of attention and reward history to this process is uncertain. We conducted two experiments to investigate whether reward learning influences figure assignment and whether this relationship was mediated by attention. Participants learned to associate certain shapes with a reward contingency: During a learning phase, they chose between two shapes on each trial, with subsets of shapes associated with high-probability win, low-probability win, high-probability loss, and low-probability loss. In a test phase, participants were given a figure-ground task, in which they indicated which of two regions that shared a contour they perceived as the figure (high-probability win and low-probability win shapes were pitted against each other, as were high-probability loss and low-probability loss shapes). The results revealed that participants had learned the reward contingencies and that, following learning, attention was reliably drawn to the optimal stimulus. Despite this, neither reward history nor the resulting attentional allocation influenced figure-ground organization.

Social comparison impacts stimulus evaluation in a competitive social learning task

When we perform an action, the outcome that follows it can change the value we place on that behaviour, making it more or less likely to be repeated in the future. However, the values that we learn are not objective: we interpret the outcomes that we receive for ourselves relative to those that share our environment, i.e. we engage in social comparison. The temporal dynamics of physiological responses to stimulus valuation in social learning tasks are poorly understood, particularly in human participants. Therefore, we recorded stimulus-locked event-related potentials with 64-channel EEG to examine stimulus valuation, following the design of a study previously used in macaques. Pairs of participants performed a social learning task in which they received outcomes sequentially for a presented stimulus (partner first) by pressing a button in response to a cue. There were two conditions: one in which stimulus values varied for the participant but output a constant rate of reward for the partner (self-variable blocks), and another condition in which this payout was reversed (other-variable blocks). We then measured participants' self-reported competitiveness. Approximately 200 ms post-stimulus, an ERP related to stimulus evaluation and attentional processing appeared to encode own stimulus value in self-variable blocks. In other-variable blocks the same pattern of activity was reversed, even though the value of the stimulus for the participant did not depend on the stimulus presented. Outcome-locked analyses further showed that attention dedicated to the partner's outcome was greater in more competitive participants. We conclude that subjective stimulus value can be reflected in early stimulus-locked ERP responses and that competitive participants may be more invested in their own performance relative to the other player, hence their increased interest in the outcome of their partner.

A tradeoff relationship between internal monitoring and external feedback during the dynamic process of reinforcement learning

Effective behavior monitoring, including internal monitoring/error detection and external monitoring/feedback, is very pivotal for reinforcement learning. However, less attention has been paid to internal monitoring and the dynamic learning performance in reinforcement learning, and there is still a heated debate on which kind of external feedback is relied on in the reinforcement learning. In order to address these questions, an adaption probabilistic selection task was used to examine the effect of the internal monitoring, external feedback and the relationship between them for approach learners and avoidance learners during dynamic learning process of reinforcement learning and behavior adaption. Error-related negativity (ERN), feedback-related negativity (FRN) and feedback-related P300 are three ERPs components, which can be used as the indexes of internal monitoring, external feedback and behavior adaption. For our results, the ERN effect of avoidance learners become large in block 3, which is earlier than approach learners (block 4). This phenomenon suggests that avoidance learners learned faster than approach learners. In addition, the FRN amplitude of avoidance learners in block 4 was significantly smaller than the other three blocks. The aforementioned results demonstrated a tradeoff relationship between the ERN and FRN effects.

Learned predictiveness acquired through experience prevails over the influence of conflicting verbal instructions in rapid selective attention

Previous studies have provided evidence that selective attention tends to prioritize the processing of stimuli that are good predictors of upcoming events over nonpredictive stimuli. Moreover, studies using eye-tracking to measure attention demonstrate that this attentional bias towards predictive stimuli is at least partially under voluntary control and can be flexibly adapted via instruction. Our experiment took a similar approach to these prior studies, manipulating participants’ experience of the predictiveness of different stimuli over the course of trial-by-trial training; we then provided explicit verbal instructions regarding stimulus predictiveness that were designed to be either consistent or inconsistent with the previously established learned predictiveness. Critically, we measured the effects of training and instruction on attention to stimuli using a dot probe task, which allowed us to assess rapid shifts of attention (unlike the eye-gaze measures used in previous studies). Results revealed a rapid attentional bias towards stimuli experienced as predictive (versus those experienced as nonpredictive), that was completely unaffected by verbal instructions. This was not due to participants’ failure to recall or use instructions appropriately, as revealed by analyses of their learning about stimuli, and their memory for instructions. Overall, these findings suggest that rapid attentional biases such as those measured by the dot probe task are more strongly influenced by our prior experience during training than by our current explicit knowledge acquired via instruction.

Oculomotor capture is influenced by expected reward value but (maybe) not predictiveness

A large body of research has shown that learning about relationships between neutral stimuli and events of significance – rewards or punishments – influences the extent to which people attend to those stimuli in the future. However, different accounts of this influence differ in terms of the critical variable that is proposed to determine learned changes in attention. We describe two experiments using eye-tracking with a rewarded visual search procedure to investigate whether attentional capture is influenced by the predictiveness of stimuli (i.e., the extent to which they provide information about upcoming events) or by their absolute associative value (i.e., the expected incentive value of the outcome that a stimulus predicts). Results demonstrated a clear influence of associative value on the likelihood that stimuli will capture eye-movements, but the evidence for a distinct influence of predictiveness was less compelling. The results of these experiments can be reconciled within a simple account under which attentional prioritization is a monotonic function of the expected, subjective value of the reward that is signalled by a stimulus.

The blocking effect in associative learning involves learned biases in rapid attentional capture

Blocking refers to the finding that less is learned about the relationship between a stimulus and an outcome if pairings are conducted in the presence of a second stimulus that has previously been established as a reliable predictor of that outcome. Attentional models of associative learning suggest that blocking reflects a reduction in the attention paid to the blocked cue. We tested this idea in three experiments in which participants were trained in an associative learning task using a blocking procedure. Attention to stimuli was measured 250 ms after onset using an adapted version of the dot probe task. This task was presented at the beginning of each learning trial (Experiments 1 and 2) or in independent trials (Experiment 3). Results show evidence of reduced attention to blocked stimuli (i.e. “attentional blocking”). In addition, this attentional bias correlated with the magnitude of blocking in associative learning, as measured by predictive-value judgments. Moreover, Experiments 2 and 3 found evidence of an influence of learning about predictiveness on memory for episodes involving stimuli. These findings are consistent with a central role of learned attentional biases in producing the blocking effect, and in the encoding of new memories.

Goal-Directed and Habit-Like Modulations of Stimulus Processing during Reinforcement Learning

Recent research has shown that perceptual processing of stimuli previously associated with high-value rewards is automatically prioritized even when rewards are no longer available. It has been hypothesized that such reward-related modulation of stimulus salience is conceptually similar to an "attentional habit." Recording event-related potentials in humans during a reinforcement learning task, we show strong evidence in favor of this hypothesis. Resistance to outcome devaluation (the defining feature of a habit) was shown by the stimulus-locked P1 component, reflecting activity in the extrastriate visual cortex. Analysis at longer latencies revealed a positive component (corresponding to the P3b, from 550-700 ms) sensitive to outcome devaluation. Therefore, distinct spatiotemporal patterns of brain activity were observed corresponding to habitual and goal-directed processes. These results demonstrate that reinforcement learning engages both attentional habits and goal-directed processes in parallel. Consequences for brain and computational models of reinforcement learning are discussed.SIGNIFICANCE STATEMENT The human attentional network adapts to detect stimuli that predict important rewards. A recent hypothesis suggests that the visual cortex automatically prioritizes reward-related stimuli, driven by cached representations of reward value; that is, stimulus-response habits. Alternatively, the neural system may track the current value of the predicted outcome. Our results demonstrate for the first time that visual cortex activity is increased for reward-related stimuli even when the rewarding event is temporarily devalued. In contrast, longer-latency brain activity was specifically sensitive to transient changes in reward value. Therefore, we show that both habit-like attention and goal-directed processes occur in the same learning episode at different latencies. This result has important consequences for computational models of reinforcement learning.

Testing the controllability of contextual cuing of visual search

Locating a target among distractors improves when the configuration of distractors consistently cues the target's location across search trials, an effect called contextual cuing of visual search (CC). The important issue of whether CC is automatic has previously been studied by asking whether it can occur implicitly (outside awareness). Here we ask the novel question: is CC of visual search controllable? In 3 experiments participants were exposed to a standard CC procedure during Phase 1. In Phase 2, they localized a new target, embedded in configurations (including the previous target) repeated from Phase 1. Despite robust contextual cuing, congruency effects - which would imply the orientation of attention towards the old target in repeated configurations - were found in none of the experiments. The results suggest that top-down control can be exerted over contextually-guided visual search.

Ambiguity produces attention shifts in category learning

It has been suggested that people and nonhuman animals protect their knowledge from interference by shifting attention toward the context when presented with information that contradicts their previous beliefs. Despite that suggestion, no studies have directly measured changes in attention while participants are exposed to an interference treatment. In the present experiments, we adapted a dot-probe task to track participants’ attention to cues and contexts while they were completing a simple category learning task. The results support the hypothesis that interference produces a change in the allocation of attention to cues and contexts.