Neural evidence reveals the rapid effects of reward history on selective attention

Reward processing in the value-driven attention network: reward signals tracking cue identity and location

Through associative reward learning, arbitrary cues acquire the ability to automatically capture visual attention. Previous studies have examined the neural correlates of value-driven attentional orienting, revealing elevated activity within a network of brain regions encompassing the visual corticostriatal loop [caudate tail, lateral occipital complex (LOC) and early visual cortex] and intraparietal sulcus (IPS). Such attentional priority signals raise a broader question concerning how visual signals are combined with reward signals during learning to create a representation that is sensitive to the confluence of the two. This study examines reward signals during the cued reward training phase commonly used to generate value-driven attentional biases. High, compared with low, reward feedback preferentially activated the value-driven attention network, in addition to regions typically implicated in reward processing. Further examination of these reward signals within the visual system revealed information about the identity of the preceding cue in the caudate tail and LOC, and information about the location of the preceding cue in IPS, while early visual cortex represented both location and identity. The results reveal teaching signals within the value-driven attention network during associative reward learning, and further suggest functional specialization within different regions of this network during the acquisition of an integrated representation of stimulus value.

Goal-Directed and Habit-Like Modulations of Stimulus Processing during Reinforcement Learning

Recent research has shown that perceptual processing of stimuli previously associated with high-value rewards is automatically prioritized even when rewards are no longer available. It has been hypothesized that such reward-related modulation of stimulus salience is conceptually similar to an "attentional habit." Recording event-related potentials in humans during a reinforcement learning task, we show strong evidence in favor of this hypothesis. Resistance to outcome devaluation (the defining feature of a habit) was shown by the stimulus-locked P1 component, reflecting activity in the extrastriate visual cortex. Analysis at longer latencies revealed a positive component (corresponding to the P3b, from 550-700 ms) sensitive to outcome devaluation. Therefore, distinct spatiotemporal patterns of brain activity were observed corresponding to habitual and goal-directed processes. These results demonstrate that reinforcement learning engages both attentional habits and goal-directed processes in parallel. Consequences for brain and computational models of reinforcement learning are discussed.SIGNIFICANCE STATEMENT The human attentional network adapts to detect stimuli that predict important rewards. A recent hypothesis suggests that the visual cortex automatically prioritizes reward-related stimuli, driven by cached representations of reward value; that is, stimulus-response habits. Alternatively, the neural system may track the current value of the predicted outcome. Our results demonstrate for the first time that visual cortex activity is increased for reward-related stimuli even when the rewarding event is temporarily devalued. In contrast, longer-latency brain activity was specifically sensitive to transient changes in reward value. Therefore, we show that both habit-like attention and goal-directed processes occur in the same learning episode at different latencies. This result has important consequences for computational models of reinforcement learning.

Counterintuitive effects of negative social feedback on attention

Which stimuli we pay attention to is strongly influenced by learning. Stimuli previously associated with reward outcomes, such as money and food, and stimuli previously associated with aversive outcomes, such as monetary loss and electric shock, automatically capture attention. Social reward (happy expressions) can bias attention towards associated stimuli, but the role of negative social feedback in biasing attentional selection remains unexplored. On the one hand, negative social feedback often serves to discourage particular behaviours. If attentional selection can be curbed much like any other behavioural preference, we might expect stimuli associated with negative social feedback to be more readily ignored. On the other hand, if negative social feedback influences attention in the same way that other aversive outcomes do, such feedback might ironically bias attention towards the stimuli it is intended to discourage selection of. In the present study, participants first completed a training phase in which colour targets were associated with negative social feedback. Then, in a subsequent test phase, these same colour stimuli served as task-irrelevant distractors during a visual search task. The results strongly support the latter interpretation in that stimuli previously associated with negative social feedback impaired search performance.

The attention habit: how reward learning shapes attentional selection

There is growing consensus that reward plays an important role in the control of attention. Until recently, reward was thought to influence attention indirectly by modulating task-specific motivation and its effects on voluntary control over selection. Such an account was consistent with the goal-directed (endogenous) versus stimulus-driven (exogenous) framework that had long dominated the field of attention research. Now, a different perspective is emerging. Demonstrations that previously reward-associated stimuli can automatically capture attention even when physically inconspicuous and task-irrelevant challenge previously held assumptions about attentional control. The idea that attentional selection can be value driven, reflecting a distinct and previously unrecognized control mechanism, has gained traction. Since these early demonstrations, the influence of reward learning on attention has rapidly become an area of intense investigation, sparking many new insights. The result is an emerging picture of how the reward system of the brain automatically biases information processing. Here, I review the progress that has been made in this area, synthesizing a wealth of recent evidence to provide an integrated, up-to-date account of value-driven attention and some of its broader implications.

Reward breaks through center-surround inhibition via anterior insula

Focusing attention on a target creates a center-surround inhibition such that distractors located close to the target do not capture attention. Recent research showed that a distractor can break through this surround inhibition when associated with reward. However, the brain basis for this reward-based attention is unclear. In this fMRI study, we presented a distractor associated with high or low reward at different distances from the target. Behaviorally the low-reward distractor did not capture attention and thus did not cause interference, whereas the high-reward distractor captured attention only when located near the target. Neural activity in extrastriate cortex mirrored the behavioral pattern. A comparison between the high-reward and the low-reward distractors presented near the target (i.e., reward-based attention) and a comparison between the high-reward distractors located near and far from the target (i.e., spatial attention) revealed a common frontoparietal network, including inferior frontal gyrus and inferior parietal sulcus as well as the visual cortex. Reward-based attention specifically activated the anterior insula (AI). Dynamic causal modelling showed that reward modulated the connectivity from AI to the frontoparietal network but not the connectivity from the frontoparietal network to the visual cortex. Across participants, the reward-based attentional effect could be predicted both by the activity in AI and by the changes of spontaneous functional connectivity between AI and ventral striatum before and after reward association. These results suggest that AI encodes reward-based salience and projects it to the stimulus-driven attentional network, which enables the reward-associated distractor to break through the surround inhibition in the visual cortex.

Value-based attentional capture influences context-dependent decision-making

Normative theories posit that value-based decision-making is context independent. However, decisions between two high-value options can be suboptimally biased by the introduction of a third low-value option. This context-dependent modulation is consistent with the divisive normalization of the value of each stimulus by the total value of all stimuli. In addition, an independent line of research demonstrates that pairing a stimulus with a high-value outcome can lead to attentional capture that can mediate the efficiency of visual information processing. Here we tested the hypothesis that value-based attentional capture interacts with value-based normalization to influence the optimality of decision-making. We used a binary-choice paradigm in which observers selected between two targets and the color of each target indicated the magnitude of their reward potential. Observers also had to simultaneously ignore a task-irrelevant distractor rendered in a color that was previously associated with a specific reward magnitude. When the color of the task-irrelevant distractor was previously associated with a high reward, observers responded more slowly and less optimally. Moreover, as the learned value of the distractor increased, electrophysiological data revealed an attenuation of the lateralized N1 and N2Pc responses evoked by the relevant choice stimuli and an attenuation of the late positive deflection (LPD). Collectively, these behavioral and electrophysiological data suggest that value-based attentional capture and value-based normalization jointly mediate the influence of context on free-choice decision-making.