Reward-based transfer from bottom-up to top-down search tasks

Understanding the mechanisms underlying cognitive control in psychosis

BACKGROUND

Cognitive control (CC) involves a top-down mechanism to flexibly respond to complex stimuli and is impaired in schizophrenia.

METHODS

This study investigated the impact of increasing complexity of CC processing in 140 subjects with psychosis and 39 healthy adults, with assessments of behavioral performance, neural regions of interest and symptom severity.

RESULTS

The lowest level of CC (Stroop task) was impaired in all patients; the intermediate level of CC (Faces task) with explicit emotional information was most impaired in patients with first episode psychosis. Patients showed activation of distinct neural CC and reward networks, but iterative learning based on the higher-order of CC during the trust game, was most impaired in chronic schizophrenia. Subjects with first episode psychosis, and patients with lower symptom load, demonstrate flexibility of the CC network to facilitate learning, which appeared compromised in the more chronic stages of schizophrenia.

CONCLUSION

These data suggest optimal windows for opportunities to introduce therapeutic interventions to improve CC.

Reward delays quitting in visual search

Reward motivates goal-directed behaviors, leading to faster reaction time (RT) and lower error rate in searching for a target in the reward condition than in the no-reward condition in target-discrimination tasks. However, it is unclear how reward influences target detection in which participants are required to judge whether a predesignated target is present or absent. Here, we asked participants to complete a target-detection search task in which the color of the search array indicated the reward availability of the current trial. Correct and faster (than a baseline) responses would be rewarded if the search array had the reward-related color. In Experiments 1A and 1B, the target was presented in 50% of the trials. Experiment 1B had the same design as Experiment 1A, except that different baselines were set for the target-present and target-absent conditions. In Experiment 2, the proportion of target presence was manipulated to be high (80%), moderate (50%), or low (20%) in different blocks of stimuli. Results showed that, across all the experiments, participants responded faster and made fewer errors in the reward than in the no-reward condition when the target was present. However, this facilitatory effect was reversed when the target was absent, showcasing a reward-induced interference. The signal detection analysis suggested that reward biased the report criterion to the "yes" response. These findings demonstrate that the impact of reward on goal-directed behavior can be detrimental and reward prolongs the search process by rendering participants reluctant to say "no" in visual search termination.

Neural correlates of value-driven spatial orienting

Reward learning has been shown to habitually guide overt spatial attention to specific regions of a scene. However, the neural mechanisms that support this bias are unknown. In the present study, participants learned to orient themselves to a particular quadrant of a scene (a high-value quadrant) to maximize monetary gains. This learning was scene-specific, with the high-value quadrant varying across different scenes. During a subsequent test phase, participants were faster at identifying a target if it appeared in the high-value quadrant (valid), and initial saccades were more likely to be made to the high-value quadrant. fMRI analyses during the test phase revealed learning-dependent priority signals in the caudate tail, superior colliculus, frontal eye field, anterior cingulate cortex, and insula, paralleling findings concerning feature-based, value-driven attention. In addition, ventral regions typically associated with scene selection and spatial information processing, including the hippocampus, parahippocampal gyrus, and temporo-occipital cortex, were also implicated. Taken together, our findings offer new insights into the neural architecture subserving value-driven attention, both extending our understanding of nodes in the attention network previously implicated in feature-based, value-driven attention and identifying a ventral network of brain regions implicated in reward's influence on scene-dependent spatial orienting.

Reducing anxiety and attentional bias with reward association learning and attentional bias modification

The current study examined the effects of a reward associative learning procedure and the traditional threat-avoidance ABM paradigm on anxiety and attentional bias. In reward training, participants were given high rewards for correct responses to neutral target and low rewards for correct responses to negative target. In reward control training, participants received no cues of rewards after their responses. High trait anxious individuals (N = 76) first completed a session of reward training or reward control training, followed by four sessions of ABM training or ABM control training. Generalized anxiety disorder symptom (GAD-7) and attentional bias in a dot-probe task were assessed during pre-and post-training. Results indicated that the effect of ABM training on reducing anxiety was only obtained in the reward training condition. Participants who received reward training showed significantly less attentional bias compared with those receiving reward control training. There was no significant training effect of ABM on atttentiona bias. Results suggested that reward training reduced general anxiety and attentional bias. Traditional ABM training reduced anxiety only when combined with reward training. Attentional bias in anxiety are modifiable through reward training.

Aesthetic appeal influences visual search performance

Aesthetic appeal of a visual image can influence performance in time-critical tasks, even if it is irrelevant to the task. This series of experiments examined whether aesthetic appeal can act as an object attribute that guides visual search. If appeal enhances the salience of the targets pre-attentively, then appealing icons would lead to more efficient searches than unappealing targets and, conversely, appeal of distractors would reduce search efficiency. Three experiments (N = 112) examined how aesthetic appeal influences performance in a classic visual search task. In each experiment, participants completed 320 visual search trials, with icons varying in rated aesthetic appeal and either visual complexity (Experiments 1 and 2) of concreteness (Experiment 3) among two, four, eight, or 11 distractor icons. While target appeal did not influence search efficiency it sped up search times in all three experiments: appealing targets led to faster response time (RT) than unappealing targets across all experiments, and compared to neutral distractors, appealing distractors slowed search RT down. These findings are the first to show that an object's aesthetic appeal influences visual search performance.

Visual Selective Attention P300 Source in Frontal-Parietal Lobe: ERP and fMRI Study

Visual selective attention can be achieved into bottom-up and top-down attention. Different selective attention tasks involve different attention control ways. The pop-out task requires more bottom-up attention, whereas the search task involves more top-down attention. P300, which is the positive potential generated by the brain in the latency of 300 ~ 600 ms after stimulus, reflects the processing of attention. There is no consensus on the P300 source. The aim of present study is to study the source of P300 elicited by different visual selective attention. We collected thirteen participants' P300 elicited by pop-out and search tasks with event-related potentials (ERP). We collected twenty-six participants' activation brain regions in pop-out and search tasks with functional magnetic resonance imaging (fMRI). And we analyzed the sources of P300 using the ERP and fMRI integration with high temporal resolution and high spatial resolution. ERP results indicated that the pop-out task induced larger P300 than the search task. P300 induced by the two tasks distributed at frontal and parietal lobes, with P300 induced by the pop-out task mainly at the parietal lobe and that induced by the search task mainly at the frontal lobe. Further ERP and fMRI integration analysis showed that neural difference sources of P300 were the right precentral gyrus, left superior frontal gyrus (medial orbital), left middle temporal gyrus, left rolandic operculum, right postcentral gyrus, and left angular gyrus. Our study suggests that the frontal and parietal lobes contribute to the P300 component of visual selective attention.

Task Difficulty Regulates How Conscious and Unconscious Monetary Rewards Boost the Performance of Working Memory: An Event-Related Potential Study

The performance of working memory can be improved by the corresponding high-value vs. low-value rewards consciously or unconsciously. However, whether conscious and unconscious monetary rewards boosting the performance of working memory is regulated by the difficulty level of working memory task is unknown. In this study, a novel paradigm that consists of a reward-priming procedure and N-back task with differing levels of difficulty was designed to inspect this complex process. In particular, both high-value and low-value coins were presented consciously or unconsciously as the reward cues, followed by the N-back task, during which electroencephalogram signals were recorded. It was discovered that the high-value reward elicited larger event-related potential (ERP) component P3 along the parietal area (reflecting the working memory load) as compared to the low-value reward for the less difficult 1-back task, no matter whether the reward was unconsciously or consciously presented. In contrast, this is not the case for the more difficult 2-back task, in which the difference in P3 amplitude between the high-value and low-value rewards was not significant for the unconscious reward case, yet manifested significance for the conscious reward processing. Interestingly, the results of the behavioral analysis also exhibited very similar patterns as ERP patterns. Therefore, this study demonstrated that the difficulty level of a task can modulate the influence of unconscious reward on the performance of working memory.

The past, present, and future of selection history

The last ten years of attention research have witnessed a revolution, replacing a theoretical dichotomy (top-down vs. bottom-up control) with a trichotomy (biased by current goals, physical salience, and selection history). This third new mechanism of attentional control, selection history, is multifaceted. Some aspects of selection history must be learned over time whereas others reflect much more transient influences. A variety of different learning experiences can shape the attention system, including reward, aversive outcomes, past experience searching for a target, target‒non-target relations, and more. In this review, we provide an overview of the historical forces that led to the proposal of selection history as a distinct mechanism of attentional control. We then propose a formal definition of selection history, with concrete criteria, and identify different components of experience-driven attention that fit within this definition. The bulk of the review is devoted to exploring how these different components relate to one another. We conclude by proposing an integrative account of selection history centered on underlying themes that emerge from our review.

Guided Search 6.0: An updated model of visual search

This paper describes Guided Search 6.0 (GS6), a revised model of visual search. When we encounter a scene, we can see something everywhere. However, we cannot recognize more than a few items at a time. Attention is used to select items so that their features can be "bound" into recognizable objects. Attention is "guided" so that items can be processed in an intelligent order. In GS6, this guidance comes from five sources of preattentive information: (1) top-down and (2) bottom-up feature guidance, (3) prior history (e.g., priming), (4) reward, and (5) scene syntax and semantics. These sources are combined into a spatial "priority map," a dynamic attentional landscape that evolves over the course of search. Selective attention is guided to the most active location in the priority map approximately 20 times per second. Guidance will not be uniform across the visual field. It will favor items near the point of fixation. Three types of functional visual field (FVFs) describe the nature of these foveal biases. There is a resolution FVF, an FVF governing exploratory eye movements, and an FVF governing covert deployments of attention. To be identified as targets or rejected as distractors, items must be compared to target templates held in memory. The binding and recognition of an attended object is modeled as a diffusion process taking > 150 ms/item. Since selection occurs more frequently than that, it follows that multiple items are undergoing recognition at the same time, though asynchronously, making GS6 a hybrid of serial and parallel processes. In GS6, if a target is not found, search terminates when an accumulating quitting signal reaches a threshold. Setting of that threshold is adaptive, allowing feedback about performance to shape subsequent searches. Simulation shows that the combination of asynchronous diffusion and a quitting signal can produce the basic patterns of response time and error data from a range of search experiments.

Awareness is necessary for attentional biases by location-reward association

Many studies have reported attentional biases based on feature-reward associations. However, the effects of location-reward associations on attentional selection remain less well-understood. Unlike feature cases, a previous study that induced participants' awareness of the location-reward association by instructing them to look for a high-reward location has suggested the critical role of goal-driven manipulations in such associations. In this study, we investigated whether the reward effect occurred without goal-driven manipulations if participants were spontaneously aware of the location-reward association. We conducted three experiments using a visual search task that included four circles where participants received rewards; one possible target location was associated with a high reward, and another with a low reward. In Experiment 1, the target was presented among distractors, and participants had to search for the target. The results showed a faster reaction time in the high-reward rather than the low-reward locations only in participants aware of the location-reward association, even if they were not required to look for the association. Moreover, in Experiment 2, we replicated the main findings of Experiment 1, even when the target had an abrupt visual onset to restrict goal-driven manipulations. Furthermore, Experiment 3 confirmed that the effect observed in Experiment 2 could not be attributed to the initial eye position. These findings suggest that goal-driven manipulations are unnecessary for inducing reward biases to high-reward locations. We concluded that awareness of the association rather than goal-driven manipulations is crucial for the location-reward effect.

Beyond Looking for the Rewarded Target: The Effects of Reward on Attention in Search Tasks

One puzzling result in training-test paradigms is that effects of reward-associated stimuli on attention are often seen in test but not in training. We focus on one study, where reward-related performance benefits occur in the training and which was discussed contentiously. By using a similar design, we conceptually replicated the results. Moreover, we investigated the underlying mechanisms and processes resulting in these reward-related performance benefits. In two experiments, using search tasks and having participants perform the tasks either with or without individually adjusted time pressure, we disentangled the mechanisms and processes contributing to the reward-related benefits. We found evidence that not only search efficiency is increased with increasing reward, but also that non-search factors contribute to the results. By also investigating response time distributions, we were able to show that reward-related performance effects increased as search time increased in demanding tasks but not in less demanding tasks. Theoretical implications of the results regarding how reward influences attentional processing are discussed.

Uncertainty modulates value-driven attentional capture

The majority of previous studies on the value modulation of attention have shown that the magnitude of value-driven attentional bias correlates with the strength of reward association. However, relatively little is known about how uncertainty affects value-based attentional bias. We investigated whether attentional capture by previously rewarded stimuli is modulated by the uncertainty of the learned value without the influence of the strength of reward association. Participants were instructed to identify the line orientation in the target color circle. Importantly, each target color was associated with a different level of uncertainty by tuning the variation in reward delivery (Experiment 1) or reward magnitude (Experiment 2). Attentional interference for uncertainty-related distractors was greater than that for certainty distractors in Experiments 1 and 2. In addition, uncertainty-induced attentional bias disappeared earlier than attentional bias for certainty. The study demonstrated that uncertainty modulates value-based attentional capture in terms of strength and persistence, even when the effect of expected value remains constant.

Learned value and predictiveness affect gaze but not figure assignment

Many factors affect figure-ground segregation, but the contributions of attention and reward history to this process is uncertain. We conducted two experiments to investigate whether reward learning influences figure assignment and whether this relationship was mediated by attention. Participants learned to associate certain shapes with a reward contingency: During a learning phase, they chose between two shapes on each trial, with subsets of shapes associated with high-probability win, low-probability win, high-probability loss, and low-probability loss. In a test phase, participants were given a figure-ground task, in which they indicated which of two regions that shared a contour they perceived as the figure (high-probability win and low-probability win shapes were pitted against each other, as were high-probability loss and low-probability loss shapes). The results revealed that participants had learned the reward contingencies and that, following learning, attention was reliably drawn to the optimal stimulus. Despite this, neither reward history nor the resulting attentional allocation influenced figure-ground organization.

Impact of relative and absolute values on selective attention

Valuable stimuli receive attentional priority. However, it is unknown whether the mechanism of the attentional priority is based on relative (e.g., higher) or absolute (e.g., 45 points) values. Therefore, we manipulated the relative and absolute values independently in a modified value-driven attentional capture paradigm. In the training phase, where associative learning occurs between color and reward value, two test target colors were each presented with another different target color (reference target colors) in separate context blocks. Therefore, each test target color had different reference points. In the test phase, the two test target colors were used as singleton distractor colors. In the training phase of Experiment 1, the absolute reward value of the test target colors was the same, but one had a higher value than its reference target color and the other had a lower value. In the test phase, the high relative value color distractor captured attention more, suggesting that the relative value of stimuli influenced selective attention. In Experiment 2 the relative value of the test target colors was the same, but the absolute value was higher for one. The high and low absolute value color distractors captured attention equally in the test phase, indicating little impact of the absolute value on selective attention. These findings suggest that the relative value, rather than absolute value, plays a critical role in the allocation of attention. Accordingly, the present study suggests that prospect theory (Kahneman & Tversky, Econometrica, 47 (2), 363-391, 1979) can be extended to earlier cognitive stages such as selective attention.

Attention for future reward

When stimuli are consistently paired with reward, attention toward these stimuli becomes biased (e.g., Abrahamse, Braem, Notebaert & Verguts, et al., Psychological Bulletin 142:693-728, 2016, https://doi.org/10.1037/bul0000047). An important premise is that participants need to repeatedly experience stimulus-reward pairings to obtain these effects (e.g., Awh, Belopolsky & Theeuwes, Trends in Cognitive Sciences 16:437-443, 2012, https://doi.org/10.1016/j.tics.2012.06.010). This idea is based on associative learning theories (e.g., Pearce & Bouton, Annual Review of Psychology 52:111-139, 2001) that suggest that exposure to stimulus-reward pairings leads to the formation of stimulus-reward associations, and a transfer of salience of the reward to the neutral stimulus. However, novel learning theories (e.g., De Houwer, Learning and Motivation 53:7-23, 2009, https://doi.org/10.1016/j.lmot.2015.11.001) suggest such effects are not necessarily the result of associative learning, but can be caused by complex knowledge and expectancies as well. In the current experiment, we first instructed participants that a correct response to one centrally presented stimulus would be followed by a high reward, whereas a correct response to another centrally presented stimulus would be paired with a low reward. Before participants executed this task, they performed a visual probe task in which these stimuli were presented as distractors. We found that attention was drawn automatically toward high-reward stimuli relative to low-reward stimuli. This implies that complex inferences and expectancies can cause automatic attentional bias, challenging associative learning models of attentional control (Abrahamse et al., 2016; Awh et al., 2012).

Reward differentially interacts with physical salience in feature-based attention

A visual feature associated with reward can capture attention when it is neither physically salient nor task relevant. Although such findings suggest that reward acts similarly as physical salience, it is unknown whether reward works independently or interactively with physical salience to modulate attentional priority. Here, we first trained participants to associate two motion directions with high and low reward. During the test, we presented superimposed but perceptually separable stimuli that consisted of coherently and randomly moving dot fields, while manipulating the physical salience (low vs. high contrast) and reward history (low vs. high reward) of the coherent stimulus. Participants were instructed to identify speed-up targets on the coherent or random stimulus. We found that reward improved target detection in the coherent stimulus regardless of the physical contrast, whereas reward disrupted target detection in the random stimulus only when the coherent stimulus was of high contrast. Our findings thus suggest that goal-directed, feature-specific selection determines the pattern of interaction between reward and physical salience, such that they contribute either independently or interactively to attentional priority. We propose two possible mechanisms that can account for the intricate patterns of influence among multiple sources of priority.

Intrusive effects of task-irrelevant information on visual selective attention: semantics and size

Attentional selection is a mechanism by which incoming sensory information is prioritized for further, detailed, and more effective, processing. Given that attended information is privileged by the sensory system, understanding and predicting what information is granted prioritization becomes an important endeavor. It has been argued that salient events as well as information that is related to the current goal of the organism (i.e., task-relevant) receive such a priority. Here, we propose that attentional prioritization is not limited to task-relevance, and discuss evidence showing that task-irrelevant, non-salient, high-level properties of unattended objects, namely object meaning and size, influence attentional allocation. Such an intrusion of non-salient, task-irrelevant, high-level information points to the need to re-conceptualize and formally modify current models of attentional guidance.

Pavlovian reward learning elicits attentional capture by reward-associated stimuli

Feature-reward association elicits value-driven attentional capture (VDAC) regardless of the task relevance of associated features. What are the necessary conditions for feature-reward associations in VDAC? Recent studies claim that VDAC is based on Pavlovian conditioning. In this study, we manipulated the temporal relationships among feature, response, and reward in reward learning to elucidate the necessary components of VDAC. We presented reward-associated features in a variety of locations in a flanker task to form a color-reward association (training phase) and then tested VDAC in a subsequent visual search task (test phase). In Experiment 1, we showed reward-associated features in a task display requiring response selection and observed VDAC, consistent with most previous studies. In Experiment 2, features presented at a fixation display before a task display also induced VDAC. Moreover, in Experiment 3, we reduced the time interval between features and rewards so that features appeared after a task display and we obtained marginally significant VDAC. However, no VDAC was observed when features and rewards were simultaneously presented in a feedback display in Experiments 4 and 5, suggesting that a direct association between feature and reward is not sufficient for VDAC. These results are in favor of the idea that response selection does not mediate feature-reward association in VDAC. Moreover, the evidence suggests that the time interval of feature and reward is flexible with some restriction in the learning of feature-reward association. The present study supports the hypothesis that theories of Pavlovian conditioning can account for feature-reward association in VDAC.

Neural correlates of attentional capture by stimuli previously associated with social reward

Our attention is strongly influenced by reward learning. Stimuli previously associated with monetary reward have been shown to automatically capture attention in both behavioral and neurophysiological studies. Stimuli previously associated with positive social feedback similarly capture attention; however, it is unknown whether such social facilitation of attention relies on similar or dissociable neural systems. Here, we used the value-driven attentional capture paradigm in an fMRI study to identify the neural correlates of attention to stimuli previously associated with social reward. The results reveal learning-dependent priority signals in the contralateral visual cortex, posterior parietal cortex, and caudate tail, similar to studies using monetary reward. An additional priority signal was consistently evident in the right middle frontal gyrus (MFG). Our findings support the notion of a common neural mechanism for directing attention on the basis of selection history that generalizes across different types of reward.

Do Reward-Related Distractors Impair Cognitive Performance? Perhaps Not

Over a hundred prior studies show that reward-related distractors capture attention. It is less clear, however, whether and when reward-related distractors affect performance on tasks that require cognitive control. In this experiment, we examined whether reward-related distractors impair performance during a demanding arithmetic task. Participants (N = 81) solved math problems, while they were exposed to task-irrelevant stimuli that were previously associated with monetary rewards (vs. not). Although we found some evidence for reward learning in the training phase, results from the test phase showed no evidence that reward-related distractors harm cognitive performance. This null effect was invariant across different versions of our task. We examined the results further with Bayesian analyses, which showed positive evidence for the null. Altogether, the present study showed that reward-related distractors did not harm performance on a mental arithmetic task. When considered together with previous studies, the present study suggests that the negative impact of reward-related distractors on cognitive control is not as straightforward as it may seem, and that more research is needed to clarify the circumstances under which reward-related distractors harm cognitive control.

Left-shifting prism adaptation boosts reward-based learning

Visuospatial cognition has an inherent lateralized bias. Individual differences in the direction and magnitude of this bias are associated with asymmetrical D2/3 dopamine binding and dopamine system genotypes. Dopamine level affects feedback-based learning and dopamine signaling asymmetry is related to differential learning from reward and punishment. High D2 binding in the left hemisphere is associated with preference for reward. Prism adaptation (PA) is a simple sensorimotor technique, which modulates visuospatial bias according to the direction of the deviation. Left-deviating prism adaptation (LPA) induces rightward bias in healthy subjects. It is therefore possible that the right side of space increases in saliency along with left hemisphere dopaminergic activity. Right-deviating prism adaptation (RPA) has been used mainly as a control condition because it does not modulate behavior in healthy individuals. Since LPA induces a rightward visuospatial bias as a result of left hemisphere modulation, and higher dopaminergic activity in the left hemisphere is associated with preference for rewarding events we hypothesized that LPA would increase the preference for learning with reward. Healthy volunteers performed a computer-based probabilistic classification task before and after LPA or RPA. Consistent with our predictions, PA altered the preference for rewarded versus punished learning, with the LPA group exhibiting increased learning from reward. These results suggest that PA modulates dopaminergic activity in a lateralized fashion.

Reward-associated distractors can harm cognitive performance

When people carry out cognitive tasks, they sometimes suffer from distractions, that is, drops in performance that occur close in time to task-irrelevant stimuli. In this research, we examine how the pursuit of rewards contributes to distractions. In two experiments, participants performed a math task (in which they could earn monetary rewards vs. not) while they were exposed to task-irrelevant stimuli (that were previously associated with monetary rewards vs. not). In Experiment 1, irrelevant cues that were previously associated with rewards (vs. not) impaired performance. In Experiment 2, this effect was only replicated when these reward-associated distractors appeared relatively early during task performance. While the results were thus somewhat mixed, they generally support the idea that reward associations can augment the negative effect of distractors on performance.

Anatomy and white matter connections of the orbitofrontal gyrus

OBJECTIVE

The orbitofrontal cortex (OFC) is understood to have a role in outcome evaluation and risk assessment and is commonly involved with infiltrative tumors. A detailed understanding of the exact location and nature of associated white matter tracts could significantly improve postoperative morbidity related to declining capacity. Through diffusion tensor imaging–based fiber tracking validated by gross anatomical dissection as ground truth, the authors have characterized these connections based on relationships to other well-known structures.

METHODS

Diffusion imaging from the Human Connectome Project for 10 healthy adult controls was used for tractography analysis. The OFC was evaluated as a whole based on connectivity with other regions. All OFC tracts were mapped in both hemispheres, and a lateralization index was calculated with resultant tract volumes. Ten postmortem dissections were then performed using a modified Klingler technique to demonstrate the location of major tracts.

RESULTS

The authors identified 3 major connections of the OFC: a bundle to the thalamus and anterior cingulate gyrus, passing inferior to the caudate and medial to the vertical fibers of the thalamic projections; a bundle to the brainstem, traveling lateral to the caudate and medial to the internal capsule; and radiations to the parietal and occipital lobes traveling with the inferior fronto-occipital fasciculus.

CONCLUSIONS

The OFC is an important center for processing visual, spatial, and emotional information. Subtle differences in executive functioning following surgery for frontal lobe tumors may be better understood in the context of the fiber-bundle anatomy highlighted by this study.

Selection history: How reward modulates selectivity of visual attention

Visual attention enables us to selectively prioritize or suppress information in the environment. Prominent models concerned with the control of visual attention differentiate between goal-directed, top-down and stimulus-driven, bottom-up control, with the former determined by current selection goals and the latter determined by physical salience. In the current review, we discuss recent studies that demonstrate that attentional selection does not need to be the result of top-down or bottom-up processing but, instead, is often driven by lingering biases due to the "history" of former attention deployments. This review mainly focuses on reward-based history effects; yet other types of history effects such as (intertrial) priming, statistical learning and affective conditioning are also discussed. We argue that evidence from behavioral, eye-movement and neuroimaging studies supports the idea that selection history modulates the topographical landscape of spatial "priority" maps, such that attention is biased toward locations having the highest activation on this map.

Reward priming eliminates color-driven affect in perception

Brain and behavior evidence suggests that colors have distinct affective properties. Here, we investigated how reward influences color-driven affect in perception. In Experiment 1, we assessed competition between blue and red patches during a temporal-order judgment (TOJ) across a range of stimulus onset asynchronies (SOAs). During the value reinforcement, reward was linked to either blue (version 1) or red (version 2) in the experiment. The same stimuli then served as test ones in the following unrewarded, unspeeded TOJ task. Our analysis showed that blue patches were consistently seen as occurring first, even when objectively appearing 2nd at short SOAs. This accelerated perception of blue over red was disrupted by prior primes related to reward (vs. neutral) but not perceptional (blue vs. red) priming. Experiment 2 replicated the findings of Experiment 1 while uncoupling action and stimulus values. These results are consistent with the blue-approach and red-avoidance motivation hypothesis and highlight an active nature of the association of reward priming and color processing. Together, the present study implies a link between reward and color affect and contributes to the understanding of how reward influences color affect in visual processing.

Learned value and object perception: Accelerated perception or biased decisions?

Learned value is known to bias visual search toward valued stimuli. However, some uncertainty exists regarding the stage of visual processing that is modulated by learned value. Here, we directly tested the effect of learned value on preattentive processing using temporal order judgments. Across four experiments, we imbued some stimuli with high value and some with low value, using a nonmonetary reward task. In Experiment 1, we replicated the value-driven distraction effect, validating our nonmonetary reward task. Experiment 2 showed that high-value stimuli, but not low-value stimuli, exhibit a prior-entry effect. Experiment 3, which reversed the temporal order judgment task (i.e., reporting which stimulus came second), showed no prior-entry effect, indicating that although a response bias may be present for high-value stimuli, they are still reported as appearing earlier. However, Experiment 4, using a simultaneity judgment task, showed no shift in temporal perception. Overall, our results support the conclusion that learned value biases perceptual decisions about valued stimuli without speeding preattentive stimulus processing.

Perceptual Competition Promotes Suppression of Reward Salience in Behavioral Selection and Neural Representation

Visual attentional selection is influenced by the value of objects. Previous studies have demonstrated that reward-associated items lead to rapid distraction and associated behavioral costs, which are difficult to override with top-down control. However, it has not been determined whether a perceptually competitive environment could render the reward-driven distraction more susceptible to top-down suppression. Here, we trained both genders of human subjects to associate two orientations with high and low magnitudes of reward. After training, we collected fMRI data while the subjects performed a categorical visual search task. The item in the reward-associated orientation served as the distractor, and the relative physical salience between the target and distractor was carefully controlled to modulate the degree of perceptual competition. The behavioral results showed faster searches in the presence of high, relative to low, reward-associated distractors. However, this effect was evident only if the physical salience of the distractor was higher than that of the target, indicating a context-dependent suppression effect of reward salience that relied on high perceptual competition. By analyzing the fMRI data in primary visual cortex, we found that the behavioral pattern of results could be predicted by the suppressed channel responses tuned to the reward-associated orientation in the distractor location, accompanied by increased responses in the midbrain dopaminergic region. Our results suggest that the learned salience of a reward plays a flexible role in solving perceptual competition, enabling the neural system to adaptively modulate the perceptual representation for behavioral optimization.SIGNIFICANCE STATEMENT The predictiveness principle in learning theory suggests that the stimulus with high predictability of reward receives priority in attentional selection. This selection bias leads to difficulties in changing approach behaviors, and thus becomes an important factor related to psychiatric disorders with attentional deficits. Here, we demonstrated that such principle is adaptively implemented in attentional suppression in visual search. We showed that the learned salience induced the suppression of the reward-associated distractor if its competition with the target was strong and could not be readily solved. This behavioral pattern was accompanied by increased midbrain fMRI activity and weakened sensory representation of the reward-associated distractor in V1. Our findings provided direct evidence that our brain flexibly uses learned regularities in attentional control.

Language Learning Enhanced by Massive Multiple Online Role-Playing Games (MMORPGs) and the Underlying Behavioral and Neural Mechanisms

Massive Multiple Online Role-Playing Games (MMORPGs) have increased in popularity among children, juveniles, and adults since MMORPGs' appearance in this digital age. MMORPGs can be applied to enhancing language learning, which is drawing researchers' attention from different fields and many studies have validated MMORPGs' positive effect on language learning. However, there are few studies on the underlying behavioral or neural mechanism of such effect. This paper reviews the educational application of the MMORPGs based on relevant macroscopic and microscopic studies, showing that gamers' overall language proficiency or some specific language skills can be enhanced by real-time online interaction with peers and game narratives or instructions embedded in the MMORPGs. Mechanisms underlying the educational assistant role of MMORPGs in second language learning are discussed from both behavioral and neural perspectives. We suggest that attentional bias makes gamers/learners allocate more cognitive resources toward task-related stimuli in a controlled or an automatic way. Moreover, with a moderating role played by activation of reward circuit, playing the MMORPGs may strengthen or increase functional connectivity from seed regions such as left anterior insular/frontal operculum (AI/FO) and visual word form area to other language-related brain areas.

What is abnormal about addiction-related attentional biases?

BACKGROUND

The phenotype of addiction includes prominent attentional biases for drug cues, which play a role in motivating drug-seeking behavior and contribute to relapse. In a separate line of research, arbitrary stimuli have been shown to automatically capture attention when previously associated with reward in non-clinical samples.

METHODS AND RESULTS

Here, I argue that these two attentional biases reflect the same cognitive process. I outline five characteristics that exemplify attentional biases for drug cues: resistant to conflicting goals, robust to extinction, linked to dorsal striatal dopamine and to biases in approach behavior, and can distinguish between individuals with and without a history of drug dependence. I then go on to describe how attentional biases for arbitrary reward-associated stimuli share all of these features, and conclude by arguing that the attentional components of addiction reflect a normal cognitive process that promotes reward-seeking behavior.

Appetitive and aversive outcome associations modulate exogenous cueing

In two experiments, we utilized an exogenous cueing task in which different-colored abrupt-onset cues were associated with an appetitive (gain of 10 cents), aversive (loss of 5 cents), or neutral (no gain or loss) outcome. Reward delivery did not depend on performance, but instead the specific exogenous cues were always followed by their corresponding outcome in a classical-conditioning-like manner. Compared to neutral cues and independent of cue-target delay, the results of Experiment 1 showed that appetitive cues strengthened attentional capture, whereas aversive cues reduced attentional capture. The data revealed that both appetitive and aversive cues initially facilitated responding at the validly cued location. At the long cue-target delays, however, this facilitation effect at the validly cued location remained present for gain-associated cues while it reversed for loss-associated cues. The results of Experiment 2 confirmed these findings by showing that both neutral and aversive cues initially facilitated responding at the cued location and that, at long cue-target delays, aversive cues elicited stronger reorienting away from the cued location as compared to neutral cues. Together these findings indicate that all abrupt-onset cues initially capture attention independent of their outcome association. Yet, if time passes, attention remains lingering at the location of gain-associated cues, whereas attention is released and reoriented away from the location of loss-associated cues. Altogether, we show that associating the color of an abrupt-onset cue with an appetitive or aversive outcome can modulate attentional deployment following exogenous cueing.

Components of reward-driven attentional capture

Recent research reported that task-irrelevant colors captured attention if these colors previously served as search targets and received high monetary reward. We showed that both monetary reward and value-independent mechanisms influenced selective attention. Participants searched for two potential target colors among distractor colors in the training phase. Subsequently, they searched for a shape singleton in a testing phase. Experiment 1 found that participants were slower in the testing phase if a distractor of a previous target color was present rather than absent. Such slowing was observed even when no monetary reward was used during training. Experiment 2 associated monetary rewards with the target colors during the training phase. Participants were faster finding the target associated with higher monetary reward. However, reward training did not yield value-dependent attentional capture in the testing phase. Attentional capture by the previous target colors was not significantly greater for the previously high-reward color than the previously low or no-reward color. These findings revealed both the power and limitations of monetary reward on attention. Although monetary reward can increase attentional priority for the high-reward target during training, subsequent attentional capture effects may not be reward-based, but reflect, in part, attentional capture by previous targets.

Gaze data reveal distinct choice processes underlying model-based and model-free reinforcement learning

Organisms appear to learn and make decisions using different strategies known as model-free and model-based learning; the former is mere reinforcement of previously rewarded actions and the latter is a forward-looking strategy that involves evaluation of action-state transition probabilities. Prior work has used neural data to argue that both model-based and model-free learners implement a value comparison process at trial onset, but model-based learners assign more weight to forward-looking computations. Here using eye-tracking, we report evidence for a different interpretation of prior results: model-based subjects make their choices prior to trial onset. In contrast, model-free subjects tend to ignore model-based aspects of the task and instead seem to treat the decision problem as a simple comparison process between two differentially valued items, consistent with previous work on sequential-sampling models of decision making. These findings illustrate a problem with assuming that experimental subjects make their decisions at the same prescribed time.

How Does Awareness Modulate Goal-Directed and Stimulus-Driven Shifts of Attention Triggered by Value Learning?

In order to behave adaptively, attention can be directed in space either voluntarily (i.e., endogenously) according to strategic goals, or involuntarily (i.e., exogenously) through reflexive capture by salient or novel events. The emotional or motivational value of stimuli can also strongly influence attentional orienting. However, little is known about how reward-related effects compete or interact with endogenous and exogenous attention mechanisms, particularly outside of awareness. Here we developed a visual search paradigm to study subliminal value-based attentional orienting. We systematically manipulated goal-directed or stimulus-driven attentional orienting and examined whether an irrelevant, but previously rewarded stimulus could compete with both types of spatial attention during search. Critically, reward was learned without conscious awareness in a preceding phase where one among several visual symbols was consistently paired with a subliminal monetary reinforcement cue. Our results demonstrated that symbols previously associated with a monetary reward received higher attentional priority in the subsequent visual search task, even though these stimuli and reward were no longer task-relevant, and despite reward being unconsciously acquired. Thus, motivational processes operating independent of conscious awareness may provide powerful influences on mechanisms of attentional selection, which could mitigate both stimulus-driven and goal-directed shifts of attention.

Modulation of spatial attention by goals, statistical learning, and monetary reward

This study documented the relative strength of task goals, visual statistical learning, and monetary reward in guiding spatial attention. Using a difficult T-among-L search task, we cued spatial attention to one visual quadrant by (i) instructing people to prioritize it (goal-driven attention), (ii) placing the target frequently there (location probability learning), or (iii) associating that quadrant with greater monetary gain (reward-based attention). Results showed that successful goal-driven attention exerted the strongest influence on search RT. Incidental location probability learning yielded a smaller though still robust effect. Incidental reward learning produced negligible guidance for spatial attention. The 95 % confidence intervals of the three effects were largely nonoverlapping. To understand these results, we simulated the role of location repetition priming in probability cuing and reward learning. Repetition priming underestimated the strength of location probability cuing, suggesting that probability cuing involved long-term statistical learning of how to shift attention. Repetition priming provided a reasonable account for the negligible effect of reward on spatial attention. We propose a multiple-systems view of spatial attention that includes task goals, search habit, and priming as primary drivers of top-down attention.

Attention's Accelerator

How do people get attention to operate at peak efficiency in high-pressure situations? We tested the hypothesis that the general mechanism that allows this is the maintenance of multiple target representations in working and long-term memory. We recorded subjects' event-related potentials (ERPs) indexing the working memory and long-term memory representations used to control attention while performing visual search. We found that subjects used both types of memories to control attention when they performed the visual search task with a large reward at stake, or when they were cued to respond as fast as possible. However, under normal circumstances, one type of target memory was sufficient for slower task performance. The use of multiple types of memory representations appears to provide converging top-down control of attention, allowing people to step on the attentional accelerator in a variety of high-pressure situations.

Introducing a Method for Calculating the Allocation of Attention in a Cognitive "Two-Armed Bandit" Procedure: Probability Matching Gives Way to Maximizing

We tested whether principles that describe the allocation of overt behavior, as in choice experiments, also describe the allocation of cognition, as in attention experiments. Our procedure is a cognitive version of the “two-armed bandit choice procedure.” The two-armed bandit procedure has been of interest to psychologistsand economists because it tends to support patterns of responding that are suboptimal. Each of two alternatives provides rewards according to fixed probabilities. The optimal solution is to choose the alternative with the higher probability of reward on each trial. However, subjects often allocate responses so that the probability of a response approximates its probability of reward. Although it is this result which has attracted most interest, probability matching is not always observed. As a function of monetary incentives, practice, and individual differences, subjects tend to deviate from probability matching toward exclusive preference, as predicted by maximizing. In our version of the two-armed bandit procedure, the monitor briefly displayed two, small adjacent stimuli that predicted correct responses according to fixed probabilities, as in a two-armed bandit procedure. We show that in this setting, a simple linear equation describes the relationship between attention and correct responses, and that the equation’s solution is the allocation of attention between the two stimuli. The calculations showed that attention allocation varied as a function of the degree to which the stimuli predicted correct responses. Linear regression revealed a strong correlation (r = 0.99) between the predictiveness of a stimulus and the probability of attending to it. Nevertheless there were deviations from probability matching, and although small, they were systematic and statistically significant. As in choice studies, attention allocation deviated toward maximizing as a function of practice, feedback, and incentives. Our approach also predicts the frequency of correct guesses and the relationship between attention allocation and response latencies. The results were consistent with these two predictions, the assumptions of the equations used to calculate attention allocation, and recent studies which show that predictiveness and reward are important determinants of attention.

Value-driven attentional capture in the auditory domain

It is now well established that the visual attention system is shaped by reward learning. When visual features are associated with a reward outcome, they acquire high priority and can automatically capture visual attention. To date, evidence for value-driven attentional capture has been limited entirely to the visual system. In the present study, I demonstrate that previously reward-associated sounds also capture attention, interfering more strongly with the performance of a visual task. This finding suggests that value-driven attention reflects a broad principle of information processing that can be extended to other sensory modalities and that value-driven attention can bias cross-modal stimulus competition.

Reward alters the perception of time

Recent findings indicate that monetary rewards have a powerful effect on cognitive performance. In order to maximize overall gain, the prospect of earning reward biases visual attention to specific locations or stimulus features improving perceptual sensitivity and processing. The question we addressed in this study is whether the prospect of reward also affects the subjective perception of time. Here, participants performed a prospective timing task using temporal oddballs. The results show that temporal oddballs, displayed for varying durations, presented in a sequence of standard stimuli were perceived to last longer when they signaled a relatively high reward compared to when they signaled no or low reward. When instead of the oddball the standards signaled reward, the perception of the temporal oddball remained unaffected. We argue that by signaling reward, a stimulus becomes subjectively more salient thereby modulating its attentional deployment and distorting how it is perceived in time.

The attention habit: how reward learning shapes attentional selection

There is growing consensus that reward plays an important role in the control of attention. Until recently, reward was thought to influence attention indirectly by modulating task-specific motivation and its effects on voluntary control over selection. Such an account was consistent with the goal-directed (endogenous) versus stimulus-driven (exogenous) framework that had long dominated the field of attention research. Now, a different perspective is emerging. Demonstrations that previously reward-associated stimuli can automatically capture attention even when physically inconspicuous and task-irrelevant challenge previously held assumptions about attentional control. The idea that attentional selection can be value driven, reflecting a distinct and previously unrecognized control mechanism, has gained traction. Since these early demonstrations, the influence of reward learning on attention has rapidly become an area of intense investigation, sparking many new insights. The result is an emerging picture of how the reward system of the brain automatically biases information processing. Here, I review the progress that has been made in this area, synthesizing a wealth of recent evidence to provide an integrated, up-to-date account of value-driven attention and some of its broader implications.

How does reward compete with goal-directed and stimulus-driven shifts of attention?

In order to behave adaptively, attention can be directed in space either voluntarily (i.e. endogenously) according to strategic goals, or involuntarily (i.e. exogenously) through reflexive capture by salient or novel events. The emotional or motivational values of stimuli can also influence attentional orienting. However, little is known about how reward-related effects compete or interact with endogenous and exogenous attention mechanisms. Here we designed a visual search paradigm in which goal-driven and stimulus-driven shifts of attention were manipulated by classic spatial cueing procedures, while an irrelevant, but previously rewarded stimulus also appeared as a distractor and hence competed with both types of spatial attention during search. Our results demonstrated that stimuli previously associated with a high monetary reward received higher attentional priority in the subsequent visual search task, even though these stimuli and reward were no longer task-relevant, mitigating the attentional orienting induced by both endogenous and exogenous cues.

Value-driven attentional priority is context specific

Attention is automatically drawn to stimulus features previously associated with reward, a phenomenon referred to as value-driven attentional capture. To date, value-driven attentional capture has been studied exclusively by manipulating stimulus-reward contingencies in an experimental setting. Although practical and intuitively appealing, this approach poses theoretical challenges to understanding the broader impact of reward on attention in everyday life. These challenges arise from the fact that associative learning between a given visual feature and reward is not limited to the context of an experiment, yet such extra-experimental learning is completely ignored in studies of value-driven attention. How is it, then, that experimentally established reward associations even influence attention, seemingly overshadowing any prior learning about particular features and rewards? And how do the effects of this experimental learning persist over long periods of time, in spite of all the intervening experiences outside of the lab that might interfere with the learning? One potential answer to these questions is that value-driven attention is context specific, such that different contexts evoke different value priors that the attention system uses to assign priority. In the present study, I directly tested this hypothesis. The results show that the same stimulus feature either does or does not capture attention, depending on whether it has been rewarded specifically in the context within which it appears. The findings provide insight into how multiple reward structures can efficiently guide attention with minimal interference.

Value-based attentional capture influences context-dependent decision-making

Normative theories posit that value-based decision-making is context independent. However, decisions between two high-value options can be suboptimally biased by the introduction of a third low-value option. This context-dependent modulation is consistent with the divisive normalization of the value of each stimulus by the total value of all stimuli. In addition, an independent line of research demonstrates that pairing a stimulus with a high-value outcome can lead to attentional capture that can mediate the efficiency of visual information processing. Here we tested the hypothesis that value-based attentional capture interacts with value-based normalization to influence the optimality of decision-making. We used a binary-choice paradigm in which observers selected between two targets and the color of each target indicated the magnitude of their reward potential. Observers also had to simultaneously ignore a task-irrelevant distractor rendered in a color that was previously associated with a specific reward magnitude. When the color of the task-irrelevant distractor was previously associated with a high reward, observers responded more slowly and less optimally. Moreover, as the learned value of the distractor increased, electrophysiological data revealed an attenuation of the lateralized N1 and N2Pc responses evoked by the relevant choice stimuli and an attenuation of the late positive deflection (LPD). Collectively, these behavioral and electrophysiological data suggest that value-based attentional capture and value-based normalization jointly mediate the influence of context on free-choice decision-making.

Converging structural and functional connectivity of orbitofrontal, dorsolateral prefrontal, and posterior parietal cortex in the human striatum

Modification of spatial attention via reinforcement learning (Lee and Shomstein, 2013) requires the integration of reward, attention, and executive processes. Corticostriatal pathways are an ideal neural substrate for this integration because these projections exhibit a globally parallel (Alexander et al., 1986), but locally overlapping (Haber, 2003), topographical organization. Here we explore whether there are unique striatal regions that exhibit convergent anatomical connections from orbitofrontal cortex, dorsolateral prefrontal cortex, and posterior parietal cortex. Deterministic fiber tractography on diffusion spectrum imaging data from neurologically healthy adults (N = 60) was used to map frontostriatal and parietostriatal projections. In general, projections from cortex were organized according to both a medial-lateral and a rostral-caudal gradient along the striatal nuclei. Within rostral aspects of the striatum, we identified two bilateral convergence zones (one in the caudate nucleus and another in the putamen) that consisted of voxels with unique projections from orbitofrontal cortex, dorsolateral prefrontal cortex, and parietal regions. The distributed cortical connectivity of these striatal convergence zones was confirmed with follow-up functional connectivity analysis from resting state fMRI data, in which a high percentage of structurally connected voxels also showed significant functional connectivity. The specificity of this convergent architecture to these regions of the rostral striatum was validated against control analysis of connectivity within the motor putamen. These results delineate a neurologically plausible network of converging corticostriatal projections that may support the integration of reward, executive control, and spatial attention that occurs during spatial reinforcement learning.