Reward Draws the Eye, Uncertainty Holds the Eye: Associative Learning Modulates Distractor Interference in Visual Search

Effects of outcome revaluation on attentional prioritisation of reward-related stimuli

Stimuli associated with rewards can acquire the ability to capture our attention independently of our goals and intentions. Here, we examined whether attentional prioritisation of reward-related cues is sensitive to changes in the value of the reward itself. To this end, we incorporated an instructed outcome devaluation (Experiment 1a), "super-valuation" (Experiment 1b), or value switch (Experiment 2) into a visual search task, using eye-tracking to examine attentional prioritisation of stimuli signalling high- and low-value rewards. In Experiments 1a and 1b, we found that prioritisation of high- and low-value stimuli was insensitive to devaluation of a previously high-value outcome, and super-valuation of a previously low-value outcome, even when participants were provided with further experience of receiving that outcome. In Experiment 2, following a value-switch manipulation, we found that prioritisation of a high-value stimulus could not be overcome with knowledge of the new values of outcomes alone. Only when provided with further experience of receiving the outcomes did patterns of attentional prioritisation of high- and low-value stimuli switch, in line with the updated values of the outcomes they signalled. To reconcile these findings, we suggest that participants were motivated to engage in effortful updating of attentional control settings when there was a relative difference between reward values at test (Experiment 2) but that previous settings were allowed to persist when both outcomes had the same value at test (Experiments 1a and 1b). These findings provide a novel framework to further understand the role of cognitive control in driving reward-modulated attention and behaviour.

Are depressive symptoms linked to a reduced pupillary response to novel positive information?-An eye tracking proof-of-concept study

Introduction

Depressive symptoms have been linked to difficulties in revising established negative beliefs in response to novel positive information. Recent predictive processing accounts have suggested that this bias in belief updating may be related to a blunted processing of positive prediction errors at the neural level. In this proof-of-concept study, pupil dilation in response to unexpected positive emotional information was examined as a psychophysiological marker of an attenuated processing of positive prediction errors associated with depressive symptoms.

Methods

Participants (N = 34) completed a modified version of the emotional Bias Against Disconfirmatory Evidence (BADE) task in which scenarios initially suggest negative interpretations that are later either confirmed or disconfirmed by additional information. Pupil dilation in response to the confirmatory and disconfirmatory information was recorded.

Results

Behavioral results showed that depressive symptoms were related to difficulties in revising negative interpretations despite disconfirmatory positive information. The eye tracking results pointed to a reduced pupil response to unexpected positive information among people with elevated depressive symptoms.

Discussion

Altogether, the present study demonstrates that the adapted emotional BADE task can be appropriate for examining psychophysiological aspects such as changes in pupil size along with behavioral responses. Furthermore, the results suggest that depression may be characterized by deviations in both behavioral (i.e., reduced updating of negative beliefs) and psychophysiological (i.e., decreased pupil dilation) responses to unexpected positive information. Future work should focus on a larger sample including clinically depressed patients to further explore these findings.

Value-driven attention and associative learning models: a computational simulation analysis

Value-driven attentional capture (VDAC) refers to a phenomenon by which stimulus features associated with greater reward value attract more attention than those associated with smaller reward value. To date, the majority of VDAC research has revealed that the relationship between reward history and attentional allocation follows associative learning rules. Accordingly, a mathematical implementation of associative learning models and multiple comparison between them can elucidate the underlying process and properties of VDAC. In this study, we implemented the Rescorla-Wagner, Mackintosh (Mac), Schumajuk-Pearce-Hall (SPH), and Esber-Haselgrove (EH) models to determine whether different models predict different outcomes when critical parameters in VDAC were adjusted. Simulation results were compared with experimental data from a series of VDAC studies by fitting two key model parameters, associative strength (V) and associability (α), using the Bayesian information criterion as a loss function. The results showed that SPH-V and EH- α outperformed other implementations of phenomena related to VDAC, such as expected value, training session, switching (or inertia), and uncertainty. Although V of models were sufficient to simulate VDAC when the expected value was the main manipulation of the experiment, α of models could predict additional aspects of VDAC, including uncertainty and resistance to extinction. In summary, associative learning models concur with the crucial aspects of behavioral data from VDAC experiments and elucidate underlying dynamics including novel predictions that need to be verified.

The effect of prediction error on overt attention and learning in humans

It has been suggested that attention modulates the speed at which cues come to predict contingent outcomes, and that attention changes with the prediction errors generated by cues. Evidence for this interaction in humans is inconsistent, with divergent findings depending on whether attention was measured with eye fixations or learning speed. We included both measures in our experiment. Initially, predictive cues (A and B) were consistently followed by one outcome (o1), while nonpredictive cues (X and Y) were followed by two randomly alternating outcomes (o1 and o2). Consistent with an effect of prediction error, participants' fixated for longer on the nonpredictive cues than on the predictive ones. Then, the cues were combined in three pairs: AX, followed by o1, and AY and BX, followed by o2. Discrimination of AX and AY depended on the previously nonpredictive cues and, given that these received more attention during initial training, it should proceed faster than discrimination of AX and BX, which depended on the previously predictive cues. However, participants learned to predict the outcomes of AY and BX at a similar rate. The fixation times were similar for the previously predictive and previously nonpredictive cues. We discuss reasons that could explain these findings.

Reducing anxiety and attentional bias with reward association learning and attentional bias modification

The current study examined the effects of a reward associative learning procedure and the traditional threat-avoidance ABM paradigm on anxiety and attentional bias. In reward training, participants were given high rewards for correct responses to neutral target and low rewards for correct responses to negative target. In reward control training, participants received no cues of rewards after their responses. High trait anxious individuals (N = 76) first completed a session of reward training or reward control training, followed by four sessions of ABM training or ABM control training. Generalized anxiety disorder symptom (GAD-7) and attentional bias in a dot-probe task were assessed during pre-and post-training. Results indicated that the effect of ABM training on reducing anxiety was only obtained in the reward training condition. Participants who received reward training showed significantly less attentional bias compared with those receiving reward control training. There was no significant training effect of ABM on atttentiona bias. Results suggested that reward training reduced general anxiety and attentional bias. Traditional ABM training reduced anxiety only when combined with reward training. Attentional bias in anxiety are modifiable through reward training.

Eye movements reveal spatiotemporal dynamics of visually-informed planning in navigation

Goal-oriented navigation is widely understood to depend upon internal maps. Although this may be the case in many settings, humans tend to rely on vision in complex, unfamiliar environments. To study the nature of gaze during visually-guided navigation, we tasked humans to navigate to transiently visible goals in virtual mazes of varying levels of difficulty, observing that they took near-optimal trajectories in all arenas. By analyzing participants’ eye movements, we gained insights into how they performed visually-informed planning. The spatial distribution of gaze revealed that environmental complexity mediated a striking trade-off in the extent to which attention was directed towards two complimentary aspects of the world model: the reward location and task-relevant transitions. The temporal evolution of gaze revealed rapid, sequential prospection of the future path, evocative of neural replay. These findings suggest that the spatiotemporal characteristics of gaze during navigation are significantly shaped by the unique cognitive computations underlying real-world, sequential decision making.

An attentional perspective on differential fear conditioning in chronic pain: The informational value of safety cues

Differences in fear conditioning between individuals suffering from chronic pain and healthy controls may indicate a learning bias that contributes to the acquisition and persistence of chronic pain. However, evidence from lab-controlled conditioning studies is sparse and previous experiments have produced inconsistent findings. Twenty-five participants suffering from chronic back pain and twenty-five controls not reporting chronic pain took part in a differential fear conditioning experiment measuring attention (eye tracking) and autonomic arousal (pupil dilation and skin conductance) elicited by visual cues predicting the presence or absence of electric shock. In contrast to the healthy control group, participants with chronic pain did not acquire differential autonomic responding to cues of threat and safety and specifically failed to acquire any attentional preference for the safety cue over irrelevant contextual cues (while such preference was intact for the threat cue). We present simulations of a reinforcement learning model to show how the pattern of data can be explained by assuming that participants with chronic pain might have experienced less positive emotion (relief) when the electric shock was absent following safety cues. Our model shows how this assumption can explain both, reduced differential responding to cues of threat and safety as well as less selective attention to the safety cue.

Emergence of complex dynamics of choice due to repeated exposures to extinction learning

Extinction learning, the process of ceasing an acquired behavior in response to altered reinforcement contingencies, is not only essential for survival in a changing environment, but also plays a fundamental role in the treatment of pathological behaviors. During therapy and other forms of training involving extinction, subjects are typically exposed to several sessions with a similar structure. The effects of this repeated exposure are not well understood. Here, we studied the behavior of pigeons across several sessions of a discrimination-learning task in context A, extinction in context B, and a return to context A to test the context-dependent return of the learned responses (ABA renewal). By focusing on individual learning curves across animals, we uncovered a session-dependent variability of behavior: (1) during extinction, pigeons preferred the unrewarded alternative choice in one-third of the sessions, predominantly during the first one. (2) In later sessions, abrupt transitions of behavior at the onset of context B emerged, and (3) the renewal effect decayed as sessions progressed. We show that the observed results can be parsimoniously accounted for by a computational model based only on associative learning between stimuli and actions. Our work thus demonstrates the critical importance of studying the trial-by-trial dynamics of learning in individual sessions, and the power of “simple” associative learning processes.

Perceptual Difficulty Persistently Increases Dominance in Binocular Rivalry-Even Without a Task

A major objective of perception is the reduction of uncertainty about the outside world. Eye-movement research has demonstrated that attention and oculomotor control can subserve the function of decreasing uncertainty in vision. Here, we ask whether a similar effect exists for awareness in binocular rivalry, when two distinct stimuli presented to the two eyes compete for awareness. We tested whether this competition can be biased by uncertainty about the stimuli and their relevance for a perceptual task. Specifically, we have stimuli that are perceptually difficult (i.e., carry high perceptual uncertainty) compete with stimuli that are perceptually easy (low perceptual uncertainty). Using a no-report paradigm and reading the dominant stimulus continuously from the observers' eye movements, we find that the perceptually difficult stimulus becomes more dominant than the easy stimulus. This difference is enhanced by the stimuli's relevance for the task. In trials with task, the difference in dominance emerges quickly, peaks before the response, and then persists throughout the trial (further 10 s). However, the difference is already present in blocks before task instruction and still observable when the stimuli have ceased to be task relevant. This shows that perceptual uncertainty persistently increases perceptual dominance, and this is magnified by task relevance.

A rise in prediction error increases attention to irrelevant cues

We investigated whether a sudden rise in prediction error widens an individual's focus of attention by increasing ocular fixations on cues that otherwise tend to be ignored. To this end, we used a discrimination learning task including cues that were either relevant or irrelevant for predicting the outcomes. Half of participants experienced contingency reversal once they had learned to predict the outcomes (reversal group, n = 30). The other half experienced the same contingencies throughout the task (control group, n = 30). As participants' prediction accuracy increased, they showed a decrease in the number of fixations directed to the irrelevant cues. Following contingency reversal, participants in the reversal group showed a drop in accuracy, indicating a rise in prediction error, and fixated on the irrelevant cues more often than participants in the control group. We discuss the results in the context of attentional theories of associative learning.

Uncertainty modulates value-driven attentional capture

The majority of previous studies on the value modulation of attention have shown that the magnitude of value-driven attentional bias correlates with the strength of reward association. However, relatively little is known about how uncertainty affects value-based attentional bias. We investigated whether attentional capture by previously rewarded stimuli is modulated by the uncertainty of the learned value without the influence of the strength of reward association. Participants were instructed to identify the line orientation in the target color circle. Importantly, each target color was associated with a different level of uncertainty by tuning the variation in reward delivery (Experiment 1) or reward magnitude (Experiment 2). Attentional interference for uncertainty-related distractors was greater than that for certainty distractors in Experiments 1 and 2. In addition, uncertainty-induced attentional bias disappeared earlier than attentional bias for certainty. The study demonstrated that uncertainty modulates value-based attentional capture in terms of strength and persistence, even when the effect of expected value remains constant.

Rescorla-Wagner Models with Sparse Dynamic Attention

The Rescorla-Wagner (R-W) model describes human associative learning by proposing that an agent updates associations between stimuli, such as events in their environment or predictive cues, proportionally to a prediction error. While this model has proven informative in experiments, it has been posited that humans selectively attend to certain cues to overcome a problem with the R-W model scaling to large cue dimensions. We formally characterize this scaling problem and provide a solution that involves limiting attention in a R-W model to a sparse set of cues. Given the universal difficulty in selecting features for prediction, sparse attention faces challenges beyond those faced by the R-W model. We demonstrate several ways in which a naive attention model can fail explain those failures and leverage that understanding to produce a Sparse Attention R-W with Inference framework (SAR-WI). The SAR-WI framework not only satisfies a constraint on the number of attended cues, it also performs as well as the R-W model on a number of natural learning tasks, can correctly infer associative strengths, and focuses attention on predictive cues while ignoring uninformative cues. Given the simplicity of proposed alterations, we hope this work informs future development and empirical validation of associative learning models that seek to incorporate sparse attention.

Pupil dilation as an implicit measure of appetitive Pavlovian learning

Appetitive Pavlovian conditioning is a learning mechanism of fundamental biological and pathophysiological significance. Nonetheless, its exploration in humans remains sparse, which is partly attributed to the lack of an established psychophysiological parameter that aptly represents conditioned responding. This study evaluated pupil diameter and other ocular response measures (gaze dwelling time, blink duration and count) as indices of conditioning. Additionally, a learning model was used to infer participants' learning progress on the basis of their pupil dilation. Twenty-nine healthy volunteers completed an appetitive differential delay conditioning paradigm with a primary reward, while the ocular response measures along with other psychophysiological (heart rate, electrodermal activity, postauricular and eyeblink reflex) and behavioral (ratings, contingency awareness) parameters were obtained to examine the relation among different measures. A significantly stronger increase in pupil diameter, longer gaze duration and shorter eyeblink duration was observed in response to the reward-predicting cue compared to the control cue. The Pearce-Hall attention model best predicted the trial-by-trial pupil diameter. This conditioned response was corroborated by a pronounced heart rate deceleration to the reward-predicting cue, while no conditioning effect was observed in the electrodermal activity or startle responses. There was no discernible correlation between the psychophysiological response measures. These results highlight the potential value of ocular response measures as sensitive indices for representing appetitive conditioning.

Expected Reward Value and Reward Uncertainty Have Temporally Dissociable Effects on Memory Formation

Anticipating rewards has been shown to enhance memory formation. Although substantial evidence implicates dopamine in this behavioral effect, the precise mechanisms remain ambiguous. Because dopamine nuclei have been associated with two distinct physiological signatures of reward prediction, we hypothesized two dissociable effects on memory formation. These two signatures are a phasic dopamine response immediately following a reward cue that encodes its expected value and a sustained, ramping response that has been demonstrated during high reward uncertainty [Fiorillo, C. D., Tobler, P. N., & Schultz, W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science, 299, 1898-1902, 2003]. Here, we show in humans that the impact of reward anticipation on memory for an event depends on its timing relative to these physiological signatures. By manipulating reward probability (100%, 50%, or 0%) and the timing of the event to be encoded (just after the reward cue versus just before expected reward outcome), we demonstrated the predicted double dissociation: Early during reward anticipation, memory formation was improved by increased expected reward value, whereas late during reward anticipation, memory formation was enhanced by reward uncertainty. Notably, although the memory benefits of high expected reward in the early interval were consolidation dependent, the memory benefits of high uncertainty in the later interval were not. These findings support the view that expected reward benefits memory consolidation via phasic dopamine release. The novel finding of a distinct memory enhancement, temporally consistent with sustained anticipatory dopamine release, points toward new mechanisms of memory modulation by reward now ripe for further investigation.

Multiple reward-cue contingencies favor expectancy over uncertainty in shaping the reward-cue attentional salience

Reward-predicting cues attract attention because of their motivational value. A debated question regards the conditions under which the cue's attentional salience is governed more by reward expectancy rather than by reward uncertainty. To help shedding light on this relevant issue, here, we manipulated expectancy and uncertainty using three levels of reward-cue contingency, so that, for example, a high level of reward expectancy (p = .8) was compared with the highest level of reward uncertainty (p = .5). In Experiment 1, the best reward-cue during conditioning was preferentially attended in a subsequent visual search task. This result was replicated in Experiment 2, in which the cues were matched in terms of response history. In Experiment 3, we implemented a hybrid procedure consisting of two phases: an omission contingency procedure during conditioning, followed by a visual search task as in the previous experiments. Crucially, during both phases, the reward-cues were never task relevant. Results confirmed that, when multiple reward-cue contingencies are explored by a human observer, expectancy is the major factor controlling both the attentional and the oculomotor salience of the reward-cue.