Prediction and Uncertainty in Associative Learning: Examining Controlled and Automatic Components of Learned Attentional Biases

Food-related exploration across the menstrual cycle

When deciding what to eat we constantly weigh different aspects of the options at hand and make trade-offs between exploiting opportunities with a known outcome (e.g., eating your usual meal) and exploring novel opportunities with a potentially better outcome (e.g., trying a new dish). Environmental factors, such as scarcity, have previously been shown to tip the balance towards either exploration or exploitation. Studies in animals have further linked female steroid hormones (including estradiol and progesterone) to exploratory behavior. Previous work in humans has suggested that food preferences and food intake also change over the menstrual cycle. However, it remains unknown whether exploratory behavior in food choices also changes across the menstrual phases in humans. In a rating phase, 112 adult women (age range 18-45 years) on or off hormonal birth control rated 40 food items on desirability. In the choice phase, participants made binary choices between food items. On some trials, a surprise box replaced one of the two food options, allowing us to examine exploratory choices. Women off hormonal birth control reported their average cycle length and time since the first day of their last menstruation. Analysis of the percentage surprise choices across the menstrual cycle indicated a small, but significant effect, with exploratory choice behavior tending to increase around the middle of/later during the cycle. This provides preliminary novel evidence that hormonal fluctuations are associated with food-related exploratory choice behavior. Importantly, these effects were observed beyond effects of other food-related factors. Future studies should investigate the nature of these effects using more direct hormonal measures.

Statistical learning of target and distractor spatial probability shape a common attentional priority computation

Converging evidence recently put forward the notion that dedicated neurocognitive mechanisms do exist for the suppression of salient, but irrelevant distractors. Along this line, it is plausible to hypothesize that, in appropriate contexts, experience-dependent forms of attentional learning might selectively induce plastic changes within this dedicated circuitry, thus allowing an independent shaping of priorities at the service of attentional filtering. Conversely, previous work suggested that statistical learning (SL) of both target and distractor spatial probability distributions converge in adjusting only the overall attentional priority of locations: in fact, in the presence of an independent manipulation, either related to the target or to the distractor only, SL induces indirect effects (e.g., changes in filtering efficiency due to an uneven distribution of targets), suggesting that SL-induced plastic changes affect a shared neural substrate. Here we tested whether, when (conflicting) target- and distractor-related manipulations are concurrently applied to the very same locations, dedicated mechanisms might support the selective encoding of spatial priority in relation to the specific attentional operation involved. In three related experiments, human healthy participants discriminated the direction of a target arrow, while ignoring a salient distractor, if present; both target and distractor spatial probability distributions were concurrently manipulated in relation to each single location. Critically, the selection bias produced by the target-related SL was marginally reduced by an adverse distractor contingency, and the suppression bias generated by the distractor-related SL was erased, or even reversed, by an adverse target contingency. Our results suggest that even conflicting target- and distractor-related SL manipulations result in the adjustment of a unique spatial priority computation, likely because the process directly relies on direct plastic alterations of shared spatial priority map(s).

Learned low priority of attention after training to suppress color singleton distractor

Allocating attention to significant events, such as a salient object, is effortless. Our brain is effective on this type of processing because doing so is generally beneficial for survival. However, a salient object could also be distracting and ignoring it costs a large amount of cognitive resource. In the present study, we conducted two behavioral experiments to investigate the effect of learned suppression of a salient color. Particularly, we were interested in the effect of learning in a new task context in which the previously suppressed color was task irrelevant. In Experiment 1, we trained the participants for five days with explicit instruction to suppress a color singleton distractor in a visual search task. We measured the effect of training with a dot probe task before and after the training. Colors in the dot probe task only served as the background and were not associated with the position of the target dot. However, we found that attention was involuntarily biased away from the previously suppressed color. In Experiment 2, the color singleton could either be the target or distractor in the visual search task, making the suppression of the color singleton inefficient for task performance. The results showed no training effect in the dot probe task after this manipulation. These findings provided direct evidence for the learned low priority of attention after training to suppress the color singleton distractor.

Testing reward-cue attentional salience: Attainment and dynamic changes

A great wealth of studies has investigated the capacity of motivationally relevant stimuli to bias attention, suggesting that reward predicting cues are prioritized even when reward is no longer delivered and when attending to such stimuli is detrimental to reward achievement. Despite multiple procedures have been adopted to unveil the mechanisms whereby reward cues gain attentional salience, some open questions remain. Indeed, mechanisms different from motivation can be responsible for the capture of attention triggered by the reward cue. In addition, we note that at present only a few studies have sought to address whether the cue attractiveness dynamically follows changes in the associated reward value. Investigating how and to what extent the salience of the reward cue is updated when motivation changes, could help shedding light on how reward‐cues attain and maintain their capacity to attract attention, and therefore on apparent irrational attentive behaviors.

Reversing the relationship between a nontarget cue and the outcome facilitates subsequent human predictive learning

Two experiments were conducted to test the effect of experiencing associative interference on later learning. A predictive learning task was used in which human participants had to evaluate whether plants would grow or not (Outcome) after being watered with different fertilizers (Cues). Experiment 1 found that the increase in the prediction error produced by following a pre-exposed nontarget cue by the outcome, facilitated subsequent acquisition of the relationship between the pre-exposed target cue and the outcome. Experiment 2 compared whether learning about the target cue was differentially affected by experiencing two types of associative interference with the nontarget cue: Pairing the pre-exposed cue with the outcome and presenting the cue without outcome after being paired with it. The experience of associative interference with nontarget cues similarly facilitated subsequent learning about the target cue, regardless of the direction of the change in the nontarget cue-outcome relationship. It is suggested that the increase in prediction error produced by the experience of associative interference may lead to a general increase in attention that facilitates subsequent learning.

Uncertainty and predictiveness modulate attention in human predictive learning

[Correction Notice: An Erratum for this article was reported online in Journal of Experimental Psychology: General on Jan 14 2021 (see record 2021-07705-001). In the article, formatting for UK Research Councils funding was omitted. The author note and copyright line now reflect the standard acknowledgment of and formatting for the funding received for this article. All versions of this article have been corrected.] Attention determines which cues receive processing and are learned about. Learning, however, leads to attentional biases. In the study of animal learning, in some circumstances, cues that have been previously predictive of their consequences are subsequently learned about more than are nonpredictive cues, suggesting that they receive more attention. In other circumstances, cues that have previously led to uncertain consequences are learned about more than are predictive cues. In human learning, there is a clear role for predictiveness, but a role for uncertainty has been less clear. Here, in a human learning task, we show that cues that led to uncertain outcomes were subsequently learned about more than were cues that were previously predictive of their outcomes. This effect occurred when there were few uncertain cues. When the number of uncertain cues was increased, attention switched to predictive cues. This pattern of results was found for cues (1) that were uncertain because they led to 2 different outcomes equally often in a nonpredictable manner and (2) that were used in a nonlinear discrimination and were not predictive individually but were predictive in combination with other cues. This suggests that both the opposing predictiveness and uncertainty effects were determined by the relationship between individual cues and outcomes rather than the predictive strength of combined cues. These results demonstrate that learning affects attention; however, the precise nature of the effect on attention depends on the level of task complexity, which reflects a potential switch between exploration and exploitation of cues. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

A rise in prediction error increases attention to irrelevant cues

We investigated whether a sudden rise in prediction error widens an individual's focus of attention by increasing ocular fixations on cues that otherwise tend to be ignored. To this end, we used a discrimination learning task including cues that were either relevant or irrelevant for predicting the outcomes. Half of participants experienced contingency reversal once they had learned to predict the outcomes (reversal group, n = 30). The other half experienced the same contingencies throughout the task (control group, n = 30). As participants' prediction accuracy increased, they showed a decrease in the number of fixations directed to the irrelevant cues. Following contingency reversal, participants in the reversal group showed a drop in accuracy, indicating a rise in prediction error, and fixated on the irrelevant cues more often than participants in the control group. We discuss the results in the context of attentional theories of associative learning.

Uncertainty modulates value-driven attentional capture

The majority of previous studies on the value modulation of attention have shown that the magnitude of value-driven attentional bias correlates with the strength of reward association. However, relatively little is known about how uncertainty affects value-based attentional bias. We investigated whether attentional capture by previously rewarded stimuli is modulated by the uncertainty of the learned value without the influence of the strength of reward association. Participants were instructed to identify the line orientation in the target color circle. Importantly, each target color was associated with a different level of uncertainty by tuning the variation in reward delivery (Experiment 1) or reward magnitude (Experiment 2). Attentional interference for uncertainty-related distractors was greater than that for certainty distractors in Experiments 1 and 2. In addition, uncertainty-induced attentional bias disappeared earlier than attentional bias for certainty. The study demonstrated that uncertainty modulates value-based attentional capture in terms of strength and persistence, even when the effect of expected value remains constant.

Reward anticipation and punishment anticipation are instantiated in the brain via opponent mechanisms

fMRI investigations have examined the extent to which reward and punishment motivation are associated with common or opponent neural systems, but such investigations have been limited by confounding variables and methodological constraints. The present study aimed to address limitations of earlier approaches and more comprehensively evaluate the extent to which neural activation associated with reward and punishment motivation reflects opponent or shared systems. Participants completed a modified monetary incentive delay task, which involved the presentation of a cue followed by a target to which participants were required to make a speeded button press. Using a factorial design, cues indicated whether monetary reward and/or loss (i.e., cues signaled probability of reward, punishment, both, or neither) could be expected depending upon response speed. Neural analyses evaluated evidence of (a) directionally opposing effects by testing for regions of differential activation for reward and punishment anticipation, (b) mutual inhibition by testing for interactive effects of reward and punishment anticipation within a factorial design, and (c) opposing effects on shared outputs via a psychophysiological interaction analysis. Evidence supporting all three criteria for opponent systems was obtained. Collectively, present findings support conceptualizing reward and punishment motivation as opponent forces influencing brain and behavior and indicate that shared activation does not suggest the operation of a common neural mechanism instantiating reward and punishment motivation.

Reward and emotion influence attentional bias in rapid serial visual presentation

Facial emotion constitutes an important source of information, and rapid processing of this information may bring adaptive advantages. Previous evidence suggests that emotional faces are sometimes prioritised for cognitive processing. Three experiments used an emotion-induced blindness task to examine whether this prioritisation occurs in a purely stimulus-driven fashion or whether it emerges only when the faces are task-relevant. Angry or neutral faces appeared as distractors in a rapid serial visual presentation sequence, shortly before a target that participants were required to identify. Either the emotion (Experiment 1) or gender (Experiments 2 and 3) of the distractor face indicated whether a correct/incorrect response to the target would produce reward/punishment, or not. The three experiments found that reward-related faces impaired subsequent target identification, replicating previous results. Target identification accuracy was also impaired following angry faces, compared with neutral faces, demonstrating an emotion-induced attentional bias. Importantly, this impairment was observed even when face emotion was entirely irrelevant to the participants' ongoing task (in Experiments 2 and 3), suggesting that rapid processing of the facial emotion might arise (at least in part) from the operation of relatively automatic cognitive-perceptual processes.

Learned predictiveness acquired through experience prevails over the influence of conflicting verbal instructions in rapid selective attention

Previous studies have provided evidence that selective attention tends to prioritize the processing of stimuli that are good predictors of upcoming events over nonpredictive stimuli. Moreover, studies using eye-tracking to measure attention demonstrate that this attentional bias towards predictive stimuli is at least partially under voluntary control and can be flexibly adapted via instruction. Our experiment took a similar approach to these prior studies, manipulating participants’ experience of the predictiveness of different stimuli over the course of trial-by-trial training; we then provided explicit verbal instructions regarding stimulus predictiveness that were designed to be either consistent or inconsistent with the previously established learned predictiveness. Critically, we measured the effects of training and instruction on attention to stimuli using a dot probe task, which allowed us to assess rapid shifts of attention (unlike the eye-gaze measures used in previous studies). Results revealed a rapid attentional bias towards stimuli experienced as predictive (versus those experienced as nonpredictive), that was completely unaffected by verbal instructions. This was not due to participants’ failure to recall or use instructions appropriately, as revealed by analyses of their learning about stimuli, and their memory for instructions. Overall, these findings suggest that rapid attentional biases such as those measured by the dot probe task are more strongly influenced by our prior experience during training than by our current explicit knowledge acquired via instruction.

Multiple reward-cue contingencies favor expectancy over uncertainty in shaping the reward-cue attentional salience

Reward-predicting cues attract attention because of their motivational value. A debated question regards the conditions under which the cue's attentional salience is governed more by reward expectancy rather than by reward uncertainty. To help shedding light on this relevant issue, here, we manipulated expectancy and uncertainty using three levels of reward-cue contingency, so that, for example, a high level of reward expectancy (p = .8) was compared with the highest level of reward uncertainty (p = .5). In Experiment 1, the best reward-cue during conditioning was preferentially attended in a subsequent visual search task. This result was replicated in Experiment 2, in which the cues were matched in terms of response history. In Experiment 3, we implemented a hybrid procedure consisting of two phases: an omission contingency procedure during conditioning, followed by a visual search task as in the previous experiments. Crucially, during both phases, the reward-cues were never task relevant. Results confirmed that, when multiple reward-cue contingencies are explored by a human observer, expectancy is the major factor controlling both the attentional and the oculomotor salience of the reward-cue.

The onset of uncertainty facilitates the learning of new associations by increasing attention to cues

Past research in animals has suggested that attention is distributed to exploit known relationships between stimuli and explore stimuli whose consequences are uncertain. While there is strong support for exploitative attention and its effects on learning in humans, the evidence for exploratory attention is less well developed. Two experiments examined whether preferential allocation of attention (as measured by eye-gaze) to cues associated with uncertainty leads to more rapid learning of new associations involving these cues in the future. In each experiment, participants first learned about compounds containing one predictive cue and one nonpredictive cue. The level of uncertainty during this first stage of training was also manipulated: cue-outcome relationships were either deterministic (certain) or probabilistic (uncertain). In a second stage, new cue-outcome relationships were trained and the uncertainty of these relationships could be resolved by learning about the previously nonpredictive cues. As a result of the manipulation of uncertainty in the first stage, some participants experienced a sudden onset of uncertainty at the start of this second stage, while others experienced a stable level of uncertainty throughout the experiment. Experiment 1 showed that the former learned novel cue-outcome associations faster than participants for whom uncertainty was constant. Furthermore, participants experiencing unexpected uncertainty showed a greater increase in attention to cues in Stage 2. Extending the training stage in Experiment 2 resulted in a larger difference in rate of learning between conditions in stage 2. We argue that this represents evidence for an effect of exploratory attention on rate of learning in humans.

The blocking effect in associative learning involves learned biases in rapid attentional capture

Blocking refers to the finding that less is learned about the relationship between a stimulus and an outcome if pairings are conducted in the presence of a second stimulus that has previously been established as a reliable predictor of that outcome. Attentional models of associative learning suggest that blocking reflects a reduction in the attention paid to the blocked cue. We tested this idea in three experiments in which participants were trained in an associative learning task using a blocking procedure. Attention to stimuli was measured 250 ms after onset using an adapted version of the dot probe task. This task was presented at the beginning of each learning trial (Experiments 1 and 2) or in independent trials (Experiment 3). Results show evidence of reduced attention to blocked stimuli (i.e. “attentional blocking”). In addition, this attentional bias correlated with the magnitude of blocking in associative learning, as measured by predictive-value judgments. Moreover, Experiments 2 and 3 found evidence of an influence of learning about predictiveness on memory for episodes involving stimuli. These findings are consistent with a central role of learned attentional biases in producing the blocking effect, and in the encoding of new memories.

Changes in Cue Configuration Reduce the Impact of Interfering Information in a Predictive Learning Task

Decades of research in extinction and interference show that contexts can play a critical role at disambiguating the meaning of cues that have been paired with different outcomes at different times. For instance, if a cue x is followed by outcome 1 in the first phase of an experiment and by outcome 2 in a second phase, responses to cue x tend to be consistent with outcome 2 when tested in a context similar to that of the second phase of the experiment. However, if participants are taken back to the original context of the first phase (i.e., ABA renewal) or to a completely new context (i.e., ABC or AAB renewal), their responses to x tend to be more consistent with outcome 1. Although the role of physical and temporal contexts has been well studied, other factors that can also modulate the selective retrieval of information after interference have received less attention. The present series of experiments shows how changes in cue configuration can modulate responding in a similar manner. Across five experiments using a human predictive learning task, we found that adding, removing or replacing elements from a compound cue that had undergone an interference treatment gave rise to a recovery of responding akin to that observed after context changes in AAB renewal. These results are consistent with those of previous studies exploring the effect of changes of cue configuration on interference. Taken together, these studies suggest that a change in cue configuration can have the functional properties of a context change, a finding with important implications for formal models of configural learning and for classical accounts of interference and information retrieval.