Two kinds of attention in Pavlovian conditioning: evidence for a hybrid model of learning

The effect of prediction error on overt attention and learning in humans

It has been suggested that attention modulates the speed at which cues come to predict contingent outcomes, and that attention changes with the prediction errors generated by cues. Evidence for this interaction in humans is inconsistent, with divergent findings depending on whether attention was measured with eye fixations or learning speed. We included both measures in our experiment. Initially, predictive cues (A and B) were consistently followed by one outcome (o1), while nonpredictive cues (X and Y) were followed by two randomly alternating outcomes (o1 and o2). Consistent with an effect of prediction error, participants' fixated for longer on the nonpredictive cues than on the predictive ones. Then, the cues were combined in three pairs: AX, followed by o1, and AY and BX, followed by o2. Discrimination of AX and AY depended on the previously nonpredictive cues and, given that these received more attention during initial training, it should proceed faster than discrimination of AX and BX, which depended on the previously predictive cues. However, participants learned to predict the outcomes of AY and BX at a similar rate. The fixation times were similar for the previously predictive and previously nonpredictive cues. We discuss reasons that could explain these findings.

Models of Dynamic Belief Updating in Psychosis-A Review Across Different Computational Approaches

To understand the dysfunctional mechanisms underlying maladaptive reasoning of psychosis, computational models of decision making have widely been applied over the past decade. Thereby, a particular focus has been on the degree to which beliefs are updated based on new evidence, expressed by the learning rate in computational models. Higher order beliefs about the stability of the environment can determine the attribution of meaningfulness to events that deviate from existing beliefs by interpreting these either as noise or as true systematic changes (volatility). Both, the inappropriate downplaying of important changes as noise (belief update too low) as well as the overly flexible adaptation to random events (belief update too high) were theoretically and empirically linked to symptoms of psychosis. Whereas models with fixed learning rates fail to adjust learning in reaction to dynamic changes, increasingly complex learning models have been adopted in samples with clinical and subclinical psychosis lately. These ranged from advanced reinforcement learning models, over fully Bayesian belief updating models to approximations of fully Bayesian models with hierarchical learning or change point detection algorithms. It remains difficult to draw comparisons across findings of learning alterations in psychosis modeled by different approaches e.g., the Hierarchical Gaussian Filter and change point detection. Therefore, this review aims to summarize and compare computational definitions and findings of dynamic belief updating without perceptual ambiguity in (sub)clinical psychosis across these different mathematical approaches. There was strong heterogeneity in tasks and samples. Overall, individuals with schizophrenia and delusion-proneness showed lower behavioral performance linked to failed differentiation between uninformative noise and environmental change. This was indicated by increased belief updating and an overestimation of volatility, which was associated with cognitive deficits. Correlational evidence for computational mechanisms and positive symptoms is still sparse and might diverge from the group finding of instable beliefs. Based on the reviewed studies, we highlight some aspects to be considered to advance the field with regard to task design, modeling approach, and inclusion of participants across the psychosis spectrum. Taken together, our review shows that computational psychiatry offers powerful tools to advance our mechanistic insights into the cognitive anatomy of psychotic experiences.

A model for learning based on the joint estimation of stochasticity and volatility

Previous research has stressed the importance of uncertainty for controlling the speed of learning, and how such control depends on the learner inferring the noise properties of the environment, especially volatility: the speed of change. However, learning rates are jointly determined by the comparison between volatility and a second factor, moment-to-moment stochasticity. Yet much previous research has focused on simplified cases corresponding to estimation of either factor alone. Here, we introduce a learning model, in which both factors are learned simultaneously from experience, and use the model to simulate human and animal data across many seemingly disparate neuroscientific and behavioral phenomena. By considering the full problem of joint estimation, we highlight a set of previously unappreciated issues, arising from the mutual interdependence of inference about volatility and stochasticity. This interdependence complicates and enriches the interpretation of previous results, such as pathological learning in individuals with anxiety and following amygdala damage.

Uncertainty and predictiveness modulate attention in human predictive learning

[Correction Notice: An Erratum for this article was reported online in Journal of Experimental Psychology: General on Jan 14 2021 (see record 2021-07705-001). In the article, formatting for UK Research Councils funding was omitted. The author note and copyright line now reflect the standard acknowledgment of and formatting for the funding received for this article. All versions of this article have been corrected.] Attention determines which cues receive processing and are learned about. Learning, however, leads to attentional biases. In the study of animal learning, in some circumstances, cues that have been previously predictive of their consequences are subsequently learned about more than are nonpredictive cues, suggesting that they receive more attention. In other circumstances, cues that have previously led to uncertain consequences are learned about more than are predictive cues. In human learning, there is a clear role for predictiveness, but a role for uncertainty has been less clear. Here, in a human learning task, we show that cues that led to uncertain outcomes were subsequently learned about more than were cues that were previously predictive of their outcomes. This effect occurred when there were few uncertain cues. When the number of uncertain cues was increased, attention switched to predictive cues. This pattern of results was found for cues (1) that were uncertain because they led to 2 different outcomes equally often in a nonpredictable manner and (2) that were used in a nonlinear discrimination and were not predictive individually but were predictive in combination with other cues. This suggests that both the opposing predictiveness and uncertainty effects were determined by the relationship between individual cues and outcomes rather than the predictive strength of combined cues. These results demonstrate that learning affects attention; however, the precise nature of the effect on attention depends on the level of task complexity, which reflects a potential switch between exploration and exploitation of cues. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Sex differences in the discriminative stimulus characteristics of a morphine occasion setter in rats

The current study investigated whether the stimulus effects of morphine can function as a positive and negative feature in a Pavlovian occasion setting drug discrimination preparation in male and female rats. Sprague-Dawley rats were assigned to a feature positive (FP) or feature negative (FN) training group and all received intermixed morphine (3.2 mg/kg, IP) or saline injections 15 min before 20-min daily training sessions. For FP rats, on morphine sessions, each of eight 15-s white noise (WN) presentations was followed by 4-s access to sucrose (0.01 ml, 26% w/v); on saline sessions, sucrose was withheld. FN rats learned the reverse contingency. FP discrimination was acquired somewhat sooner than FN discrimination, and females, but not males, became sensitized to the locomotor effects of morphine, which did not influence conditioned responding. Rats then entered dose generalization testing. There was no sex difference in dose generalization for FN groups (ED₅₀ 1.26 for males and 1.57 for females). Yet for FP rats, the dose response curve for females was shifted to the right compared to males (ED₅₀ 0.54 for males and 1.94 for females). FP females exhibited enhanced responding at a dose higher than that of their original training. These findings reveal the need to reassess our notions of drug stimuli that guide appropriate associative behaviours from the perspective of sex differences.

Leveraging Basic Science for the Clinic-From Bench to Bedside

IMPORTANCE

The tools and insights of behavioral neuroscience grow apace, yet their clinical application is lagging.

OBSERVATIONS

This article suggests that associative learning theory may be the algorithmic bridge to connect a burgeoning understanding of the brain with the challenges to the mind with which all clinicians and researchers are concerned.

CONCLUSIONS AND RELEVANCE

Instead of giving up, talking past one another, or resting on the laurels of face validity, a consilient and collaborative approach is suggested: visiting laboratory meetings and clinical rounds and attempting to converse in the language of behavior and cognition to better understand and ultimately treat patients.

A systems-neuroscience model of phasic dopamine

We describe a neurobiologically informed computational model of phasic dopamine signaling to account for a wide range of findings, including many considered inconsistent with the simple reward prediction error (RPE) formalism. The central feature of this PVLV framework is a distinction between a primary value (PV) system for anticipating primary rewards (Unconditioned Stimuli [USs]), and a learned value (LV) system for learning about stimuli associated with such rewards (CSs). The LV system represents the amygdala, which drives phasic bursting in midbrain dopamine areas, while the PV system represents the ventral striatum, which drives shunting inhibition of dopamine for expected USs (via direct inhibitory projections) and phasic pausing for expected USs (via the lateral habenula). Our model accounts for data supporting the separability of these systems, including individual differences in CS-based (sign-tracking) versus US-based learning (goal-tracking). Both systems use competing opponent-processing pathways representing evidence for and against specific USs, which can explain data dissociating the processes involved in acquisition versus extinction conditioning. Further, opponent processing proved critical in accounting for the full range of conditioned inhibition phenomena, and the closely related paradigm of second-order conditioning. Finally, we show how additional separable pathways representing aversive USs, largely mirroring those for appetitive USs, also have important differences from the positive valence case, allowing the model to account for several important phenomena in aversive conditioning. Overall, accounting for all of these phenomena strongly constrains the model, thus providing a well-validated framework for understanding phasic dopamine signaling. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Responding to preconditioned cues is devaluation sensitive and requires orbitofrontal cortex during cue-cue learning

The orbitofrontal cortex (OFC) is necessary for inferring value in tests of model-based reasoning, including in sensory preconditioning. This involvement could be accounted for by representation of value or by representation of broader associative structure. We recently reported neural correlates of such broader associative structure in OFC during the initial phase of sensory preconditioning (Sadacca et al., 2018). Here, we used optogenetic inhibition of OFC to test whether these correlates might be necessary for value inference during later probe testing. We found that inhibition of OFC during cue-cue learning abolished value inference during the probe test, inference subsequently shown in control rats to be sensitive to devaluation of the expected reward. These results demonstrate that OFC must be online during cue-cue learning, consistent with the argument that the correlates previously observed are not simply downstream readouts of sensory processing and instead contribute to building the associative model supporting later behavior.

Reward anticipation and punishment anticipation are instantiated in the brain via opponent mechanisms

fMRI investigations have examined the extent to which reward and punishment motivation are associated with common or opponent neural systems, but such investigations have been limited by confounding variables and methodological constraints. The present study aimed to address limitations of earlier approaches and more comprehensively evaluate the extent to which neural activation associated with reward and punishment motivation reflects opponent or shared systems. Participants completed a modified monetary incentive delay task, which involved the presentation of a cue followed by a target to which participants were required to make a speeded button press. Using a factorial design, cues indicated whether monetary reward and/or loss (i.e., cues signaled probability of reward, punishment, both, or neither) could be expected depending upon response speed. Neural analyses evaluated evidence of (a) directionally opposing effects by testing for regions of differential activation for reward and punishment anticipation, (b) mutual inhibition by testing for interactive effects of reward and punishment anticipation within a factorial design, and (c) opposing effects on shared outputs via a psychophysiological interaction analysis. Evidence supporting all three criteria for opponent systems was obtained. Collectively, present findings support conceptualizing reward and punishment motivation as opponent forces influencing brain and behavior and indicate that shared activation does not suggest the operation of a common neural mechanism instantiating reward and punishment motivation.

The development and present status of the SOP model of associative learning

The Sometimes Opponent Processes (SOP) model in its original form was especially calculated to address how expected unconditioned stimulus (US) and conditioned stimulus (CS) are rendered less effective than their novel counterparts in Pavlovian conditioning. Its several elaborations embracing the essential notion have extended the scope of the model to integrate a much greater number of phenomena of Pavlovian conditioning. Here, we trace the development of the model and add further thoughts about its extension and refinement.

The effects of extinction-aroused attention on context conditioning

Two experiments assessed the effects of extinguishing a conditioned cue on subsequent context conditioning. Each experiment used a different video-game method where sensors predicted attacking spaceships and participants responded to the sensor in a way that prepared them for the upcoming attack. In Experiment 1 extinction of a cue which signaled a spaceship-attack outcome facilitated subsequent learning when the attack occurred unsignaled. In Experiment 2 extinction of a cue facilitated subsequent learning, regardless of whether the spaceship outcome was the same or different as used in the earlier training. In neither experiment did the extinction context become inhibitory. Results are discussed in terms of current associative theories of attention and conditioning.

Oculomotor capture is influenced by expected reward value but (maybe) not predictiveness

A large body of research has shown that learning about relationships between neutral stimuli and events of significance – rewards or punishments – influences the extent to which people attend to those stimuli in the future. However, different accounts of this influence differ in terms of the critical variable that is proposed to determine learned changes in attention. We describe two experiments using eye-tracking with a rewarded visual search procedure to investigate whether attentional capture is influenced by the predictiveness of stimuli (i.e., the extent to which they provide information about upcoming events) or by their absolute associative value (i.e., the expected incentive value of the outcome that a stimulus predicts). Results demonstrated a clear influence of associative value on the likelihood that stimuli will capture eye-movements, but the evidence for a distinct influence of predictiveness was less compelling. The results of these experiments can be reconciled within a simple account under which attentional prioritization is a monotonic function of the expected, subjective value of the reward that is signalled by a stimulus.

Recovery of attention with renewal

One experiment determined the relationship between renewed associative strength and attention. Following cue1-outcome pairings in Context A, cue1 was extinguished in Context B while cue2 was conditioned. On test cue2 was chosen as a predictor of the outcome in Context B. Both cues were chosen equally often as predictors in Context A. Consistent with attributing attention to effective associative strength (as noted in a previous study), participants could locate only cue2 in Context B while both were located in Context A, regardless of having been chosen as a predictor. Attention varied as a function of both cues' associative strengths across contexts.

Prediction and Uncertainty in Associative Learning: Examining Controlled and Automatic Components of Learned Attentional Biases

It has been suggested that attention is guided by two factors that operate during associative learning: a predictiveness principle, by which attention is allocated to the best predictors of outcomes, and an uncertainty principle, by which attention is allocated to learn about the less known features of the environment. Recent studies have shown that predictiveness-driven attention can operate rapidly and in an automatic way to exploit known relationships. The corresponding characteristics of uncertainty-driven attention, on the other hand, remain unexplored. In two experiments we examined whether both predictiveness and uncertainty modulate attentional processing in an adaptation of the dot probe task. This task provides a measure of automatic orientation to cues during associative learning. The stimulus onset asynchrony of the probe display was manipulated in order to explore temporal characteristics of predictiveness- and uncertainty-driven attentional effects. Results showed that the predictive status of cues determined selective attention, with faster attentional capture to predictive than to non-predictive cues. In contrast, the level of uncertainty slowed down responses to the probe regardless of the predictive status of the cues. Both predictiveness- and uncertainty-driven attentional effects were very rapid (at 250 ms from cue onset) and were automatically activated.

Reward Draws the Eye, Uncertainty Holds the Eye: Associative Learning Modulates Distractor Interference in Visual Search

Stimuli in our sensory environment differ with respect to their physical salience but moreover may acquire motivational salience by association with reward. If we repeatedly observed that reward is available in the context of a particular cue but absent in the context of another cue the former typically attracts more attention than the latter. However, we also may encounter cues uncorrelated with reward. A cue with 50% reward contingency may induce an average reward expectancy but at the same time induces high reward uncertainty. In the current experiment we examined how both values, reward expectancy and uncertainty, affected overt attention. Two different colors were established as predictive cues for low reward and high reward respectively. A third color was followed by high reward on 50% of the trials and thus induced uncertainty. Colors then were introduced as distractors during search for a shape target, and we examined the relative potential of the color distractors to capture and hold the first fixation. We observed that capture frequency corresponded to reward expectancy while capture duration corresponded to uncertainty. The results may suggest that within trial reward expectancy is represented at an earlier time window than uncertainty.

Learned changes in outcome associability

When a cue reliably predicts an outcome, the associability of that cue will change. Associative theories of learning propose this change will persist even when the same cue is paired with a different outcome. These theories, however, do not extend the same privilege to an outcome; an outcome's learning history is deemed to have no bearing on subsequent new learning involving that outcome. Two experiments were conducted which sought to investigate this assumption inherent in these theories using a serial letter-prediction task. In both experiments, participants were exposed, in Stage 1, to a predictable outcome ('X') and an unpredictable outcome ('Z'). In Stage 2, participants were exposed to the same outcomes preceded by novel cues which were equally predictive of both outcomes. Both experiments revealed that participants' learning towards the previously predictable outcome was more rapid in Stage 2 than the previously unpredicted outcome. The implications of these results for theories of associative learning are discussed.

Active avoidance requires a serial basal amygdala to nucleus accumbens shell circuit

Freezing is a species-typical defensive reaction to conditioned threats. While the neural circuitry of aversive Pavlovian behavior has been extensively studied, less is known about the circuitry underlying more active responses to danger. Here we show that the flow of information between the basal amygdala (BA) and the nucleus accumbens (NAcc) is necessary for signaled active avoidance behavior. Rats trained to avoid shock by shuttling during an auditory conditioned stimulus showed increased expression of the activity-dependent protein c-Fos in the NAcc, specifically the shell subregion (NAccSh). Silencing neural activity in the NAccSh, but not in the adjacent NAcc core, disrupted avoidance behavior. Disconnection of the BA and the NAccSh was just as effective at disrupting avoidance behavior as bilateral NAccSh inactivations, suggesting learned avoidance behavior requires an intact BA-NAccSh circuit. Together, these data highlight an essential role for the amygdalar projection to the ventral striatum in aversively motivated actions.

Relative prediction error and protection from attentional blink in human associative learning

The relationship between predictive learning and attentional processing was investigated in two experiments. During a learning procedure participants viewed rapid serial visual presentation (RSVP) of stimuli in the context of a choice-reaction-time (CRT) task. Salient stimuli in the RSVP streams were either predictive or non-predictive for the outcome of the CRT task. Following this procedure we measured attentional blink (AB) to the predictive and non-predictive stimuli. In Experiment 1, despite the use of a large sample and checks demonstrating the validity of the learning procedure and the AB measure, we did not observe reduced AB for predictive stimuli. In contrast, in Experiment 2, where the predictive stimuli occurred alongside salient non-predictive comparison stimuli, we did find less AB for predictive than for non-predictive stimuli. Our results support an attentional model of learning in which relative prediction error is used to increase learning rates for good predictors and reduce learning rates for poor predictors and provide confirmation of the AB learning effect.

Individual Differences in Discriminatory Fear Learning under Conditions of Ambiguity: A Vulnerability Factor for Anxiety Disorders?

Complex fear learning procedures might be better suited than the common differential fear-conditioning paradigm for detecting individual differences related to vulnerability for anxiety disorders. Two such procedures are the blocking procedure and the protection-from-overshadowing procedure. Their comparison allows for the examination of discriminatory fear learning under conditions of ambiguity. The present study examined the role of individual differences in such discriminatory fear learning. We hypothesized that heightened trait anxiety would be related to a deficit in discriminatory fear learning. Participants gave US-expectancy ratings as an index for the threat value of individual CSs following blocking and protection-from-overshadowing training. The difference in threat value at test between the protected-from-overshadowing conditioned stimulus (CS) and the blocked CS was negatively correlated with scores on a self-report tension-stress scale that approximates facets of generalized anxiety disorder (GAD), the Depression Anxiety Stress Scale-Stress (DASS-S), but not with other individual difference variables. In addition, a behavioral test showed that only participants scoring high on the DASS-S avoided the protected-from-overshadowing CS. This observed deficit in discriminatory fear learning for participants with high levels of tension-stress might be an underlying mechanism for fear overgeneralization in diffuse anxiety disorders such as GAD.

Risk-responsive orbitofrontal neurons track acquired salience

Decision making is impacted by uncertainty and risk (i.e., variance). Activity in the orbitofrontal cortex, an area implicated in decision making, covaries with these quantities. However, this activity could reflect the heightened salience of situations in which multiple outcomes-reward and reward omission-are expected. To resolve these accounts, rats were trained to respond to cues predicting 100%, 67%, 33%, or 0% reward. Consistent with prior reports, some orbitofrontal neurons fired differently in anticipation of uncertain (33% and 67%) versus certain (100% and 0%) reward. However, over 90% of these neurons also fired differently prior to 100% versus 0% reward (or baseline) or prior to 33% versus 67% reward. These responses are inconsistent with risk but fit well with the representation of acquired salience linked to the sum of cue-outcome and cue-no-outcome associative strengths. These results expand our understanding of how the orbitofrontal cortex might regulate learning and behavior.

Special issue on computational models of classical conditioning guest editors' introduction

In the present special issue, the performance of current computational models of classical conditioning was evaluated under three requirements: (1) Models were to be tested against a list of previously agreed-upon phenomena; (2) the parameters were fixed across simulations; and (3) the simulations used to test the models had to be made available. These requirements resulted in three major products: (a) a list of fundamental classical-conditioning results for which there is a consensus about their reliability; (b) the necessary information to evaluate each of the models on the basis of its ordinal successes in accounting for the experimental data; and (c) a repository of computational models ready to generate simulations. We believe that the contents of this issue represent the 2012 state of the art in computational modeling of classical conditioning and provide a way to find promising avenues for future model development.

An exploration of the feature-positive effect in adult humans

Experiment 1 compared the acquisition of a feature-positive and a feature-negative discrimination in humans. In the former, an outcome was signaled by two stimuli together, but not by one of these stimuli alone. In the latter, the outcome was signaled by one stimulus alone, but not by two stimuli together. Using a within-group design, the experiment revealed that the feature-positive discrimination was acquired more readily than the feature-negative discrimination. Experiment 2 tested an explanation for these results, based on the Rescorla-Wagner theory, by examining how novel discriminations, based on a combination of a feature-positive and a feature-negative discrimination, were solved. The results did not accord with predictions from the theory. Alternative explanations for the results are considered.

Changes in attention to relevant and irrelevant stimuli during spatial learning

Rats were trained in 2 experiments to find a submerged platform that was situated in 1 of 2 of the 4 corners of a rectangular pool with a curved long wall. Different landmarks occupied 2 of the corners on every trial, and the platform was always situated near a landmark. For the place group in each experiment, the location of the platform was indicated by the shape of the pool and stimuli outside the pool (place cues), but not the landmarks within the pool. For the landmark groups, the landmarks, not the place cues, indicated where the platform could be found. During Stage 2, 2 of the place cues were relevant, and 2 of the landmarks were irrelevant, for a new discrimination. The place cues better controlled searching for the platform in the place group than in the landmark group when the place cues had initially been relevant by signaling the presence (Experiment 1) or the absence (Experiment 2) of the platform. The results show that animals pay more attention to relevant than irrelevant cues.

On the nature of CS and US representations in Pavlovian learning

A significant problem in the study of Pavlovian conditioning is characterizing the nature of the representations of events that enter into learning. This issue has been explored extensively with regard to the question of what features of the unconditioned stimulus enter into learning, but considerably less work has been directed to the question of characterizing the nature of the conditioned stimulus. This article introduces a multilayered connectionist network approach to understanding how "perceptual" or "conceptual" representations of the conditioned stimulus might emerge from conditioning and participate in various learning phenomena. The model is applied to acquired equivalence/distinctiveness of cue effects, as well as a variety of conditional discrimination learning tasks (patterning, biconditional, ambiguous occasion setting, feature discriminations). In addition, studies that have examined what aspects of the unconditioned stimulus enter into learning are also reviewed. Ultimately, it is concluded that adopting a multilayered connectionist network perspective of Pavlovian learning provides us with a richer way in which to view basic learning processes, but a number of key theoretical problems remain to be solved, particularly as they relate to the integration of what we know about the nature of the representations of conditioned and unconditioned stimuli.

Reconciling the influence of predictiveness and uncertainty on stimulus salience: a model of attention in associative learning

Theories of selective attention in associative learning posit that the salience of a cue will be high if the cue is the best available predictor of reinforcement (high predictiveness). In contrast, a different class of attentional theory stipulates that the salience of a cue will be high if the cue is an inaccurate predictor of reinforcement (high uncertainty). Evidence in support of these seemingly contradictory propositions has led to: (i) the development of hybrid attentional models that assume the coexistence of separate, predictiveness-driven and uncertainty-driven mechanisms of changes in cue salience; and (ii) a surge of interest in identifying the neural circuits underpinning these mechanisms. Here, we put forward a formal attentional model of learning that reconciles the roles of predictiveness and uncertainty in salience modification. The issues discussed are relevant to psychologists, behavioural neuroscientists and neuroeconomists investigating the roles of predictiveness and uncertainty in behaviour.