The Role of Associative History in Models of Associative Learning: A Selective Review and a Hybrid Model

"Blocking-like" effects in attentional set-shifting: Redundant cues facilitate shifting in male rats with medial prefrontal cortex inactivation

Without a functioning prefrontal cortex, humans and other animals are impaired in measures of cognitive control and behavioral flexibility, including attentional set-shifting. However, the reason for this is unclear with evidence suggesting both impaired and enhanced attentional shifting. We inhibited the medial prefrontal cortex (mPFC) of rats while they performed a modified version of an attentional set-shifting task to explore the nature of this apparent contradiction. Twelve adult male Lister hooded rats received AAV5-CaMKIIa-hM4D(Gi)-mCherry viral vector bilaterally into mPFC to express inhibitory 'Designer Receptors Exclusively Activated by Designer Drugs' (iDREADDs). The receptors were activated by systemic clozapine N-oxide (CNO) to inhibit mPFC function. The rats were tested in the standard attentional set-shifting task four times: twice after i.p. administration and twice after oral administration of vehicle or CNO (10 mg/kg). They were then tested twice in a modified task, with or without oral CNO. The modified task had an extra stage before the extradimensional shift, in which the relevant exemplars remained relevant and new exemplars that were fully predictive but redundant replaced the previous irrelevant exemplars. These exemplars then became relevant at the subsequent ED stage. In the standard task, mPFC inactivation impaired attentional set-shifting, consistent with previous findings. However, in the modified task, mPFC inactivation abolished ED shift-costs. The results support the suggestion that the mPFC is needed for the downregulation of attention that prevents learning about redundant and irrelevant stimuli. With mPFC inactivated, the rat learns more rapidly when previously redundant exemplars become the only relevant information.

Methamphetamine-induced adaptation of learning rate dynamics depend on baseline performance

The ability to calibrate learning according to new information is a fundamental component of an organism's ability to adapt to changing conditions. Yet, the exact neural mechanisms guiding dynamic learning rate adjustments remain unclear. Catecholamines appear to play a critical role in adjusting the degree to which we use new information over time, but individuals vary widely in the manner in which they adjust to changes. Here, we studied the effects of a low dose of methamphetamine (MA), and individual differences in these effects, on probabilistic reversal learning dynamics in a within-subject, double-blind, randomized design. Participants first completed a reversal learning task during a drug-free baseline session to provide a measure of baseline performance. Then they completed the task during two sessions, one with MA (20 mg oral) and one with placebo (PL). First, we showed that, relative to PL, MA modulates the ability to dynamically adjust learning from prediction errors. Second, this effect was more pronounced in participants who performed poorly at baseline. These results present novel evidence for the involvement of catecholaminergic transmission on learning flexibility and highlights that baseline performance modulates the effect of the drug.

Impairment in acquisition of conditioned fear in people with depressive symptoms

Background

Depression is one of the primary global public health issues, and there has been a dramatic increase in depression levels among young people over the past decade. The neuroplasticity theory of depression postulates that a malfunction in neural plasticity, which is responsible for learning, memory, and adaptive behavior, is the primary source of the disorder's clinical manifestations. Nevertheless, the impact of depression symptoms on associative learning remains underexplored.

Methods

We used the differential fear conditioning paradigm to investigate the effects of depressive symptoms on fear acquisition and extinction learning. Skin conductance response (SCR) is an objective evaluation indicator, and ratings of nervousness, likeability, and unconditioned stimuli (US) expectancy are subjective evaluation indicators. In addition, we used associability generated by a computational reinforcement learning model to characterize the skin conductance response.

Results

The findings indicate that individuals with depressive symptoms exhibited significant impairment in fear acquisition learning compared to those without depressive symptoms based on the results of the skin conductance response. Moreover, in the discrimination fear learning task, the skin conductance response was positively correlated with associability, as estimated by the hybrid model in the group without depressive symptoms. Additionally, the likeability rating scores improved post-extinction learning in the group without depressive symptoms, and no such increase was observed in the group with depressive symptoms.

Conclusion

The study highlights that individuals with pronounced depressive symptoms exhibit impaired fear acquisition and extinction learning, suggesting a possible deficit in associative learning. Employing the hybrid model to analyze the learning process offers a deeper insight into the associative learning processes of humans, thus allowing for improved comprehension and treatment of these mental health problems.

Bayesian reinforcement learning: A basic overview

We and other animals learn because there is some aspect of the world about which we are uncertain. This uncertainty arises from initial ignorance, and from changes in the world that we do not perfectly know; the uncertainty often becomes evident when our predictions about the world are found to be erroneous. The Rescorla-Wagner learning rule, which specifies one way that prediction errors can occasion learning, has been hugely influential as a characterization of Pavlovian conditioning and, through its equivalence to the delta rule in engineering, in a much wider class of learning problems. Here, we review the embedding of the Rescorla-Wagner rule in a Bayesian context that is precise about the link between uncertainty and learning, and thereby discuss extensions to such suggestions as the Kalman filter, structure learning, and beyond, that collectively encompass a wider range of uncertainties and accommodate a wider assortment of phenomena in conditioning.

Latent-state and model-based learning in PTSD

Post-traumatic stress disorder (PTSD) is characterized by altered emotional and behavioral responding following a traumatic event. In this article, we review the concepts of latent-state and model-based learning (i.e., learning and inferring abstract task representations) and discuss their relevance for clinical and neuroscience models of PTSD. Recent data demonstrate evidence for brain and behavioral biases in these learning processes in PTSD. These new data potentially recast excessive fear towards trauma cues as a problem in learning and updating abstract task representations, as opposed to traditional conceptualizations focused on stimulus-specific learning. Biases in latent-state and model-based learning may also be a common mechanism targeted in common therapies for PTSD. We highlight key knowledge gaps that need to be addressed to further elaborate how latent-state learning and its associated neurocircuitry mechanisms function in PTSD and how to optimize treatments to target these processes.

Neural substrates of parallel devaluation-sensitive and devaluation-insensitive Pavlovian learning in humans

We aim to differentiate the brain regions involved in the learning and encoding of Pavlovian associations sensitive to changes in outcome value from those that are not sensitive to such changes by combining a learning task with outcome devaluation, eye-tracking, and functional magnetic resonance imaging in humans. Contrary to theoretical expectation, voxels correlating with reward prediction errors in the ventral striatum and subgenual cingulate appear to be sensitive to devaluation. Moreover, regions encoding state prediction errors appear to be devaluation insensitive. We can also distinguish regions encoding predictions about outcome taste identity from predictions about expected spatial location. Regions encoding predictions about taste identity seem devaluation sensitive while those encoding predictions about an outcome's spatial location seem devaluation insensitive. These findings suggest the existence of multiple and distinct associative mechanisms in the brain and help identify putative neural correlates for the parallel expression of both devaluation sensitive and insensitive conditioned behaviors.

Value-driven attention and associative learning models: a computational simulation analysis

Value-driven attentional capture (VDAC) refers to a phenomenon by which stimulus features associated with greater reward value attract more attention than those associated with smaller reward value. To date, the majority of VDAC research has revealed that the relationship between reward history and attentional allocation follows associative learning rules. Accordingly, a mathematical implementation of associative learning models and multiple comparison between them can elucidate the underlying process and properties of VDAC. In this study, we implemented the Rescorla-Wagner, Mackintosh (Mac), Schumajuk-Pearce-Hall (SPH), and Esber-Haselgrove (EH) models to determine whether different models predict different outcomes when critical parameters in VDAC were adjusted. Simulation results were compared with experimental data from a series of VDAC studies by fitting two key model parameters, associative strength (V) and associability (α), using the Bayesian information criterion as a loss function. The results showed that SPH-V and EH- α outperformed other implementations of phenomena related to VDAC, such as expected value, training session, switching (or inertia), and uncertainty. Although V of models were sufficient to simulate VDAC when the expected value was the main manipulation of the experiment, α of models could predict additional aspects of VDAC, including uncertainty and resistance to extinction. In summary, associative learning models concur with the crucial aspects of behavioral data from VDAC experiments and elucidate underlying dynamics including novel predictions that need to be verified.

Schizotypy dimensions do not predict overshadowing

When two cues are presented together and reliably predict an outcome (AB-O1) an "overshadowing" effect is typically observed. That is, the relationship between these cues and the outcome is learned about less well than a cue presented on its own with an outcome (e.g., C - O1). The current study sought to explore the relationship between overshadowing and the positive and negative dimensions of schizotypy. A total of 256 participants completed an overshadowing procedure embedded within a causal judgement task and the Short Oxford-Liverpool Inventory of Feelings and Experiences (O-LIFE) which measured the different dimensions of schizotypy. A unilateral overshadowing effect was observed, however, none of the dimensions of schizotypy predicted the magnitude of this effect. These results are the first to demonstrate this finding using an appropriately powered sample and reveal that a tendency to experience symptoms of schizophrenia does not impact upon the overshadowing effect.

Anxiety as a disorder of uncertainty: implications for understanding maladaptive anxiety, anxious avoidance, and exposure therapy

In cognitive-behavioral conceptualizations of anxiety, exaggerated threat expectancies underlie maladaptive anxiety. This view has led to successful treatments, notably exposure therapy, but is not consistent with the empirical literature on learning and choice alterations in anxiety. Empirically, anxiety is better described as a disorder of uncertainty learning. How disruptions in uncertainty lead to impairing avoidance and are treated with exposure-based methods, however, is unclear. Here, we integrate concepts from neurocomputational learning models with clinical literature on exposure therapy to propose a new framework for understanding maladaptive uncertainty functioning in anxiety. Specifically, we propose that anxiety disorders are fundamentally disorders of uncertainty learning and that successful treatments, particularly exposure therapy, work by remediating maladaptive avoidance from dysfunctional explore/exploit decisions in uncertain, potentially aversive situations. This framework reconciles several inconsistencies in the literature and provides a path forward to better understand and treat anxiety.

The effect of prediction error on overt attention and learning in humans

It has been suggested that attention modulates the speed at which cues come to predict contingent outcomes, and that attention changes with the prediction errors generated by cues. Evidence for this interaction in humans is inconsistent, with divergent findings depending on whether attention was measured with eye fixations or learning speed. We included both measures in our experiment. Initially, predictive cues (A and B) were consistently followed by one outcome (o1), while nonpredictive cues (X and Y) were followed by two randomly alternating outcomes (o1 and o2). Consistent with an effect of prediction error, participants' fixated for longer on the nonpredictive cues than on the predictive ones. Then, the cues were combined in three pairs: AX, followed by o1, and AY and BX, followed by o2. Discrimination of AX and AY depended on the previously nonpredictive cues and, given that these received more attention during initial training, it should proceed faster than discrimination of AX and BX, which depended on the previously predictive cues. However, participants learned to predict the outcomes of AY and BX at a similar rate. The fixation times were similar for the previously predictive and previously nonpredictive cues. We discuss reasons that could explain these findings.

Two-stage reinforcement learning task predicts psychological traits

External sources of information influence human actions. However, psychological traits (PTs), considered internal variables, also play a crucial role in decision making. PTs are stable across time and contexts and define the set of behavioral repertoires that individuals express. Here, we explored how multiple metrics of adaptive behavior under uncertainty related to several PTs. Participants solved a reversal-learning task with volatile contingencies, from which we characterized a detailed behavioral profile based on their response sequences. We then tested the relationship between this multimetric behavioral profile and scores obtained from self-report psychological questionnaires. The PT measurements were based on the Hierarchical Taxonomy Of Psychopathology (HiTOP) model. By using multiple linear regression models (MLRMs), we found that the learning curves predicted important differences in the PTs and task response times. We confirmed the significance of these relationships by using random permutations of the predictors of the MLRM. Therefore, the behavioral profile configurations predicted the PTs and served as a "fingerprint" to identify participants with a high certainty level. We discuss briefly how this characterization and approach could contribute to better nosological classifications.

Latent Inhibition in Schizophrenia and Schizotypy

Background

The Salience Hypothesis posits that aberrations in the assignment of salience culminate in hallucinations and unusual beliefs, the "positive symptoms" of schizophrenia. Evidence for this comes from studies on latent inhibition (LI), referring to the phenomenon that prior exposure to a stimulus impedes learning about the relationship between that stimulus and an outcome.

Design

This article reviewed all published studies examining the relationship between LI and both schizophrenia and schizotypy.

Results

Contemporary literature suggests that LI is attenuated in both people with schizophrenia and those loading highly on measures of schizotypy, the multidimensional derivative of schizophrenia. This suggests that these individuals assign greater salience to stimuli than healthy controls and people scoring low on measures of schizotypy, respectively. However, several confounds limit these conclusions. Studies on people with schizophrenia are limited by the confounding effects of psychotropic medications, idiosyncratic parsing of samples, variation in dependent variables, and lack of statistical power. Moreover, LI paradigms are limited by the confounding effects of learned irrelevance, conditioned inhibition, negative priming, and novel pop-out effects.

Conclusions

This review concludes with the recommendation that researchers develop novel paradigms that overcome these limitations to evaluate the predictions of the Salience Hypothesis.

NMDA receptor-mediated processing in inferior frontal gyrus facilitates acquisition and extinction learning and strengthens renewal

While the renewal effect of extinction is considered to be invoked by attention to context during the extinction phase, there is also evidence that processing during initial learning (acquisition) may be important for later renewal. A noradrenergic agonist and a dopaminergic antagonist, administered before acquisition, did not affect renewal, however, the effects of NMDAergic neurotransmission in this regard are as yet unknown. In a previous study, administration of a single dose of the NMDA agonist D-cycloserine (DCS) before extinction learning facilitated extinction in the context of acquisition (AAA), but had no effect upon renewal. In the present fMRI study, DCS was administered prior to the initial acquisition of a predictive learning task, in order to investigate whether NMDA receptor (NMDAR) stimulation at this timepoint will modulate overall learning as well as the level of renewal, while increasing activation in the extinction- and renewal-relevant brain regions of inferior frontal gyrus (iFG) and hippocampus (HC). DCS facilitated acquisition, as well as extinction learning in the context of acquisition (AAA), and raised the level of ABA renewal. While BOLD activation during acquisition did not differ between treatment groups, activation in bilateral iFG showed a double dissociation during processing of AAA extinction trials, with DCS-mediated higher activation in right iFG and deactivation in left iFG. In contrast, placebo showed higher activation in left iFG and deactivation in right iFG. During the test (recall) phase, left iFG and right anterior hippocampus activation was increased in DCS participants who showed renewal, with activation in this region correlating with the ABA renewal level. The results demonstrate that NMDA receptor stimulation can facilitate both initial learning and extinction of associations, and in this way has an impact upon the resultant level of renewal. In particular NMDAergic processing in iFG appears relevant for the facilitation of AAA extinction and ABA recall in the test phase.

The human insula processes both modality-independent and pain-selective learning signals

Prediction errors (PEs) are generated when there are differences between an expected and an actual event or sensory input. The insula is a key brain region involved in pain processing, and studies have shown that the insula encodes the magnitude of an unexpected outcome (unsigned PEs). In addition to signaling this general magnitude information, PEs can give specific information on the direction of this deviation-i.e., whether an event is better or worse than expected. It is unclear whether the unsigned PE responses in the insula are selective for pain or reflective of a more general processing of aversive events irrespective of modality. It is also unknown whether the insula can process signed PEs at all. Understanding these specific mechanisms has implications for understanding how pain is processed in the brain in both health and in chronic pain conditions. In this study, 47 participants learned associations between 2 conditioned stimuli (CS) with 4 unconditioned stimuli (US; painful heat or loud sound, of one low and one high intensity each) while undergoing functional magnetic resonance imaging (fMRI) and skin conductance response (SCR) measurements. We demonstrate that activation in the anterior insula correlated with unsigned intensity PEs, irrespective of modality, indicating an unspecific aversive surprise signal. Conversely, signed intensity PE signals were modality specific, with signed PEs following pain but not sound located in the dorsal posterior insula, an area implicated in pain intensity processing. Previous studies have identified abnormal insula function and abnormal learning as potential causes of pain chronification. Our findings link these results and suggest that a misrepresentation of learning relevant PEs in the insular cortex may serve as an underlying factor in chronic pain.

Value estimation and latent-state update-related neural activity during fear conditioning predict posttraumatic stress disorder symptom severity

Learning theories of posttraumatic stress disorder (PTSD) purport that fear-learning processes, such as those that support fear acquisition and extinction, are impaired. Computational models designed to capture specific processes involved in fear learning have primarily assessed model-free, or trial-and-error, reinforcement learning (RL). Although previous studies indicated that aspects of model-free RL are disrupted among individuals with PTSD, research has yet to identify whether model-based RL, which is inferential and contextually driven, is impaired. Given empirical evidence of aberrant contextual modulation of fear in PTSD, the present study sought to identify whether model-based RL processes are altered during fear conditioning among women with interpersonal violence (IPV)-related PTSD (n = 85) using computational modeling. Model-free, hybrid, and model-based RL models were applied to skin conductance responses (SCR) collected during fear acquisition and extinction, and the model-based RL model was found to provide the best fit to the SCR data. Parameters from the model-based RL model were carried forward to neuroimaging analyses (voxel-wise and independent component analysis). Results revealed that reduced activity within visual processing regions during model-based updating uniquely predicted higher PTSD symptoms. Additionally, after controlling for model-based updating, greater value estimation encoding within the left frontoparietal network during fear acquisition and reduced value estimation encoding within the dorsomedial prefrontal cortex during fear extinction predicted greater PTSD symptoms. Results provide evidence of disrupted RL processes in women with assault-related PTSD, which may contribute to impaired fear and safety learning, and, furthermore, may relate to treatment response (e.g., poorer response to exposure therapy).

A model for learning based on the joint estimation of stochasticity and volatility

Previous research has stressed the importance of uncertainty for controlling the speed of learning, and how such control depends on the learner inferring the noise properties of the environment, especially volatility: the speed of change. However, learning rates are jointly determined by the comparison between volatility and a second factor, moment-to-moment stochasticity. Yet much previous research has focused on simplified cases corresponding to estimation of either factor alone. Here, we introduce a learning model, in which both factors are learned simultaneously from experience, and use the model to simulate human and animal data across many seemingly disparate neuroscientific and behavioral phenomena. By considering the full problem of joint estimation, we highlight a set of previously unappreciated issues, arising from the mutual interdependence of inference about volatility and stochasticity. This interdependence complicates and enriches the interpretation of previous results, such as pathological learning in individuals with anxiety and following amygdala damage.

Context, attention, and the switch between habit and goal-direction in behavior

This article reviews recent findings from the author’s laboratory that may provide new insights into how habits are made and broken. Habits are extensively practiced behaviors that are automatically evoked by antecedent cues and performed without their goal (or reinforcer) “in mind.” Goal-directed actions, in contrast, are instrumental behaviors that are performed because their goal is remembered and valued. New results suggest that actions may transition to habit after extended practice when conditions encourage reduced attention to the behavior. Consistent with theories of attention and learning, a behavior may command less attention (and become habitual) as its reinforcer becomes well-predicted by cues in the environment; habit learning is prevented if presentation of the reinforcer is uncertain. Other results suggest that habits are not permanent, and that goal-direction can be restored by several environmental manipulations, including exposure to unexpected reinforcers or context change. Habits are more context-dependent than goal-directed actions are. Habit learning causes retroactive interference in a way that is reminiscent of extinction: It inhibits, but does not erase, goal-direction in a context-dependent way. The findings have implications for the understanding of habitual and goal-directed control of behavior as well as disordered behaviors like addictions.

Reversing the relationship between a nontarget cue and the outcome facilitates subsequent human predictive learning

Two experiments were conducted to test the effect of experiencing associative interference on later learning. A predictive learning task was used in which human participants had to evaluate whether plants would grow or not (Outcome) after being watered with different fertilizers (Cues). Experiment 1 found that the increase in the prediction error produced by following a pre-exposed nontarget cue by the outcome, facilitated subsequent acquisition of the relationship between the pre-exposed target cue and the outcome. Experiment 2 compared whether learning about the target cue was differentially affected by experiencing two types of associative interference with the nontarget cue: Pairing the pre-exposed cue with the outcome and presenting the cue without outcome after being paired with it. The experience of associative interference with nontarget cues similarly facilitated subsequent learning about the target cue, regardless of the direction of the change in the nontarget cue-outcome relationship. It is suggested that the increase in prediction error produced by the experience of associative interference may lead to a general increase in attention that facilitates subsequent learning.

μ Opioid Antagonist Naltrexone Partially Abolishes the Antidepressant Placebo Effect and Reduces Orbitofrontal Cortex Encoding of Reinforcement

BACKGROUND

Like placebo analgesia, the antidepressant placebo effect appears to involve cortical and subcortical endogenous opioid signaling, yet the mechanism through which opioid release affects mood remains unclear. The orbitofrontal cortex (OFC)-which integrates various attributes of a stimulus to predict associated outcomes-has been implicated in placebo effects and is rich in μ opioid receptors. We hypothesized that naltrexone blockade of μ opioid receptors would blunt OFC-dependent antidepressant placebo effects.

METHODS

Twenty psychotropic-free patients with major depressive disorder completed a randomized, double-blind, placebo-controlled crossover study of 1 oral dose of 50 mg of naltrexone or matching placebo immediately before completing 2 sessions of the antidepressant placebo functional magnetic resonance imaging task. This task manipulates placebo-associated expectancies and their reinforcement while assessing expected and actual mood improvement.

RESULTS

Behaviorally, manipulations of antidepressant placebo expectancies and their reinforcement had positive, interactive effects on participants' expectancy and mood ratings. The high-expectancy condition recruited the dorsolateral and ventrolateral prefrontal cortex, as well as dorsal attention stream regions. Interestingly, increased dorsolateral and ventrolateral prefrontal cortex brain responses appeared to attenuate the antidepressant placebo effect. The administration of 1 oral dose of naltrexone, compared with placebo, partially abolished the interaction of the expectancy and reinforcement manipulation on mood and blocked reinforcement-induced responses in the right central OFC.

CONCLUSIONS

Our results show preliminary evidence for the role of μ opioid central OFC modulation in antidepressant placebo effects by positively biasing the value of placebo based on reinforcement and enhancing subsequent hedonic experiences.

L-DOPA modulates activity in the vmPFC, nucleus accumbens, and VTA during threat extinction learning in humans

Learning to be safe is central for adaptive behaviour when threats are no longer present. Detecting the absence of an expected threat is key for threat extinction learning and an essential process for the behavioural treatment of anxiety-related disorders. One possible mechanism underlying extinction learning is a dopaminergic mismatch signal that encodes the absence of an expected threat. Here we show that such a dopamine-related pathway underlies extinction learning in humans. Dopaminergic enhancement via administration of L-DOPA (vs. Placebo) was associated with reduced retention of differential psychophysiological threat responses at later test, which was mediated by activity in the ventromedial prefrontal cortex that was specific to extinction learning. L-DOPA administration enhanced signals at the time-point of an expected, but omitted threat in extinction learning within the nucleus accumbens, which were functionally coupled with the ventral tegmental area and the amygdala. Computational modelling of threat expectancies further revealed prediction error encoding in nucleus accumbens that was reduced when L-DOPA was administered. Our results thereby provide evidence that extinction learning is influenced by L-DOPA and provide a mechanistic perspective to augment extinction learning by dopaminergic enhancement in humans.

An attentional perspective on differential fear conditioning in chronic pain: The informational value of safety cues

Differences in fear conditioning between individuals suffering from chronic pain and healthy controls may indicate a learning bias that contributes to the acquisition and persistence of chronic pain. However, evidence from lab-controlled conditioning studies is sparse and previous experiments have produced inconsistent findings. Twenty-five participants suffering from chronic back pain and twenty-five controls not reporting chronic pain took part in a differential fear conditioning experiment measuring attention (eye tracking) and autonomic arousal (pupil dilation and skin conductance) elicited by visual cues predicting the presence or absence of electric shock. In contrast to the healthy control group, participants with chronic pain did not acquire differential autonomic responding to cues of threat and safety and specifically failed to acquire any attentional preference for the safety cue over irrelevant contextual cues (while such preference was intact for the threat cue). We present simulations of a reinforcement learning model to show how the pattern of data can be explained by assuming that participants with chronic pain might have experienced less positive emotion (relief) when the electric shock was absent following safety cues. Our model shows how this assumption can explain both, reduced differential responding to cues of threat and safety as well as less selective attention to the safety cue.

Uncertainty and predictiveness modulate attention in human predictive learning

[Correction Notice: An Erratum for this article was reported online in Journal of Experimental Psychology: General on Jan 14 2021 (see record 2021-07705-001). In the article, formatting for UK Research Councils funding was omitted. The author note and copyright line now reflect the standard acknowledgment of and formatting for the funding received for this article. All versions of this article have been corrected.] Attention determines which cues receive processing and are learned about. Learning, however, leads to attentional biases. In the study of animal learning, in some circumstances, cues that have been previously predictive of their consequences are subsequently learned about more than are nonpredictive cues, suggesting that they receive more attention. In other circumstances, cues that have previously led to uncertain consequences are learned about more than are predictive cues. In human learning, there is a clear role for predictiveness, but a role for uncertainty has been less clear. Here, in a human learning task, we show that cues that led to uncertain outcomes were subsequently learned about more than were cues that were previously predictive of their outcomes. This effect occurred when there were few uncertain cues. When the number of uncertain cues was increased, attention switched to predictive cues. This pattern of results was found for cues (1) that were uncertain because they led to 2 different outcomes equally often in a nonpredictable manner and (2) that were used in a nonlinear discrimination and were not predictive individually but were predictive in combination with other cues. This suggests that both the opposing predictiveness and uncertainty effects were determined by the relationship between individual cues and outcomes rather than the predictive strength of combined cues. These results demonstrate that learning affects attention; however, the precise nature of the effect on attention depends on the level of task complexity, which reflects a potential switch between exploration and exploitation of cues. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

The Emergence of Richly Organized Semantic Knowledge from Simple Statistics: A Synthetic Review

As adults, we draw upon our ample knowledge about the world to support such vital cognitive feats as using language, reasoning, retrieving knowledge relevant to our current goals, planning for the future, adapting to unexpected events, and navigating through the environment. Our knowledge readily supports these feats because it is not merely a collection of stored facts, but rather functions as an organized, semantic network of concepts connected by meaningful relations. How do the relations that fundamentally organize semantic concepts emerge with development? Here, we cast a spotlight on a potentially powerful but often overlooked driver of semantic organization: Rich statistical regularities that are ubiquitous in both language and visual input. In this synthetic review, we show that a driving role for statistical regularities is convergently supported by evidence from diverse fields, including computational modeling, statistical learning, and semantic development. Finally, we identify a number of key avenues of future research into how statistical regularities may drive the development of semantic organization.

Safety learning and the Pavlovian conditioned inhibition of fear in humans: Current state and future directions

Safety learning occurs when an otherwise neutral stimulus comes to signal the absence of threat, allowing organisms to use safety information to inhibit fear and anxiety in nonthreatening environments. Although it continues to emerge as a topic of relevance in biological and clinical psychology, safety learning remains inconsistently defined and under-researched. Here, we analyse the Pavlovian conditioned inhibition paradigm and its application to the study of safety learning in humans. We discuss existing studies; address outstanding theoretical considerations; and identify prospects for its further application. Though Pavlovian conditioned inhibition presents a theoretically sound model of safety learning, it has been investigated infrequently, with decade-long interims between some studies, and notable methodological variability. Consequently, we argue that the full potential of conditioned inhibition as a model for human safety learning remains untapped, and propose that it could be revisited as a framework for addressing timely questions in the behavioural and clinical sciences.

Prediction in Autism Spectrum Disorder: A Systematic Review of Empirical Evidence

According to a recent influential proposal, several phenotypic features of autism spectrum disorder (ASD) may be accounted for by differences in predictive skills between individuals with ASD and neurotypical individuals. In this systematic review, we describe results from 47 studies that have empirically tested this hypothesis. We assess the results based on two observable aspects of prediction: learning a pairing between an antecedent and a consequence and responding to an antecedent in a predictive manner. Taken together, these studies suggest distinct differences in both predictive learning and predictive response. Studies documenting differences in learning predictive pairings indicate challenges in detecting such relationships especially when predictive features of an antecedent have low salience or consistency, and studies showing differences in habituation and perceptual adaptation suggest low-level predictive processing differences in ASD. These challenges may account for the observed differences in the influence of predictive priors, in spontaneous predictive movement or gaze, and in social prediction. An important goal for future research will be to better define and constrain the broad domain-general hypothesis by testing multiple types of prediction within the same individuals. Additional promising avenues include studying prediction within naturalistic contexts and assessing the effect of prediction-based intervention on supporting functional outcomes for individuals with ASD. LAY SUMMARY: Researchers have suggested that many features of autism spectrum disorder (ASD) may be explained by differences in the prediction skills of people with ASD. We review results from 47 studies. These studies suggest that ASD may be associated with differences in the learning of predictive pairings (e.g., learning cause and effect) and in low-level predictive processing in the brain (e.g., processing repeated sounds). These findings lay the groundwork for research that can improve our understanding of ASD and inform interventions. Autism Res 2021, 14: 604-630. © 2021 International Society for Autism Research and Wiley Periodicals LLC.

Signed and unsigned reward prediction errors dynamically enhance learning and memory

Memory helps guide behavior, but which experiences from the past are prioritized? Classic models of learning posit that events associated with unpredictable outcomes as well as, paradoxically, predictable outcomes, deploy more attention and learning for those events. Here, we test reinforcement learning and subsequent memory for those events, and treat signed and unsigned reward prediction errors (RPEs), experienced at the reward-predictive cue or reward outcome, as drivers of these two seemingly contradictory signals. By fitting reinforcement learning models to behavior, we find that both RPEs contribute to learning by modulating a dynamically changing learning rate. We further characterize the effects of these RPE signals on memory and show that both signed and unsigned RPEs enhance memory, in line with midbrain dopamine and locus-coeruleus modulation of hippocampal plasticity, thereby reconciling separate findings in the literature.

A global framework for a systemic view of brain modeling

The brain is a complex system, due to the heterogeneity of its structure, the diversity of the functions in which it participates and to its reciprocal relationships with the body and the environment. A systemic description of the brain is presented here, as a contribution to developing a brain theory and as a general framework where specific models in computational neuroscience can be integrated and associated with global information flows and cognitive functions. In an enactive view, this framework integrates the fundamental organization of the brain in sensorimotor loops with the internal and the external worlds, answering four fundamental questions (what, why, where and how). Our survival-oriented definition of behavior gives a prominent role to pavlovian and instrumental conditioning, augmented during phylogeny by the specific contribution of other kinds of learning, related to semantic memory in the posterior cortex, episodic memory in the hippocampus and working memory in the frontal cortex. This framework highlights that responses can be prepared in different ways, from pavlovian reflexes and habitual behavior to deliberations for goal-directed planning and reasoning, and explains that these different kinds of responses coexist, collaborate and compete for the control of behavior. It also lays emphasis on the fact that cognition can be described as a dynamical system of interacting memories, some acting to provide information to others, to replace them when they are not efficient enough, or to help for their improvement. Describing the brain as an architecture of learning systems has also strong implications in Machine Learning. Our biologically informed view of pavlovian and instrumental conditioning can be very precious to revisit classical Reinforcement Learning and provide a basis to ensure really autonomous learning.

Hearing hooves, thinking zebras: A review of the inverse base-rate effect

People often fail to use base-rate information appropriately in decision-making. This is evident in the inverse base-rate effect, a phenomenon in which people tend to predict a rare outcome for a new and ambiguous combination of cues. While the effect was first reported in 1988, it has recently seen a renewed interest from researchers concerned with learning, attention and decision-making. However, some researchers have raised concerns that the effect arises in specific circumstances and is unlikely to provide insight into general learning and decision-making processes. In this review, we critically evaluate the evidence for and against the main explanations that have been proposed to explain the effect, and identify where this evidence is currently weak. We argue that concerns about the effect are not well supported by the data. Instead, the evidence supports the conclusion that the effect is a result of general mechanisms that provides a useful opportunity to understand the processes involved in learning and decision making. We discuss gaps in our knowledge and some promising avenues for future research, including the relevance of the effect to models of attentional change in learning, an area where the phenomenon promises to contribute new insights.

The Role of the Rodent Lateral Orbitofrontal Cortex in Simple Pavlovian Cue-Outcome Learning Depends on Training Experience

The orbitofrontal cortex (OFC) is a critical structure in the flexible control of value-based behaviors. OFC dysfunction is typically only detected when task or environmental contingencies change, against a backdrop of apparently intact initial acquisition and behavior. While intact acquisition following OFC lesions in simple Pavlovian cue-outcome conditioning is often predicted by models of OFC function, this predicted null effect has not been thoroughly investigated. Here, we test the effects of lesions and temporary muscimol inactivation of the rodent lateral OFC on the acquisition of a simple single cue-outcome relationship. Surprisingly, pretraining lesions significantly enhanced acquisition after overtraining, whereas post-training lesions and inactivation significantly impaired acquisition. This impaired acquisition to the cue reflects a disruption of behavioral control and not learning since the cue could also act as an effective blocking stimulus in an associative blocking procedure. These findings suggest that even simple cue-outcome representations acquired in the absence of OFC function are impoverished. Therefore, while OFC function is often associated with flexible behavioral control in complex environments, it is also involved in very simple Pavlovian acquisition where complex cue-outcome relationships are irrelevant to task performance.

Posttraumatic Stress Disorder and the Developing Adolescent Brain

Posttraumatic stress disorder (PTSD) in adolescents is common and debilitating. In contrast to adult PTSD, relatively little is known about the neurobiology of adolescent PTSD, nor about how current treatments may alter adolescent neurodevelopment to allow recovery from PTSD. Improving our understanding of biological mechanisms of adolescent PTSD, taken in the context of neurodevelopment, is crucial for developing novel and personalized treatment approaches. In this review, we highlight prevailing constructs of PTSD and current findings on these domains in adolescent PTSD. Notably, little data exist in adolescent PTSD for prominent adult PTSD constructs, including threat learning and attentional threat bias. Most work to date has examined general threat processing, emotion regulation, and their neural substrates. These studies suggest that adolescent PTSD, while phenomenologically similar to adult PTSD, shows unique neurodevelopmental substrates that may impair recovery but could also be targeted in the context of adolescent neuroplasticity to improve outcomes. Both cross-sectional and longitudinal data suggest abnormal frontolimbic development compared with typically developing youths, a pattern that may differ from resilient youths. Whether current treatments such as trauma-focused psychotherapy engage these targets and restore healthy neurodevelopment remains an open question. We end our review by highlighting emerging areas and knowledge gaps that could be addressed to better characterize the biology underlying adolescent PTSD. Emerging studies in computational modeling of decision making, caregiver-related transmission of traumatic stress, and other areas may offer new targets that could harness adolescent neurobehavioral plasticity to improve resilience and recovery for some of our most vulnerable youths.

A rise in prediction error increases attention to irrelevant cues

We investigated whether a sudden rise in prediction error widens an individual's focus of attention by increasing ocular fixations on cues that otherwise tend to be ignored. To this end, we used a discrimination learning task including cues that were either relevant or irrelevant for predicting the outcomes. Half of participants experienced contingency reversal once they had learned to predict the outcomes (reversal group, n = 30). The other half experienced the same contingencies throughout the task (control group, n = 30). As participants' prediction accuracy increased, they showed a decrease in the number of fixations directed to the irrelevant cues. Following contingency reversal, participants in the reversal group showed a drop in accuracy, indicating a rise in prediction error, and fixated on the irrelevant cues more often than participants in the control group. We discuss the results in the context of attentional theories of associative learning.

Reinforcement history shapes primary visual cortical responses: An SSVEP study

Efficient learning requires allocating limited attentional resources to meaningful stimuli and away from irrelevant stimuli. This prioritization may occur via covert attention, evident in the activity of the visual cortex. We used steady-state visual evoked potentials (SSVEPs) to assess whether associability-driven changes in stimulus processing were evident in visuocortical responses. Participants were trained on a learned-predictiveness protocol, whereby one stimulus on each trial accurately predicted the correct response for that trial, and the other was irrelevant. In a second phase the task was arranged so that all cues were objectively predictive. Participants' overt attention (eye gaze) was affected by each cue's reinforcement history, as was their covert attention (SSVEP responses). These biases persisted into Phase 2 when all stimuli were objectively predictive, thereby demonstrating that learned attentional processes are evident in basic sensory processing, and exert an effect on covert attention above and beyond the effects of overt gaze bias.

A systems-neuroscience model of phasic dopamine

We describe a neurobiologically informed computational model of phasic dopamine signaling to account for a wide range of findings, including many considered inconsistent with the simple reward prediction error (RPE) formalism. The central feature of this PVLV framework is a distinction between a primary value (PV) system for anticipating primary rewards (Unconditioned Stimuli [USs]), and a learned value (LV) system for learning about stimuli associated with such rewards (CSs). The LV system represents the amygdala, which drives phasic bursting in midbrain dopamine areas, while the PV system represents the ventral striatum, which drives shunting inhibition of dopamine for expected USs (via direct inhibitory projections) and phasic pausing for expected USs (via the lateral habenula). Our model accounts for data supporting the separability of these systems, including individual differences in CS-based (sign-tracking) versus US-based learning (goal-tracking). Both systems use competing opponent-processing pathways representing evidence for and against specific USs, which can explain data dissociating the processes involved in acquisition versus extinction conditioning. Further, opponent processing proved critical in accounting for the full range of conditioned inhibition phenomena, and the closely related paradigm of second-order conditioning. Finally, we show how additional separable pathways representing aversive USs, largely mirroring those for appetitive USs, also have important differences from the positive valence case, allowing the model to account for several important phenomena in aversive conditioning. Overall, accounting for all of these phenomena strongly constrains the model, thus providing a well-validated framework for understanding phasic dopamine signaling. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Uncertainty-driven regulation of learning and exploration in adolescents: A computational account

Healthy adults flexibly adapt their learning strategies to ongoing changes in uncertainty, a key feature of adaptive behaviour. However, the developmental trajectory of this ability is yet unknown, as developmental studies have not incorporated trial-to-trial variation in uncertainty in their analyses or models. To address this issue, we compared adolescents’ and adults’ trial-to-trial dynamics of uncertainty, learning rate, and exploration in two tasks that assess learning in noisy but otherwise stable environments. In an estimation task—which provides direct indices of trial-specific learning rate—both age groups reduced their learning rate over time, as self-reported uncertainty decreased. Accordingly, the estimation data in both groups was better explained by a Bayesian model with dynamic learning rate (Kalman filter) than by conventional reinforcement-learning models. Furthermore, adolescents’ learning rates asymptoted at a higher level, reflecting an over-weighting of the most recent outcome, and the estimated Kalman-filter parameters suggested that this was due to an overestimation of environmental volatility. In a choice task, both age groups became more likely to choose the higher-valued option over time, but this increase in choice accuracy was smaller in the adolescents. In contrast to the estimation task, we found no evidence for a Bayesian expectation-updating process in the choice task, suggesting that estimation and choice tasks engage different learning processes. However, our modeling results of the choice task suggested that both age groups reduced their degree of exploration over time, and that the adolescents explored overall more than the adults. Finally, age-related differences in exploration parameters from fits to the choice data were mediated by participants’ volatility parameter from fits to the estimation data. Together, these results suggest that adolescents overestimate the rate of environmental change, resulting in elevated learning rates and increased exploration, which may help understand developmental changes in learning and decision-making.

To successfully learn the value of stimuli and actions, people should take into account their current (un)certainty about these values: Learning rates and exploration should be high when one’s value estimates are highly uncertain (in the beginning of learning), and decrease over time as evidence accumulates and uncertainty decreases. Recent studies have shown that healthy adults flexibly adapt their learning strategies based on ongoing changes in uncertainty, consistent with normative learning. However, the development of this ability prior to adulthood is yet unknown, as developmental learning studies have not considered trial-to-trial changes in uncertainty. Here, we show that adolescents, as compared to adults, showed a smaller decrease in both learning rate and exploration over time. Computational modeling revealed that both of these effects were due to adolescents overestimating the amount of environmental volatility, which made them more sensitive to recent relative to older evidence. The overestimation of volatility during adolescence may represent the rapidly changing environmental demands during this developmental period, and can help understand the surge in real-life risk taking and exploratory behaviours characteristic of adolescents.

The presence of joint predictors generates conjunctive predictions

The cognitive system readily detects statistical relationships where a cue predicts a specific outcome. What is less known is how the mind generates predictions when multiple cues are presented simultaneously that predict different outcomes. Here, we examine the nature of such predictions. Specifically, we examine whether the presence of joint cues leads to conjunctive predictions that represent the overlap between the outcomes associated with the cues, or disjunctive predictions that represent the total possible outcomes. To test this, we used a visual search paradigm where participants first viewed a cue and then searched for a target in an array. Each cue predicted where the target would appear in one of the four quadrants in the array. After learning the cue-location associations, two cues were presented simultaneously, and participants searched for the target where the target now appeared in each quadrant with equal probability. We found that participants were faster to find the target in conjunctive locations over disjunctive ones upon seeing two cues (Experiment 1). This attentional prioritization was not solely driven by the smaller spatial scope of conjunctive locations (Experiment 2), and was present when two cues were presented in two feature dimensions in a single object (Experiment 3). These results were consistent with a conjunctive account, where the presence of joint cues led to a conjunctive prediction that represented the overlap of the different outcomes associated with each cue. The study sheds a new light on how the mind makes predictions in novel contexts based on existing knowledge.

A-learning: A new formulation of associative learning theory

We present a new mathematical formulation of associative learning focused on non-human animals, which we call A-learning. Building on current animal learning theory and machine learning, A-learning is composed of two learning equations, one for stimulus-response values and one for stimulus values (conditioned reinforcement). A third equation implements decision-making by mapping stimulus-response values to response probabilities. We show that A-learning can reproduce the main features of: instrumental acquisition, including the effects of signaled and unsignaled non-contingent reinforcement; Pavlovian acquisition, including higher-order conditioning, omission training, autoshaping, and differences in form between conditioned and unconditioned responses; acquisition of avoidance responses; acquisition and extinction of instrumental chains and Pavlovian higher-order conditioning; Pavlovian-to-instrumental transfer; Pavlovian and instrumental outcome revaluation effects, including insight into why these effects vary greatly with training procedures and with the proximity of a response to the reinforcer. We discuss the differences between current theory and A-learning, such as its lack of stimulus-stimulus and response-stimulus associations, and compare A-learning with other temporal-difference models from machine learning, such as Q-learning, SARSA, and the actor-critic model. We conclude that A-learning may offer a more convenient view of associative learning than current mathematical models, and point out areas that need further development.

A simple model for learning in volatile environments

Sound principles of statistical inference dictate that uncertainty shapes learning. In this work, we revisit the question of learning in volatile environments, in which both the first and second-order statistics of observations dynamically evolve over time. We propose a new model, the volatile Kalman filter (VKF), which is based on a tractable state-space model of uncertainty and extends the Kalman filter algorithm to volatile environments. The proposed model is algorithmically simple and encompasses the Kalman filter as a special case. Specifically, in addition to the error-correcting rule of Kalman filter for learning observations, the VKF learns volatility according to a second error-correcting rule. These dual updates echo and contextualize classical psychological models of learning, in particular hybrid accounts of Pearce-Hall and Rescorla-Wagner. At the computational level, compared with existing models, the VKF gives up some flexibility in the generative model to enable a more faithful approximation to exact inference. When fit to empirical data, the VKF is better behaved than alternatives and better captures human choice data in two independent datasets of probabilistic learning tasks. The proposed model provides a coherent account of learning in stable or volatile environments and has implications for decision neuroscience research.

Rescorla-Wagner Models with Sparse Dynamic Attention

The Rescorla-Wagner (R-W) model describes human associative learning by proposing that an agent updates associations between stimuli, such as events in their environment or predictive cues, proportionally to a prediction error. While this model has proven informative in experiments, it has been posited that humans selectively attend to certain cues to overcome a problem with the R-W model scaling to large cue dimensions. We formally characterize this scaling problem and provide a solution that involves limiting attention in a R-W model to a sparse set of cues. Given the universal difficulty in selecting features for prediction, sparse attention faces challenges beyond those faced by the R-W model. We demonstrate several ways in which a naive attention model can fail explain those failures and leverage that understanding to produce a Sparse Attention R-W with Inference framework (SAR-WI). The SAR-WI framework not only satisfies a constraint on the number of attended cues, it also performs as well as the R-W model on a number of natural learning tasks, can correctly infer associative strengths, and focuses attention on predictive cues while ignoring uninformative cues. Given the simplicity of proposed alterations, we hope this work informs future development and empirical validation of associative learning models that seek to incorporate sparse attention.

Behavior Stability and Individual Differences in Pavlovian Extended Conditioning

How stable and general is behavior once maximum learning is reached? To answer this question and understand post-acquisition behavior and its related individual differences, we propose a psychological principle that naturally extends associative models of Pavlovian conditioning to a dynamical oscillatory model where subjects have a greater memory capacity than usually postulated, but with greater forecast uncertainty. This results in a greater resistance to learning in the first few sessions followed by an over-optimal response peak and a sequence of progressively damped response oscillations. We detected the first peak and trough of the new learning curve in our data, but their dispersion was too large to also check the presence of oscillations with smaller amplitude. We ran an unusually long experiment with 32 rats over 3,960 trials, where we excluded habituation and other well-known phenomena as sources of variability in the subjects' performance. Using the data of this and another Pavlovian experiment by Harris et al. (2015), as an illustration of the principle we tested the theory against the basic associative single-cue Rescorla–Wagner (RW) model. We found evidence that the RW model is the best non-linear regression to data only for a minority of the subjects, while its dynamical extension can explain the almost totality of data with strong to very strong evidence. Finally, an analysis of short-scale fluctuations of individual responses showed that they are described by random white noise, in contrast with the colored-noise findings in human performance.

Cue-Evoked Dopamine Promotes Conditioned Responding during Learning

Dopamine neurons mediate the association of conditioned stimuli (CS) with reward (unconditioned stimuli, US) by signaling the discrepancy between predicted and actual reward during the US. Some theoretical models suggest that learning is also influenced by the salience or associability of the CS. A hallmark of CS associability models is that they can explain latent inhibition, i.e., the observation that novel CS are more effectively learned than familiar CS. Novel CS are known to activate dopamine neurons, but whether those responses affect associative learning has not been investigated. Here, we used fiber photometry to characterize dopamine responses to inconsequential familiar and novel stimuli. Using bidirectional optogenetic modulation during conditioning, we then show that CS-evoked dopamine promotes conditioned responses. This suggests that Pavlovian conditioning is influenced by CS dopamine, in addition to US reward prediction errors. Accordingly, the absence of dopamine responses to familiar CS might explain their slower learning in latent inhibition.

The redundancy effect is related to a lack of conditioned inhibition: Evidence from a task in which excitation and inhibition are symmetrical

Rescorla and Wagner’s model of learning describes excitation and inhibition as symmetrical opposites. However, tasks used in human causal learning experiments, such as the allergist task, generally involve learning about cues leading to the presence or absence of the outcome, which may not reflect this assumption. This is important when considering learning effects which provide a challenge to this model, such as the redundancy effect. The redundancy effect describes higher causal ratings for the blocked cue X than for the uncorrelated cue Y in the design A+/AX+/BY+/CY–, the opposite pattern to that predicted by the Rescorla–Wagner model, which predicts higher associative strength for Y than for X. Crucially, this prediction depends on cue C gaining some inhibitory associative strength. In this article, we used a task in which cues could have independent inhibitory effects on the outcome, to investigate whether a lack of inhibition was related to the redundancy effect. In Experiment 1, inhibition for C was not detected in the allergist task, supporting this possibility. Three further experiments using the alternative task showed that a lack of inhibition was related to the redundancy effect: the redundancy effect was smaller when C was rated as inhibitory. Individual variation in the strength of inhibition for C also determined the size of the redundancy effect. Given that weak inhibition was detected in the alternative scenario but not in the allergist task, we recommend carefully choosing the type of task used to investigate associative learning phenomena, as it may influence results.

Quantifying how much attention rodents allocate to motivationally-salient objects with a novel object preference test

The allocation of attention can be modulated by the emotional value of a stimulus. In order to understand the biasing influence of emotion on attention allocation further, we require an animal test of how motivational salience modulates attention. In mice, female odour triggers arousal and elicits emotional responses in males. Here, we determined the extent to which objects labelled with female odour modulated the attention of C57BL/6J male mice. Seven experiments were conducted, using a modified version of the spontaneous Novel Object Recognition task. Attention was operationalised as differential exploration time of identical objects that were labelled with either female mouse odour (O+), a non-social odour, almond odour (O^a) or not labelled with any odour (O-). In some experiments we tested trial unique (novel) objects than never carried an odour (X-). Using this novel object preference test we found that when single objects were presented, as well as when two objects were presented simultaneously (so competed with each other for attention), O+ received preferential attention compared to O-. This result was independent of whether O+ was at a novel or familiar location. When compared with O^a at a novel location, O+ at a familiar location attracted more attention. Compared to X-, O+ received more exploration only when placed at a novel location, but attention to O+ and X- was equivalent when they were placed in a familiar location. These results suggest that C57BL/6J male mice weigh up aspects of odour, object novelty and special novelty for motivational salience, and that, in some instances, female odour elicits more attention (object exploration) compared to other object properties. The findings of this study pave the way to using motivationally-significant odours to modulate the cognitive processes that give rise to differential attention to objects.

Combining error-driven models of associative learning with evidence accumulation models of decision-making

As people learn a new skill, performance changes along two fundamental dimensions: Responses become progressively faster and more accurate. In cognitive psychology, these facets of improvement have typically been addressed by separate classes of theories. Reductions in response time (RT) have usually been addressed by theories of skill acquisition, whereas increases in accuracy have been explained by associative learning theories. To date, relatively little work has examined how changes in RT relate to changes in response accuracy, and whether these changes can be accounted for quantitatively within a single theoretical framework. The current work examines joint changes in accuracy and RT in a probabilistic category learning task. We report a model-based analysis of changes in the shapes of RT distributions for different category responses at the level of individual stimuli over the course of learning. We show that changes in performance are determined solely by changes in the quality of information entering the decision process. We then develop a new model that combines an associative learning front end with a sequential sampling model of the decision process, showing that the model provides a good account of all aspects of the learning data. We conclude by discussing potential extensions of the model and future directions for theoretical development that are opened up by our findings.

A flexible and generalizable model of online latent-state learning

Many models of classical conditioning fail to describe important phenomena, notably the rapid return of fear after extinction. To address this shortfall, evidence converged on the idea that learning agents rely on latent-state inferences, i.e. an ability to index disparate associations from cues to rewards (or penalties) and infer which index (i.e. latent state) is presently active. Our goal was to develop a model of latent-state inferences that uses latent states to predict rewards from cues efficiently and that can describe behavior in a diverse set of experiments. The resulting model combines a Rescorla-Wagner rule, for which updates to associations are proportional to prediction error, with an approximate Bayesian rule, for which beliefs in latent states are proportional to prior beliefs and an approximate likelihood based on current associations. In simulation, we demonstrate the model's ability to reproduce learning effects both famously explained and not explained by the Rescorla-Wagner model, including rapid return of fear after extinction, the Hall-Pearce effect, partial reinforcement extinction effect, backwards blocking, and memory modification. Lastly, we derive our model as an online algorithm to maximum likelihood estimation, demonstrating it is an efficient approach to outcome prediction. Establishing such a framework is a key step towards quantifying normative and pathological ranges of latent-state inferences in various contexts.

The fate of redundant cues in human predictive learning: The outcome ratio effect

In four experiments, participants were shown a sequence of pairs of pictures of food and asked to predict whether each pair signalled an allergic reaction in a hypothetical patient. The pairs of pictures were used to present two simple discriminations that differed in their outcome ratio. A rich discrimination, 3AX+ BX−, involved three trials in which the compound of two foods, AX, was followed by a reaction, for every trial in which the compound BX was not followed by the outcome. A lean discrimination, CY+ 3DY− was based on the opposite outcome ratio. Upon the completion of this training, participants were asked to rate how likely an individual food would be followed by the allergic reaction. In each experiment, the rating for X was stronger than for Y. This outcome ratio effect poses a challenge for theories of learning that assume changes in associative strength are governed by a common error term, based on the significance of all the cues present on a trial. Instead, the results are consistent with the assumption that changes in associative strength are governed by an individual error term, based on the significance of a single cue.

Context dependency as a Function of Prediction Error-Based Attention

Context dependence of information has been shown to be based, at least in part, on the attention contexts received at the time of training. Recent research suggests that attention to irrelevant contexts may be a byproduct of the activation of a general exploratory attentional mechanism prompted by high prediction errors associated with situations of uncertainty. Alternatively, low prediction errors may engage an attentional mechanism of exploitation in situations in which contexts play a relevant role. A selective review discusses the potential of this approach to explain context switch effects from an attentional perspective.

Execution-based and verbal code-based stimulus-response associations: proportion manipulations reveal conflict adaptation processes in item-specific priming

Stimulus-response (S-R) associations consist of two independent components: Stimulus-classification (S-C) and stimulus-action (S-A) associations. Here, we examined whether these S-C and S-A associations were modulated by cognitive control operations. In two item-specific priming experiments, we systematically manipulated the proportion of trials in which item-specific S-C and/or S-A mappings repeated or switched between the single encoding (prime) and single retrieval (probe) instance of each stimulus (i.e., each stimulus appeared only twice). Thus, we assessed the influence of a list-level proportion switch manipulation on the strength of item-specific S-C and S-A associations. Participants responded slower and committed more errors when item-specific S-C or S-A mappings switched rather than repeated between prime and probe (i.e., S-C/S-A switch effects). S-C switch effects were larger when S-C repetitions rather than switches were frequent on the list-level. Similarly, S-A switch effects were modulated by S-A switch proportion. Most importantly, our findings rule out contingency learning and temporal learning as explanations of the observed results and point towards a conflict adaptation mechanism that selectively adapts the encoding and/or retrieval for each S-R component. Finally, we outline how cognitive control over S-R associations operates in the context of item-specific priming.