A flexible and generalizable model of online latent-state learning

Prediction error determines how memories are organized in the brain

How is new information organized in memory? According to latent state theories, this is determined by the level of surprise, or prediction error, generated by the new information: a small prediction error leads to the updating of existing memory, large prediction error leads to encoding of a new memory. We tested this idea using a protocol in which rats were first conditioned to fear a stimulus paired with shock. The stimulus was then gradually extinguished by progressively reducing the shock intensity until the stimulus was presented alone. Consistent with latent state theories, this gradual extinction protocol (small prediction errors) was better than standard extinction (large prediction errors) in producing long-term suppression of fear responses, and the benefit of gradual extinction was due to updating of the conditioning memory with information about extinction. Thus, prediction error determines how new information is organized in memory, and latent state theories adequately describe the ways in which this occurs.

Using electrodermal activity to estimate fear learning differences in anxiety: A multiverse analysis

Meta-analyses indicate differences in Pavlovian fear responses between anxious and non-anxious individuals using electrodermal activity (EDA). Recent research, however, has cast doubt on whether these effects are robust to different analytic choices. Using the multiverse approach conceived by Steegen et al. (2016), we surveyed analytic choices typically implemented in clinical fear conditioning research by conducting 1240 analyses reflecting different choice permutations. Only 1.45% of our analyses produced theoretically congruent statistically significant effects, and the strength and direction of the estimated effects varied substantially across EDA processing methods. We conclude that EDA-estimated fear learning differences are vulnerable to researcher degrees of freedom and make recommendations regarding which analytical choices should be approached with a high degree of caution.

Integrating and fragmenting memories under stress and alcohol

Stress can powerfully influence the way we form memories, particularly the extent to which they are integrated or situated within an underlying spatiotemporal and broader knowledge architecture. These different representations in turn have significant consequences for the way we use these memories to guide later behavior. Puzzlingly, although stress has historically been argued to promote fragmentation, leading to disjoint memory representations, more recent work suggests that stress can also facilitate memory binding and integration. Understanding the circumstances under which stress fosters integration will be key to resolving this discrepancy and unpacking the mechanisms by which stress can shape later behavior. Here, we examine memory integration at multiple levels: linking together the content of an individual experience, threading associations between related but distinct events, and binding an experience into a pre-existing schema or sense of causal structure. We discuss neural and cognitive mechanisms underlying each form of integration as well as findings regarding how stress, aversive learning, and negative affect can modulate each. In this analysis, we uncover that stress can indeed promote each level of integration. We also show how memory integration may apply to understanding effects of alcohol, highlighting extant clinical and preclinical findings and opportunities for further investigation. Finally, we consider the implications of integration and fragmentation for later memory-guided behavior, and the importance of understanding which type of memory representation is potentiated in order to design appropriate interventions.

Latent-state and model-based learning in PTSD

Post-traumatic stress disorder (PTSD) is characterized by altered emotional and behavioral responding following a traumatic event. In this article, we review the concepts of latent-state and model-based learning (i.e., learning and inferring abstract task representations) and discuss their relevance for clinical and neuroscience models of PTSD. Recent data demonstrate evidence for brain and behavioral biases in these learning processes in PTSD. These new data potentially recast excessive fear towards trauma cues as a problem in learning and updating abstract task representations, as opposed to traditional conceptualizations focused on stimulus-specific learning. Biases in latent-state and model-based learning may also be a common mechanism targeted in common therapies for PTSD. We highlight key knowledge gaps that need to be addressed to further elaborate how latent-state learning and its associated neurocircuitry mechanisms function in PTSD and how to optimize treatments to target these processes.

Decoding context memories for threat in large-scale neural networks

Humans are often tasked with determining the degree to which a given situation poses threat. Salient cues present during prior events help bring online memories for context, which plays an informative role in this process. However, it is relatively unknown whether and how individuals use features of the environment to retrieve context memories for threat, enabling accurate inferences about the current level of danger/threat (i.e. retrieve appropriate memory) when there is a degree of ambiguity surrounding the present context. We leveraged computational neuroscience approaches (i.e. independent component analysis and multivariate pattern analyses) to decode large-scale neural network activity patterns engaged during learning and inferring threat context during a novel functional magnetic resonance imaging task. Here, we report that individuals accurately infer threat contexts under ambiguous conditions through neural reinstatement of large-scale network activity patterns (specifically striatum, salience, and frontoparietal networks) that track the signal value of environmental cues, which, in turn, allows reinstatement of a mental representation, primarily within a ventral visual network, of the previously learned threat context. These results provide novel insight into distinct, but overlapping, neural mechanisms by which individuals may utilize prior learning to effectively make decisions about ambiguous threat-related contexts as they navigate the environment.

Placing Approach-Avoidance Conflict Within the Framework of Multi-objective Reinforcement Learning

Many psychiatric disorders are marked by impaired decision-making during an approach-avoidance conflict. Current experiments elicit approach-avoidance conflicts in bandit tasks by pairing an individual's actions with consequences that are simultaneously desirable (reward) and undesirable (harm). We frame approach-avoidance conflict tasks as a multi-objective multi-armed bandit. By defining a general decision-maker as a limiting sequence of actions, we disentangle the decision process from learning. Each decision maker can then be identified as a multi-dimensional point representing its long-term average expected outcomes, while different decision making models can be associated by the geometry of their 'feasible region', the set of all possible long term performances on a fixed task. We introduce three example decision-makers based on popular reinforcement learning models and characterize their feasible regions, including whether they can be Pareto optimal. From this perspective, we find that existing tasks are unable to distinguish between the three examples of decision-makers. We show how to design new tasks whose geometric structure can be used to better distinguish between decision-makers. These findings are expected to guide the design of approach-avoidance conflict tasks and the modeling of resulting decision-making behavior.

NMDA Receptors in the Basolateral Amygdala Complex Are Engaged for Pavlovian Fear Conditioning When an Animal's Predictions about Danger Are in Error

It is widely accepted that Pavlovian fear conditioning requires activation of NMDA receptors (NMDARs) in the basolateral amygdala complex (BLA). However, it was recently shown that activation of NMDAR in the BLA is only required for fear conditioning when danger occurs unexpectedly; it is not required for fear conditioning when danger occurs as expected. This study tested the hypothesis that NMDARs in the BLA are engaged for Pavlovian fear conditioning when an animal's predictions regarding danger are in error. In each experiment, rats (females in Experiment 1 and males in Experiments 2-5) were conditioned to fear one stimulus, S1, when it was paired with foot-shock (S1→shock), and 48 h later, a second stimulus, S2, when it was presented in sequence with the already-conditioned S1 and foot-shock (S2→S1→shock). Conditioning to S2 occurred under a BLA infusion of the NMDAR antagonist, D-AP5 or vehicle. The subsequent tests of freezing to S2 alone and S1 alone revealed that the antagonist had no effect on conditioning to S2 when the shock occurred exactly as predicted by the S1, but disrupted this conditioning when the shock occurred earlier/later than predicted by S1, or at a stronger/weaker intensity. These results imply that errors in the timing or intensity of a predicted foot-shock engage NMDARs in the BLA for Pavlovian fear conditioning. They are discussed in relation to theories which propose a role for prediction error in determining how experiences are organized in memory and how activation of NMDAR in the BLA might contribute to this organization.SIGNIFICANCE STATEMENT This study is significant in showing that prediction error determines how a new experience is encoded with respect to a past experience and, thereby, whether NMDA receptors (NMDARs) in the basolateral amygdala complex (BLA) encode the new experience. When prediction error is small (e.g., danger occurs as and when expected), the new experience is encoded together with a past experience as part of the same "mental model," and NMDAR activation in the BLA is not needed for this encoding. By contrast, when prediction error is large (e.g., danger occurs at an unexpected intensity or time), the new experience is encoded separately from the past experience as part of a new mental model, and NMDAR activation in the BLA is needed for this encoding.

Enhancing reinforcement learning models by including direct and indirect pathways improves performance on striatal dependent tasks

A major advance in understanding learning behavior stems from experiments showing that reward learning requires dopamine inputs to striatal neurons and arises from synaptic plasticity of cortico-striatal synapses. Numerous reinforcement learning models mimic this dopamine-dependent synaptic plasticity by using the reward prediction error, which resembles dopamine neuron firing, to learn the best action in response to a set of cues. Though these models can explain many facets of behavior, reproducing some types of goal-directed behavior, such as renewal and reversal, require additional model components. Here we present a reinforcement learning model, TD2Q, which better corresponds to the basal ganglia with two Q matrices, one representing direct pathway neurons (G) and another representing indirect pathway neurons (N). Unlike previous two-Q architectures, a novel and critical aspect of TD2Q is to update the G and N matrices utilizing the temporal difference reward prediction error. A best action is selected for N and G using a softmax with a reward-dependent adaptive exploration parameter, and then differences are resolved using a second selection step applied to the two action probabilities. The model is tested on a range of multi-step tasks including extinction, renewal, discrimination; switching reward probability learning; and sequence learning. Simulations show that TD2Q produces behaviors similar to rodents in choice and sequence learning tasks, and that use of the temporal difference reward prediction error is required to learn multi-step tasks. Blocking the update rule on the N matrix blocks discrimination learning, as observed experimentally. Performance in the sequence learning task is dramatically improved with two matrices. These results suggest that including additional aspects of basal ganglia physiology can improve the performance of reinforcement learning models, better reproduce animal behaviors, and provide insight as to the role of direct- and indirect-pathway striatal neurons.

Quantifying aberrant approach-avoidance conflict in psychopathology: A review of computational approaches

Making effective decisions during approach-avoidance conflict is critical in daily life. Aberrant decision-making during approach-avoidance conflict is evident in a range of psychological disorders, including anxiety, depression, trauma-related disorders, substance use disorders, and alcohol use disorders. To help clarify etiological pathways and reveal novel intervention targets, clinical research into decision-making is increasingly adopting a computational psychopathology approach. This approach uses mathematical models that can identify specific decision-making related processes that are altered in mental health disorders. In our review, we highlight foundational approach-avoidance conflict research, followed by more in-depth discussion of computational approaches that have been used to model behavior in these tasks. Specifically, we describe the computational models that have been applied to approach-avoidance conflict (e.g., drift-diffusion, active inference, and reinforcement learning models), and provide resources to guide clinical researchers who may be interested in applying computational modeling. Finally, we identify notable gaps in the current literature and potential future directions for computational approaches aimed at identifying mechanisms of approach-avoidance conflict in psychopathology.

Human generalization of internal representations through prototype learning with goal-directed attention

The world is overabundant with feature-rich information obscuring the latent causes of experience. How do people approximate the complexities of the external world with simplified internal representations that generalize to novel examples or situations? Theories suggest that internal representations could be determined by decision boundaries that discriminate between alternatives, or by distance measurements against prototypes and individual exemplars. Each provide advantages and drawbacks for generalization. We therefore developed theoretical models that leverage both discriminative and distance components to form internal representations via action-reward feedback. We then developed three latent-state learning tasks to test how humans use goal-oriented discrimination attention and prototypes/exemplar representations. The majority of participants attended to both goal-relevant discriminative features and the covariance of features within a prototype. A minority of participants relied only on the discriminative feature. Behaviour of all participants could be captured by parameterizing a model combining prototype representations with goal-oriented discriminative attention.

Value estimation and latent-state update-related neural activity during fear conditioning predict posttraumatic stress disorder symptom severity

Learning theories of posttraumatic stress disorder (PTSD) purport that fear-learning processes, such as those that support fear acquisition and extinction, are impaired. Computational models designed to capture specific processes involved in fear learning have primarily assessed model-free, or trial-and-error, reinforcement learning (RL). Although previous studies indicated that aspects of model-free RL are disrupted among individuals with PTSD, research has yet to identify whether model-based RL, which is inferential and contextually driven, is impaired. Given empirical evidence of aberrant contextual modulation of fear in PTSD, the present study sought to identify whether model-based RL processes are altered during fear conditioning among women with interpersonal violence (IPV)-related PTSD (n = 85) using computational modeling. Model-free, hybrid, and model-based RL models were applied to skin conductance responses (SCR) collected during fear acquisition and extinction, and the model-based RL model was found to provide the best fit to the SCR data. Parameters from the model-based RL model were carried forward to neuroimaging analyses (voxel-wise and independent component analysis). Results revealed that reduced activity within visual processing regions during model-based updating uniquely predicted higher PTSD symptoms. Additionally, after controlling for model-based updating, greater value estimation encoding within the left frontoparietal network during fear acquisition and reduced value estimation encoding within the dorsomedial prefrontal cortex during fear extinction predicted greater PTSD symptoms. Results provide evidence of disrupted RL processes in women with assault-related PTSD, which may contribute to impaired fear and safety learning, and, furthermore, may relate to treatment response (e.g., poorer response to exposure therapy).

Frontoparietal network activity during model-based reinforcement learning updates is reduced among adolescents with severe sexual abuse

Trauma and trauma-related disorders are characterized by impaired learning processes, including reinforcement learning (RL). Identifying which aspects of learning are altered by trauma is critical endeavor, as this may reveal key mechanisms of impairment and potential intervention targets. There are at least two types of RL that have been delineated using computational modeling: model-free and model-based RL. Although these RL processes differentially predict decision-making behavior, most research has examined the impact of trauma on model-free RL. Currently unclear whether model-based RL, which involves building abstract and nuanced representations of stimulus-outcome relationships, is impaired among individuals with a history of trauma. The present study sought to test the hypothesis of impaired model-based RL among adolescent females exposed to assaultive trauma. Participants (n = 60; 29 without a history of assault and 31 with a history of assault with and without PTSD) completed a three-arm bandit task during fMRI acquisition. Two computational models compared the degree to which participants' task behavior fit the use of a model-free versus model-based RL strategy. Although a history of assaultive trauma did not predict poorer model-based RL, greater sexual abuse severity predicted less use of model-based compared to model-free RL. Additionally, severe sexual abuse predicted less left frontoparietal network encoding of model-based RL updates. Altered model-based RL, which supports goal-directed behavior, may be an important route through which clinical impairment emerges among individuals with a history of severe sexual abuse and should be examined further in future studies.

A salience misattribution model for addictive-like behaviors

Adapting to the changing environment is a key component of optimal decision-making. Internal-models that accurately represent and selectively update from behaviorally relevant/salient stimuli may facilitate adaptive behaviors. Anterior cingulate cortex (ACC) and dopaminergic systems may produce these adaptive internal-models through selective updates from behaviorally relevant stimuli. Dysfunction of ACC and dopaminergic systems could therefore produce misaligned internal-models where updates are disproportionate to the salience of the cues. An aspect of addictive-like behaviors is reduced adaptation and, ACC and dopaminergic systems typically exhibit dysfunction in drug-dependents. We argue that ACC and dopaminergic dysfunction in dependents may produce misaligned internal-models such that drug-related stimuli are misattributed with a higher salience compared to non-drug related stimuli. Hence, drug-related rewarding stimuli generate over-weighted updates to the internal-model, while negative feedback and non-drug related rewarding stimuli generate down-weighted updates. This misaligned internal-model may therefore incorrectly reinforce maladaptive drug-related behaviors. We use the proposed framework to discuss ways behavior may be made more adaptive and how the framework may be supported or falsified experimentally.

Posttraumatic Stress Disorder and the Developing Adolescent Brain

Posttraumatic stress disorder (PTSD) in adolescents is common and debilitating. In contrast to adult PTSD, relatively little is known about the neurobiology of adolescent PTSD, nor about how current treatments may alter adolescent neurodevelopment to allow recovery from PTSD. Improving our understanding of biological mechanisms of adolescent PTSD, taken in the context of neurodevelopment, is crucial for developing novel and personalized treatment approaches. In this review, we highlight prevailing constructs of PTSD and current findings on these domains in adolescent PTSD. Notably, little data exist in adolescent PTSD for prominent adult PTSD constructs, including threat learning and attentional threat bias. Most work to date has examined general threat processing, emotion regulation, and their neural substrates. These studies suggest that adolescent PTSD, while phenomenologically similar to adult PTSD, shows unique neurodevelopmental substrates that may impair recovery but could also be targeted in the context of adolescent neuroplasticity to improve outcomes. Both cross-sectional and longitudinal data suggest abnormal frontolimbic development compared with typically developing youths, a pattern that may differ from resilient youths. Whether current treatments such as trauma-focused psychotherapy engage these targets and restore healthy neurodevelopment remains an open question. We end our review by highlighting emerging areas and knowledge gaps that could be addressed to better characterize the biology underlying adolescent PTSD. Emerging studies in computational modeling of decision making, caregiver-related transmission of traumatic stress, and other areas may offer new targets that could harness adolescent neurobehavioral plasticity to improve resilience and recovery for some of our most vulnerable youths.

Potential roles of the rodent medial prefrontal cortex in conflict resolution between multiple decision-making systems

Mammalian decision-making is mediated by the interaction of multiple, neurally and computationally separable decision systems. Having multiple systems requires a mechanism to manage conflict and converge onto the selection of singular actions. A long history of evidence has pointed to the prefrontal cortex as a central component in processing the interactions between distinct decision systems and resolving conflicts among them. In this chapter we review four theories of how that interaction might occur and identify how the medial prefrontal cortex in the rodent may be involved in each theory. We then present experimental predictions implied by the neurobiological data in the context of each theory as a starting point for future investigation of medial prefrontal cortex and decision-making.

Rescorla-Wagner Models with Sparse Dynamic Attention

The Rescorla-Wagner (R-W) model describes human associative learning by proposing that an agent updates associations between stimuli, such as events in their environment or predictive cues, proportionally to a prediction error. While this model has proven informative in experiments, it has been posited that humans selectively attend to certain cues to overcome a problem with the R-W model scaling to large cue dimensions. We formally characterize this scaling problem and provide a solution that involves limiting attention in a R-W model to a sparse set of cues. Given the universal difficulty in selecting features for prediction, sparse attention faces challenges beyond those faced by the R-W model. We demonstrate several ways in which a naive attention model can fail explain those failures and leverage that understanding to produce a Sparse Attention R-W with Inference framework (SAR-WI). The SAR-WI framework not only satisfies a constraint on the number of attended cues, it also performs as well as the R-W model on a number of natural learning tasks, can correctly infer associative strengths, and focuses attention on predictive cues while ignoring uninformative cues. Given the simplicity of proposed alterations, we hope this work informs future development and empirical validation of associative learning models that seek to incorporate sparse attention.