Explaining compound generalization in associative and causal learning through rational principles of dimensional generalization

Bayesian reinforcement learning: A basic overview

We and other animals learn because there is some aspect of the world about which we are uncertain. This uncertainty arises from initial ignorance, and from changes in the world that we do not perfectly know; the uncertainty often becomes evident when our predictions about the world are found to be erroneous. The Rescorla-Wagner learning rule, which specifies one way that prediction errors can occasion learning, has been hugely influential as a characterization of Pavlovian conditioning and, through its equivalence to the delta rule in engineering, in a much wider class of learning problems. Here, we review the embedding of the Rescorla-Wagner rule in a Bayesian context that is precise about the link between uncertainty and learning, and thereby discuss extensions to such suggestions as the Kalman filter, structure learning, and beyond, that collectively encompass a wider range of uncertainties and accommodate a wider assortment of phenomena in conditioning.

Affect-congruent attention modulates generalized reward expectations

Positive and negative affective states are respectively associated with optimistic and pessimistic expectations regarding future reward. One mechanism that might underlie these affect-related expectation biases is attention to positive- versus negative-valence features (e.g., attending to the positive reviews of a restaurant versus its expensive price). Here we tested the effects of experimentally induced positive and negative affect on feature-based attention in 120 participants completing a compound-generalization task with eye-tracking. We found that participants' reward expectations for novel compound stimuli were modulated in an affect-congruent way: positive affect induction increased reward expectations for compounds, whereas negative affect induction decreased reward expectations. Computational modelling and eye-tracking analyses each revealed that these effects were driven by affect-congruent changes in participants' allocation of attention to high- versus low-value features of compounds. These results provide mechanistic insight into a process by which affect produces biases in generalized reward expectations.

Prefrontal cortex state representations shape human credit assignment

People learn adaptively from feedback, but the rate of such learning differs drastically across individuals and contexts. Here, we examine whether this variability reflects differences in what is learned. Leveraging a neurocomputational approach that merges fMRI and an iterative reward learning task, we link the specificity of credit assignment-how well people are able to appropriately attribute outcomes to their causes-to the precision of neural codes in the prefrontal cortex (PFC). Participants credit task-relevant cues more precisely in social compared vto nonsocial contexts, a process that is mediated by high-fidelity (i.e., distinct and consistent) state representations in the PFC. Specifically, the medial PFC and orbitofrontal cortex work in concert to match the neural codes from feedback to those at choice, and the strength of these common neural codes predicts credit assignment precision. Together this work provides a window into how neural representations drive adaptive learning.

Subsampling of cues in associative learning

Theories of learning distinguish between elemental and configural stimulus processing depending on whether stimuli are processed independently or as whole configurations. Evidence for elemental processing comes from findings of summation in animals where a compound of two dissimilar stimuli is deemed to be more predictive than each stimulus alone, whereas configural processing is supported by experiments using similar stimuli in which summation is not found. However, in humans the summation effect is robust and impervious to similarity manipulations. In three experiments in human predictive learning, we show that summation can be obliterated when partially reinforced cues are added to the summands in training and tests. This lack of summation only holds when the partially reinforced cues are similar to the reinforced cues (experiment 1) and seems to depend on participants sampling only the most salient cue in each trial (experiments 2a and 2b) in a sequential visual search process. Instead of attributing our and others' instances of lack of summation to the customary idea of configural processing, we offer a formal subsampling rule that might be applied to situations in which the stimuli are hard to parse from each other.

Temporal and spatial contiguity are necessary for competition between events

Over the last 50 years, cue competition phenomena have shaped theoretical developments in animal and human learning. However, recent failures to observe competition effects in standard conditioning procedures, as well as the lengthy and ongoing debate surrounding cue competition in the spatial learning literature, have cast doubts on the generality of these phenomena. In the present study, we manipulated temporal contiguity between simultaneously trained predictors and outcomes (Experiments 1-4), and spatial contiguity between landmarks and goals in spatial learning (Supplemental Experiments 1 and 2; Experiment 5). Across different parametric variations, we observed overshadowing when temporal and spatial contiguity were strong, but no overshadowing when contiguity was weak. Thus, across temporal and spatial domains, we observed that contiguity is necessary for competition to occur, and that competition between cues presented simultaneously during learning is absent when these cues were either spatially or temporally discontiguous from the outcome. Consequently, we advance a model in which the contiguity between events is accounted for and which explains these results and reconciles the previously contradictory findings observed in spatial learning. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Reinforcement learning and its connections with neuroscience and psychology

Reinforcement learning methods have recently been very successful at performing complex sequential tasks like playing Atari games, Go and Poker. These algorithms have outperformed humans in several tasks by learning from scratch, using only scalar rewards obtained through interaction with their environment. While there certainly has been considerable independent innovation to produce such results, many core ideas in reinforcement learning are inspired by phenomena in animal learning, psychology and neuroscience. In this paper, we comprehensively review a large number of findings in both neuroscience and psychology that evidence reinforcement learning as a promising candidate for modeling learning and decision making in the brain. In doing so, we construct a mapping between various classes of modern RL algorithms and specific findings in both neurophysiological and behavioral literature. We then discuss the implications of this observed relationship between RL, neuroscience and psychology and its role in advancing research in both AI and brain science.

The effects of spatial stability and cue type on spatial learning: Implications for theories of parallel memory systems

Some theories of spatial learning predict that associative rules apply under only limited circumstances. For example, learning based on a boundary has been claimed to be immune to cue competition effects because boundary information is the basis for the formation of a cognitive map, whilst landmark learning does not involve cognitive mapping. This is referred to as the cue type hypothesis. However, it has also been claimed that cue stability is a prerequisite for the formation of a cognitive map, meaning that whichever cue type was perceived as stable would enter a cognitive map and thus be immune to cue competition, while unstable cues will be subject to cue competition, regardless of cue type. In experiments 1 and 2 we manipulated the stability of boundary and landmark cues when learning the location of two hidden goals. One goal location was constant with respect to the boundary, and the other constant with respect to the landmark cues. For both cue types, the presence of distal orientation cues provided directional information. For half the participants the landmark cues were unstable relative to the boundary and orientation cues, whereas for the remainder of the participants the boundary was unstable relative to landmarks and orientation cues. In a second stage of training, all cues remained stable so that both goal locations could be learned with respect to both landmark and boundary information. According to the cue type hypothesis, boundary information should block learning about landmarks regardless of cue stability. According to the cue stability hypothesis, however, landmarks should block learning about the boundary when the landmarks appear stable relative to the boundary. Regardless of cue type or stability the results showed reciprocal blocking, contrary to both formulations of incidental cognitive mapping. Experiment 3 established that the results of Experiments 1 and 2 could not be explained in terms of difficulty in learning certain locations with respect to different cue types. In a final experiment, following training in which both landmarks and boundary cues signalled two goal locations, a new goal location was established with respect to the landmark cues, before testing with the boundary, which had never been used to define the new goal location. The results of this novel test of the interaction between boundary and landmark cues indicated that new learning with respect to the landmark had a profound effect on navigation with respect to the boundary, counter to the predictions of incidental cognitive mapping of boundaries.

Learning with reinforcement prediction errors in a model of the Drosophila mushroom body

Effective decision making in a changing environment demands that accurate predictions are learned about decision outcomes. In Drosophila, such learning is orchestrated in part by the mushroom body, where dopamine neurons signal reinforcing stimuli to modulate plasticity presynaptic to mushroom body output neurons. Building on previous mushroom body models, in which dopamine neurons signal absolute reinforcement, we propose instead that dopamine neurons signal reinforcement prediction errors by utilising feedback reinforcement predictions from output neurons. We formulate plasticity rules that minimise prediction errors, verify that output neurons learn accurate reinforcement predictions in simulations, and postulate connectivity that explains more physiological observations than an experimentally constrained model. The constrained and augmented models reproduce a broad range of conditioning and blocking experiments, and we demonstrate that the absence of blocking does not imply the absence of prediction error dependent learning. Our results provide five predictions that can be tested using established experimental methods.

Elemental and configural threat learning bias extinction generalization

Emotional experiences often contain a multitude of details that may be represented in memory as individual elements or integrated into a single representation. How details associated with a negative emotional event are represented in memory can have important implications for extinction strategies designed to reduce emotional responses. For example, is extinguishing one cue associated with an aversive outcome sufficient to reduce learned behavior to other cues present at the time of learning that were not directly extinguished? Here, we used a between-subjects multi-day threat conditioning and extinction task to assess whether participants generalize extinction from one cue to unextinguished cues. On Day 1, one group of participants learned that a compound conditioned stimulus, composed of a tone and colored square, predicted an uncomfortable shock to the wrist (Compound group). A second group learned that the tone and square separately predicted shock (Separate group). On Day 2, participants in both groups were exposed to the tone in the absence of shocks (cue extinction). On Day 3, we tested whether extinction generalized from the extinguished to the unextinguished cue, as well as to a compound composed of both cues. Results showed that configural and elemental learning had unique and opposite effects on extinction generalization. Subjects who initially learned that a compound cue predicted shock successfully generalized extinction learning from the tone to the square, but exhibited threat relapse to the compound cue. In contrast, subjects who initially learned that each cue individually predicted shock did not generalize extinction learning from the tone to the square, but threat responses to the compound were low. These results highlight the importance of whether details of an aversive event are represented as integrated or separated memories, as these representations affect the success or limits of extinction generalization.

Human Representation Learning

The central theme of this review is the dynamic interaction between information selection and learning. We pose a fundamental question about this interaction: How do we learn what features of our experiences are worth learning about? In humans, this process depends on attention and memory, two cognitive functions that together constrain representations of the world to features that are relevant for goal attainment. Recent evidence suggests that the representations shaped by attention and memory are themselves inferred from experience with each task. We review this evidence and place it in the context of work that has explicitly characterized representation learning as statistical inference. We discuss how inference can be scaled to real-world decisions by approximating beliefs based on a small number of experiences. Finally, we highlight some implications of this inference process for human decision-making in social environments.

Fixation-pattern similarity analysis reveals adaptive changes in face-viewing strategies following aversive learning

Animals can effortlessly adapt their behavior by generalizing from past aversive experiences, allowing to avoid harm in novel situations. We studied how visual information was sampled by eye-movements during this process called fear generalization, using faces organized along a circular two-dimensional perceptual continuum. During learning, one face was conditioned to predict a harmful event, whereas the most dissimilar face stayed neutral. This introduced an adversity gradient along one specific dimension, while the other, unspecific dimension was defined solely by perceptual similarity. Aversive learning changed scanning patterns selectively along the adversity-related dimension, but not the orthogonal dimension. This effect was mainly located within the eye region of faces. Our results provide evidence for adaptive changes in viewing strategies of faces following aversive learning. This is compatible with the view that these changes serve to sample information in a way that allows discriminating between safe and adverse for a better threat prediction.

Beyond the "Conceptual Nervous System": Can computational cognitive neuroscience transform learning theory?

In the last century, learning theory has been dominated by an approach assuming that associations between hypothetical representational nodes can support the acquisition of knowledge about the environment. The similarities between this approach and connectionism did not go unnoticed to learning theorists, with many of them explicitly adopting a neural network approach in the modeling of learning phenomena. Skinner famously criticized such use of hypothetical neural structures for the explanation of behavior (the "Conceptual Nervous System"), and one aspect of his criticism has proven to be correct: theory underdetermination is a pervasive problem in cognitive modeling in general, and in associationist and connectionist models in particular. That is, models implementing two very different cognitive processes often make the exact same behavioral predictions, meaning that important theoretical questions posed by contrasting the two models remain unanswered. We show through several examples that theory underdetermination is common in the learning theory literature, affecting the solvability of some of the most important theoretical problems that have been posed in the last decades. Computational cognitive neuroscience (CCN) offers a solution to this problem, by including neurobiological constraints in computational models of behavior and cognition. Rather than simply being inspired by neural computation, CCN models are built to reflect as much as possible about the actual neural structures thought to underlie a particular behavior. They go beyond the "Conceptual Nervous System" and offer a true integration of behavioral and neural levels of analysis.

Occasion setting

Occasion setting refers to the ability of 1 stimulus, an occasion setter, to modulate the efficacy of the association between another, conditioned stimulus (CS) and an unconditioned stimulus (US) or reinforcer. Occasion setters and simple CSs are readily distinguished. For example, occasion setters are relatively immune to extinction and counterconditioning, and their combination and transfer functions differ substantially from those of simple CSs. Similarly, the acquisition of occasion setting is favored when stimuli are separated by longer intervals, by empty trace intervals, and are of different modalities, whereas the opposite conditions typically favor the acquisition of simple associations. Furthermore, the simple conditioning and occasion setting properties of a single stimulus can be independent, for example, that stimulus may simultaneously predict the occurrence of a reinforcer and indicate that another stimulus will not be reinforced. Many behavioral phenomena that are intractable to simple associative analysis are better understood within an occasion setting framework. Besides capturing the distinction between direct and modulatory control common to many arenas in neuroscience, occasion setting provides a model for the hierarchical organization of memory for events and event relations, and for contextual control more broadly. Although early lesion studies further differentiated between occasion setting and simple conditioning functions, little is known about the neurobiology of occasion setting. Modern techniques for precise manipulation and monitoring of neuronal activity in multiple brain regions are ideally suited for disentangling contributions of simple conditioning and occasion setting in associative learning. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Selective sampling and inductive inference: Drawing inferences based on observed and missing evidence

We propose and test a Bayesian model of property induction with evidence that has been selectively sampled leading to "censoring" or exclusion of potentially relevant data. A core model prediction is that identical evidence samples can lead to different patterns of inductive inference depending on the censoring mechanisms that cause some instances to be excluded. This prediction was confirmed in four experiments examining property induction following exposure to identical samples that were subject to different sampling frames. Each experiment found narrower generalization of a novel property when the sample instances were selected because they shared a common property (property sampling) than when they were selected because they belonged to the same category (category sampling). In line with model predictions, sampling frame effects were moderated by the addition of explicit negative evidence (Experiment 1), sample size (Experiment 2) and category base rates (Experiments 3-4). These data show that reasoners are sensitive to constraints on the sampling process when making property inferences; they consider both the observed evidence and the reasons why certain types of evidence has not been observed.

Holistic Reinforcement Learning: The Role of Structure and Attention

Compact representations of the environment allow humans to behave efficiently in a complex world. Reinforcement learning models capture many behavioral and neural effects but do not explain recent findings showing that structure in the environment influences learning. In parallel, Bayesian cognitive models predict how humans learn structured knowledge but do not have a clear neurobiological implementation. We propose an integration of these two model classes in which structured knowledge learned via approximate Bayesian inference acts as a source of selective attention. In turn, selective attention biases reinforcement learning towards relevant dimensions of the environment. An understanding of structure learning will help to resolve the fundamental challenge in decision science: explaining why people make the decisions they do.

Novel representations that support rule-based categorization are acquired on-the-fly during category learning

Humans learn categorization rules that are aligned with separable dimensions through a rule-based learning system, which makes learning faster and easier to generalize than categorization rules that require integration of information from different dimensions. Recent research suggests that learning to categorize objects along a completely novel dimension changes its perceptual representation, making it more separable and discriminable. Here, we asked whether such newly learned dimensions could support rule-based category learning. One group received extensive categorization training and a second group did not receive such training. Later, both groups were trained in a task that made use of the category-relevant dimension, and then tested in an analogical transfer task (Experiment 1) and a button-switch interference task (Experiment 2). We expected that only the group with extensive pre-training (with well-learned dimensional representations) would show evidence of rule-based behavior in these tasks. Surprisingly, both groups performed as expected from rule-based learning. A third experiment tested whether a single session (less than 1 h) of training in a categorization task would facilitate learning in a task requiring executive function. There was a substantial learning advantage for a group with brief pre-training with the relevant dimension. We hypothesize that extensive experience with separable dimensions is not required for rule-based category learning; rather, the rule-based system may learn representations "on the fly" that allow rule application. We discuss what kind of neurocomputational model might explain these data best.

The Effect of Outcome Probability on Generalization in Predictive Learning

Abstract. People apply what they learn from experience not only to the experienced stimuli, but also to novel stimuli. But what determines how widely people generalize what they have learned? Using a predictive learning paradigm, we examined the hypothesis that a low (vs. high) probability of an outcome following a predicting stimulus would widen generalization. In three experiments, participants learned which stimulus predicted an outcome (S+) and which stimulus did not (S−) and then indicated how much they expected the outcome after each of eight novel stimuli ranging in perceptual similarity to S+ and S−. The stimuli were rings of different sizes and the outcome was a picture of a lightning bolt. As hypothesized, a lower probability of the outcome widened generalization. That is, novel stimuli that were similar to S+ (but not to S−) produced expectations for the outcome that were as high as those associated with S+.

Cue Competition and Incidental Learning: No Blocking or Overshadowing in the Colour-Word Contingency Learning Procedure Without Instructions to Learn

Overshadowing and blocking are two important findings that are frequently used to constrain models of associative learning. Overshadowing is the finding that learning about a cue (referred to as X) is reduced when that cue is always accompanied by a second cue (referred to as A) during the learning phase (AX). Blocking is the finding that after learning a stimulus-outcome relation for one stimulus (A), learning about a second stimulus (X) is reduced when the second stimulus is always accompanied by the first stimulus (AX). It remains unclear whether overshadowing and blocking result from explicit decision processes (e.g., “I know that A predicts the outcome, so I am not sure whether X does, too”), or whether cue competition is built directly into low-level association formation processes. In that vein, the present work examined whether overshadowing and/or blocking are present in an incidental learning procedure, where the predictive stimuli (words or shapes) are irrelevant to the cover task and merely correlated with the task-relevant stimulus dimension (colour). In two large online studies, we observed no evidence for overshadowing or blocking in this setup: (a) no evidence for an overshadowing cost was observed with compound (word-shape) cues relative to single cue learning conditions, and (b) contingency learning effects for blocked stimuli did not differ from those for blocking stimuli. However, when participants were given the explicit instructions to learn contingencies, evidence for blocking and overshadowing was observed. Together, these results suggest that contingencies of blocked/overshadowed stimuli are learned incidentally, but are suppressed by explicit decision processes due to knowledge of the contingencies for the blocking/overshadowing stimuli.

Rethinking dopamine as generalized prediction error

Midbrain dopamine neurons are commonly thought to report a reward prediction error (RPE), as hypothesized by reinforcement learning (RL) theory. While this theory has been highly successful, several lines of evidence suggest that dopamine activity also encodes sensory prediction errors unrelated to reward. Here, we develop a new theory of dopamine function that embraces a broader conceptualization of prediction errors. By signalling errors in both sensory and reward predictions, dopamine supports a form of RL that lies between model-based and model-free algorithms. This account remains consistent with current canon regarding the correspondence between dopamine transients and RPEs, while also accounting for new data suggesting a role for these signals in phenomena such as sensory preconditioning and identity unblocking, which ostensibly draw upon knowledge beyond reward predictions.

Linking signal detection theory and encoding models to reveal independent neural representations from neuroimaging data

Many research questions in visual perception involve determining whether stimulus properties are represented and processed independently. In visual neuroscience, there is great interest in determining whether important object dimensions are represented independently in the brain. For example, theories of face recognition have proposed either completely or partially independent processing of identity and emotional expression. Unfortunately, most previous research has only vaguely defined what is meant by “independence,” which hinders its precise quantification and testing. This article develops a new quantitative framework that links signal detection theory from psychophysics and encoding models from computational neuroscience, focusing on a special form of independence defined in the psychophysics literature: perceptual separability. The new theory allowed us, for the first time, to precisely define separability of neural representations and to theoretically link behavioral and brain measures of separability. The framework formally specifies the relation between these different levels of perceptual and brain representation, providing the tools for a truly integrative research approach. In particular, the theory identifies exactly what valid inferences can be made about independent encoding of stimulus dimensions from the results of multivariate analyses of neuroimaging data and psychophysical studies. In addition, commonly used operational tests of independence are re-interpreted within this new theoretical framework, providing insights on their correct use and interpretation. Finally, we apply this new framework to the study of separability of brain representations of face identity and emotional expression (neutral/sad) in a human fMRI study with male and female participants.

A common question in vision research is whether certain stimulus properties, like face identity and expression, are represented and processed independently. We develop a theoretical framework that allowed us, for the first time, to link behavioral and brain measures of independence. Unlike previous approaches, our framework formally specifies the relation between these different levels of perceptual and brain representation, providing the tools for a truly integrative research approach in the study of independence. This allows to identify what kind of inferences can be made about brain representations from multivariate analyses of neuroimaging data or psychophysical studies. We apply this framework to the study of independent processing of face identity and expression.

Bayesian updating during development predicts genotypic differences in plasticity

Interactions between genotypes and environments are central to evolutionary genetics, but such interactions are typically described, rather than predicted from theory. Recent Bayesian models of development generate specific predictions about genotypic differences in developmental plasticity (changes in the value of a given trait as a result of a given experience) based on genotypic differences in the value of the trait that is expressed by naïve subjects. We used these models to make a priori predictions about the effects of an aversive olfactory conditioning regime on the response of Drosophila melanogaster larvae to the odor of ethyl acetate. As predicted, across 116 genotypes initial trait values were related to plasticity. Genotypes most strongly attracted to the odor of ethyl acetate when naïve reduced their attraction scores more as a result of the aversive training regime than those less attracted to the same odor when naïve. Thus, as predicted, the variance across genotypes in attraction scores was higher before than after the shared experience. These results support predictions generated by Bayesian models of development and indicate that such models can be successfully used to investigate how variation across genotypes in information derived from ancestors combines with personal experience to differentially affect developmental plasticity in response to specific types of experience.

Exploring the Effect of Stimulus Similarity on the Summation Effect in Causal Learning

Several contemporary models anticipate that the summation effect is modulated by the similarity between the cues forming a compound. Here, we explore this hypothesis in a series of causal learning experiments. Participants were presented with two visual cues that separately predicted a common outcome and later asked for the outcome predicted by the compound of the two cues. Similarity was varied between groups through changes in shape, spatial position, color, configuration, and rotation. In variance with the predictions of these models, we observed similar and strong levels of summation in both groups across all manipulations of similarity. The effect, however, was significantly reduced by manipulations intended to impact assumptions about the causal independence of the cues forming the compound, but this reduction was independent of stimulus similarity. These results are problematic for similarity-based models and can be more readily explained by rational approaches to causal learning.

The Successor Representation: Its Computational Logic and Neural Substrates

Reinforcement learning is the process by which an agent learns to predict long-term future reward. We now understand a great deal about the brain's reinforcement learning algorithms, but we know considerably less about the representations of states and actions over which these algorithms operate. A useful starting point is asking what kinds of representations we would want the brain to have, given the constraints on its computational architecture. Following this logic leads to the idea of the successor representation, which encodes states of the environment in terms of their predictive relationships with other states. Recent behavioral and neural studies have provided evidence for the successor representation, and computational studies have explored ways to extend the original idea. This paper reviews progress on these fronts, organizing them within a broader framework for understanding how the brain negotiates tradeoffs between efficiency and flexibility for reinforcement learning.

Neural Computations Underlying Causal Structure Learning

Behavioral evidence suggests that beliefs about causal structure constrain associative learning, determining which stimuli can enter into association, as well as the functional form of that association. Bayesian learning theory provides one mechanism by which structural beliefs can be acquired from experience, but the neural basis of this mechanism is poorly understood. We studied this question with a combination of behavioral, computational, and neuroimaging techniques. Male and female human subjects learned to predict an outcome based on cue and context stimuli while being scanned using fMRI. Using a model-based analysis of the fMRI data, we show that structure learning signals are encoded in posterior parietal cortex, lateral prefrontal cortex, and the frontal pole. These structure learning signals are distinct from associative learning signals. Moreover, representational similarity analysis and information mapping revealed that the multivariate patterns of activity in posterior parietal cortex and anterior insula encode the full posterior distribution over causal structures. Variability in the encoding of the posterior across subjects predicted variability in their subsequent behavioral performance. These results provide evidence for a neural architecture in which structure learning guides the formation of associations.SIGNIFICANCE STATEMENT Animals are able to infer the hidden structure behind causal relations between stimuli in the environment, allowing them to generalize this knowledge to stimuli they have never experienced before. A recently published computational model based on this idea provided a parsimonious account of a wide range of phenomena reported in the animal learning literature, suggesting a dedicated neural mechanism for learning this hidden structure. Here, we validate this model by measuring brain activity during a task that involves both structure learning and associative learning. We show that a distinct network of regions supports structure learning and that the neural signal corresponding to beliefs about structure predicts future behavioral performance.

Failures to replicate blocking are surprising and informative-Reply to Soto (2018)

The blocking effect has inspired numerous associative learning theories and is widely cited in the literature. We recently reported a series of 15 experiments that failed to obtain a blocking effect in rodents. On the basis of those consistent failures, we claimed that there is a lack of insight into the boundary conditions for blocking. In his commentary, Soto (2018) argued that contemporary associative learning theory does provide a specific boundary condition for the occurrence of blocking, namely the use of same- versus different-modality stimuli. Given that in 10 of our 15 experiments same-modality stimuli were used, he claims that our failure to observe a blocking effect is unsurprising. We disagree with that claim, because of theoretical, empirical, and statistical problems with his analysis. We also address 2 other possible reasons for a lack of blocking that are referred to in Soto's (2018) analysis, related to generalization and salience, and dissect the potential importance of both. Although Soto's (2018) analyses raise a number of interesting points, we see more merit in an empirically guided analysis and call for empirical testing of boundary conditions on blocking. (PsycINFO Database Record

Does mental context drift or shift?

Theories of episodic memory have generally proposed that individual memory traces are linked together by a representation of context that drifts slowly over time. Recent data challenge the notion that contextual drift is always slow and passive. In particular, changes in one's external environment or internal model induce discontinuities in memory that are reflected in sudden changes in neural activity, suggesting that context can shift abruptly. Furthermore, context change effects are sensitive to top-down goals, suggesting that contextual drift may be an active process. These findings call for revising models of the role of context in memory, in order to account for abrupt contextual shifts and the controllable nature of context change.

The computational nature of memory modification

Retrieving a memory can modify its influence on subsequent behavior. We develop a computational theory of memory modification, according to which modification of a memory trace occurs through classical associative learning, but which memory trace is eligible for modification depends on a structure learning mechanism that discovers the units of association by segmenting the stream of experience into statistically distinct clusters (latent causes). New memories are formed when the structure learning mechanism infers that a new latent cause underlies current sensory observations. By the same token, old memories are modified when old and new sensory observations are inferred to have been generated by the same latent cause. We derive this framework from probabilistic principles, and present a computational implementation. Simulations demonstrate that our model can reproduce the major experimental findings from studies of memory modification in the Pavlovian conditioning literature.

DOI:http://dx.doi.org/10.7554/eLife.23763.001

Our memories contain our expectations about the world that we can retrieve to make predictions about the future. For example, most people would expect a chocolate bar to taste good, because they have previously learned to associate chocolate with pleasure. When a surprising event occurs, such as tasting an unpalatable chocolate bar, the brain therefore faces a dilemma. Should it update the existing memory and overwrite the association between chocolate and pleasure? Or should it create an additional memory? In the latter case, the brain would form a new association between chocolate and displeasure that competes with, but does not overwrite, the original one between chocolate and pleasure.

Previous studies have shown that surprising events tend to create new memories unless the existing memory is briefly reactivated before the surprising event occurs. In other words, retrieving old memories makes them more malleable. Gershman et al. have now developed a computational model for how the brain decides whether to update an old memory or create a new one. The idea at the heart of the model is that the brain will attempt to infer what caused the surprising event. The reason the chocolate bar tastes unpalatable, for example, might be because it was old and had spoiled. Every time the brain infers a new possible cause for a surprising event, it will create an additional memory to store this new set of expectations. In the future we will know that spoiled chocolate bars taste bad.

However, if the brain cannot infer a new cause for the surprising event – because, for example, there appears to be nothing unusual about the unpalatable chocolate bar – it will instead opt to update the existing memory. The next time we buy a chocolate bar, we will have slightly lower expectations about how good it will taste. The dilemma of whether to update an existing memory or create a new one thus boils down to the question: is the surprising event the consequence of a new cause or an old one? This theory implies that retrieving a memory nudges the brain to infer that its associated cause is once again active and, since this is an old cause, it means that the memory will be eligible for updating.

Many experiments have been performed on the topic of modifying memories, but this is the first computational model that offers a unifying explanation for the results. The next step is to work out how to apply the model, which is phrased in abstract terms, to networks of neurons that are more biologically realistic.

DOI:http://dx.doi.org/10.7554/eLife.23763.002

Empirical evaluation of a virtual laboratory approach to teach lactate dehydrogenase enzyme kinetics

Background

Personalised instruction is increasingly recognised as crucial for efficacious learning today. Our seminal work delineates and elaborates on the principles, development and implementation of a specially-designed adaptive, virtual laboratory.

Aims

We strived to teach laboratory skills associated with lactate dehydrogenase (LDH) enzyme kinetics to 2nd-year biochemistry students using our adaptive learning platform. Pertinent specific aims were to:(1)

design/implement a web-based lesson to teach lactate dehydrogenase(LDH) enzyme kinetics to 2nd-year biochemistry students

(2)

determine its efficacious in improving students' comprehension of enzyme kinetics

(3)

assess their perception of its usefulness/manageability(vLab versus Conventional Tutorial)

Methods

Our tools were designed using HTML5 technology. We hosted the program on an adaptive e-learning platform (AeLP). Provisions were made to interactively impart informed laboratory skills associated with measuring LDH enzyme kinetics. A series of e-learning methods were created. Tutorials were generated for interactive teaching and assessment.

Results

The learning outcomes herein were on par with that from a conventional classroom tutorial. Student feedback showed that the majority of students found the vLab learning experience “valuable”; and the vLab format/interface “well-designed”. However, there were a few technical issues with the 1st roll-out of the platform.

Conclusions

Our pioneering effort resulted in productive learning with the vLab, with parity with that from a conventional tutorial. Our contingent discussion emphasises not only the cornerstone advantages, but also the shortcomings of the AeLP method utilised. We conclude with an astute analysis of possible extensions and applications of our methodology.

•

We developed & implemented a specially-designed adaptive virtual-laboratory [vLab].

•

Laboratory lactate dehydrogenase kinetics were taught to 2nd-year biochem students.

•

The vLab was designed using HTML5 and hosted on an adaptive e-learning platform.

•

The learning outcomes were on par with that from a conventional classroom tutorial.

Evaluation of ambiguous associations in the amygdala by learning the structure of the environment

Recognizing predictive relationships is critical for survival, but an understanding of the underlying neural mechanisms remains elusive. In particular, it is unclear how the brain distinguishes predictive relationships from spurious ones when evidence about a relationship is ambiguous, or how it computes predictions given such uncertainty. To better understand this process, we introduced ambiguity into an associative learning task by presenting aversive outcomes both in the presence and in the absence of a predictive cue. Electrophysiological and optogenetic approaches revealed that amygdala neurons directly regulated and tracked the effects of ambiguity on learning. Contrary to established accounts of associative learning, however, interference from competing associations was not required to assess an ambiguous cue-outcome contingency. Instead, animals' behavior was explained by a normative account that evaluates different models of the environment's statistical structure. These findings suggest an alternative view of amygdala circuits in resolving ambiguity during aversive learning.

Rethinking Extinction

Extinction serves as the leading theoretical framework and experimental model to describe how learned behaviors diminish through absence of anticipated reinforcement. In the past decade, extinction has moved beyond the realm of associative learning theory and behavioral experimentation in animals and has become a topic of considerable interest in the neuroscience of learning, memory, and emotion. Here, we review research and theories of extinction, both as a learning process and as a behavioral technique, and consider whether traditional understandings warrant a re-examination. We discuss the neurobiology, cognitive factors, and major computational theories, and revisit the predominant view that extinction results in new learning that interferes with expression of the original memory. Additionally, we reconsider the limitations of extinction as a technique to prevent the relapse of maladaptive behavior and discuss novel approaches, informed by contemporary theoretical advances, that augment traditional extinction methods to target and potentially alter maladaptive memories.

A Unifying Probabilistic View of Associative Learning

Two important ideas about associative learning have emerged in recent decades: (1) Animals are Bayesian learners, tracking their uncertainty about associations; and (2) animals acquire long-term reward predictions through reinforcement learning. Both of these ideas are normative, in the sense that they are derived from rational design principles. They are also descriptive, capturing a wide range of empirical phenomena that troubled earlier theories. This article describes a unifying framework encompassing Bayesian and reinforcement learning theories of associative learning. Each perspective captures a different aspect of associative learning, and their synthesis offers insight into phenomena that neither perspective can explain on its own.

Reinforcement learning in multidimensional environments relies on attention mechanisms

In recent years, ideas from the computational field of reinforcement learning have revolutionized the study of learning in the brain, famously providing new, precise theories of how dopamine affects learning in the basal ganglia. However, reinforcement learning algorithms are notorious for not scaling well to multidimensional environments, as is required for real-world learning. We hypothesized that the brain naturally reduces the dimensionality of real-world problems to only those dimensions that are relevant to predicting reward, and conducted an experiment to assess by what algorithms and with what neural mechanisms this "representation learning" process is realized in humans. Our results suggest that a bilateral attentional control network comprising the intraparietal sulcus, precuneus, and dorsolateral prefrontal cortex is involved in selecting what dimensions are relevant to the task at hand, effectively updating the task representation through trial and error. In this way, cortical attention mechanisms interact with learning in the basal ganglia to solve the "curse of dimensionality" in reinforcement learning.

Why are some dimensions integral? Testing two hypotheses through causal learning experiments

Compound generalization and dimensional generalization are traditionally studied independently by different groups of researchers, who have proposed separate theories to explain results from each area. A recent extension of Shepard's rational theory of dimensional generalization allows an explanation of data from both areas within a single framework. However, the conceptualization of dimensional integrality in this theory (the direction hypothesis) is different from that favored by Shepard in his original theory (the correlation hypothesis). Here, we report two experiments that test differential predictions of these two notions of integrality. Each experiment takes a design from compound generalization and translates it into a design for dimensional generalization by replacing discrete stimulus components with dimensional values. Experiment 1 showed that an effect analogous to summation is found in dimensional generalization with separable dimensions, but the opposite effect is found with integral dimensions. Experiment 2 showed that the analogue of a biconditional discrimination is solved faster when stimuli vary in integral dimensions than when stimuli vary in separable dimensions. These results, which are analogous to more "non-linear" processing with integral than with separable dimensions, were predicted by the direction hypothesis, but not by the correlation hypothesis. This confirms the assumptions of the unified rational theory of stimulus generalization and reveals interesting links between compound and dimensional generalization phenomena.