Statistical prediction of the future impairs episodic encoding of the present

Neural Activity Differentiates Novel and Learned Event Boundaries

People parse continuous experiences at natural breakpoints called event boundaries, which is important for understanding an environment's causal structure and for responding to uncertainty within it. However, it remains unclear how different forms of uncertainty affect the parsing of continuous experiences and how such uncertainty influences the brain's processing of ongoing events. We exposed human participants of both sexes (N = 34) to a continuous sequence of semantically meaningless images. We generated sequences from random walks through a graph that grouped images into temporal communities. After learning, we asked participants to segment another sequence at natural breakpoints (event boundaries). Participants segmented the sequence at learned transitions between communities, as well as at novel transitions, suggesting that people can segment temporally extended experiences into events based on learned structure as well as prediction error. Greater segmentation at novel boundaries was associated with enhanced parietal scalp electroencephalography (EEG) activity between 250 and 450 ms after the stimulus onset. Multivariate classification of EEG activity showed that novel and learned boundaries evoked distinct patterns of neural activity, particularly theta band power in posterior electrodes. Learning also led to distinct neural representations for stimuli within the temporal communities, while neural activity at learned boundary nodes showed predictive evidence for the adjacent community. The data show that people segment experiences at both learned and novel boundaries and suggest that learned event boundaries trigger retrieval of information about the upcoming community that could underlie anticipation of the next event in a sequence.

Timescales of learning in prefrontal cortex

The lateral prefrontal cortex (PFC) in humans and other primates is critical for immediate, goal-directed behaviour and working memory, which are classically considered distinct from the cognitive and neural circuits that support long-term learning and memory. Over the past few years, a reconsideration of this textbook perspective has emerged, in that different timescales of memory-guided behaviour are in constant interaction during the pursuit of immediate goals. Here, we will first detail how neural activity related to the shortest timescales of goal-directed behaviour (which requires maintenance of current states and goals in working memory) is sculpted by long-term knowledge and learning - that is, how the past informs present behaviour. Then, we will outline how learning across different timescales (from seconds to years) drives plasticity in the primate lateral PFC, from single neuron firing rates to mesoscale neuroimaging activity patterns. Finally, we will review how, over days and months of learning, dense local and long-range connectivity patterns in PFC facilitate longer-lasting changes in population activity by changing synaptic weights and recruiting additional neural resources to inform future behaviour. Our Review sheds light on how the machinery of plasticity in PFC circuits facilitates the integration of learned experiences across time to best guide adaptive behaviour.

Predictions enable top-down pattern separation in the macaque face-processing hierarchy

Distinguishing faces requires well distinguishable neural activity patterns. Contextual information may separate neural representations, leading to enhanced identity recognition. Here, we use functional magnetic resonance imaging to investigate how predictions derived from contextual information affect the separability of neural activity patterns in the macaque face-processing system, a 3-level processing hierarchy in ventral visual cortex. We find that in the presence of predictions, early stages of this hierarchy exhibit well separable and high-dimensional neural geometries resembling those at the top of the hierarchy. This is accompanied by a systematic shift of tuning properties from higher to lower areas, endowing lower areas with higher-order, invariant representations instead of their feedforward tuning properties. Thus, top-down signals dynamically transform neural representations of faces into separable and high-dimensional neural geometries. Our results provide evidence how predictive context transforms flexible representational spaces to optimally use the computational resources provided by cortical processing hierarchies for better and faster distinction of facial identities.

Acute Stress Effects on Statistical Learning and Episodic Memory

Stress is widely considered to negatively impact hippocampal function, thus impairing episodic memory. However, the hippocampus is not merely the seat of episodic memory. Rather, it also (via distinct circuitry) supports statistical learning. On the basis of rodent work suggesting that stress may impair the hippocampal pathway involved in episodic memory while sparing or enhancing the pathway involved in statistical learning, we developed a behavioral experiment to investigate the effects of acute stress on both episodic memory and statistical learning in humans. Participants were randomly assigned to one of three conditions: stress (socially evaluated cold pressor) immediately before learning, stress ∼15 min before learning, or no stress. In the learning task, participants viewed a series of trial-unique scenes (allowing for episodic encoding of each image) in which certain scene categories reliably followed one another (allowing for statistical learning of associations between paired categories). Memory was assessed 24 hr later to isolate stress effects on encoding/learning rather than retrieval. We found modest support for our hypothesis that acute stress can amplify statistical learning: Only participants stressed ∼15 min in advance exhibited reliable evidence of learning across multiple measures. Furthermore, stress-induced cortisol levels predicted statistical learning retention 24 hr later. In contrast, episodic memory did not differ by stress condition, although we did find preliminary evidence that acute stress promoted memory for statistically predictable information and attenuated competition between statistical and episodic encoding. Together, these findings provide initial insights into how stress may differentially modulate learning processes within the hippocampus.

Finding Pattern in the Noise: Persistent Implicit Statistical Knowledge Impacts the Processing of Unpredictable Stimuli

Humans can extract statistical regularities of the environment to predict upcoming events. Previous research recognized that implicitly acquired statistical knowledge remained persistent and continued to influence behavior even when the regularities were no longer present in the environment. Here, in an fMRI experiment, we investigated how the persistence of statistical knowledge is represented in the brain. Participants (n = 32) completed a visual, four-choice, RT task consisting of statistical regularities. Two types of blocks constantly alternated with one another throughout the task: predictable statistical regularities in one block type and unpredictable ones in the other. Participants were unaware of the statistical regularities and their changing distribution across the blocks. Yet, they acquired the statistical regularities and showed significant statistical knowledge at the behavioral level not only in the predictable blocks but also in the unpredictable ones, albeit to a smaller extent. Brain activity in a range of cortical and subcortical areas, including early visual cortex, the insula, the right inferior frontal gyrus, and the right globus pallidus/putamen contributed to the acquisition of statistical regularities. The right insula, inferior frontal gyrus, and hippocampus as well as the bilateral angular gyrus seemed to play a role in maintaining this statistical knowledge. The results altogether suggest that statistical knowledge could be exploited in a relevant, predictable context as well as transmitted to and retrieved in an irrelevant context without a predictable structure.

Hebbian learning can explain rhythmic neural entrainment to statistical regularities

In many domains, learners extract recurring units from continuous sequences. For example, in unknown languages, fluent speech is perceived as a continuous signal. Learners need to extract the underlying words from this continuous signal and then memorize them. One prominent candidate mechanism is statistical learning, whereby learners track how predictive syllables (or other items) are of one another. Syllables within the same word predict each other better than syllables straddling word boundaries. But does statistical learning lead to memories of the underlying words-or just to pairwise associations among syllables? Electrophysiological results provide the strongest evidence for the memory view. Electrophysiological responses can be time-locked to statistical word boundaries (e.g., N400s) and show rhythmic activity with a periodicity of word durations. Here, I reproduce such results with a simple Hebbian network. When exposed to statistically structured syllable sequences (and when the underlying words are not excessively long), the network activation is rhythmic with the periodicity of a word duration and activation maxima on word-final syllables. This is because word-final syllables receive more excitation from earlier syllables with which they are associated than less predictable syllables that occur earlier in words. The network is also sensitive to information whose electrophysiological correlates were used to support the encoding of ordinal positions within words. Hebbian learning can thus explain rhythmic neural activity in statistical learning tasks without any memory representations of words. Learners might thus need to rely on cues beyond statistical associations to learn the words of their native language. RESEARCH HIGHLIGHTS: Statistical learning may be utilized to identify recurring units in continuous sequences (e.g., words in fluent speech) but may not generate explicit memory for words. Exposure to statistically structured sequences leads to rhythmic activity with a period of the duration of the underlying units (e.g., words). I show that a memory-less Hebbian network model can reproduce this rhythmic neural activity as well as putative encodings of ordinal positions observed in earlier research. Direct tests are needed to establish whether statistical learning leads to declarative memories for words.

Prediction error and event segmentation in episodic memory

Organizing the continuous flow of experiences into meaningful events is a crucial prerequisite for episodic memory. Prediction error and event segmentation both play important roles in supporting the genesis of meaningful mnemonic representations of events. We review theoretical contributions discussing the relationship between prediction error and event segmentation, as well as literature on episodic memory related to prediction error and event segmentation. We discuss the extent of overlap of mechanisms underlying memory emergence through prediction error and event segmentation, with a specific focus on attention and working memory. Finally, we identify areas in research that are currently developing and suggest future directions. We provide an overview of mechanisms underlying memory formation through predictions, violations of predictions, and event segmentation.

Predictable and unpredictable deviance detection in the human hippocampus and amygdala

Our brains extract structure from the environment and form predictions given past experience. Predictive circuits have been identified in wide-spread cortical regions. However, the contribution of medial temporal structures in predictions remains under-explored. The hippocampus underlies sequence detection and is sensitive to novel stimuli, sufficient to gain access to memory, while the amygdala to novelty. Yet, their electrophysiological profiles in detecting predictable and unpredictable deviant auditory events remain unknown. Here, we hypothesized that the hippocampus would be sensitive to predictability, while the amygdala to unexpected deviance. We presented epileptic patients undergoing presurgical monitoring with standard and deviant sounds, in predictable or unpredictable contexts. Onsets of auditory responses and unpredictable deviance effects were detected earlier in the temporal cortex compared with the amygdala and hippocampus. Deviance effects in 1-20 Hz local field potentials were detected in the lateral temporal cortex, irrespective of predictability. The amygdala showed stronger deviance in the unpredictable context. Low-frequency deviance responses in the hippocampus (1-8 Hz) were observed in the predictable but not in the unpredictable context. Our results reveal a distributed network underlying the generation of auditory predictions and suggest that the neural basis of sensory predictions and prediction error signals needs to be extended.

Multiple Memory Subsystems: Reconsidering Memory in the Mind and Brain

The multiple-memory-systems framework-that distinct types of memory are supported by distinct brain systems-has guided learning and memory research for decades. However, recent work challenges the one-to-one mapping between brain structures and memory types central to this taxonomy, with key memory-related structures supporting multiple functions across substructures. Here we integrate cross-species findings in the hippocampus, striatum, and amygdala to propose an updated framework of multiple memory subsystems (MMSS). We provide evidence for two organizational principles of the MMSS theory: First, opposing memory representations are colocated in the same brain structures; second, parallel memory representations are supported by distinct structures. We discuss why this burgeoning framework has the potential to provide a useful revision of classic theories of long-term memory, what evidence is needed to further validate the framework, and how this novel perspective on memory organization may guide future research.

A neural network model of hippocampal contributions to category learning

In addition to its critical role in encoding individual episodes, the hippocampus is capable of extracting regularities across experiences. This ability is central to category learning, and a growing literature indicates that the hippocampus indeed makes important contributions to this form of learning. Using a neural network model that mirrors the anatomy of the hippocampus, we investigated the mechanisms by which the hippocampus may support novel category learning. We simulated three category learning paradigms and evaluated the network's ability to categorize and recognize specific exemplars in each. We found that the trisynaptic pathway within the hippocampus-connecting entorhinal cortex to dentate gyrus, CA3, and CA1-was critical for remembering exemplar-specific information, reflecting the rapid binding and pattern separation capabilities of this circuit. The monosynaptic pathway from entorhinal cortex to CA1, in contrast, specialized in detecting the regularities that define category structure across exemplars, supported by the use of distributed representations and a relatively slower learning rate. Together, the simulations provide an account of how the hippocampus and its constituent pathways support novel category learning.

Cue predictiveness and uncertainty determine cue representation during visual statistical learning

This study investigated how humans process probabilistic-associated information when encountering varying levels of uncertainty during implicit visual statistical learning. A novel probabilistic cueing validation paradigm was developed to probe the representation of cues with high (75% probability), medium (50%), low (25%), or zero levels of predictiveness in response to preceding targets that appeared with high (75%), medium (50%), or low (25%) transitional probabilities (TPs). Experiments 1 and 2 demonstrated a significant negative association between cue probe identification accuracy and cue predictiveness when these cues appeared after high-TP but not medium-TP or low-TP targets, establishing exploration-like cue processing triggered by lower-uncertainty rather than high-uncertainty inputs. Experiment 3 ruled out the confounding factor of probe repetition and extended this finding by demonstrating (1) enhanced representation of low-predictive and zero-predictive but not high-predictive cues across blocks after high-TP targets and (2) enhanced representation of high-predictive but not low-predictive and zero-predictive cues across blocks after low-TP targets for learners who exhibited above-chance awareness of cue-target transition. These results suggest that during implicit statistical learning, input characteristics alter cue-processing mechanisms, such that exploration-like and exploitation-like mechanisms are triggered by lower-uncertainty and higher-uncertainty cue-target sequences, respectively.

Ignoring the alternatives: The N400 is sensitive to stimulus preactivation alone

The N400 component of the event-related brain potential is a neural signal of processing difficulty. In the language domain, it is widely believed to be sensitive to the degree to which a given word or its semantic features have been preactivated in the brain based on the preceding context. However, it has also been shown that the brain often preactivates many words in parallel. It is currently unknown whether the N400 is also affected by the preactivations of alternative words other than the stimulus that is actually presented. This leaves a weak link in the derivation chain-how can we use the N400 to understand the mechanisms of preactivation if we do not know what it indexes? This study directly addresses this gap. We estimate the extent to which all words in a lexicon are preactivated in a given context using the predictions of contemporary large language models. We then directly compare two competing possibilities: that the amplitude of the N400 is sensitive only to the extent to which the stimulus is preactivated, and that it is also sensitive to the preactivation states of the alternatives. We find evidence of the former. This result allows for better grounded inferences about the mechanisms underlying the N400, lexical preactivation in the brain, and language processing more generally.

Separate but not independent: Behavioral pattern separation and statistical learning are differentially affected by aging

Our brains are capable of discriminating similar inputs (pattern separation) and rapidly generalizing across inputs (statistical learning). Are these two processes dissociable in behavior? Here, we asked whether cognitive aging affects them in a differential or parallel manner. Older and younger adults were tested on their ability to discriminate between similar trisyllabic words and to extract trisyllabic words embedded in a continuous speech stream. Older adults demonstrated intact statistical learning on an implicit, reaction time-based measure and an explicit, familiarity-based measure of learning. However, they performed poorly in discriminating similar items presented in isolation, both for episodically-encoded items and for statistically-learned regularities. These results indicate that pattern separation and statistical learning are dissociable and differentially affected by aging. The acquisition of implicit representations of statistical regularities operates robustly into old age, whereas pattern separation influences the expression of statistical learning with high representational fidelity and is subject to age-related decline.

Signal processing in the vagus nerve: Hypotheses based on new genetic and anatomical evidence

Each organism must regulate its internal state in a metabolically efficient way as it interacts in space and time with an ever-changing and only partly predictable world. Success in this endeavor is largely determined by the ongoing communication between brain and body, and the vagus nerve is a crucial structure in that dialogue. In this review, we introduce the novel hypothesis that the afferent vagus nerve is engaged in signal processing rather than just signal relay. New genetic and structural evidence of vagal afferent fiber anatomy motivates two hypotheses: (1) that sensory signals informing on the physiological state of the body compute both spatial and temporal viscerosensory features as they ascend the vagus nerve, following patterns found in other sensory architectures, such as the visual and olfactory systems; and (2) that ascending and descending signals modulate one another, calling into question the strict segregation of sensory and motor signals, respectively. Finally, we discuss several implications of our two hypotheses for understanding the role of viscerosensory signal processing in predictive energy regulation (i.e., allostasis) as well as the role of metabolic signals in memory and in disorders of prediction (e.g., mood disorders).

The effect of load on spatial statistical learning

Statistical learning (SL), the extraction of regularities embedded in the environment, is often viewed as a fundamental and effortless process. However, whether spatial SL requires resources, or it can operate in parallel to other demands, is still not clear. To examine this issue, we tested spatial SL using the standard lab experiment under concurrent demands: high- and low-cognitive load (Experiment 1) and, spatial memory load (Experiment 2) during the familiarization phase. We found that any type of high-load demands during the familiarization abolished learning. Experiment 3 compared SL under spatial low-load and no-load. We found robust learning in the no-load condition that was dramatically reduced in the low-load condition. Finally, we compared a no-load condition with a very low-load, infrequent dot-probe condition that posed minimal demands while still requiring attention to the display (Experiment 4). The results showed, once again, that any concurrent task during the familiarization phase largely impaired spatial SL. Taken together, we conclude that spatial SL requires resources, a finding that challenges the view that the extraction of spatial regularities is automatic and implicit and suggests that this fundamental learning process is not as effortless as was typically assumed. We further discuss the practical and methodological implications of these findings.

The effect of prediction error on episodic memory encoding is modulated by the outcome of the predictions

Expectations can lead to prediction errors of varying degrees depending on the extent to which the information encountered in the environment conforms with prior knowledge. While there is strong evidence on the computationally specific effects of such prediction errors on learning, relatively less evidence is available regarding their effects on episodic memory. Here, we had participants work on a task in which they learned context/object-category associations of different strengths based on the outcomes of their predictions. We then used a reinforcement learning model to derive subject-specific trial-to-trial estimates of prediction error at encoding and link it to subsequent recognition memory. Results showed that model-derived prediction errors at encoding influenced subsequent memory as a function of the outcome of participants' predictions (correct vs. incorrect). When participants correctly predicted the object category, stronger prediction errors (as a consequence of weak expectations) led to enhanced memory. In contrast, when participants incorrectly predicted the object category, stronger prediction errors (as a consequence of strong expectations) led to impaired memory. These results highlight the important moderating role of choice outcome that may be related to interactions between the hippocampal and striatal dopaminergic systems.

Competitive Neurocognitive Processes Following Bereavement

Bereavement is a common human experience that often involves significant impacts on psychological, emotional and even cognitive functioning. Though various psychological theories have been proposed to conceptualize the grief process, our current understanding of the underlying neurocognitive mechanisms of grief is limited. The present paper proposes a neurocognitive model to understand phenomena in typical grief, which links loss-related reactions to underlying learning and executive processes. We posit that the competitive relationship between the basal ganglia (BG) and circuitry involving the medial temporal lobe (MTL) underlies common cognitive experiences in grief such as a sense of "brain fog." Due to the intense stressor of bereavement, we suggest that these two systems' usually flexible interactive relationship become imbalanced. The resulting temporary dominance of either the BG or the MTL system is then manifested in perceived cognitive changes. Understanding the underlying neurocognitive mechanism in grief could inform ways to best support bereaved individuals.

Cross-Modal Facilitation of Episodic Memory by Sequential Action Execution

Throughout our lives, the actions we produce are often highly familiar and repetitive (e.g., commuting to work). However, layered upon these routine actions are novel, episodic experiences. Substantial research has shown that prior knowledge can facilitate learning of conceptually related new information. But despite the central role our behavior plays in real-world experience, it remains unclear how engagement in a familiar sequence of actions influences memory for unrelated, nonmotor information coincident with those actions. To investigate this, we had healthy young adults encode novel items while simultaneously following a sequence of actions (key presses) that was either predictable and well-learned or random. Across three experiments (N = 80 each), we found that temporal order memory, but not item memory, was significantly enhanced for novel items encoded while participants executed predictable compared with random action sequences. These results suggest that engaging in familiar behaviors during novel learning scaffolds within-event temporal memory, an essential feature of episodic experiences.

Correspondence of functional connectivity gradients across human isocortex, cerebellum, and hippocampus

Gradient mapping is an important technique to summarize high dimensional biological features as low dimensional manifold representations in exploring brain structure-function relationships at various levels of the cerebral cortex. While recent studies have characterized the major gradients of functional connectivity in several brain structures using this technique, very few have systematically examined the correspondence of such gradients across structures under a common systems-level framework. Using resting-state functional magnetic resonance imaging, here we show that the organizing principles of the isocortex, and those of the cerebellum and hippocampus in relation to the isocortex, can be described using two common functional gradients. We suggest that the similarity in functional connectivity gradients across these structures can be meaningfully interpreted within a common computational framework based on the principles of predictive processing. The present results, and the specific hypotheses that they suggest, represent an important step toward an integrative account of brain function.

Hebbian, correlational learning provides a memory-less mechanism for Statistical Learning irrespective of implementational choices: Reply to Tovar and Westermann (2022)

Statistical learning relies on detecting the frequency of co-occurrences of items and has been proposed to be crucial for a variety of learning problems, notably to learn and memorize words from fluent speech. Endress and Johnson (2021) (hereafter EJ) recently showed that such results can be explained based on simple memory-less correlational learning mechanisms such as Hebbian Learning. Tovar and Westermann (2022) (hereafter TW) reproduced these results with a different Hebbian model. We show that the main differences between the models are whether temporal decay acts on both the connection weights and the activations (in TW) or only on the activations (in EJ), and whether interference affects weights (in TW) or activations (in EJ). Given that weights and activations are linked through the Hebbian learning rule, the networks behave similarly. However, in contrast to TW, we do not believe that neurophysiological data are relevant to adjudicate between abstract psychological models with little biological detail. Taken together, both models show that different memory-less correlational learning mechanisms provide a parsimonious account of Statistical Learning results. They are consistent with evidence that Statistical Learning might not allow learners to learn and retain words, and Statistical Learning might support predictive processing instead.

Temporal Dynamics of Competition between Statistical Learning and Episodic Memory in Intracranial Recordings of Human Visual Cortex

The function of long-term memory is not just to reminisce about the past, but also to make predictions that help us behave appropriately and efficiently in the future. This predictive function of memory provides a new perspective on the classic question from memory research of why we remember some things but not others. If prediction is a key outcome of memory, then the extent to which an item generates a prediction signifies that this information already exists in memory and need not be encoded. We tested this principle using human intracranial EEG as a time-resolved method to quantify prediction in visual cortex during a statistical learning task and link the strength of these predictions to subsequent episodic memory behavior. Epilepsy patients of both sexes viewed rapid streams of scenes, some of which contained regularities that allowed the category of the next scene to be predicted. We verified that statistical learning occurred using neural frequency tagging and measured category prediction with multivariate pattern analysis. Although neural prediction was robust overall, this was driven entirely by predictive items that were subsequently forgotten. Such interference provides a mechanism by which prediction can regulate memory formation to prioritize encoding of information that could help learn new predictive relationships.SIGNIFICANCE STATEMENT When faced with a new experience, we are rarely at a loss for what to do. Rather, because many aspects of the world are stable over time, we rely on past experiences to generate expectations that guide behavior. Here we show that these expectations during a new experience come at the expense of memory for that experience. From intracranial recordings of visual cortex, we decoded what humans expected to see next in a series of photographs based on patterns of neural activity. Photographs that generated strong neural expectations were more likely to be forgotten in a later behavioral memory test. Prioritizing the storage of experiences that currently lead to weak expectations could help improve these expectations in future encounters.

Remembering the pattern: A longitudinal case study on statistical learning in spatial navigation and memory consolidation

Distinct brain systems are thought to support statistical learning over different timescales. Regularities encountered during online perceptual experience can be acquired rapidly by the hippocampus. Further processing during offline consolidation can establish these regularities gradually in cortical regions, including the medial prefrontal cortex (mPFC). These mechanisms of statistical learning may be critical during spatial navigation, for which knowledge of the structure of an environment can facilitate future behavior. Rapid acquisition and prolonged retention of regularities have been investigated in isolation, but how they interact in the context of spatial navigation is unknown. We had the rare opportunity to study the brain systems underlying both rapid and gradual timescales of statistical learning using intracranial electroencephalography (iEEG) longitudinally in the same patient over a period of three weeks. As hypothesized, spatial patterns were represented in the hippocampus but not mPFC for up to one week after statistical learning and then represented in the mPFC but not hippocampus two and three weeks after statistical learning. Taken together, these findings suggest that the hippocampus may contribute to the initial extraction of regularities prior to cortical consolidation.

Experiencing Surprise: The Temporal Dynamics of Its Impact on Memory

To efficiently process information, the brain shifts between encoding and retrieval states, prioritizing bottom-up or top-down processing accordingly. Expectation violation before or during learning has been shown to trigger an adaptive encoding mechanism, resulting in better memory for unexpected events. Using fMRI, we explored (1) whether this encoding mechanism is also triggered during retrieval, and if so, (2) what the temporal dynamics of its mnemonic consequences are. Male and female participants studied object images, then, with new objects, they learned a contingency between a cue and a semantic category. Rule-abiding (expected) and violating (unexpected) targets and similar foils were used at test. We found interactions between previous and current similar events' expectation, such that when an expected event followed a similar but unexpected event, its performance was boosted, underpinned by activation in the hippocampus, midbrain, and occipital cortex. In contrast, a sequence of two unexpected similar events also triggered occipital engagement; however, this did not enhance memory performance. Taken together, our findings suggest that when the goal is to retrieve, encountering surprising events engages an encoding mechanism, supported by bottom-up processing, that may enhance memory for future related events.SIGNIFICANCE STATEMENT Optimizing the balance between new learning and the retrieval of existing knowledge is an ongoing process, at the core of human cognition. Previous research into memory encoding suggests experiencing surprise leads to the prioritization of the learning of new memories, forming an adaptive encoding mechanism. We examined whether this mechanism is also engaged when the current goal is to retrieve information. Our results demonstrate that an expectation-driven shift toward an encoding state, supported by enhanced perceptual processing, is beneficial for the correct identification of subsequent expected similar events. These findings have important implications for our understanding of the temporal dynamics of the adaptive encoding of information into memory.

Contextual Expectations Shape Cortical Reinstatement of Sensory Representations

When making a turn at a familiar intersection, we know what items and landmarks will come into view. These perceptual expectations, or predictions, come from our knowledge of the context; however, it is unclear how memory and perceptual systems interact to support the prediction and reactivation of sensory details in cortex. To address this, human participants learned the spatial layout of animals positioned in a cross maze. During fMRI, participants of both sexes navigated between animals to reach a target, and in the process saw a predictable sequence of five animal images. Critically, to isolate activity patterns related to item predictions, rather than bottom-up inputs, one-fourth of trials ended early, with a blank screen presented instead. Using multivariate pattern similarity analysis, we reveal that activity patterns in early visual cortex, posterior medial regions, and the posterior hippocampus showed greater similarity when seeing the same item compared with different items. Further, item effects in posterior hippocampus were specific to the sequence context. Critically, activity patterns associated with seeing an item in visual cortex and posterior medial cortex, were also related to activity patterns when an item was expected, but omitted, suggesting sequence predictions were reinstated in these regions. Finally, multivariate connectivity showed that patterns in the posterior hippocampus at one position in the sequence were related to patterns in early visual cortex and posterior medial cortex at a later position. Together, our results support the idea that hippocampal representations facilitate sensory processing by modulating visual cortical activity in anticipation of expected items.SIGNIFICANCE STATEMENT Our visual world is a series of connected events, where we can predict what we might see next based on our recent past. Understanding the neural circuitry and mechanisms of the perceptual and memory systems that support these expectations is fundamental to revealing how we perceive and act in our world. Using brain imaging, we studied what happens when we expect to see specific visual items, and how such expectations relate to top-down memory signals. We find both visual and memory systems reflect item predictions, and moreover, we show that hippocampal activity supports predictions of future expected items. This demonstrates that the hippocampus acts to predict upcoming items, and reinstates such predictions in cortex.

Statistical Learning in Vision

Vision and learning have long been considered to be two areas of research linked only distantly. However, recent developments in vision research have changed the conceptual definition of vision from a signal-evaluating process to a goal-oriented interpreting process, and this shift binds learning, together with the resulting internal representations, intimately to vision. In this review, we consider various types of learning (perceptual, statistical, and rule/abstract) associated with vision in the past decades and argue that they represent differently specialized versions of the fundamental learning process, which must be captured in its entirety when applied to complex visual processes. We show why the generalized version of statistical learning can provide the appropriate setup for such a unified treatment of learning in vision, what computational framework best accommodates this kind of statistical learning, and what plausible neural scheme could feasibly implement this framework. Finally, we list the challenges that the field of statistical learning faces in fulfilling the promise of being the right vehicle for advancing our understanding of vision in its entirety. Expected final online publication date for the Annual Review of Vision Science, Volume 8 is September 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Models of psychedelic drug action: modulation of cortical-subcortical circuits

Classic psychedelic drugs such as psilocybin and lysergic acid diethylamide (LSD) have recaptured the imagination of both science and popular culture, and may have efficacy in treating a wide range of psychiatric disorders. Human and animal studies of psychedelic drug action in the brain have demonstrated the involvement of the serotonin 2A (5-HT2A) receptor and the cerebral cortex in acute psychedelic drug action, but different models have evolved to try to explain the impact of 5-HT2A activation on neural systems. Two prominent models of psychedelic drug action (the cortico-striatal thalamo-cortical, or CSTC, model and relaxed beliefs under psychedelics, or REBUS, model) have emphasized the role of different subcortical structures as crucial in mediating psychedelic drug effects. We describe these models and discuss gaps in knowledge, inconsistencies in the literature and extensions of both models. We then introduce a third circuit-level model involving the claustrum, a thin strip of grey matter between the insula and the external capsule that densely expresses 5-HT2A receptors (the cortico-claustro-cortical, or CCC, model). In this model, we propose that the claustrum entrains canonical cortical network states, and that psychedelic drugs disrupt 5-HT2A-mediated network coupling between the claustrum and the cortex, leading to attenuation of canonical cortical networks during psychedelic drug effects. Together, these three models may explain many phenomena of the psychedelic experience, and using this framework, future research may help to delineate the functional specificity of each circuit to the action of both serotonergic and non-serotonergic hallucinogens.

A neural network model of when to retrieve and encode episodic memories

Recent human behavioral and neuroimaging results suggest that people are selective in when they encode and retrieve episodic memories. To explain these findings, we trained a memory-augmented neural network to use its episodic memory to support prediction of upcoming states in an environment where past situations sometimes reoccur. We found that the network learned to retrieve selectively as a function of several factors, including its uncertainty about the upcoming state. Additionally, we found that selectively encoding episodic memories at the end of an event (but not mid-event) led to better subsequent prediction performance. In all of these cases, the benefits of selective retrieval and encoding can be explained in terms of reducing the risk of retrieving irrelevant memories. Overall, these modeling results provide a resource-rational account of why episodic retrieval and encoding should be selective and lead to several testable predictions.

The Neural Representation of Events Is Dominated by Elements that Are Most Reliably Present

An episodic memory is specific to an event that occurred at a particular time and place. However, the elements that comprise the event-the location, the people present, and their actions and goals-might be shared with numerous other similar events. Does the brain preferentially represent certain elements of a remembered event? If so, which elements dominate its neural representation: those that are shared across similar events, or the novel elements that define a specific event? We addressed these questions by using a novel experimental paradigm combined with fMRI. Multiple events were created involving conversations between two individuals using the format of a television chat show. Chat show "hosts" occurred repeatedly across multiple events, whereas the "guests" were unique to only one event. Before learning the conversations, participants were scanned while viewing images or names of the (famous) individuals to be used in the study to obtain person-specific activity patterns. After learning all the conversations over a week, participants were scanned for a second time while they recalled each event multiple times. We found that during recall, person-specific activity patterns within the posterior midline network were reinstated for the hosts of the shows but not the guests, and that reinstatement of the hosts was significantly stronger than the reinstatement of the guests. These findings demonstrate that it is the more generic, familiar, and predictable elements of an event that dominate its neural representation compared with the more idiosyncratic, event-defining, elements.

Prediction errors disrupt hippocampal representations and update episodic memories

The brain supports adaptive behavior by generating predictions, learning from errors, and updating memories to incorporate new information. Prediction error, or surprise, triggers learning when reality contradicts expectations. Prior studies have shown that the hippocampus signals prediction errors, but the hypothesized link to memory updating has not been demonstrated. In a human functional MRI study, we elicited mnemonic prediction errors by interrupting familiar narrative videos immediately before the expected endings. We found that prediction errors reversed the relationship between univariate hippocampal activation and memory: greater hippocampal activation predicted memory preservation after expected endings, but memory updating after surprising endings. In contrast to previous studies, we show that univariate activation was insufficient for understanding hippocampal prediction error signals. We explain this surprising finding by tracking both the evolution of hippocampal activation patterns and the connectivity between the hippocampus and neuromodulatory regions. We found that hippocampal activation patterns stabilized as each narrative episode unfolded, suggesting sustained episodic representations. Prediction errors disrupted these sustained representations and the degree of disruption predicted memory updating. The relationship between hippocampal activation and subsequent memory depended on concurrent basal forebrain activation, supporting the idea that cholinergic modulation regulates attention and memory. We conclude that prediction errors create conditions that favor memory updating, prompting the hippocampus to abandon ongoing predictions and make memories malleable.

Mnemonic prediction errors promote detailed memories

When our experience violates our predictions, it is adaptive to update our knowledge to promote a more accurate representation of the world and facilitate future predictions. Theoretical models propose that these mnemonic prediction errors should be encoded into a distinct memory trace to prevent interference with previous, conflicting memories. We investigated this proposal by repeatedly exposing participants to pairs of sequentially presented objects (A → B), thus evoking expectations. Then, we violated participants' expectations by replacing the second object in the pairs with a novel object (A → C). The following item memory test required participants to discriminate between identical old items and similar lures, thus testing detailed and distinctive item memory representations. In two experiments, mnemonic prediction errors enhanced item memory: Participants correctly identified more old items as old when those items violated expectations during learning, compared with items that did not violate expectations. This memory enhancement for C items was only observed when participants later showed intact memory for the related A → B pairs, suggesting that strong predictions are required to facilitate memory for violations. Following up on this, a third experiment reduced prediction strength prior to violation and subsequently eliminated the memory advantage of violations. Interestingly, mnemonic prediction errors did not increase gist-based mistakes of identifying old items as similar lures or identifying similar lures as old. Enhanced item memory in the absence of gist-based mistakes suggests that violations enhanced memory for items' details, which could be mediated via distinct memory traces. Together, these results advance our knowledge of how mnemonic prediction errors promote memory formation.

Previous Motor Actions Outweigh Sensory Information in Sensorimotor Statistical Learning

Humans can use their previous experience in form of statistical priors to improve decisions. It is, however, unclear how such priors are learned and represented. Importantly, it has remained elusive whether prior learning is independent of the sensorimotor system involved in the learning process or not, as both modality-specific and modality-general learning have been reported in the past. Here, we used a saccadic eye movement task to probe the learning and representation of a spatial prior across a few trials. In this task, learning occurs in an unsupervised manner and through encountering trial-by-trial visual hints drawn from a distribution centered on the target location. Using a model-comparison approach, we found that participants’ prior knowledge is largely represented in the form of their previous motor actions, with minimal influence from the previously seen visual hints. By using two different motor contexts for response (looking either at the estimated target location, or exactly opposite to it), we could further compare whether prior experience obtained in one motor context can be transferred to the other. Although learning curves were highly similar, and participants seemed to use the same strategy for both response types, they could not fully transfer their knowledge between contexts, as performance and confidence ratings dropped after a switch of the required response. Together, our results suggest that humans preferably use the internal representations of their previous motor actions, rather than past incoming sensory information, to form statistical sensorimotor priors on the timescale of a few trials.

Evidence of hippocampal learning in human infants

The hippocampus is essential for human memory.¹ The protracted maturation of memory capacities from infancy through early childhood^2-4 is thus often attributed to hippocampal immaturity.^5-7 The hippocampus of human infants has been characterized in terms of anatomy,^8,9 but its function has never been tested directly because of technical challenges.^10,11 Here, we use recently developed methods for task-based fMRI in awake human infants¹² to test the hypothesis that the infant hippocampus supports statistical learning.^13-15 Hippocampal activity increased with exposure to visual sequences of objects when the temporal order contained regularities to be learned, compared to when the order was random. Despite the hippocampus doubling in anatomical volume across infancy, learning-related functional activity bore no relationship to age. This suggests that the hippocampus is recruited for statistical learning at the youngest ages in our sample, around 3 months. Within the hippocampus, statistical learning was clearer in anterior than posterior divisions. This is consistent with the theory that statistical learning occurs in the monosynaptic pathway,¹⁶ which is more strongly represented in the anterior hippocampus.^17,18 The monosynaptic pathway develops earlier than the trisynaptic pathway, which is linked to episodic memory,^19,20 raising the possibility that the infant hippocampus participates in statistical learning before it forms durable memories. Beyond the hippocampus, the medial prefrontal cortex showed statistical learning, consistent with its role in adult memory integration²¹ and generalization.²² These results suggest that the hippocampus supports the vital ability of infants to extract the structure of their environment through experience.

Statistical learning occurs during practice while high-order rule learning during rest period

Knowing when the brain learns is crucial for both the comprehension of memory formation and consolidation and for developing new training and neurorehabilitation strategies in healthy and patient populations. Recently, a rapid form of offline learning developing during short rest periods has been shown to account for most of procedural learning, leading to the hypothesis that the brain mainly learns during rest between practice periods. Nonetheless, procedural learning has several subcomponents not disentangled in previous studies investigating learning dynamics, such as acquiring the statistical regularities of the task, or else the high-order rules that regulate its organization. Here we analyzed 506 behavioral sessions of implicit visuomotor deterministic and probabilistic sequence learning tasks, allowing the distinction between general skill learning, statistical learning, and high-order rule learning. Our results show that the temporal dynamics of apparently simultaneous learning processes differ. While high-order rule learning is acquired offline, statistical learning is evidenced online. These findings open new avenues on the short-scale temporal dynamics of learning and memory consolidation and reveal a fundamental distinction between statistical and high-order rule learning, the former benefiting from online evidence accumulation and the latter requiring short rest periods for rapid consolidation.

When forgetting fosters learning: A neural network model for statistical learning

Learning often requires splitting continuous signals into recurring units, such as the discrete words constituting fluent speech; these units then need to be encoded in memory. A prominent candidate mechanism involves statistical learning of co-occurrence statistics like transitional probabilities (TPs), reflecting the idea that items from the same unit (e.g., syllables within a word) predict each other better than items from different units. TP computations are surprisingly flexible and sophisticated. Humans are sensitive to forward and backward TPs, compute TPs between adjacent items and longer-distance items, and even recognize TPs in novel units. We explain these hallmarks of statistical learning with a simple model with tunable, Hebbian excitatory connections and inhibitory interactions controlling the overall activation. With weak forgetting, activations are long-lasting, yielding associations among all items; with strong forgetting, no associations ensue as activations do not outlast stimuli; with intermediate forgetting, the network reproduces the hallmarks above. Forgetting thus is a key determinant of these sophisticated learning abilities. Further, in line with earlier dissociations between statistical learning and memory encoding, our model reproduces the hallmarks of statistical learning in the absence of a memory store in which items could be placed.

When the Memory System Gets Ahead of Itself

Humans are adept at learning and exploiting statistical regularities to predict future events from current experience. A recent paper by Sherman and Turk-Browne demonstrates that statistical regularities bias the hippocampus toward representing future states over current experience and reduce the degree to which current experience is encoded into memory.