Replay of very early encoding representations during recollection

Testing cognitive theories with multivariate pattern analysis of neuroimaging data

Multivariate pattern analysis (MVPA) has emerged as a powerful method for the analysis of functional magnetic resonance imaging, electroencephalography and magnetoencephalography data. The new approaches to experimental design and hypothesis testing afforded by MVPA have made it possible to address theories that describe cognition at the functional level. Here we review a selection of studies that have used MVPA to test cognitive theories from a range of domains, including perception, attention, memory, navigation, emotion, social cognition and motor control. This broad view reveals properties of MVPA that make it suitable for understanding the 'how' of human cognition, such as the ability to test predictions expressed at the item or event level. It also reveals limitations and points to future directions.

Neural reinstatement of context memory in adults with autism spectrum disorder

Autism spectrum disorder (ASD) is associated with episodic memory impairment. However, episodic memories include a variety of contextual details, and it is difficult to solely rely on behavioral data to assess how specifically (i.e. event-specific reinstatement) an event is remembered. We applied encoding-retrieval representational similarity (ERS) analysis to EEG data to assess event-specific ERS for object-context associations in a sample of 34 adults (17 with, 17 without ASD). Participants studied objects presented alongside 2 contextual features: scene/color, and attention was directed toward one object-context relationship. At retrieval, memory was assessed for the object and both contexts. Behavioral results revealed no group differences in item or context memory performance. ERS results revealed group temporal differences in reinstatement. Results may indicate differences in both encoding (i.e. fewer perceptual details) and retrieval (i.e. ineffectively skipping through memory fragments) in ASD and should be further investigated in studies modulating the perceptual detail required for memory decisions. Results highlight the utility of ERS as a methodology used to evaluate episodic reinstatement even in the absence of behavioral differences in memory performance.

Examining the engram encoding specificity hypothesis in mice

According to the encoding specificity hypothesis, memory is best recalled by retrieval cues that overlap with training cues. Human studies generally support this hypothesis. However, memories are thought to be stored in neuronal ensembles (engrams), and retrieval cues are thought to reactivate neurons in an engram to induce memory recall. Here, we visualized engrams in mice to test whether retrieval cues that overlap with training cues produce maximal memory recall via high engram reactivation (engram encoding specificity hypothesis). Using variations of cued threat conditioning (pairing conditioned stimulus [CS] with footshock), we manipulated encoding and retrieval conditions along multiple domains, including pharmacological state, external sensory cue, and internal optogenetic cue. Maximal engram reactivation and memory recall occurred when retrieval conditions closely matched training conditions. These findings provide a biological basis for the encoding specificity hypothesis and highlight the important interaction between stored information (engram) and cues available at memory retrieval (ecphory).

Distinct encoding and post-encoding representational formats contribute to episodic sequence memory formation

In episodic encoding, an unfolding experience is rapidly transformed into a memory representation that binds separate episodic elements into a memory form to be later recollected. However, it is unclear how brain activity changes over time to accommodate the encoding of incoming information. This study aimed to investigate the dynamics of the representational format that contributed to memory formation of sequential episodes. We combined representational similarity analysis and multivariate decoding approaches on EEG data to compare whether "category-level" or "item-level" representations supported memory formation during the online encoding of a picture triplet sequence and offline, in the period that immediately followed encoding. The findings revealed a gradual integration of category-level representation during the online encoding of the picture sequence and a rapid item-based neural reactivation of the encoded sequence at the episodic offset. However, we found that only memory reinstatement at episodic offset was associated with successful memory retrieval from long-term memory. These results suggest that post-encoding memory reinstatement is crucial for the rapid formation of unique memory for episodes that unfold over time. Overall, the study sheds light on the dynamics of representational format changes that take place during the formation of episodic memories.

Spatiotemporal Precision of Neuroimaging in Psychiatry

Aberrant patterns of cognition, perception, and behavior seen in psychiatric disorders are thought to be driven by a complex interplay of neural processes that evolve at a rapid temporal scale. Understanding these dynamic processes in vivo in humans has been hampered by a trade-off between spatial and temporal resolutions inherent to current neuroimaging technology. A recent trend in psychiatric research has been the use of high temporal resolution imaging, particularly magnetoencephalography, often in conjunction with sophisticated machine learning decoding techniques. Developments here promise novel insights into the spatiotemporal dynamics of cognitive phenomena, including domains relevant to psychiatric illnesses such as reward and avoidance learning, memory, and planning. This review considers recent advances afforded by exploiting this increased spatiotemporal precision, with specific reference to applications that seek to drive a mechanistic understanding of psychopathology and the realization of preclinical translation.

How our understanding of memory replay evolves

Memory reactivations and replay, widely reported in the hippocampus and cortex across species, have been implicated in memory consolidation, planning, and spatial and skill learning. Technological advances in electrophysiology, calcium imaging, and human neuroimaging techniques have enabled neuroscientists to measure large-scale neural activity with increasing spatiotemporal resolution and have provided opportunities for developing robust analytic methods to identify memory replay. In this article, we first review a large body of historically important and representative memory replay studies from the animal and human literature. We then discuss our current understanding of memory replay functions in learning, planning, and memory consolidation and further discuss the progress in computational modeling that has contributed to these improvements. Next, we review past and present analytic methods for replay analyses and discuss their limitations and challenges. Finally, looking ahead, we discuss some promising analytic methods for detecting nonstereotypical, behaviorally nondecodable structures from large-scale neural recordings. We argue that seamless integration of multisite recordings, real-time replay decoding, and closed-loop manipulation experiments will be essential for delineating the role of memory replay in a wide range of cognitive and motor functions.

Sustained neural representations of personally familiar people and places during cued recall

The recall and visualization of people and places from memory is an everyday occurrence, yet the neural mechanisms underpinning this phenomenon are not well understood. In particular, the temporal characteristics of the internal representations generated by active recall are unclear. Here, we used magnetoencephalography (MEG) and multivariate pattern analysis to measure the evolving neural representation of familiar places and people across the whole brain when human participants engage in active recall. To isolate self-generated imagined representations, we used a retro-cue paradigm in which participants were first presented with two possible labels before being cued to recall either the first or second item. We collected personalized labels for specific locations and people familiar to each participant. Importantly, no visual stimuli were presented during the recall period, and the retro-cue paradigm allowed the dissociation of responses associated with the labels from those corresponding to the self-generated representations. First, we found that following the retro-cue it took on average ∼1000 ms for distinct neural representations of freely recalled people or places to develop. Second, we found distinct representations of personally familiar concepts throughout the 4 s recall period. Finally, we found that these representations were highly stable and generalizable across time. These results suggest that self-generated visualizations and recall of familiar places and people are subserved by a stable neural mechanism that operates relatively slowly when under conscious control.

A neural network account of memory replay and knowledge consolidation

Replay can consolidate memories through offline neural reactivation related to past experiences. Category knowledge is learned across multiple experiences, and its subsequent generalization is promoted by consolidation and replay during rest and sleep. However, aspects of replay are difficult to determine from neuroimaging studies. We provided insights into category knowledge replay by simulating these processes in a neural network which approximated the roles of the human ventral visual stream and hippocampus. Generative replay, akin to imagining new category instances, facilitated generalization to new experiences. Consolidation-related replay may therefore help to prepare us for the future as much as remember the past. Generative replay was more effective in later network layers functionally similar to the lateral occipital cortex than layers corresponding to early visual cortex, drawing a distinction between neural replay and its relevance to consolidation. Category replay was most beneficial for newly acquired knowledge, suggesting replay helps us adapt to changes in our environment. Finally, we present a novel mechanism for the observation that the brain selectively consolidates weaker information, namely a reinforcement learning process in which categories were replayed according to their contribution to network performance. This reinforces the idea of consolidation-related replay as an active rather than passive process.

From remembering to reconstruction: The transformative neural representation of episodic memory

Although memory has long been recognized as a generative process, neural research of memory in recent decades has been predominantly influenced by Tulving's "mental time traveling" perspective and focused on the reactivation and consolidation of encoded memory representations. With the development of multiple powerful analytical approaches to characterize the contents and formats of neural representations, recent studies are able to provide detailed examinations of the representations at various processing stages and have provided exciting new insights into the transformative nature of episodic memory. These studies have revealed the rapid, substantial, and continuous transformation of memory representation during the encoding, maintenance, consolidation, and retrieval of both single and multiple events, as well as event sequences. These transformations are characterized by the abstraction, integration, differentiation, and reorganization of memory representations, enabling the long-term retention and generalization of memory. These studies mark a significant shift in perspective from remembering to reconstruction, which might better reveal the nature of memory and its roles in supporting more effective learning, adaptive decision-making, and creative problem solving.

Differences in memory for what, where, and when components of recently formed episodes

An integral feature of human memory is the ability to recall past events. What distinguishes such episodic memory from semantic or associative memory is the joint encoding and retrieval of "what," "where," and "when" (WWW) for such events. Surprisingly, little work has addressed whether all three components of WWW are retrieved with equal fidelity when remembering episodes. To study this question, we created a novel task where human participants identified matched or mismatched still images sampled from recently viewed synthetic movies. The mismatch images only probe one of the three WWW components at a time, allowing us to separately test accuracies for each component of the episodes. Crucially, each WWW component in the movies is easily distinguishable in isolation, thereby making any differences in accuracy between components due to how they are joined in memory. We find that memory for "when" has the lowest accuracy, with it being the component most influenced by primacy and recency. Furthermore, the memory of "when" is most susceptible to interference due to changes in task load. These findings suggest that episodes are not stored and retrieved as a coherent whole but instead their components are either stored or retrieved differentially as part of an active reconstruction process. NEW & NOTEWORTHY When we store and subsequently retrieve episodes, does the brain encode them holistically or in separate parts that are later reconstructed? Using a task where participants study abstract episodes and on any given trial are probed on the what, where, and when components, we find mnemonic differences between them. Accuracy for "when" memory is the lowest, as it is most influenced by primacy, recency, and interference, suggesting that episodes are not treated holistically by the brain.

Pattern reinstatement and attentional control overlap during episodic long-term memory retrieval

Episodic long-term memory (eLTM) retrieval involves the reinstatement of neural patterns from the encoding phase. However, recent evidence suggests that comparable cortical activity patterns can also be linked to attentional control processes on the level of memory representations. The current investigation assesses these two processes independently based on alpha-beta-band activity in the electroencephalogram (EEG). During encoding, subjects were presented with an object on a certain position on the screen and had to imagine it on a new position. In each trial, either the task-irrelevant presentation position or the task-relevant imagination position was lateralized. In the retrieval phase, subjects first made an old/new judgement based on centrally presented objects and then reported the imagination position. Pattern reinstatement should be reflected in similar lateralized alpha-beta activity during encoding and retrieval. Conversely, the influence of attentional control processes during retrieval would be associated with the suppression of alpha-beta power contralateral to the to-be-reported imagination position and with the increase of activity contralateral to the irrelevant presentation position. Our results support this latter pattern. This shows that an experimental differentiation between selective attention and pattern reinstatement processes is necessary when studying the neural basis of eLTM retrieval.

Greater sleep variance related to decrements in memory performance and event-specific neural similarity: a racially/ethnically diverse lifespan sample

Successful memory performance depends on overlap between neural representations at encoding and retrieval. With older age, neural similarity, memory performance, and sleep quality decline. Regardless of age, racial/ethnic minorities tend to experience poor sleep, which may contribute to poor memory. Previous studies have not investigated memory performance, neural similarity, sleep quality, and age in diverse participants. Here, we recruited racially/ethnically diverse adults across the lifespan and examined night-to-night sleep quality in relation to memory performance and encoding-retrieval similarity. We employed item-specific, representational similarity analysis (not confounded by effort, word perception, or differences in electroencephalography signal amplitude) to assess neural similarity for intact and recombined paired associates. Greater sleep variance and poorer memory performance were more strongly associated with older age. Interestingly, sleep variance was positively associated with neural similarity for intact pairs. This relationship was stronger with younger age and for racial/ethnic minorities. For recombined pairs, greater sleep variance was associated with reduced neural similarity. Thus, varied sleep may induce greater reliance on familiarity, while consistent sleep may support recollection.

Neural Reinstatement of Overlapping Memories in Young and Older Adults

When we update our episodic memories with new information, mnemonic competition between old and new memories may result because of the presence of shared features. Behavioral studies suggest that this competition can lead to proactive interference, resulting in unsuccessful memory updating, particularly for older adults. It is difficult with behavioral data alone to measure the reactivation of old, overlapping memories during retrieval and its impact on memory for new memories. Here, we applied encoding-retrieval representational similarity (ERS) analysis to EEG data to estimate event-specific encoding-related neural reinstatement of old associations during the retrieval of new ones and its impact on memory for new associations in young and older adults. Our results showed that older adults' new associative memory performance was more negatively impacted by proactive interference from old memories than that of young adults. In both age groups, ERS for old associative memories was greater for trials for which new associative memories were forgotten than remembered. In contrast, ERS for new associative memories was greater when they were remembered than forgotten. In addition, older adults showed relatively attenuated target (i.e., new associates) and lure (i.e., old associates) ERS effects compared to younger adults. Collectively, these results suggest that the neural reinstatement of interfering memories during retrieval contributes to proactive interference across age, whereas overall attenuated ERS effect in older adults might reflect their reduced memory fidelity.

Contributions of representational distinctiveness and stability to memory performance and age differences

Long-standing theories of cognitive aging suggest that memory decline is associated with age-related differences in the way information is neurally represented. Multivariate pattern similarity analyses enabled researchers to take a representational perspective on brain and cognition, and allowed them to study the properties of neural representations that support successful episodic memory. Two representational properties have been identified as crucial for memory performance, namely the distinctiveness and the stability of neural representations. Here, we review studies that used multivariate analysis tools for different neuroimaging techniques to clarify how these representational properties relate to memory performance across adulthood. While most evidence on age differences in neural representations involved stimulus category information , recent studies demonstrated that particularly item-level stability and specificity of activity patterns are linked to memory success and decline during aging. Overall, multivariate methods offer a versatile tool for our understanding of age differences in the neural representations underlying memory.

Fronto-medial theta coordinates posterior maintenance of working memory content

How does the human brain manage multiple bits of information to guide goal-directed behavior? Successful working memory (WM) functioning has consistently been linked to oscillatory power in the theta frequency band (4–8 Hz) over fronto-medial cortex (fronto-medial theta [FMT]). Specifically, FMT is thought to reflect the mechanism of an executive sub-system that coordinates maintenance of memory contents in posterior regions. However, direct evidence for the role of FMT in controlling specific WM content is lacking. Here, we collected high-density electroencephalography (EEG) data while participants engaged in WM-dependent tasks and then used multivariate decoding methods to examine WM content during the maintenance period. Engagement of WM was accompanied by a focal increase in FMT. Importantly, decoding of WM content was driven by posterior sites, which, in turn, showed increased functional theta coupling with fronto-medial channels. Finally, we observed a significant slowing of FMT frequency with increasing WM load, consistent with the hypothesized broadening of a theta “duty cycle” to accommodate additional WM items. Together, these findings demonstrate that frontal theta orchestrates posterior maintenance of WM content. Moreover, the observed frequency slowing elucidates the function of FMT oscillations by specifically supporting phase-coding accounts of WM.

•

FMT power supports WM functions

•

During WM performance, posterior/parietal regions are coupled with FMT

•

Multivariate decoding of WM content is mediated by these same posterior channels

•

Frontal theta frequency slows with WM load supporting phase-coding models

Encoding contexts are incidentally reinstated during competitive retrieval and track the temporal dynamics of memory interference

The ability to remember an episode from our past is often hindered by competition from similar events. For example, if we want to remember the article a colleague recommended during the last lab meeting, we may need to resolve interference from other article recommendations from the same colleague. This study investigates if the contextual features specifying the encoding episodes are incidentally reinstated during competitive memory retrieval. Competition between memories was created through the AB/AC interference paradigm. Individual word-pairs were presented embedded in a slowly drifting real-word-like context. Multivariate pattern analysis (MVPA) of high temporal-resolution electroencephalographic (EEG) data was used to investigate context reactivation during memory retrieval. Behaviorally, we observed proactive (but not retroactive) interference; that is, performance for AC competitive retrieval was worse compared with a control DE noncompetitive retrieval, whereas AB retrieval did not suffer from competition. Neurally, proactive interference was accompanied by an early reinstatement of the competitor context and interference resolution was associated with the ensuing reinstatement of the target context. Together, these findings provide novel evidence showing that the encoding contexts of competing discrete events are incidentally reinstated during competitive retrieval and that such reinstatement tracks retrieval competition and subsequent interference resolution.

Transformative neural representations support long-term episodic memory

Memory is often conceived as a dynamic process that involves substantial transformations of mental representations. However, the neural mechanisms underlying these transformations and their role in memory formation and retrieval have only started to be elucidated. Combining intracranial EEG recordings with deep neural network models, we provide a detailed picture of the representational transformations from encoding to short-term memory maintenance and long-term memory retrieval that underlie successful episodic memory. We observed substantial representational transformations during encoding. Critically, more pronounced semantic representational formats predicted better subsequent long-term memory, and this effect was mediated by more consistent item-specific representations across encoding events. The representations were further transformed right after stimulus offset, and the representations during long-term memory retrieval were more similar to those during short-term maintenance than during encoding. Our results suggest that memory representations pass through multiple stages of transformations to achieve successful long-term memory formation and recall.

Tendency to ruminate and anxiety are associated with altered alpha and beta oscillatory power dynamics during memory for contextual details

Rumination occurs when an individual becomes mentally stuck and cannot redirect attention away from an unwanted thought demonstrating cognitive inflexibility. Cognitive flexibility is important for various cognitive functions, including episodic memory. Trait rumination is a partial mediator in the relationship between depression and overgeneral episodic memory, suggesting that rumination may negatively influence memory for contextual details. Oscillations in the alpha (8-12 Hz) and beta (13-30 Hz) frequency bands are crucial for various cognitive functions (e.g., attention control and episodic memory) and may help to explain the relationship between trait rumination and memory for contextual details. Our study uses EEG recorded during a source memory task to assess how alpha and beta oscillations during memory for contextual details may change as a function of trait rumination, anxiety, and depression level (n = 43). The source memory task instructs participants to remember objects and their associated contextual details. Memory for contextual details is lessened for participants higher in trait rumination paired with higher trait anxiety. Oscillations were analyzed in posterior parietal/occipital regions. During encoding, an interaction of nonclinical depression level and rumination predicts higher alpha power for items that were later not successfully remembered. During test, depression and rumination interact and predict higher alpha power for both successful and unsuccessful memory. These results suggest that trait anxiety, depression, and rumination impact accuracy and alpha oscillatory dynamics during contextual memory via changes in attention control.

Dynamics of fMRI patterns reflect sub-second activation sequences and reveal replay in human visual cortex

Neural computations are often fast and anatomically localized. Yet, investigating such computations in humans is challenging because non-invasive methods have either high temporal or spatial resolution, but not both. Of particular relevance, fast neural replay is known to occur throughout the brain in a coordinated fashion about which little is known. We develop a multivariate analysis method for functional magnetic resonance imaging that makes it possible to study sequentially activated neural patterns separated by less than 100 ms with precise spatial resolution. Human participants viewed five images individually and sequentially with speeds up to 32 ms between items. Probabilistic pattern classifiers were trained on activation patterns in visual and ventrotemporal cortex during individual image trials. Applied to sequence trials, probabilistic classifier time courses allow the detection of neural representations and their order. Order detection remains possible at speeds up to 32 ms between items (plus 100 ms per item). The frequency spectrum of the sequentiality metric distinguishes between sub- versus supra-second sequences. Importantly, applied to resting-state data our method reveals fast replay of task-related stimuli in visual cortex. This indicates that non-hippocampal replay occurs even after tasks without memory requirements and shows that our method can be used to detect such spontaneously occurring replay.

Competitive state of movements during planning predicts sequence performance

Humans can learn and produce skilled movement sequences from memory, yet the nature of sequence planning is not well understood. Previous computational and neurophysiological work suggests that movements in a sequence are planned as parallel graded activations and selected for output through competition. However, the relevance of this planning pattern to sequence production fluency and accuracy, as opposed to the temporal structure of sequences, is unclear. To resolve this question, we assessed the relative availability of constituent movements behaviorally during the preparation of motor sequences from memory. In three separate multisession experiments, healthy participants were trained to retrieve and produce four-element finger press sequences with particular timing according to an abstract sequence cue. We evaluated reaction time (RT) and error rate as markers of movement availability to constituent movement probes. Our results demonstrate that longer preparation time produces more pronounced differences in availability between adjacent sequence elements, whereas no effect was found for sequence speed or temporal grouping. Further, participants with larger position-dependent differences in movement availability tended to initiate correct sequences faster and with a higher temporal accuracy. Our results suggest that competitive preactivation is established gradually during sequence planning and predicts sequence skill, rather than the temporal structure of the motor sequence.NEW & NOTEWORTHY Sequence planning is an integral part of motor sequence control. Here, we demonstrate that the competitive state of sequential movements during sequence planning can be read out behaviorally through movement probes. We show that position-dependent differences in movement availability during planning reflect sequence preparedness and skill but not the timing of the planned sequence. Behavioral access to the preparatory state of movements may serve as a marker of sequence planning capacity.

In your phase: neural phase synchronisation underlies visual imagery of faces

Mental imagery is the process through which we retrieve and recombine information from our memory to elicit the subjective impression of “seeing with the mind’s eye”. In the social domain, we imagine other individuals while recalling our encounters with them or modelling alternative social interactions in future. Many studies using imaging and neurophysiological techniques have shown several similarities in brain activity between visual imagery and visual perception, and have identified frontoparietal, occipital and temporal neural components of visual imagery. However, the neural connectivity between these regions during visual imagery of socially relevant stimuli has not been studied. Here we used electroencephalography to investigate neural connectivity and its dynamics between frontal, parietal, occipital and temporal electrodes during visual imagery of faces. We found that voluntary visual imagery of faces is associated with long-range phase synchronisation in the gamma frequency range between frontoparietal electrode pairs and between occipitoparietal electrode pairs. In contrast, no effect of imagery was observed in the connectivity between occipitotemporal electrode pairs. Gamma range synchronisation between occipitoparietal electrode pairs predicted subjective ratings of the contour definition of imagined faces. Furthermore, we found that visual imagery of faces is associated with an increase of short-range frontal synchronisation in the theta frequency range, which temporally preceded the long-range increase in the gamma synchronisation. We speculate that the local frontal synchrony in the theta frequency range might be associated with an effortful top-down mnemonic reactivation of faces. In contrast, the long-range connectivity in the gamma frequency range along the fronto-parieto-occipital axis might be related to the endogenous binding and subjective clarity of facial visual features.

Cross-species neuroscience: closing the explanatory gap

Neuroscience has seen substantial development in non-invasive methods available for investigating the living human brain. However, these tools are limited to coarse macroscopic measures of neural activity that aggregate the diverse responses of thousands of cells. To access neural activity at the cellular and circuit level, researchers instead rely on invasive recordings in animals. Recent advances in invasive methods now permit large-scale recording and circuit-level manipulations with exquisite spatio-temporal precision. Yet, there has been limited progress in relating these microcircuit measures to complex cognition and behaviour observed in humans. Contemporary neuroscience thus faces an explanatory gap between macroscopic descriptions of the human brain and microscopic descriptions in animal models. To close the explanatory gap, we propose adopting a cross-species approach. Despite dramatic differences in the size of mammalian brains, this approach is broadly justified by preserved homology. Here, we outline a three-armed approach for effective cross-species investigation that highlights the need to translate different measures of neural activity into a common space. We discuss how a cross-species approach has the potential to transform basic neuroscience while also benefiting neuropsychiatric drug development where clinical translation has, to date, seen minimal success. This article is part of the theme issue 'Key relationships between non-invasive functional neuroimaging and the underlying neuronal activity'.

Assessing Rhythmic Visual Entrainment and Reinstatement of Brain Oscillations to Modulate Memory Performance

The human brain’s ability to store information and remember past events is thought to be orchestrated by the synchronization of neuronal oscillations in various frequency bands. A vast amount of research has found that neural oscillations in the theta (∼4–7 Hz) and alpha (∼8–12 Hz) bands play an important role in memory formation. More specifically, it has been suggested that memory performance benefits if the same oscillatory pattern is present during encoding and retrieval. However, the causal relevance of these oscillations is not well understood. Rhythmic sensory stimulation is thought to entrain ongoing brain oscillations and modulate associated functions (e.g., memory formation). In the present study, we used rhythmic visual stimulation at 6 and 10 Hz to experimentally modulate the memory encoding process in a recognition memory task. In addition, we reinstated oscillatory activity from the encoding episode during retrieval, which has been hypothesized to result in memory performance improvements compared to non-reinstated conditions and incongruent reinstatement. Contrary to our hypothesis, we find no effect of neural entrainment during encoding on subsequent memory performance. Likewise, memory retrieval does not benefit from neural reinstatement. The results are discussed with respect to methodological challenges of rhythmic sensory stimulation as a means to alter cognitive processes and induce context-dependent memory effects.

Episodic memory retrieval success is associated with rapid replay of episode content

Retrieval of everyday experiences is fundamental for informing our future decisions. The fine-grained neurophysiological mechanisms that support such memory retrieval are largely unknown. We studied participants who first experienced, without repetition, unique multicomponent 40-80-s episodes. One day later, they engaged in cued retrieval of these episodes while undergoing magnetoencephalography. By decoding individual episode elements, we found that trial-by-trial successful retrieval was supported by the sequential replay of episode elements, with a temporal compression factor of >60. The direction of replay supporting retrieval, either backward or forward, depended on whether the task goal was to retrieve elements of an episode that followed or preceded, respectively, a retrieval cue. This sequential replay was weaker in very-high-performing participants, in whom instead we found evidence for simultaneous clustered reactivation. Our results demonstrate that memory-mediated decisions are supported by a rapid replay mechanism that can flexibly shift in direction in response to task goals.

Cue-triggered activity replay in human early visual cortex

The recall of learned temporal sequences by a visual cue is an important form of experience-based neural plasticity. Here we observed such reactivation in awake human visual cortex using intracranial recording. After repeated exposure to a moving dot, a flash of the dot was able to trigger neural reactivation in the downstream receptive field along the motion path. This effect was observed only when the cue appeared near the receptive field. The estimated traveling speed was faster compared to the activation induced by the real motion. We suggest a range-limited, time-compressed reactivation as a result of repeated visual exposure in awake human visual cortex.

Visual Imagery and Perception Share Neural Representations in the Alpha Frequency Band

To behave adaptively with sufficient flexibility, biological organisms must cognize beyond immediate reaction to a physically present stimulus. For this, humans use visual mental imagery [1, 2], the ability to conjure up a vivid internal experience from memory that stands in for the percept of the stimulus. Visually imagined contents subjectively mimic perceived contents, suggesting that imagery and perception share common neural mechanisms. Using multivariate pattern analysis on human electroencephalography (EEG) data, we compared the oscillatory time courses of mental imagery and perception of objects. We found that representations shared between imagery and perception emerged specifically in the alpha frequency band. These representations were present in posterior, but not anterior, electrodes, suggesting an origin in parieto-occipital cortex. Comparison of the shared representations to computational models using representational similarity analysis revealed a relationship to later layers of deep neural networks trained on object representations, but not auditory or semantic models, suggesting representations of complex visual features as the basis of commonality. Together, our results identify and characterize alpha oscillations as a cortical signature of representations shared between visual mental imagery and perception.

•

Perception and imagery share neural representations in the alpha frequency band

•

Shared representations stem from parieto-occipital sources

•

Modeling suggests contents of shared representations are complex visual features

Representation of probabilistic outcomes during risky decision-making

Goal-directed behaviour requires prospectively retrieving and evaluating multiple possible action outcomes. While a plethora of studies suggested sequential retrieval for deterministic choice outcomes, it remains unclear whether this is also the case when integrating multiple probabilistic outcomes of the same action. We address this question by capitalising on magnetoencephalography (MEG) in humans who made choices in a risky foraging task. We train classifiers to distinguish MEG field patterns during presentation of two probabilistic outcomes (reward, loss), and then apply these to decode such patterns during deliberation. First, decoded outcome representations have a temporal structure, suggesting alternating retrieval of the outcomes. Moreover, the probability that one or the other outcome is being represented depends on loss magnitude, but not on loss probability, and it predicts the chosen action. In summary, we demonstrate decodable outcome representations during probabilistic decision-making, which are sequentially structured, depend on task features, and predict subsequent action.

Electrophysiological signatures of memory reactivation in humans

The reactivation of neural activity that was present during the encoding of an event is assumed to be essential for human episodic memory retrieval and the consolidation of memories during sleep. Pioneering animal work has already established a crucial role of memory reactivation to prepare and guide behaviour. Research in humans is now delineating the neural processes involved in memory reactivation during both wakefulness and sleep as well as their functional significance. Focusing on the electrophysiological signatures of memory reactivation in humans during both memory retrieval and sleep-related consolidation, this review provides an overview of the state of the art in the field. We outline recent advances, methodological developments and open questions and specifically highlight commonalities and differences in the neuronal signatures of memory reactivation during the states of wakefulness and sleep.

This article is part of the Theo Murphy meeting issue ‘Memory reactivation: replaying events past, present and future’.

Revealing the information contents of memory within the stimulus information representation framework

The information contents of memory are the cornerstone of the most influential models in cognition. To illustrate, consider that in predictive coding, a prediction implies that specific information is propagated down from memory through the visual hierarchy. Likewise, recognizing the input implies that sequentially accrued sensory evidence is successfully matched with memorized information (categorical knowledge). Although the existing models of prediction, memory, sensory representation and categorical decision are all implicitly cast within an information processing framework, it remains a challenge to precisely specify what this information is, and therefore where, when and how the architecture of the brain dynamically processes it to produce behaviour. Here, we review a framework that addresses these challenges for the studies of perception and categorization–stimulus information representation (SIR). We illustrate how SIR can reverse engineer the information contents of memory from behavioural and brain measures in the context of specific cognitive tasks that involve memory. We discuss two specific lessons from this approach that generally apply to memory studies: the importance of task, to constrain what the brain does, and of stimulus variations, to identify the specific information contents that are memorized, predicted, recalled and replayed.

This article is part of the Theo Murphy meeting issue ‘Memory reactivation: replaying events past, present and future’.

Number and time in acquisition, extinction and recovery

We measured rate of acquisition, trials to extinction, cumulative responses in extinction, and the spontaneous recovery of anticipatory hopper poking in a Pavlovian protocol with mouse subjects. We varied by factors of 4 number of sessions, trials per session, intersession interval, and span of training (number of days over which training extended). We find that different variables affect each measure: Rate of acquisition [1/(trials to acquisition)] is faster when there are fewer trials per session. Terminal rate of responding is faster when there are more total training trials. Trials to extinction and amount of responding during extinction are unaffected by these variables. The number of training trials has no effect on recovery in a 4-trial probe session 21 days after extinction. However, recovery is greater when the span of training is greater, regardless of how many sessions there are within that span. Our results and those of others suggest that the numbers and durations and spacings of longer-duration "episodes" in a conditioning protocol (sessions and the spans in days of training and extinction) are important variables and that different variables affect different aspects of subjects' behavior. We discuss the theoretical and clinical implications of these and related findings and conclusions-for theories of conditioning and for neuroscience.

The neurobiological foundation of memory retrieval

Memory retrieval involves the interaction between external sensory or internally generated cues and stored memory traces (or engrams) in a process termed 'ecphory'. While ecphory has been examined in human cognitive neuroscience research, its neurobiological foundation is less understood. To the extent that ecphory involves 'reawakening' of engrams, leveraging recently developed technologies that can identify and manipulate engrams in rodents provides a fertile avenue for examining retrieval at the level of neuronal ensembles. Here we evaluate emerging neuroscientific research of this type, using cognitive theory as a guiding principle to organize and interpret initial findings. Our Review highlights the critical interaction between engrams and retrieval cues (environmental or artificial) for memory accessibility and retrieval success. These findings also highlight the intimate relationship between the mechanisms important in forming engrams and those important in their recovery, as captured in the cognitive notion of 'encoding specificity'. Finally, we identify several questions that currently remain unanswered.

Alpha-band oscillations track the retrieval of precise spatial representations from long-term memory

A hallmark of episodic memory is the phenomenon of mentally reexperiencing the details of past events, and a well-established concept is that the neuronal activity that mediates encoding is reinstated at retrieval. Evidence for reinstatement has come from multiple modalities, including functional magnetic resonance imaging and electroencephalography (EEG). These EEG studies have shed light on the time course of reinstatement but have been limited to distinguishing between a few categories. The goal of this work was to use recently developed experimental and technical approaches, namely continuous report tasks and inverted encoding models, to determine which frequencies of oscillatory brain activity support the retrieval of precise spatial memories. In experiment 1, we establish that an inverted encoding model applied to multivariate alpha topography tracks the retrieval of precise spatial memories. In experiment 2, we demonstrate that the frequencies and patterns of multivariate activity at study are similar to the frequencies and patterns observed during retrieval. These findings highlight the broad potential for using encoding models to characterize long-term memory retrieval.NEW & NOTEWORTHY Previous EEG work has shown that category-level information observed during encoding is recapitulated during memory retrieval, but studies with this time-resolved method have not demonstrated the reinstatement of feature-specific patterns of neural activity during retrieval. Here we show that EEG alpha-band activity tracks the retrieval of spatial representations from long-term memory. Moreover, we find considerable overlap between the frequencies and patterns of activity that track spatial memories during initial study and at retrieval.

Reinstating verbal memories with virtual contexts: Myth or reality?

When learning new information, contextual information about the encoding situation is stored in addition to the focal memory content. Later, these strings of extra information can help retrieve the learned content as demonstrated by experiments where contextual cues from an encoding situation facilitate remembering and improve memory performance when reinstated during retrieval. This context-dependent memory effect has been investigated over the course of several decades and has been demonstrated with many different types of contexts. Based on this, the widely held belief is that context-dependent memory is a strong and robust effect, with transferable substance for everyday learning and potential clinical applications. Here we report the results of a multi-study design investigating the influence of reinstated visual contexts on memory performance. Data from 120 participants were included in three studies comprising a variety of visual cues. We show convincingly that even rich, salient and fully surrounding visual contexts provided by virtual reality are not sufficient to induce effects of context-dependency in a free recall memory task. We also investigated contextual modulation of oscillatory brain activity in order to test the effect of reinstated neural contexts, which failed to evoke a robust effect when re-tested in an internal conceptual replication study. Moreover, a Bayesian sequential statistical analysis revealed moderate to strong evidence against the hypothesis that reinstatement of visual contexts benefits free recall memory tasks indicating that effects are small and may not be suitable for transfer into everyday learning.

Evidence that neural information flow is reversed between object perception and object reconstruction from memory

Remembering is a reconstructive process, yet little is known about how the reconstruction of a memory unfolds in time in the human brain. Here, we used reaction times and EEG time-series decoding to test the hypothesis that the information flow is reversed when an event is reconstructed from memory, compared to when the same event is initially being perceived. Across three experiments, we found highly consistent evidence supporting such a reversed stream. When seeing an object, low-level perceptual features were discriminated faster behaviourally, and could be decoded from brain activity earlier, than high-level conceptual features. This pattern reversed during associative memory recall, with reaction times and brain activity patterns now indicating that conceptual information was reconstructed more rapidly than perceptual details. Our findings support a neurobiologically plausible model of human memory, suggesting that memory retrieval is a hierarchical, multi-layered process that prioritises semantically meaningful information over perceptual details.

Neural Pattern Classification Tracks Transfer-Appropriate Processing in Episodic Memory

The transfer-appropriate processing (TAP) account holds that episodic memory depends on the overlap between encoding and retrieval processing. In the current study, we employed multivariate pattern analysis (MVPA) of electroencephalography to examine the relevance of spontaneously engaged visual processing during encoding for later retrieval. Human participants encoded word-picture associations, where the picture could be a famous face, a landmark, or an object. At test, we manipulated the retrieval demands by asking participants to retrieve either visual or verbal information about the pictures. MVPA revealed classification between picture categories during early perceptual stages of encoding (∼170 ms). Importantly, these visual category-specific neural patterns were predictive of later episodic remembering, but the direction of the relationship was contingent on the particular retrieval demand of the memory task: a benefit for the visual and a cost for the verbal. A reinstatement of the category-specific neural patterns established during encoding was observed during retrieval, and again the relationship with behavior varied with retrieval demands. Reactivation of visual representations during retrieval was associated with better memory in the visual task, but with lower performance in the verbal task. Our findings support and extend the TAP account by demonstrating that processing of particular aspects during memory formation can also have detrimental effects on later episodic remembering when other aspects of the event are called-for and shed new light on encoding and retrieval interactions in episodic memory.

Serial representation of items during working memory maintenance at letter-selective cortical sites

A key component of working memory is the ability to remember multiple items simultaneously. To understand how the human brain maintains multiple items in memory, we examined direct brain recordings of neural oscillations from neurosurgical patients as they performed a working memory task. We analyzed the data to identify the neural representations of individual memory items by identifying recording sites with broadband gamma activity that varied according to the identity of the letter a subject viewed. Next, we tested a previously proposed model of working memory, which had hypothesized that the neural representations of individual memory items sequentially occurred at different phases of the theta/alpha cycle. Consistent with this model, the phase of the theta/alpha oscillation when stimulus-related gamma activity occurred during maintenance reflected the order of list presentation. These results suggest that working memory is organized by a cortical phase code coordinated by coupled theta/alpha and gamma oscillations and, more broadly, provide support for the serial representation of items in working memory.

Single-Trial EEG Analysis Predicts Memory Retrieval and Reveals Source-Dependent Differences

We used pattern classifiers to extract features related to recognition memory retrieval from the temporal information in single-trial electroencephalography (EEG) data during attempted memory retrieval. Two-class classification was conducted on correctly remembered trials with accurate context (or source) judgments vs. correctly rejected trials. The average accuracy for datasets recorded in a single session was 61% while the average accuracy for datasets recorded in two separate sessions was 56%. To further understand the basis of the classifier’s performance, two other pattern classifiers were trained on different pairs of behavioral conditions. The first of these was designed to use information related to remembering the item and the second to use information related to remembering the contextual information (or source) about the item. Mollison and Curran (2012) had earlier shown that subjects’ familiarity judgments contributed to improved memory of spatial contextual information but not of extrinsic associated color information. These behavioral results were similarly reflected in the event-related potential (ERP) known as the FN400 (an early frontal effect relating to familiarity) which revealed differences between correct and incorrect context memories in the spatial but not color conditions. In our analyses we show that a classifier designed to distinguish between correct and incorrect context memories, more strongly involves early activity (400–500 ms) over the frontal channels for the location distinctions, than for the extrinsic color associations. In contrast, the classifier designed to classify memory for the item (without memory for the context), had more frontal channel involvement for the color associated experiments than for the spatial experiments. Taken together these results argue that location may be bound more tightly with the item than an extrinsic color association. The multivariate classification approach also showed that trial-by-trial variation in EEG corresponding to these ERP components were predictive of subjects’ behavioral responses. Additionally, the multivariate classification approach enabled analysis of error conditions that did not have sufficient trials for standard ERP analyses. These results suggested that false alarms were primarily attributable to item memory (as opposed to memory of associated context), as commonly predicted, but with little previous corroborating EEG evidence.

Alpha Oscillations during Incidental Encoding Predict Subsequent Memory for New "Foil" Information

People can employ adaptive strategies to increase the likelihood that previously encoded information will be successfully retrieved. One such strategy is to constrain retrieval toward relevant information by reimplementing the neurocognitive processes that were engaged during encoding. Using EEG, we examined the temporal dynamics with which constraining retrieval toward semantic versus nonsemantic information affects the processing of new “foil” information encountered during a memory test. Time–frequency analysis of EEG data acquired during an initial study phase revealed that semantic compared with nonsemantic processing was associated with alpha decreases in a left frontal electrode cluster from around 600 msec after stimulus onset. Successful encoding of semantic versus nonsemantic foils during a subsequent memory test was related to decreases in alpha oscillatory activity in the same left frontal electrode cluster, which emerged relatively late in the trial at around 1000–1600 msec after stimulus onset. Across participants, left frontal alpha power elicited by semantic processing during the study phase correlated significantly with left frontal alpha power associated with semantic foil encoding during the memory test. Furthermore, larger left frontal alpha power decreases elicited by semantic foil encoding during the memory test predicted better subsequent semantic foil recognition in an additional surprise foil memory test, although this effect did not reach significance. These findings indicate that constraining retrieval toward semantic information involves reimplementing semantic encoding operations that are mediated by alpha oscillations and that such reimplementation occurs at a late stage of memory retrieval, perhaps reflecting additional monitoring processes.

Event Boundaries Trigger Rapid Memory Reinstatement of the Prior Events to Promote Their Representation in Long-Term Memory

Although everyday experiences unfold continuously over time, shifts in context, or event boundaries, can influence how those events come to be represented in memory [1-4]. Specifically, mnemonic binding across sequential representations is more challenging at context shifts, such that successful temporal associations are more likely to be formed within than across contexts [1, 2, 5-9]. However, in order to preserve a subjective sense of continuity, it is important that the memory system bridge temporally adjacent events, even if they occur in seemingly distinct contexts. Here, we used pattern similarity analysis to scalp electroencephalographic (EEG) recordings during a sequential learning task [2, 3] in humans and showed that the detection of event boundaries triggered a rapid memory reinstatement of the just-encoded sequence episode. Memory reactivation was detected rapidly (∼200-800 ms from the onset of the event boundary) and was specific to context shifts that were preceded by an event sequence with episodic content. Memory reinstatement was not observed during the sequential encoding of events within an episode, indicating that memory reactivation was induced specifically upon context shifts. Finally, the degree of neural similarity between neural responses elicited during sequence encoding and at event boundaries correlated positively with participants' ability to later link across sequences of events, suggesting a critical role in binding temporally adjacent events in long-term memory. Current results shed light onto the neural mechanisms that promote episodic encoding not only for information within the event, but also, importantly, in the ability to link across events to create a memory representation of continuous experience.

Top-down modulation of alpha power and pattern similarity for threatening representations in visual short-term memory

Recent studies have shown that top-down attention biases task-relevant representations in visual short-term memory (VSTM). Accumulating evidence has also revealed the modulatory effects of emotional arousal on attentional processing. However, it remains unclear how top-down attention interacts with emotional memoranda in VSTM. In this study, we investigated the mechanisms of alpha oscillations and their spatiotemporal characteristics that underlie top-down attention to threatening representations during VSTM maintenance with electroencephalography. Participants were instructed to remember a threatening object and a neutral object in a cued variant delayed response task. Retrospective cues (retro-cues) were presented to direct attention to the hemifield of a threatening object (i.e., cue-to-threat trials) or a neutral object (i.e., cue-to-neutral trials) during a retention interval prior to the probe test. We found a significant retro-cue-related alpha lateralisation over posterior regions during VSTM maintenance. The novel finding was that the magnitude of alpha lateralisation was greater for cue-to-threat objects compared to cue-to-neutral ones. These results indicated that directing attention towards threatening representations compared to neutral representations could result in greater regulation of alpha activity contralateral to the cued hemifield. Importantly, we estimated the spatiotemporal pattern similarity in alpha activity and found significantly higher similarity indexes for the posterior regions relative to the anterior regions and for the cue-to-threat objects relative to cue-to-neutral objects over the posterior regions. Together, our findings provided the oscillatory evidence of greater top-down modulations of alpha lateralisation and spatiotemporal pattern similarity for attending to threatening representations in VSTM.

Distinct Neurophysiological Mechanisms Support the Online Formation of Individual and Across-Episode Memory Representations

Individual experiences often overlap in their content, presenting opportunities for rapid generalization across them. In this study, we show in 2 independent experiments that integrative encoding-the ability to form individual and across memory representations during online encoding-is supported by 2 distinct neurophysiological responses. Brain potential is increased gradually during encoding and fit to a trial level memory measure for individual episodes, whereas neural oscillations in the theta range (4-6 Hz) emerge later during learning and predict participants' generalization performance in a subsequent test. These results suggest that integrative encoding requires the recruitment of 2 separate neural mechanisms that, despite their co-occurrence in time, differ in their underlying neural dynamics, reflect different brain learning rates and are supportive of the formation of opposed memory representations, individual versus across-event episodes.

Working Memory Replay Prioritizes Weakly Attended Events

One view of working memory posits that maintaining a series of events requires their sequential and equal mnemonic replay. Another view is that the content of working memory maintenance is prioritized by attention. We decoded the dynamics for retaining a sequence of items using magnetoencephalography, wherein participants encoded sequences of three stimuli depicting a face, a manufactured object, or a natural item and maintained them in working memory for 5000 ms. Memory for sequence position and stimulus details were probed at the end of the maintenance period. Decoding of brain activity revealed that one of the three stimuli dominated maintenance independent of its sequence position or category; and memory was enhanced for the selectively replayed stimulus. Analysis of event-related responses during the encoding of the sequence showed that the selectively replayed stimuli were determined by the degree of attention at encoding. The selectively replayed stimuli had the weakest initial encoding indexed by weaker visual attention signals at encoding. These findings do not rule out sequential mnemonic replay but reveal that attention influences the content of working memory maintenance by prioritizing replay of weakly encoded events. We propose that the prioritization of weakly encoded stimuli protects them from interference during the maintenance period, whereas the more strongly encoded stimuli can be retrieved from long-term memory at the end of the delay period.

Transformed Neural Pattern Reinstatement during Episodic Memory Retrieval

Contemporary models of episodic memory posit that remembering involves the reenactment of encoding processes. Although encoding-retrieval similarity has been consistently reported and linked to memory success, the nature of neural pattern reinstatement is poorly understood. Using high-resolution fMRI on human subjects, our results obtained clear evidence for item-specific pattern reinstatement in the frontoparietal cortex, even when the encoding-retrieval pairs shared no perceptual similarity. No item-specific pattern reinstatement was found in the ventral visual cortex. Importantly, the brain regions and voxels carrying item-specific representation differed significantly between encoding and retrieval, and the item specificity for encoding-retrieval similarity was smaller than that for encoding or retrieval, suggesting different nature of representations between encoding and retrieval. Moreover, cross-region representational similarity analysis suggests that the encoded representation in the ventral visual cortex was reinstated in the frontoparietal cortex during retrieval. Together, these results suggest that, in addition to reinstatement of the originally encoded pattern in the brain regions that perform encoding processes, retrieval may also involve the reinstatement of a transformed representation of the encoded information. These results emphasize the constructive nature of memory retrieval that helps to serve important adaptive functions.SIGNIFICANCE STATEMENT Episodic memory enables humans to vividly reexperience past events, yet how this is achieved at the neural level is barely understood. A long-standing hypothesis posits that memory retrieval involves the faithful reinstatement of encoding-related activity. We tested this hypothesis by comparing the neural representations during encoding and retrieval. We found strong pattern reinstatement in the frontoparietal cortex, but not in the ventral visual cortex, that represents visual details. Critically, even within the same brain regions, the nature of representation during retrieval was qualitatively different from that during encoding. These results suggest that memory retrieval is not a faithful replay of past event but rather involves additional constructive processes to serve adaptive functions.