Transformative neural representations support long-term episodic memory

Re-evaluating human MTL in working memory: insights from intracranial recordings

The study of human working memory (WM) holds significant importance in neuroscience; yet, exploring the role of the medial temporal lobe (MTL) in WM has been limited by the technological constraints of noninvasive methods. Recent advancements in human intracranial neural recordings have indicated the involvement of the MTL in WM processes. These recordings show that different regions of the MTL are involved in distinct aspects of WM processing and also dynamically interact with each other and the broader brain network. These findings support incorporating the MTL into models of the neural basis of WM. This integration can better reflect the complex neural mechanisms underlying WM and enhance our understanding of WM's flexibility, adaptability, and precision.

Convolutional neural networks uncover the dynamics of human visual memory representations over time

The ability to accurately retrieve visual details of past events is a fundamental cognitive function relevant for daily life. While a visual stimulus contains an abundance of information, only some of it is later encoded into long-term memory representations. However, an ongoing challenge has been to isolate memory representations that integrate various visual features and uncover their dynamics over time. To address this question, we leveraged a novel combination of empirical and computational frameworks based on the hierarchal structure of convolutional neural networks and their correspondence to human visual processing. This enabled to reveal the contribution of different levels of visual representations to memory strength and their dynamics over time. Visual memory strength was measured with distractors selected based on their shared similarity to the target memory along low or high layers of the convolutional neural network hierarchy. The results show that visual working memory relies similarly on low and high-level visual representations. However, already after a few minutes and on to the next day, visual memory relies more strongly on high-level visual representations. These findings suggest that visual representations transform from a distributed to a stronger high-level conceptual representation, providing novel insights into the dynamics of visual memory over time.

Distinct mechanisms and functions of episodic memory

The concept of episodic memory (EM) faces significant challenges by two claims: EM might not be a distinct memory system, and EM might be an epiphenomenon of a more general capacity for mental time travel (MTT). Nevertheless, the observations leading to these arguments do not preclude the existence of a mechanically and functionally distinct EM system. First, modular systems, like cognition, can have distinct subsystems that may not be distinguishable in the system's final output. EM could be such a subsystem, even though its effects may be difficult to distinguish from those of other subsystems. Second, EM could have a distinct and consistent low-level function, which is used in diverse high-level functions such as MTT. This article introduces the scenario construction framework, proposing that EM crucially rests on memory traces containing the gist of an episodic experience. During retrieval, EM traces trigger the reconstruction of semantic representations, which were active during the remembered episode, and are further enriched with semantic information, to generate a scenario of the past experience. This conceptualization of EM is consistent with studies on the neural basis of EM and resolves the two challenges while retaining the key properties associated with EM. This article is part of the theme issue 'Elements of episodic memory: lessons from 40 years of research'.

Semantic Content in Face Representation: Essential for Proficient Recognition of Unfamiliar Faces by Good Recognizers

Face recognition is adapted to achieve goals of social interactions, which rely on further processing of the semantic information of faces, beyond visual computations. Here, we explored the semantic content of face representation apart from visual component, and tested their relations to face recognition performance. Specifically, we propose that enhanced visual or semantic coding could underlie the advantage of familiar over unfamiliar faces recognition, as well as the superior recognition of skilled face recognizers. We asked participants to freely describe familiar/unfamiliar faces using words or phrases, and converted these descriptions into semantic vectors. Face semantics were transformed into quantifiable face vectors by aggregating these word/phrase vectors. We also extracted visual features from a deep convolutional neural network and obtained the visual representation of familiar/unfamiliar faces. Semantic and visual representations were used to predict perceptual representation generated from a behavior rating task separately in different groups (bad/good face recognizers in familiar-face/unfamiliar-face conditions). Comparisons revealed that although long-term memory facilitated visual feature extraction for familiar faces compared to unfamiliar faces, good recognizers compensated for this disparity by incorporating more semantic information for unfamiliar faces, a strategy not observed in bad recognizers. This study highlights the significance of semantics in recognizing unfamiliar faces.

Coordinated NREM sleep oscillations among hippocampal subfields modulate synaptic plasticity in humans

The integration of hippocampal oscillations during non-rapid eye movement (NREM) sleep is crucial for memory consolidation. However, how cardinal sleep oscillations bind across various subfields of the human hippocampus to promote information transfer and synaptic plasticity remains unclear. Using human intracranial recordings from 25 epilepsy patients, we find that hippocampal subfields, including DG/CA3, CA1, and SUB, all exhibit significant delta and spindle power during NREM sleep. The DG/CA3 displays strong coupling between delta and ripple oscillations with all the other hippocampal subfields. In contrast, the regions of CA1 and SUB exhibit more precise coordination, characterized by event-level triple coupling between delta, spindle, and ripple oscillations. Furthermore, we demonstrate that the synaptic plasticity within the hippocampal circuit, as indexed by delta-wave slope, is linearly modulated by spindle power. In contrast, ripples act as a binary switch that triggers a sudden increase in delta-wave slope. Overall, these results suggest that different subfields of the hippocampus regulate one another through diverse layers of sleep oscillation synchronization, collectively facilitating information processing and synaptic plasticity during NREM sleep.

Maintenance and transformation of representational formats during working memory prioritization

Visual working memory depends on both material-specific brain areas in the ventral visual stream (VVS) that support the maintenance of stimulus representations and on regions in the prefrontal cortex (PFC) that control these representations. How executive control prioritizes working memory contents and whether this affects their representational formats remains an open question, however. Here, we analyzed intracranial EEG (iEEG) recordings in epilepsy patients with electrodes in VVS and PFC who performed a multi-item working memory task involving a retro-cue. We employed Representational Similarity Analysis (RSA) with various Deep Neural Network (DNN) architectures to investigate the representational format of prioritized VWM content. While recurrent DNN representations matched PFC representations in the beta band (15-29 Hz) following the retro-cue, they corresponded to VVS representations in a lower frequency range (3-14 Hz) towards the end of the maintenance period. Our findings highlight the distinct coding schemes and representational formats of prioritized content in VVS and PFC.

The dynamic reconfiguration of the functional network during episodic memory task predicts the memory performance

Episodic memory is essential for forming and retaining personal experiences, representing a fundamental aspect of human cognition. Traditional studies of episodic memory have typically used static analysis methods, viewing the brain as an unchanging entity and overlooking its dynamic properties over time. In this study, we utilized dynamic functional connectivity analysis on fMRI data from healthy adults performing an episodic memory task. We quantified integration and recruitment metrics and examined their correlation with memory performance using Pearson correlation. During encoding, integration across the entire brain, especially within the frontoparietal subnetwork, was significantly correlated with memory performance. During retrieval, recruitment becomes significantly associated with memory performance in visual subnetwork, somatomotor subnetwork, and ventral attention subnetwork. At the nodal level, a significant negative correlation was observed between memory scores and integration of the anterior cingulate gyrus, precentral gyrus, and inferior frontal gyrus within the frontoparietal network during encoding task. During retrieval task, a significant negative correlation was found between memory scores and recruitment in the left progranular cortex and right transverse gyral ventral, whereas positive correlations were seen in the right posterior inferior temporal, left middle temporal, right frontal operculum, and left operculum nodes. Moreover, the dynamic reconfiguration of the functional network was predictive of predict memory performance, as demonstrated by a significant correlation between actual and predicted memory scores. These findings advance our understanding network mechanisms underlying memory processes and developing intervention approaches for memory-related disorders as they shed light on critical factors involved in cognitive processes and provide a deeper understanding of the underlying mechanisms driving cognitive function.

Goal-directed attention transforms both working and long-term memory representations in the human parietal cortex

The abundance of distractors in the world poses a major challenge to our brain's limited processing capacity, but little is known about how selective attention modulates stimulus representations in the brain to reduce interference and support durable target memory. Here, we collected functional magnetic resonance imaging (fMRI) data in a selective attention task in which target and distractor pictures of different visual categories were simultaneously presented. Participants were asked to selectively process the target according to the effective cue, either before the encoding period (i.e., perceptual attention) or the maintenance period (i.e., reflective attention). On the next day, participants were asked to perform a memory recognition task in the scanner in which the targets, distractors, and novel items were presented in a pseudorandom order. Behavioral results showed that perceptual attention was better at enhancing target memory and reducing distractor memory than reflective attention, although the overall memory capacity (memory for both target and distractor) was comparable. Using multiple-voxel pattern analysis of the neural data, we found more robust target representation and weaker distractor representation in working memory for perceptual attention than for reflective attention. Interestingly, perceptual attention partially shifted the regions involved in maintaining the target representation from the visual cortex to the parietal cortex. Furthermore, the targets and distractors simultaneously presented in the perceptual attention condition showed reduced pattern similarity in the parietal cortex during retrieval compared to items not presented together. This neural pattern repulsion positively correlated with individuals' recognition of both targets and distractors. These results emphasize the critical role of selective attention in transforming memory representations to reduce interference and improve long-term memory performance.

Images with harder-to-reconstruct visual representations leave stronger memory traces

Much of what we remember is not because of intentional selection, but simply a by-product of perceiving. This raises a foundational question about the architecture of the mind: how does perception interface with and influence memory? Here, inspired by a classic proposal relating perceptual processing to memory durability, the level-of-processing theory, we present a sparse coding model for compressing feature embeddings of images, and show that the reconstruction residuals from this model predict how well images are encoded into memory. In an open memorability dataset of scene images, we show that reconstruction error not only explains memory accuracy, but also response latencies during retrieval, subsuming, in the latter case, all of the variance explained by powerful vision-only models. We also confirm a prediction of this account with 'model-driven psychophysics'. This work establishes reconstruction error as an important signal interfacing perception and memory, possibly through adaptive modulation of perceptual processing.

Visual Recognition Memory of Scenes Is Driven by Categorical, Not Sensory, Visual Representations

When we perceive a scene, our brain processes various types of visual information simultaneously, ranging from sensory features, such as line orientations and colors, to categorical features, such as objects and their arrangements. Whereas the role of sensory and categorical visual representations in predicting subsequent memory has been studied using isolated objects, their impact on memory for complex scenes remains largely unknown. To address this gap, we conducted an fMRI study in which female and male participants encoded pictures of familiar scenes (e.g., an airport picture) and later recalled them, while rating the vividness of their visual recall. Outside the scanner, participants had to distinguish each seen scene from three similar lures (e.g., three airport pictures). We modeled the sensory and categorical visual features of multiple scenes using both early and late layers of a deep convolutional neural network. Then, we applied representational similarity analysis to determine which brain regions represented stimuli in accordance with the sensory and categorical models. We found that categorical, but not sensory, representations predicted subsequent memory. In line with the previous result, only for the categorical model, the average recognition performance of each scene exhibited a positive correlation with the average visual dissimilarity between the item in question and its respective lures. These results strongly suggest that even in memory tests that ostensibly rely solely on visual cues (such as forced-choice visual recognition with similar distractors), memory decisions for scenes may be primarily influenced by categorical rather than sensory representations.

Representational formats of human memory traces

Neural representations are internal brain states that constitute the brain's model of the external world or some of its features. In the presence of sensory input, a representation may reflect various properties of this input. When perceptual information is no longer available, the brain can still activate representations of previously experienced episodes due to the formation of memory traces. In this review, we aim at characterizing the nature of neural memory representations and how they can be assessed with cognitive neuroscience methods, mainly focusing on neuroimaging. We discuss how multivariate analysis techniques such as representational similarity analysis (RSA) and deep neural networks (DNNs) can be leveraged to gain insights into the structure of neural representations and their different representational formats. We provide several examples of recent studies which demonstrate that we are able to not only measure memory representations using RSA but are also able to investigate their multiple formats using DNNs. We demonstrate that in addition to slow generalization during consolidation, memory representations are subject to semantization already during short-term memory, by revealing a shift from visual to semantic format. In addition to perceptual and conceptual formats, we describe the impact of affective evaluations as an additional dimension of episodic memories. Overall, these studies illustrate how the analysis of neural representations may help us gain a deeper understanding of the nature of human memory.

Decoding face recognition abilities in the human brain

Why are some individuals better at recognizing faces? Uncovering the neural mechanisms supporting face recognition ability has proven elusive. To tackle this challenge, we used a multimodal data-driven approach combining neuroimaging, computational modeling, and behavioral tests. We recorded the high-density electroencephalographic brain activity of individuals with extraordinary face recognition abilities-super-recognizers-and typical recognizers in response to diverse visual stimuli. Using multivariate pattern analyses, we decoded face recognition abilities from 1 s of brain activity with up to 80% accuracy. To better understand the mechanisms subtending this decoding, we compared representations in the brains of our participants with those in artificial neural network models of vision and semantics, as well as with those involved in human judgments of shape and meaning similarity. Compared to typical recognizers, we found stronger associations between early brain representations of super-recognizers and midlevel representations of vision models as well as shape similarity judgments. Moreover, we found stronger associations between late brain representations of super-recognizers and representations of the artificial semantic model as well as meaning similarity judgments. Overall, these results indicate that important individual variations in brain processing, including neural computations extending beyond purely visual processes, support differences in face recognition abilities. They provide the first empirical evidence for an association between semantic computations and face recognition abilities. We believe that such multimodal data-driven approaches will likely play a critical role in further revealing the complex nature of idiosyncratic face recognition in the human brain.

Harmonic memory signals in the human cerebral cortex induced by semantic relatedness of words

When we memorize multiple words simultaneously, semantic relatedness among those words assists memory. For example, the information about "apple", "banana," and "orange" will be connected via a common concept of "fruits" and become easy to retain and recall. Neural mechanisms underlying this semantic integration in verbal working memory remain unclear. Here I used electroencephalography (EEG) and investigated neural signals when healthy human participants memorized five nouns semantically related (Sem trial) or not (NonSem trial). The regularity of oscillatory signals (8-30 Hz) during the retention period was found to be lower in NonSem than Sem trials, indicating that memorizing words unrelated to each other induced a non-harmonic (irregular) waveform in the temporal cortex. These results suggest that (i) semantic features of a word are retained as a set of neural oscillations at specific frequencies and (ii) memorizing words sharing a common semantic feature produces harmonic brain responses through a resonance or integration (sharing) of the oscillatory signals.

What is it like to do a visuo-spatial working memory task: A qualitative phenomenological study of the visual span task

Working memory is typically measured with specifically designed psychological tasks. When evaluating the validity of working memory tasks, we commonly focus on the reliability of the outcome measurements. Only rarely do we focus on how participants experience these tasks. Accounting for lived experience of working memory task may help us better understand variability in working memory performance and conscious experience in general. We replicated recently established protocols for the phenomenological investigation of working memory using the visual span task. We collected subjective reports from eighteen healthy participants (10 women) aged 21 to 35 years. We observed that working memory can be phenomenologically characterized at three different time scales: background feelings, strategies, and tactics. On the level of tactics, we identified transmodality (i.e., how one modality of lived experience can be transformed into another one) as the central phenomenological dynamic at play during working memory task performance.

Static and dynamic coding in distinct cell types during associative learning in the prefrontal cortex

The prefrontal cortex maintains information in memory through static or dynamic population codes depending on task demands, but whether the population coding schemes used are learning-dependent and differ between cell types is currently unknown. We investigate the population coding properties and temporal stability of neurons recorded from male macaques in two mapping tasks during and after stimulus-response associative learning, and then we use a Strategy task with the same stimuli and responses as control. We identify a heterogeneous population coding for stimuli, responses, and novel associations: static for putative pyramidal cells and dynamic for putative interneurons that show the strongest selectivity for all the variables. The population coding of learned associations shows overall the highest stability driven by cell types, with interneurons changing from dynamic to static coding after successful learning. The results support that prefrontal microcircuitry expresses mixed population coding governed by cell types and changes its stability during associative learning.

Item-specific neural representations during human sleep support long-term memory

Understanding how individual memories are reactivated during sleep is essential in theorizing memory consolidation. Here, we employed the targeted memory reactivation (TMR) paradigm to unobtrusively replaying auditory memory cues during human participants' slow-wave sleep (SWS). Using representational similarity analysis (RSA) on cue-elicited electroencephalogram (EEG), we found temporally segregated and functionally distinct item-specific neural representations: the early post-cue EEG activity (within 0 to 2,000 ms) contained comparable item-specific representations for memory cues and control cues, signifying effective processing of auditory cues. Critically, the later EEG activity (2,500 to 2,960 ms) showed greater item-specific representations for post-sleep remembered items than for forgotten and control cues, indicating memory reprocessing. Moreover, these later item-specific neural representations were supported by concurrently increased spindles, particularly for items that had not been tested prior to sleep. These findings elucidated how external memory cues triggered item-specific neural representations during SWS and how such representations were linked to successful long-term memory. These results will benefit future research aiming to perturb specific memory episodes during sleep.

Dynamic neural representations of memory and space during human ambulatory navigation

Our ability to recall memories of personal experiences is an essential part of daily life. These episodic memories often involve movement through space and thus require continuous encoding of one's position relative to the surrounding environment. The medial temporal lobe (MTL) is thought to be critically involved, based on studies in freely moving rodents and stationary humans. However, it remains unclear if and how the MTL represents both space and memory especially during physical navigation, given challenges associated with deep brain recordings in humans during movement. We recorded intracranial electroencephalographic (iEEG) activity while participants completed an ambulatory spatial memory task within an immersive virtual reality environment. MTL theta activity was modulated by successful memory retrieval or spatial positions within the environment, depending on dynamically changing behavioral goals. Altogether, these results demonstrate how human MTL oscillations can represent both memory and space in a temporally flexible manner during freely moving navigation.

Intersubject similarity in neural representations underlies shared episodic memory content

Individuals generally form their unique memories from shared experiences, yet the neural representational mechanisms underlying this subjectiveness of memory are poorly understood. The current study addressed this important question from the cross-subject neural representational perspective, leveraging a large functional magnetic resonance imaging dataset (n = 415) of a face-name associative memory task. We found that individuals' memory abilities were predicted by their synchronization to the group-averaged, canonical trial-by-trial activation level and, to a lesser degree, by their similarity to the group-averaged representational patterns during encoding. More importantly, the memory content shared between pairs of participants could be predicted by their shared local neural activation pattern, particularly in the angular gyrus and ventromedial prefrontal cortex, even after controlling for differences in memory abilities. These results uncover neural representational mechanisms for individualized memory and underscore the constructive nature of episodic memory.

Cognitive neuroscience perspective on memory: overview and summary

This paper explores memory from a cognitive neuroscience perspective and examines associated neural mechanisms. It examines the different types of memory: working, declarative, and non-declarative, and the brain regions involved in each type. The paper highlights the role of different brain regions, such as the prefrontal cortex in working memory and the hippocampus in declarative memory. The paper also examines the mechanisms that underlie the formation and consolidation of memory, including the importance of sleep in the consolidation of memory and the role of the hippocampus in linking new memories to existing cognitive schemata. The paper highlights two types of memory consolidation processes: cellular consolidation and system consolidation. Cellular consolidation is the process of stabilizing information by strengthening synaptic connections. System consolidation models suggest that memories are initially stored in the hippocampus and are gradually consolidated into the neocortex over time. The consolidation process involves a hippocampal-neocortical binding process incorporating newly acquired information into existing cognitive schemata. The paper highlights the role of the medial temporal lobe and its involvement in autobiographical memory. Further, the paper discusses the relationship between episodic and semantic memory and the role of the hippocampus. Finally, the paper underscores the need for further research into the neurobiological mechanisms underlying non-declarative memory, particularly conditioning. Overall, the paper provides a comprehensive overview from a cognitive neuroscience perspective of the different processes involved in memory consolidation of different types of memory.

Adaptive Memory of a Neuromorphic Transistor with Multi-Sensory Signal Fusion

One of the ultimate goals of artificial intelligence is to achieve the capability of memory evolution and adaptability to changing environments, which is termed adaptive memory. To realize adaptive memory in artificial neuromorphic devices, artificial synapses with multi-sensing capability are required to collect and analyze various sensory cues from the external changing environments. However, due to the lack of platforms for mediating multiple sensory signals, most artificial synapses have been mainly limited to unimodal or bimodal sensory devices. Herein, we present a multi-modal artificial sensory synapse (MASS) based on an organic synapse to realize sensory fusion and adaptive memory. The MASS receives optical, electrical, and pressure information and in turn generates typical synaptic behaviors, mimicking the multi-sensory neurons in the brain. Sophisticated synaptic behaviors, such as Pavlovian dogs, writing/erasing, signal accumulation, and offset, were emulated to demonstrate the joint efforts of bimodal sensory cues. Moreover, associative memory can be formed in the device and be subsequently reshaped by signals from a third perception, mimicking modification of the memory and cognition when encountering a new environment. Our MASS provides a step toward next-generation artificial neural networks with an adaptive memory capability.

Continuously changing memories: a framework for proactive and non-linear consolidation

The traditional view of long-term memory is that memory traces mature in a predetermined 'linear' process: their neural substrate shifts from rapidly plastic medial temporal regions towards stable neocortical networks. We propose that memories remain malleable, not by repeated reinstantiations of this linear process but instead via dynamic routes of proactive and non-linear consolidation: memories change, their trajectory is flexible and reversible, and their physical basis develops continuously according to anticipated demands. Studies demonstrating memory updating, increasing hippocampal dependence to support adaptive use, and rapid neocortical plasticity provide evidence for continued non-linear consolidation. Although anticipated demand can affect all stages of memory formation, the extent to which it shapes the physical memory trace repeatedly and proactively will require further dedicated research.

Reactivating ordinal position information from auditory sequence memory in human brains

Retaining a sequence of events in their order is a core ability of many cognitive functions, such as speech recognition, movement control, and episodic memory. Although content representations have been widely studied in working memory (WM), little is known about how ordinal position information of an auditory sequence is retained in the human brain as well as its coding characteristics. In fact, there is still a lack of an efficient approach to directly accessing the stored ordinal position code during WM retention. Here, 31 participants performed an auditory sequence WM task with their brain activities recorded using electroencephalography (EEG). We developed new triggering events that could successfully reactivate neural representations of ordinal position during the delay period. Importantly, the ordinal position reactivation is further related to recognition behavior, confirming its indexing of WM storage. Furthermore, the ordinal position code displays an intriguing “stable-dynamic” format, i.e. undergoing the same dynamic neutral trajectory in the multivariate neural space during both encoding and retention (whenever reactivated). Overall, our results provide an effective approach to accessing the behaviorally-relevant ordinal position information in auditory sequence WM and reveal its new temporal characteristics.

Semantically congruent audiovisual integration with modal-based attention accelerates auditory short-term memory retrieval

Evidence has shown that multisensory integration benefits to unisensory perception performance are asymmetric and that auditory perception performance can receive more multisensory benefits, especially when the attention focus is directed toward a task-irrelevant visual stimulus. At present, whether the benefits of semantically (in)congruent multisensory integration with modal-based attention for subsequent unisensory short-term memory (STM) retrieval are also asymmetric remains unclear. Using a delayed matching-to-sample paradigm, the present study investigated this issue by manipulating the attention focus during multisensory memory encoding. The results revealed that both visual and auditory STM retrieval reaction times were faster under semantically congruent multisensory conditions than under unisensory memory encoding conditions. We suggest that coherent multisensory representation formation might be optimized by restricted multisensory encoding and can be rapidly triggered by subsequent unisensory memory retrieval demands. Crucially, auditory STM retrieval is exclusively accelerated by semantically congruent multisensory memory encoding, indicating that the less effective sensory modality of memory retrieval relies more on the coherent prior formation of a multisensory representation optimized by modal-based attention.

Higher-dimensional neural representations predict better episodic memory

Episodic memory enables humans to encode and later vividly retrieve information about our rich experiences, yet the neural representations that support this mental capacity are poorly understood. Using a large fMRI dataset (n = 468) of face-name associative memory tasks and principal component analysis to examine neural representational dimensionality (RD), we found that the human brain maintained a high-dimensional representation of faces through hierarchical representation within and beyond the face-selective regions. Critically, greater RD was associated with better subsequent memory performance both within and across participants, and this association was specific to episodic memory but not general cognitive abilities. Furthermore, the frontoparietal activities could suppress the shared low-dimensional fluctuations and reduce the correlations of local neural responses, resulting in greater RD. RD was not associated with the degree of item-specific pattern similarity, and it made complementary contributions to episodic memory. These results provide a mechanistic understanding of the role of RD in supporting accurate episodic memory.