Shared Representational Formats for Information Maintained in Working Memory and Information Retrieved from Long-Term Memory

Contributions of repeated learning to memory in humans: insights from single-neuron recordings in the hippocampus and amygdala

Despite the well-established phenomenon of improved memory performance through repeated learning, studies investigating the associated neural mechanisms have yielded complex and sometimes contradictory findings, and direct evidence from human neuronal recordings has been lacking. This study employs single-neuron recordings with exceptional spatial-temporal resolution, combined with representational similarity analysis, to explore the neural dynamics within the hippocampus and amygdala during repeated learning. Our results demonstrate that in the hippocampus, repetition enhances both representational specificity and fidelity, with these features predicting learning times. Conversely, the amygdala exhibits heightened representational specificity and fidelity during initial learning but does not show improvement with repetition, suggesting functional specialization of the hippocampus and amygdala during different stages of the learning repetition. Specifically, the hippocampus appears to contribute to sustained engagement necessary for benefiting from repeated learning, while the amygdala may play a role in the representation of novel items. These findings contribute to a comprehensive understanding of the intricate interplay between these brain regions in memory processes. Significance statement  For over a century, understanding how repetition contributes to memory enhancement has captivated researchers, yet direct neuronal evidence has been lacking, with a primary focus on the hippocampus and a neglect of the neighboring amygdala. Employing advanced single-neuron recordings and analytical techniques, this study unveils a nuanced functional specialization within the amygdala-hippocampal circuit during various learning repetition. The results highlight the hippocampus's role in sustaining engagement for improved memory with repetition, contrasting with the amygdala's superior ability in representing novel items. This exploration not only deepens our comprehension of memory enhancement intricacies but also sheds light on potential interventions to optimize learning and memory processes.

Feedback scales the spatial tuning of cortical responses during visual memory

Perception, working memory, and long-term memory each evoke neural responses in visual cortex, suggesting that memory uses encoding mechanisms shared with perception. While previous research has largely focused on how perception and memory are similar, we hypothesized that responses in visual cortex would differ depending on the origins of the inputs. Using fMRI, we quantified spatial tuning in visual cortex while participants (both sexes) viewed, maintained in working memory, or retrieved from long-term memory a peripheral target. In each of these conditions, BOLD responses were spatially tuned and were aligned with the target's polar angle in all measured visual field maps including V1. As expected given the increasing sizes of receptive fields, polar angle tuning during perception increased in width systematically up the visual hierarchy from V1 to V2, V3, hV4, and beyond. In stark contrast, the widths of tuned responses were broad across the visual hierarchy during working memory and long-term memory, matched to the widths in perception in later visual field maps but much broader in V1. This pattern is consistent with the idea that mnemonic responses in V1 stem from top-down sources. Moreover, these tuned responses when biased (clockwise or counterclockwise of target) predicted matched biases in memory, suggesting that the readout of maintained and reinstated mnemonic responses influences memory guided behavior. We conclude that feedback constrains spatial tuning during memory, where earlier visual maps inherit broader tuning from later maps thereby impacting the precision of memory.

Representational formats of human memory traces

Neural representations are internal brain states that constitute the brain's model of the external world or some of its features. In the presence of sensory input, a representation may reflect various properties of this input. When perceptual information is no longer available, the brain can still activate representations of previously experienced episodes due to the formation of memory traces. In this review, we aim at characterizing the nature of neural memory representations and how they can be assessed with cognitive neuroscience methods, mainly focusing on neuroimaging. We discuss how multivariate analysis techniques such as representational similarity analysis (RSA) and deep neural networks (DNNs) can be leveraged to gain insights into the structure of neural representations and their different representational formats. We provide several examples of recent studies which demonstrate that we are able to not only measure memory representations using RSA but are also able to investigate their multiple formats using DNNs. We demonstrate that in addition to slow generalization during consolidation, memory representations are subject to semantization already during short-term memory, by revealing a shift from visual to semantic format. In addition to perceptual and conceptual formats, we describe the impact of affective evaluations as an additional dimension of episodic memories. Overall, these studies illustrate how the analysis of neural representations may help us gain a deeper understanding of the nature of human memory.

Prediction error and event segmentation in episodic memory

Organizing the continuous flow of experiences into meaningful events is a crucial prerequisite for episodic memory. Prediction error and event segmentation both play important roles in supporting the genesis of meaningful mnemonic representations of events. We review theoretical contributions discussing the relationship between prediction error and event segmentation, as well as literature on episodic memory related to prediction error and event segmentation. We discuss the extent of overlap of mechanisms underlying memory emergence through prediction error and event segmentation, with a specific focus on attention and working memory. Finally, we identify areas in research that are currently developing and suggest future directions. We provide an overview of mechanisms underlying memory formation through predictions, violations of predictions, and event segmentation.

Prefrontal Cortex Codes Representations of Target Identity and Feature Uncertainty

Many objects in the real world have features that vary over time, creating uncertainty in how they will look in the future. This uncertainty makes statistical knowledge about the likelihood of features critical to attention demanding processes such as visual search. However, little is known about how the uncertainty of visual features is integrated into predictions about search targets in the brain. In the current study, we test the idea that regions prefrontal cortex code statistical knowledge about search targets before the onset of search. Across 20 human participants (13 female; 7 male), we observe target identity in the multivariate pattern and uncertainty in the overall activation of dorsolateral prefrontal cortex (DLPFC) and inferior frontal junction (IFJ) in advance of the search display. This indicates that the target identity (mean) and uncertainty (variance) of the target distribution are coded independently within the same regions. Furthermore, once the search display appears the univariate IFJ signal scaled with the distance of the actual target from the expected mean, but more so when expected variability was low. These results inform neural theories of attention by showing how the prefrontal cortex represents both the identity and expected variability of features in service of top-down attentional control.SIGNIFICANCE STATEMENT Theories of attention and working memory posit that when we engage in complex cognitive tasks our performance is determined by how precisely we remember task-relevant information. However, in the real world the properties of objects change over time, creating uncertainty about many aspects of the task. There is currently a gap in our understanding of how neural systems represent this uncertainty and combine it with target identity information in anticipation of attention demanding cognitive tasks. In this study, we show that the prefrontal cortex represents identity and uncertainty as unique codes before task onset. These results advance theories of attention by showing that the prefrontal cortex codes both target identity and uncertainty to implement top-down attentional control.

Neural evidence for advantaged representation of first items in memory

Visual areas activated during perception can retain specific information held in memory without the presence of physical stimuli via distributed activity patterns. Neuroimaging studies have shown that the delay-period representation of information in visual areas is modulated by factors such as memory load and task demands, raising the possibility of serial position as another potential modulator. Specifically, enhanced representation of first items during the post-encoding delay period may serve as a mechanism underlying the well-established but not well-understood primacy effect - the mnemonic advantage of first items. To test this hypothesis, 13 males and 16 females performed a human fMRI task, wherein each trial consisted of the sequential encoding of two stimuli (a famous face and landscape, order counterbalanced), followed by a distracting task, a delay period, and then a cued recall of one of the items. Participants exhibited the expected behavioral primacy effect, manifested as faster recall of the first items. In order to elucidate the still debated neural underpinnings of this effect, using multivariate decoding, a classifier was trained on data collected during encoding to differentiate stimulus categories (i.e., faces vs. landscapes) and tested on data collected during the post-encoding period. Greater reactivation of first versus second items was observed in the ventral occipito-temporal cortex during the entire post-encoding period but not during encoding. Moreover, trial-level analyses revealed that the degree of first-item neural advantage during the post-encoding delay predicted the behavioral primacy effect. These findings highlight the role of item reinstatement in ventral occipito-temporal cortex in the primacy effect and are discussed in the context of the uniqueness of the very first item and event boundaries, illuminating putative neural mechanisms underlying the effect.

The intersection of the retrieval state and internal attention

Large-scale brain states or distributed patterns of brain activity modulate downstream processing and behavior. Sustained attention and memory retrieval states impact subsequent memory, yet how these states relate to one another is unclear. I hypothesize that internal attention is a central process of the retrieval state. The alternative is that the retrieval state specifically reflects a controlled, episodic retrieval mode, engaged only when intentionally accessing events situated within a spatiotemporal context. To test my hypothesis, I developed a mnemonic state classifier independently trained to measure retrieval state evidence and applied this classifier to a spatial attention task. I find that retrieval state evidence increases during delay and response intervals when participants are maintaining spatial information. Critically, retrieval state evidence is positively related to the amount of maintained spatial location information and predicts target detection reaction times. Together, these findings support the hypothesis that internal attention is a central process of the retrieval state.

A gradual transition from veridical to categorical representations along the visual hierarchy during working memory, but not perception

The ability to stably maintain visual information over brief delays is central to cognitive functioning. One possible way to achieve robust working memory maintenance is by having multiple concurrent mnemonic representations across multiple cortical loci. For example, early visual cortex might contribute to storage by representing information in a "sensory-like" format, while intraparietal sulcus uses a format transformed away from sensory driven responses. As an explicit test of mnemonic code transformations along the visual hierarchy, we quantitatively modeled the progression of veridical-to-categorical orientation representations in human participants. Participants directly viewed, or held in mind, an oriented grating pattern, and the similarity between fMRI activation patterns for different orientations was calculated throughout retinotopic cortex. During direct perception, similarity was clustered around cardinal orientations, while during working memory the obliques were represented more similarly. We modeled these similarity patterns based on the known distribution of orientation information in the natural world: The "veridical" model uses an efficient coding framework to capture hypothesized representations during visual perception. The "categorical" model assumes that different "psychological distances" between orientations result in orientation categorization relative to cardinal axes. During direct perception, the veridical model explained the data well in early visual areas, while the categorical model did worse. During working memory, the veridical model only explained some of the data, while the categorical model gradually gained explanatory power for increasingly anterior retinotopic regions. These findings suggest that directly viewed images are represented veridically, but once visual information is no longer tethered to the sensory world, there is a gradual progression to more categorical mnemonic formats along the visual hierarchy.

Long-term memory and working memory compete and cooperate to guide attention

Multiple types of memory guide attention: Both long-term memory (LTM) and working memory (WM) effectively guide visual search. Furthermore, both types of memories can capture attention automatically, even when detrimental to performance. It is less clear, however, how LTM and WM cooperate or compete to guide attention in the same task. In a series of behavioral experiments, we show that LTM and WM reliably cooperate to guide attention: Visual search is faster when both memories cue attention to the same spatial location (relative to when only one memory can guide attention). LTM and WM competed to guide attention in more limited circumstances: Competition only occurred when these memories were in different dimensions - particularly when participants searched for a shape and held an accessory color in mind. Finally, we found no evidence for asymmetry in either cooperation or competition: There was no evidence that WM helped (or hindered) LTM-guided search more than the other way around. This lack of asymmetry was found despite differences in LTM-guided and WM-guided search overall, and differences in how two LTMs and two WMs compete or cooperate with each other to guide attention. This work suggests that, even if only one memory is currently task-relevant, WM and LTM can cooperate to guide attention; they can also compete when distracting features are salient enough. This work elucidates interactions between WM and LTM during attentional guidance, adding to the literature on costs and benefits to attention from multiple active memories.

Long-term memory retrieval bypasses working memory

For decades, it has been assumed that when humans retrieve information from long-term memory (LTM), information need first to be brought back into working memory (WM). However, as WM capacity is limited, it is unclear what happens if information from LTM needs to be retrieved while WM is fully engaged? To address this question, observers had to retrieve colors from LTM while WM storage capacity was fully engaged. The behavioral results showed that retrieving information from LTM is possible even when WM capacity is fully occupied. Additional evidence from electroencephalogram (EEG) confirmed that WM was fully engaged as the suppression of alpha oscillation reached its maximum when memorizing the maximum amount of information into WM; yet the suppression in alpha oscillation was even further amplified when items were retrieved simultaneously from LTM, providing a neural signature of additional LTM retrieval capacity above and beyond the maximum WM capacity. Together, our findings indicate that information retrieved from LTM does not always have to be brought back into WM, but instead might be accessed through a different mechanism when WM is fully engaged.

Pattern reinstatement and attentional control overlap during episodic long-term memory retrieval

Episodic long-term memory (eLTM) retrieval involves the reinstatement of neural patterns from the encoding phase. However, recent evidence suggests that comparable cortical activity patterns can also be linked to attentional control processes on the level of memory representations. The current investigation assesses these two processes independently based on alpha-beta-band activity in the electroencephalogram (EEG). During encoding, subjects were presented with an object on a certain position on the screen and had to imagine it on a new position. In each trial, either the task-irrelevant presentation position or the task-relevant imagination position was lateralized. In the retrieval phase, subjects first made an old/new judgement based on centrally presented objects and then reported the imagination position. Pattern reinstatement should be reflected in similar lateralized alpha-beta activity during encoding and retrieval. Conversely, the influence of attentional control processes during retrieval would be associated with the suppression of alpha-beta power contralateral to the to-be-reported imagination position and with the increase of activity contralateral to the irrelevant presentation position. Our results support this latter pattern. This shows that an experimental differentiation between selective attention and pattern reinstatement processes is necessary when studying the neural basis of eLTM retrieval.