Perception and Memory Reinstatement Engage Overlapping Face-Selective Regions within Human Ventral Temporal Cortex

Humans have the remarkable ability to vividly retrieve sensory details of past events. According to the theory of sensory reinstatement, during remembering, brain regions specialized for processing specific sensory stimuli are reactivated to support content-specific retrieval. Recently, several studies have emphasized transformations in the spatial organization of these reinstated activity patterns. Specifically, studies of scene stimuli suggest a clear anterior shift in the location of retrieval activations compared with the activity observed during perception. However, it is not clear that such transformations occur universally, with inconsistent evidence for other important stimulus categories, particularly faces. One challenge in addressing this question is the careful delineation of face-selective cortices, which are interdigitated with other selective regions, in configurations that spatially differ across individuals. Therefore, we conducted a multisession neuroimaging study to first carefully map individual participants' (nine males and seven females) face-selective regions within ventral temporal cortex (VTC), followed by a second session to examine the activity patterns within these regions during face memory encoding and retrieval. While face-selective regions were expectedly engaged during face perception at encoding, memory retrieval engagement exhibited a more selective and constricted reinstatement pattern within these regions, but did not show any consistent direction of spatial transformation (e.g., anteriorization). We also report on unique human intracranial recordings from VTC under the same experimental conditions. These findings highlight the importance of considering the complex configuration of category-selective cortex in elucidating principles shaping the neural transformations that occur from perception to memory.

Pupil size tracks cue-trace interactions during episodic memory retrieval

Our ability to remember past events requires not only storing enduring engrams or memory traces of these events, but also successfully reactivating these latent traces in response to appropriate cues at the time of retrieval-a process that has been termed ecphory. However, relatively little is known about the processes that facilitate the dynamic interactions between retrieval cues and stored memory traces that are critical for successful recognition and recollection. Recently, an intriguing link between pupil dilation and recognition memory has been identified, with studied items eliciting greater pupil dilation than unstudied items during retrieval. However, the processes contributing to this "pupillary old/new effect" remain unresolved, with current explanations suggesting that it reflects the strength of the underlying memory trace. Here, we explore the novel hypothesis that the pupillary old/new effect does not index memory strength alone, but rather reflects the facilitation of cue-trace interactions during episodic memory retrieval that may be supported by activity within the pupil-linked locus coeruleus-noradrenergic (LC-NA) arousal system. First, we show that the magnitude of pupil dilation is influenced by the degree of overlap between cue and trace information. Second, we find that the magnitude of pupil dilation reflects the amount of study contextual information reinstated during retrieval. These findings provide a novel framework for understanding the pupillary old/new effect, and identify a potential role for the LC-NA system in recognition memory retrieval.

Neural similarity between overlapping events at learning differentially affects reinstatement across the cortex

Episodic memory often involves high overlap between the actors, locations, and objects of everyday events. Under some circumstances, it may be beneficial to distinguish, or differentiate, neural representations of similar events to avoid interference at recall. Alternatively, forming overlapping representations of similar events, or integration, may aid recall by linking shared information between memories. It is currently unclear how the brain supports these seemingly conflicting functions of differentiation and integration. We used multivoxel pattern similarity analysis (MVPA) of fMRI data and neural-network analysis of visual similarity to examine how highly overlapping naturalistic events are encoded in patterns of cortical activity, and how the degree of differentiation versus integration at encoding affects later retrieval. Participants performed an episodic memory task in which they learned and recalled naturalistic video stimuli with high feature overlap. Visually similar videos were encoded in overlapping patterns of neural activity in temporal, parietal, and occipital regions, suggesting integration. We further found that encoding processes differentially predicted later reinstatement across the cortex. In visual processing regions in occipital cortex, greater differentiation at encoding predicted later reinstatement. Higher-level sensory processing regions in temporal and parietal lobes showed the opposite pattern, whereby highly integrated stimuli showed greater reinstatement. Moreover, integration in high-level sensory processing regions during encoding predicted greater accuracy and vividness at recall. These findings provide novel evidence that encoding-related differentiation and integration processes across the cortex have divergent effects on later recall of highly similar naturalistic events.

Mapping multidimensional content representations to neural and behavioral expressions of episodic memory

Human neuroimaging studies have shown that the contents of episodic memories are represented in distributed patterns of neural activity. However, these studies have mostly been limited to decoding simple, unidimensional properties of stimuli. Semantic encoding models, in contrast, offer a means for characterizing the rich, multidimensional information that comprises episodic memories. Here, we extensively sampled four human fMRI subjects to build semantic encoding models and then applied these models to reconstruct content from natural scene images as they were viewed and recalled from memory. First, we found that multidimensional semantic information was successfully reconstructed from activity patterns across visual and lateral parietal cortices, both when viewing scenes and when recalling them from memory. Second, whereas visual cortical reconstructions were much more accurate when images were viewed versus recalled from memory, lateral parietal reconstructions were comparably accurate across visual perception and memory. Third, by applying natural language processing methods to verbal recall data, we showed that fMRI-based reconstructions reliably matched subjects' verbal descriptions of their memories. In fact, reconstructions from ventral temporal cortex more closely matched subjects' own verbal recall than other subjects' verbal recall of the same images. Fourth, encoding models reliably transferred across subjects: memories were successfully reconstructed using encoding models trained on data from entirely independent subjects. Together, these findings provide evidence for successful reconstructions of multidimensional and idiosyncratic memory representations and highlight the differential sensitivity of visual cortical and lateral parietal regions to information derived from the external visual environment versus internally-generated memories.

Individual-specific memory reinstatement patterns within human face-selective cortex

Humans have the remarkable ability to vividly retrieve sensory details of past events. According to the theory of sensory reinstatement, during remembering, brain regions involved in the sensory processing of prior events are reactivated to support this perception of the past. Recently, several studies have emphasized potential transformations in the spatial organization of reinstated activity patterns. In particular, studies of scene stimuli suggest a clear anterior shift in the location of retrieval activations compared with those during perception. However, it is not clear that such transformations occur universally, with evidence lacking for other important stimulus categories, particularly faces. Critical to addressing these questions, and to studies of reinstatement more broadly, is the growing importance of considering meaningful variations in the organization of sensory systems across individuals. Therefore, we conducted a multi-session neuroimaging study to first carefully map individual participants face-selective regions within ventral temporal cortex (VTC), followed by a second session to examine the correspondence of activity patterns during face memory encoding and retrieval. Our results showed distinct configurations of face-selective regions within the VTC across individuals. While a significant degree of overlap was observed between face perception and memory encoding, memory retrieval engagement exhibited a more selective and constricted reinstatement pattern within these regions. Importantly, these activity patterns were consistently tied to individual-specific neural substrates, but did not show any consistent direction of spatial transformation (e.g., anteriorization). To provide further insight to these findings, we also report on unique human intracranial recordings from VTC under the same experimental conditions. Our findings highlight the importance of considering individual variations in functional neuroanatomy in the context of assessing the nature of cortical reinstatement. Consideration of such factors will be important for establishing general principles shaping the neural transformations that occur from perception to memory.

Examining the engram encoding specificity hypothesis in mice

According to the encoding specificity hypothesis, memory is best recalled by retrieval cues that overlap with training cues. Human studies generally support this hypothesis. However, memories are thought to be stored in neuronal ensembles (engrams), and retrieval cues are thought to reactivate neurons in an engram to induce memory recall. Here, we visualized engrams in mice to test whether retrieval cues that overlap with training cues produce maximal memory recall via high engram reactivation (engram encoding specificity hypothesis). Using variations of cued threat conditioning (pairing conditioned stimulus [CS] with footshock), we manipulated encoding and retrieval conditions along multiple domains, including pharmacological state, external sensory cue, and internal optogenetic cue. Maximal engram reactivation and memory recall occurred when retrieval conditions closely matched training conditions. These findings provide a biological basis for the encoding specificity hypothesis and highlight the important interaction between stored information (engram) and cues available at memory retrieval (ecphory).

Typicality in the brain during semantic and episodic memory decisions

Concept typicality is a key semantic dimension supporting the categorical organization of items based on their features, such that typical items share more features with other members of their category than atypical items, which are more distinctive. Typicality effects manifest in better accuracy and faster response times during categorization tasks, but higher performance for atypical items in episodic memory tasks, due to their distinctiveness. At a neural level, typicality has been linked to the anterior temporal lobe (ATL) and the inferior frontal gyrus (IFG) in semantic decision tasks, but patterns of brain activity during episodic memory tasks remain to be understood. We investigated the neural correlates of typicality in semantic and episodic memory to determine the brain regions associated with semantic typicality and uncover effects arising when items are reinstated during retrieval. In an fMRI study, 26 healthy young subjects first performed a category verification task on words representing typical and atypical concepts (encoding), and then completed a recognition memory task (retrieval). In line with previous literature, we observed higher accuracy and faster response times for typical items in the category verification task, while atypical items were better recognized in the episodic memory task. During category verification, univariate analyses revealed a greater involvement of the angular gyrus for typical items and the inferior frontal gyrus for atypical items. During the correct recognition of old items, regions belonging to the core recollection network were activated. We then compared the similarity of the representations from encoding to retrieval (ERS) using Representation Similarity Analyses. Results showed that typical items were reinstated more than atypical ones in several regions including the left precuneus and left anterior temporal lobe (ATL). This suggests that the correct retrieval of typical items requires finer-grained processing, evidenced by greater item-specific reinstatement, which is needed to resolve their confusability with other members of the category due to their higher feature similarity. Our findings confirm the centrality of the ATL in the processing of typicality while extending it to memory retrieval.

Decoding of human identity by computer vision and neuronal vision

Extracting meaning from a dynamic and variable flow of incoming information is a major goal of both natural and artificial intelligence. Computer vision (CV) guided by deep learning (DL) has made significant strides in recognizing a specific identity despite highly variable attributes. This is the same challenge faced by the nervous system and partially addressed by the concept cells-neurons exhibiting selective firing in response to specific persons/places, described in the human medial temporal lobe (MTL) ⁠. Yet, access to neurons representing a particular concept is limited due to these neurons' sparse coding. It is conceivable, however, that the information required for such decoding is present in relatively small neuronal populations. To evaluate how well neuronal populations encode identity information in natural settings, we recorded neuronal activity from multiple brain regions of nine neurosurgical epilepsy patients implanted with depth electrodes, while the subjects watched an episode of the TV series "24". First, we devised a minimally supervised CV algorithm (with comparable performance against manually-labeled data) to detect the most prevalent characters (above 1% overall appearance) in each frame. Next, we implemented DL models that used the time-varying population neural data as inputs and decoded the visual presence of the four main characters throughout the episode. This methodology allowed us to compare "computer vision" with "neuronal vision"-footprints associated with each character present in the activity of a subset of neurons-and identify the brain regions that contributed to this decoding process. We then tested the DL models during a recognition memory task following movie viewing where subjects were asked to recognize clip segments from the presented episode. DL model activations were not only modulated by the presence of the corresponding characters but also by participants' subjective memory of whether they had seen the clip segment, and by the associative strengths of the characters in the narrative plot. The described approach can offer novel ways to probe the representation of concepts in time-evolving dynamic behavioral tasks. Further, the results suggest that the information required to robustly decode concepts is present in the population activity of only tens of neurons even in brain regions beyond MTL.

Recollection and prior knowledge recruit the left angular gyrus during recognition

The human angular gyrus (AG) is implicated in recollection, or the ability to retrieve detailed memory content from a specific episode. A separate line of research examining the neural bases of more general mnemonic representations, extracted over multiple episodes, also highlights the AG as a core region of interest. To reconcile these separate views of AG function, the present fMRI experiment used a Remember-Know paradigm with famous (prior knowledge) and non-famous (no prior knowledge) faces to test whether AG activity could be modulated by both task-specific recollection and general prior knowledge within the same individuals. Increased BOLD activity in the left AG was observed during both recollection in the absence of prior knowledge (recollected > non-recollected or correctly rejected non-famous faces) and when prior knowledge was accessed in the absence of experiment-specific recollection (famous > non-famous correct rejections). This pattern was most prominent for the left AG as compared to the broader inferior parietal lobe. Recollection-related responses in the left AG increased with encoding duration and prior knowledge, despite prior knowledge being incidental to the recognition decision. Overall, the left AG appears sensitive to both task-specific recollection and the incidental access of general prior knowledge, thus broadening our notions of the kinds of mnemonic representations that drive activity in this region.

Dual-axes of functional organisation across lateral parietal cortex: the angular gyrus forms part of a multi-modal buffering system

Decades of neuropsychological and neuroimaging evidence have implicated the lateral parietal cortex (LPC) in a myriad of cognitive domains, generating numerous influential theoretical models. However, these theories fail to explain why distinct cognitive activities appear to implicate common neural regions. Here we discuss a unifying model in which the angular gyrus forms part of a wider LPC system with a core underlying neurocomputational function; the multi-sensory buffering of spatio-temporally extended representations. We review the principles derived from computational modelling with neuroimaging task data and functional and structural connectivity measures that underpin the unified neurocomputational framework. We propose that although a variety of cognitive activities might draw on shared underlying machinery, variations in task preference across angular gyrus, and wider LPC, arise from graded changes in the underlying structural connectivity of the region to different input/output information sources. More specifically, we propose two primary axes of organisation: a dorsal-ventral axis and an anterior-posterior axis, with variations in task preference arising from underlying connectivity to different core cognitive networks (e.g. the executive, language, visual, or episodic memory networks).

Patterns of episodic content and specificity predicting subjective memory vividness

The ability to remember and internally represent events is often accompanied by a subjective sense of "vividness". Vividness measures are frequently used to evaluate the experience of remembering and imagining events, yet little research has considered the objective attributes of event memories that underlie this subjective judgment, and individual differences in this mapping. Here, we tested how the content and specificity of event memories support subjectively vivid recollection. Over three experiments, participants encoded events containing a theme word and three distinct elements - a person, a place, and an object. In a memory test, memory for event elements was assessed at two levels of specificity - semantic gist (names) and perceptual details (lure discrimination). We found a strong correspondence between memory vividness and memory for gist information that did not vary by which elements were contained in memory. There was a smaller, additive benefit of remembering specific perceptual details on vividness, which, in one study, was driven by memory for place details. Moreover, we found individual differences in the relationship between memory vividness and objective memory attributes primarily along the specificity dimension, such that one cluster of participants used perceptual detail to inform memory vividness whereas another cluster was more driven by gist information. Therefore, while gist memory appears to drive vividness on average, there were idiosyncrasies in this pattern across participants. When assessing subjective ratings of memory and imagination, research should consider how these ratings map onto objective memory attributes in the context of their study design and population.

Decreased associative processing and memory confidence in aphantasia

Visual imagery and mental reconstruction of scenes are considered core components of episodic memory retrieval. Individuals with absent visual imagery (aphantasia) score lower on tests of autobiographical memory, suggesting that aphantasia may be associated with differences in episodic and associative processing. In this online study, we tested aphantasic participants and controls on associative recognition and memory confidence for three types of associations encoded incidentally: associations between visual-visual and audio-visual stimulus pairs, associations between an object and its location on the screen, and intraitem associations. Aphantasic participants had a lower rate of high-confidence hits in all associative memory tests compared with controls. Performance on auditory-visual associations was correlated with individual differences in a measure of object imagery in the aphantasic group but not in controls. No overall group difference in memory performance was found, indicating that visual imagery selectively contributes to memory confidence. Analysis of the encoding task revealed that aphantasics made fewer associative links between the stimuli, suggesting a role for visual imagery in associative processing of visual and auditory input. These data enhance our understanding of visual imagery contributions to associative memory and further characterize the cognitive profile of aphantasia.

Functional imaging analyses reveal prototype and exemplar representations in a perceptual single-category task

Similarity-based categorization can be performed by memorizing category members as exemplars or by abstracting the central tendency of the category – the prototype. In similarity-based categorization of stimuli with clearly identifiable dimensions from two categories, prototype representations were previously located in the hippocampus and the ventromedial prefrontal cortex (vmPFC) and exemplar representations in areas supporting visual memory. However, the neural implementation of exemplar and prototype representations in perceptual similarity-based categorization of single categories is unclear. To investigate these representations, we applied model-based univariate and multivariate analyses of functional imaging data from a dot-pattern paradigm-based task. Univariate prototype and exemplar representations occurred bilaterally in visual areas. Multivariate analyses additionally identified prototype representations in parietal areas and exemplar representations in the hippocampus. Bayesian analyses supported the non-presence of prototype representations in the hippocampus and the vmPFC. We additionally demonstrate that some individuals form both representation types simultaneously, probably granting flexibility in categorization strategies.

Model-based univariate and multivariate analyses of fMRI data from 62 healthy participants in a dot-pattern paradigm-based task provide further insight into the neural basis of similarity-based categorization.

Brief memory reactivations induce learning in the numeric domain

Learning of arithmetic facts such as the multiplication table requires time-consuming, repeated practice. In light of evidence indicating that reactivation of encoded memories can modulate learning and memory processes at the synaptic, system and behavioral levels, we asked whether brief memory reactivations can induce human learning in the numeric domain. Adult participants performed a number-fact retrieval task in which they learned arbitrary numeric facts. Following encoding and a baseline test, 3 passive, brief reactivation sessions of only 40 s each were conducted on separate days. Learning was evaluated in a retest session. Results showed reactivations induced learning, with improved performance at retest relative to baseline test. Furthermore, performance was superior compared to a control group performing test-retest sessions without reactivations, who showed significant memory deterioration. A standard practice group completed active-retrieval sessions on 3 separate days, and showed significant learning gains. Interestingly, while these gains were higher than those of the reactivations group, subjects showing reactivation-induced learning were characterized by superior efficiency relative to standard practice subjects, with higher rate of improvement per practice time. A follow-up long-term retention experiment showed that 30 days following initial practice, weekly brief reactivations reduced forgetting, with participants performing superior to controls undergoing the same initial practice without reactivations. Overall, the results demonstrate that brief passive reactivations induce efficient learning and reduce forgetting within a numerical context. Time-efficient practice in the numeric domain carries implications for enhancement of learning strategies in daily-life settings.

Stronger memory representation after memory reinstatement during retrieval in the human hippocampus

Memory retrieval allows us to reinstate previously encoded information but is also considered to contribute to memory enhancement. Retrieval-induced enhancement may involve processing to strengthen memory traces, but neural processing beyond reinstatement during retrieval remains elusive. Here, we show that hippocampal processing, different from memory reinstatement, exists during retrieval in the human brain. By tracking changes in the response patterns in the selected hippocampal and cortical regions over time during retrieval based on functional MRI, we found that the representation of associative memory in CA3/DG became stronger even after cortical memory reinstatement, while CA1 showed significant memory representation at retrieval onset with the cortical reinstatement, but not afterwards. This tendency was not observed in the condition without active retrieval. Moreover, subsequent long-term memory performance depended on the delayed CA3/DG representation during retrieval. These findings suggest that CA3/DG contributes to neural processing beyond memory reinstatement during retrieval, which may lead to memory enhancement.

Neural signatures associated with temporal compression in the verbal retelling of past events

When we retell our past experiences, we aim to reproduce some version of the original events; this reproduced version is often temporally compressed relative to the original. However, it is currently unclear how this compression manifests in brain activity. One possibility is that a compressed retrieved memory manifests as a neural pattern which is more dissimilar to the original, relative to a more detailed or vivid memory. However, we argue that measuring raw dissimilarity alone is insufficient, as it confuses a variety of interesting and uninteresting changes. To address this problem, we examine brain pattern changes that are consistent across people. We show that temporal compression in individuals’ retelling of past events predicts systematic encoding-to-recall transformations in several higher associative regions. These findings elucidate how neural representations are not simply reactivated, but can also be transformed due to temporal compression during a universal form of human memory expression: verbal retelling.

Brain patterns measured while participants first watched a movie in the fMRI scanner, then recalled the movie’s key narrative features, demonstrate that temporal compression in individuals’ retelling of past events predicts encoding-to-recall transformations.

Category-sensitive incidental reinstatement in medial temporal lobe subregions during word recognition

During associative retrieval, the brain reinstates neural representations that were present during encoding. The human medial temporal lobe (MTL), with its subregions hippocampus (HC), perirhinal cortex (PRC), and parahippocampal cortex (PHC), plays a central role in neural reinstatement. Previous studies have given compelling evidence for reinstatement in the MTL during explicitly instructed associative retrieval. High-confident recognition may be similarly accompanied by recollection of associated information from the encoding context. It is unclear, however, whether high-confident recognition memory elicits reinstatement in the MTL even in the absence of an explicit instruction to retrieve associated information. Here, we addressed this open question using high-resolution fMRI. Twenty-eight male and female human volunteers engaged in a recognition memory task for words that they had previously encoded together with faces and scenes. Using complementary univariate and multivariate approaches, we show that MTL subregions including the PRC, PHC, and HC differentially reinstate category-sensitive representations during high-confident word recognition, even though no explicit instruction to retrieve the associated category was given. This constitutes novel evidence that high-confident recognition memory is accompanied by incidental reinstatement of associated category information in MTL subregions, and supports a functional model of the MTL that emphasizes content-sensitive representations during both encoding and retrieval.

Neural reactivation and judgements of vividness reveal separable contributions to mnemonic representation

Mnemonic representations vary in fidelity, sharpness, and strength-qualities that can be examined using both introspective judgements of mental states and objective measures of brain activity. Subjective and objective measures are both valid ways of "reading out" the content of someone's internal mnemonic states, each with different strengths and weaknesses. St-Laurent and colleagues (2015) compared the neural correlates of memory vividness ratings with patterns of neural reactivation evoked during memory recall and found considerable overlap between the two, suggesting a common neural basis underlying these different markers of representational quality. Here we extended this work with meta-analytic methods by pooling together four neuroimaging datasets in order to contrast the neural substrates of neural reactivation and those of vividness judgements. While reactivation and vividness judgements correlated positively with one another and were associated with common univariate activity in the dorsal attention network and anterior hippocampus, some notable differences were also observed. Vividness judgments were tied to stronger activation in the striatum and dorsal attention network, together with activity suppression in default mode network nodes. We also observed a trend for reactivation to be more closely associated with early visual cortex activity. A mediation analysis found support for the hypothesis that neural reactivation is necessary for memory vividness, with activity in the anterior hippocampus associated with greater reactivation. Our results suggest that neural reactivation and vividness judgements reflect common mnemonic processes but differ in the extent to which they engage effortful, attentional processes. Additionally, the similarity between reactivation and vividness appears to arise, partly, through hippocampal engagement during memory retrieval.

The Neural Representation of Events Is Dominated by Elements that Are Most Reliably Present

An episodic memory is specific to an event that occurred at a particular time and place. However, the elements that comprise the event-the location, the people present, and their actions and goals-might be shared with numerous other similar events. Does the brain preferentially represent certain elements of a remembered event? If so, which elements dominate its neural representation: those that are shared across similar events, or the novel elements that define a specific event? We addressed these questions by using a novel experimental paradigm combined with fMRI. Multiple events were created involving conversations between two individuals using the format of a television chat show. Chat show "hosts" occurred repeatedly across multiple events, whereas the "guests" were unique to only one event. Before learning the conversations, participants were scanned while viewing images or names of the (famous) individuals to be used in the study to obtain person-specific activity patterns. After learning all the conversations over a week, participants were scanned for a second time while they recalled each event multiple times. We found that during recall, person-specific activity patterns within the posterior midline network were reinstated for the hosts of the shows but not the guests, and that reinstatement of the hosts was significantly stronger than the reinstatement of the guests. These findings demonstrate that it is the more generic, familiar, and predictable elements of an event that dominate its neural representation compared with the more idiosyncratic, event-defining, elements.

I remember it like it was yesterday: Age-related differences in the subjective experience of remembering

It has been frequently described that older adults subjectively report the vividness of their memories as being as high, or even higher, than young adults, despite poorer objective memory performance. Here, we review studies that examined age-related differences in the subjective experience of memory vividness. By examining vividness calibration and resolution, studies using different types of approaches converge to suggest that older adults overestimate the intensity of their vividness ratings relative to young adults, and that they rely on retrieved memory details to a lesser extent to judge vividness. We discuss potential mechanisms underlying these observations. Inflation of memory vividness with regard to the richness of memory content may stem from age-differences in vividness criterion or scale interpretation and psycho-social factors. The reduced reliance on episodic memory details in older adults may stem from age-related differences in how they monitor these details to make their vividness ratings. Considered together, these findings emphasize the importance of examining age-differences in memory vividness using different analytical methods and they provide valuable evidence that the subjective experience of remembering is more than the reactivation of memory content. In this vein, we recommend that future studies explore the links between memory vividness and other subjective memory scales (e.g., ratings of details or memory confidence) in healthy aging and/or other populations, as it could be used as a window to better characterize the cognitive processes that underpin the subjective assessment of the quality of recollected events.

Neural Correlates of Subsequent Memory-Related Gaze Reinstatement

Mounting evidence linking gaze reinstatement-the recapitulation of encoding-related gaze patterns during retrieval-to behavioral measures of memory suggests that eye movements play an important role in mnemonic processing. Yet, the nature of the gaze scanpath, including its informational content and neural correlates, has remained in question. In this study, we examined eye movement and neural data from a recognition memory task to further elucidate the behavioral and neural bases of functional gaze reinstatement. Consistent with previous work, gaze reinstatement during retrieval of freely viewed scene images was greater than chance and predictive of recognition memory performance. Gaze reinstatement was also associated with viewing of informationally salient image regions at encoding, suggesting that scanpaths may encode and contain high-level scene content. At the brain level, gaze reinstatement was predicted by encoding-related activity in the occipital pole and BG, neural regions associated with visual processing and oculomotor control. Finally, cross-voxel brain pattern similarity analysis revealed overlapping subsequent memory and subsequent gaze reinstatement modulation effects in the parahippocampal place area and hippocampus, in addition to the occipital pole and BG. Together, these findings suggest that encoding-related activity in brain regions associated with scene processing, oculomotor control, and memory supports the formation, and subsequent recapitulation, of functional scanpaths. More broadly, these findings lend support to scanpath theory's assertion that eye movements both encode, and are themselves embedded in, mnemonic representations.

Enhanced reinstatement of naturalistic event memories due to hippocampal-network-targeted stimulation

Episodic memory involves the reinstatement of distributed patterns of brain activity present when events were initially experienced. The hippocampus is thought to coordinate reinstatement via its interactions with a network of brain regions, but this hypothesis has not been causally tested in humans. The current study directly tested the involvement of the hippocampal network in reinstatement using network-targeted noninvasive stimulation. We measured reinstatement of multi-voxel patterns of functional magnetic resonance imaging (fMRI) activity during encoding and retrieval of naturalistic video clips depicting everyday activities. Reinstatement of video-specific activity patterns was robust in posterior parietal and occipital areas previously implicated in event reinstatement. Theta-burst stimulation targeting the hippocampal network increased video-specific reinstatement of fMRI activity patterns in occipital cortex and improved memory accuracy relative to stimulation of a control out-of-network location. Furthermore, stimulation targeting the hippocampal network influenced the trial-by-trial relationship between hippocampal activity during encoding and later reinstatement in occipital cortex. These findings implicate the hippocampal network in the reinstatement of spatially distributed patterns of event-specific activity and identify a role for the hippocampus in encoding complex naturalistic events that later undergo cortical reinstatement.

Shared event memory in aging: Across-participants similarity of vividness judgements decreases with age

When they remember the same events, humans recollect common episodic traces. For making vividness judgements, older adults rely less than young adults on retrieved episodic details. Here, we examined the similarity of the subjective experience of remembering and the associated memory content across participants and we investigated age-effects. Young and older adults studied pictures associated with labels. At retrieval, participants judged the vividness of their memories and recalled pictures details. We examined the similarity of vividness judgements and memory recall across-participants. Across-participants similarity in vividness judgements was higher in young than in older adults, while no age-difference in the similarity of the richness of memory recall between participants was found. Together, these findings suggest that older adults' vividness ratings are less similar from one participant to another than those of young adults, which may be explained by how older adults use memory details to frame their sense of memory vividness.

Eye movements during visual imagery and perception show spatial correspondence but have unique temporal signatures

Eye fixation patterns during mental imagery are similar to those during perception of the same picture, suggesting that oculomotor mechanisms play a role in mental imagery (i.e., the "looking at nothing" effect). Previous research has focused on the spatial similarities of eye movements during perception and mental imagery. The primary aim of this study was to assess whether the spatial similarity translates to the temporal domain. We used recurrence quantification analysis (RQA) to assess the temporal structure of eye fixations in visual perception and mental imagery and we compared the temporal as well as the spatial characteristics in mental imagery with perception by means of Bayesian hierarchical regression models. We further investigated how person and picture-specific characteristics contribute to eye movement behavior in mental imagery. Working memory capacity and mental imagery abilities were assessed to either predict gaze dynamics in visual imagery or to moderate a possible correspondence between spatial or temporal gaze dynamics in perception and mental imagery. We were able to show the spatial similarity of fixations between visual perception and imagery and we provide first evidence for its moderation by working memory capacity. Interestingly, the temporal gaze dynamics in mental imagery were unrelated to those in perception and their variance between participants was not explained by variance in visuo-spatial working memory capacity or vividness of mental images. The semantic content of the imagined pictures was the only meaningful predictor of temporal gaze dynamics. The spatial correspondence reflects shared spatial structure of mental images and perceived pictures, while the unique temporal gaze behavior could be driven by generation, maintenance and protection processes specific to visual imagery. The unique temporal gaze dynamics offer a window to new insights into the genuine process of mental imagery independent of its similarity to perception.

Transforming the Concept of Memory Reactivation

Reactivation refers to the phenomenon wherein patterns of neural activity expressed during perceptual experience are re-expressed at a later time, a putative neural marker of memory. Reactivation of perceptual content has been observed across many cortical areas and correlates with objective and subjective expressions of memory in humans. However, because reactivation emphasizes similarities between perceptual and memory-based representations, it obscures differences in how perceptual events and memories are represented. Here, we highlight recent evidence of systematic differences in how (and where) perceptual events and memories are represented in the brain. We argue that neural representations of memories are best thought of as spatially transformed versions of perceptual representations. We consider why spatial transformations occur and identify critical questions for future research.

Age-related alterations of default mode network in autobiographical memory: Recent versus remote events

Previous studies have shown that the vividness of autobiographical memory decreases over time, and older adults often retrieve fewer details than young adults. However, the age-by-temporal distance (i.e., recent versus remote events) effect on autobiographical memory and underlying neural mechanisms are less understood. We recruited 25 young adults and 27 older adults to perform an fMRI-adapted autobiographical memory task with different temporal distances. The results showed that older adults' vividness ratings were generally higher than that of young adults, but were less sensitive to temporal distances. For neural imaging, an age-by-temporal distance effect was found in the left precuneus, manifested as young adults had more activation for recent events than for remote events, whereas no temporal distance effect was found in older adults. Interestingly, for older adults, the temporal distance effect was reflected by functional connectivity within the default mode network (DMN), with a stronger anterior DMN-posterior DMN coupling for remote events than for recent events, whereas no temporal distance difference on functional connectivity was found in young adults. The results suggest that older adults exhibit age-related neural differences in both activation and functional connectivity during the processing of autobiographical memory with different temporal distances, shedding new light for the understanding of the relationship between the DMN, autobiographical memory, and aging.

Among the Two Kinds of Metacognitive Evaluation, Only One Is Predictive of Illusory Object Perception

The relationship between expectation-induced hallucination proneness and self-confidence in performance was studied in a visual perception task. Participants were prompted either to recognize briefly shown faces as male or female or to rate the subjective vividness of a square surrounding the face. Importantly, in a few critical trials, the square was absent. Upon completion, participants rated their performance in the face recognition task; they were also asked whether they were sure that their estimation was correct. Out of 35 participants, 33 "hallucinated" on at least one trial, rating the square as visible when it was actually absent. Negative correlation between hallucination proneness and self-confidence in performance (metacognitive rating) was found: The more hallucinations a participant experienced, the less confident he/she was in his/her performance in the face recognition task. Most subjects underestimated their performance; higher ratings were also more accurate. Thus, higher hallucination proneness was associated with more inaccurate ratings of one's own perception. However, confidence in self-ratings as measured by the second follow-up question was unrelated to both, hallucination proneness and self-confidence in performance, supporting the view that there is no unitary mechanism of metacognitive evaluations and extending this view to the domain of visual hallucinatory perception.

How Multiple Retrievals Affect Neural Reactivation in Young and Older Adults

OBJECTIVES

Aging can reduce the specificity with which memory episodes are represented as distributed patterns of brain activity. It remains unclear, however, whether repeated encoding and retrieval of stimuli modulate this decline. Memory repetition is thought to promote semanticization, a transformative process during which episodic memory becomes gradually decontextualized and abstracted. Because semantic memory is considered more resilient to aging than context-rich episodic memory, we hypothesized that repeated retrieval would affect cortical reinstatement differently in young versus older adults.

METHODS

We reanalyzed data from young and older adults undergoing functional magnetic resonance imaging while repeatedly viewing and recalling short videos. We derived trial-unique multivariate measures of similarity between video-specific brain activity patterns elicited at perception and at recall, which we compared between age groups at each repetition.

RESULTS

With repetition, memory representation became gradually more distinct from perception in young adults, as reinstatement specificity converged downward toward levels observed in the older group. In older adults, alternative representations that were item-specific but orthogonal to patterns elicited at perception became more salient with repetition.

DISCUSSION

Repetition transformed dominant patterns of memory representation away and orthogonally from perception in young and older adults, respectively. Although distinct, both changes are consistent with repetition-induced semanticization.

Eye Movement Reinstatement and Neural Reactivation During Mental Imagery

Half a century ago, Donald Hebb posited that mental imagery is a constructive process that emulates perception. Specifically, Hebb claimed that visual imagery results from the reactivation of neural activity associated with viewing images. He also argued that neural reactivation and imagery benefit from the re-enactment of eye movement patterns that first occurred at viewing (fixation reinstatement). To investigate these claims, we applied multivariate pattern analyses to functional MRI (fMRI) and eye tracking data collected while healthy human participants repeatedly viewed and visualized complex images. We observed that the specificity of neural reactivation correlated positively with vivid imagery and with memory for stimulus image details. Moreover, neural reactivation correlated positively with fixation reinstatement, meaning that image-specific eye movements accompanied image-specific patterns of brain activity during visualization. These findings support the conception of mental imagery as a simulation of perception, and provide evidence consistent with the supportive role of eye movement in neural reactivation.

Deconstructing the Posterior Medial Episodic Network

Our ability to remember or imagine specific events involves the construction of complex mental representations, a process that engages cortical and hippocampal regions in a core posterior medial (PM) brain network. Existing theoretical approaches have described the overarching contributions of the PM network, but less is known about how episodic content is represented and transformed throughout this system. Here, we review evidence of key functional interactions among PM regions and their relation to the core cognitive operations and representations supporting episodic construction. Recent demonstrations of intranetwork functional diversity are integrated with existing accounts to inform a network-based model of episodic construction, in which PM regions flexibly share and manipulate event information to support the variable phenomenology of episodic memory and simulation.

Feature-specific neural reactivation during episodic memory

We present a multi-voxel analytical approach, feature-specific informational connectivity (FSIC), that leverages hierarchical representations from a neural network to decode neural reactivation in fMRI data collected while participants performed an episodic visual recall task. We show that neural reactivation associated with low-level (e.g. edges), high-level (e.g. facial features), and semantic (e.g. “terrier”) features occur throughout the dorsal and ventral visual streams and extend into the frontal cortex. Moreover, we show that reactivation of both low- and high-level features correlate with the vividness of the memory, whereas only reactivation of low-level features correlates with recognition accuracy when the lure and target images are semantically similar. In addition to demonstrating the utility of FSIC for mapping feature-specific reactivation, these findings resolve the contributions of low- and high-level features to the vividness of visual memories and challenge a strict interpretation the posterior-to-anterior visual hierarchy.

Memory recollection involves reactivation of neural activity that occurred during the recalled experience. Here, the authors show that neural reactivation can be decomposed into visual-semantic features, is widely synchronized throughout the brain, and predicts memory vividness and accuracy.

The neural influence of autobiographical memory related to the parent-child relationship on psychological health in adulthood

The recollection of childhood memories is affected by the subjects involved, such as father and mother, and by the context. This study aimed to clarify the neural influence of autobiographical memory related to the parent-child relationship on psychological health in adulthood. Twenty-nine healthy volunteers participated in a functional magnetic resonance imaging experiment using a childhood memory recollection task, in which they appraised the emotion a parent would have provided in a given situation. Whole-brain univariate and psychophysiological interaction analyses were performed. Neuroimaging results indicated notable involvement of the caudal anterior cingulate cortex and precuneus in autobiographical memory related to the parent-child relationship, and their activities were closely associated with the level of depression and self-esteem, respectively. The functional connectivity results indicated increased connectivity between the caudal anterior cingulate cortex and fusiform gyrus for the father-positive condition compared to the mother-positive condition and there was a positive correlation between the strength of connectivity between the two regions and the anxiety level. Our findings suggest the processing of negative affect and the personalness of autobiographical memories are distinctly engaged depending on the parent in question and the situational valence. The present study illuminates the impact of autobiographical memory processes on various dimensions of psychological health.

Representation of viewed and recalled film clips in patterns of brain activity in a person with developmental amnesia

As clear memories transport us back into the past, the brain also revives prior patterns of neural activity, a phenomenon known as neural reactivation. While growing evidence indicates a link between neural reactivation and typical variations in memory performance in healthy individuals, it is unclear how and to what extent reactivation is disrupted by a memory disorder. The current study characterizes neural reactivation in a case of amnesia using Multivoxel Pattern Analysis (MVPA). We tested NC, an individual with developmental amnesia linked to a diencephalic stroke, and 19 young adult controls on a functional magnetic resonance imaging (fMRI) task during which participants viewed and recalled short videos multiple times. An encoding classifier trained and tested to identify videos based on brain activity patterns elicited at perception revealed superior classification in NC. The enhanced consistency in stimulus representation we observed in NC at encoding was accompanied by an absence of multivariate repetition suppression, which occurred over repeated viewing in the controls. Another recall classifier trained and tested to identify videos during mental replay indicated normal levels of classification in NC, despite his poor memory for stimulus content. However, a cross-condition classifier trained on perception trials and tested on mental replay trials-a strict test of reactivation-revealed significantly poorer classification in NC. Thus, while NC's brain activity was consistent and stimulus-specific during mental replay, this specificity did not reflect the reactivation of patterns elicited at perception to the same extent as controls. Fittingly, we identified brain regions for which activity supported stimulus representation during mental replay to a greater extent in NC than in controls. This activity was not modeled on perception, suggesting that compensatory patterns of representation based on generic knowledge can support consistent mental constructs when memory is faulty. Our results reveal several ways in which amnesia impacts distributed patterns of stimulus representation during encoding and retrieval.

Neural correlates of retrieval-based enhancement of autobiographical memory in older adults

Lifelog photo review is considered to enhance the recall of personal events. While a sizable body of research has explored the neural basis of autobiographical memory (AM), there is limited neural evidence on the retrieval-based enhancement effect on event memory among older adults in the real-world environment. This study examined the neural processes of AM as was modulated by retrieval practice through lifelog photo review in older adults. In the experiment, blood-oxygen-level dependent response during subjects’ recall of recent events was recorded, where events were cued by photos that may or may not have been exposed to a priori retrieval practice (training). Subjects remembered more episodic details under the trained relative to non-trained condition. Importantly, the neural correlates of AM was exhibited by (1) dissociable cortical areas related to recollection and familiarity, and (2) a positive correlation between the amount of recollected episodic details and cortical activation within several lateral temporal and parietal regions. Further analysis of the brain activation pattern at a few regions of interest within the core remember network showed a training_condition × event_detail interaction effect, suggesting that the boosting effect of retrieval practice depended on the level of recollected event details.

Multiscale Integration of Contextual Information During a Naturalistic Task

Everyday experience requires rapid and automatic integration of incoming stimuli with previously stored knowledge. Prior knowledge can help to construct a general "situation model" of the event, as well as aid comprehension of an ongoing narrative. Using fMRI in healthy adult humans, we investigated processing of videos whose locations and characters were always familiar but whose narratives were either a continuation or noncontinuation of an earlier video (high context (HC) or low context (LC), respectively). Responses in parahippocampal gyrus and retrosplenial cortex were composed of an initial transient, locked to the video onsets, followed by a period of lower amplitude activation that was greater in the LC condition. This may reflect rapid processing of core components of situation models such as location and characters and more gradual incorporation of their narrative themes. By contrast, activity increases in left hemisphere middle temporal gyrus (MTG), angular gyrus, and inferior frontal gyrus were maintained throughout the videos and were higher for HC versus LC videos. Further, activity in the left MTG peaked earlier in the HC condition. We suggest that these regions support representations of the specific interlinked concepts necessary to comprehend an ongoing narrative, which are already established for the HC videos.

The neurobiological foundation of memory retrieval

Memory retrieval involves the interaction between external sensory or internally generated cues and stored memory traces (or engrams) in a process termed 'ecphory'. While ecphory has been examined in human cognitive neuroscience research, its neurobiological foundation is less understood. To the extent that ecphory involves 'reawakening' of engrams, leveraging recently developed technologies that can identify and manipulate engrams in rodents provides a fertile avenue for examining retrieval at the level of neuronal ensembles. Here we evaluate emerging neuroscientific research of this type, using cognitive theory as a guiding principle to organize and interpret initial findings. Our Review highlights the critical interaction between engrams and retrieval cues (environmental or artificial) for memory accessibility and retrieval success. These findings also highlight the intimate relationship between the mechanisms important in forming engrams and those important in their recovery, as captured in the cognitive notion of 'encoding specificity'. Finally, we identify several questions that currently remain unanswered.

Imagery-Mediated Verbal Learning Depends on Vividness-Familiarity Interactions: The Possible Role of Dualistic Resting State Network Activity Interference

Using secondary database analysis, we tested whether the (implicit) familiarity of eliciting noun-cues and the (explicit) vividness of corresponding imagery exerted additive or interactive influences on verbal learning, as measured by the probability of incidental noun recall and image latency times (RTs). Noun-cues with incongruent levels of vividness and familiarity (high/low; low/high, respectively) at encoding were subsequently associated at retrieval with the lowest recall probabilities, while noun-cues related with congruent levels (high/high; low/low) were associated with higher recall probabilities. RTs in the high vividness and high familiarity grouping were significantly faster than all other subsets (low/low, low/high, high/low) which did not significantly differ among each other. The findings contradict: (1) associative theories predicting positive monotonic relationships between memory strength and learning; and (2) non-monotonic plasticity hypothesis (NMPH), aiming at generalizing the non-monotonic relationship between a neuron's excitation level and its synaptic strength to broad neural networks. We propose a dualistic neuropsychological model of memory consolidation that mimics the global activity in two large resting-state networks (RSNs), the default mode network (DMN) and the task-positive-network (TPN). Based on this model, we suggest that incongruence and congruence between vividness and familiarity reflect, respectively, competition and synergy between DMN and TPN activity. We argue that competition or synergy between these RSNs at the time of stimulus encoding disproportionately influences long term semantic memory consolidation in healthy controls. These findings could assist in developing neurophenomenological markers of core memory deficits currently hypothesized to be shared across multiple psychopathological conditions.

Memories Fade: The Relationship Between Memory Vividness and Remembered Visual Salience

Past events, particularly emotional experiences, are often vividly recollected. However, it remains unclear how qualitative information, such as low-level visual salience, is reconstructed and how the precision and bias of this information relate to subjective memory vividness. Here, we tested whether remembered visual salience contributes to vivid recollection. In three experiments, participants studied emotionally negative and neutral images that varied in luminance and color saturation, and they reconstructed the visual salience of each image in a subsequent test. Results revealed, unexpectedly, that memories were recollected as less visually salient than they were encoded, demonstrating a novel memory-fading effect, whereas negative emotion increased subjective memory vividness and the precision with which visual features were encoded. Finally, memory vividness tracked both the precision and remembered salience (bias) of visual information. These findings provide evidence that low-level visual information fades in memory and contributes to the experience of vivid recollection.

The experience of vivid autobiographical reminiscence is supported by subjective content representations in the precuneus

The human posteromedial cortex, which includes core regions of the default mode network (DMN), is thought to play an important role in episodic memory. However, the nature and functional role of representations in these brain regions remain unspecified. Nine participants (all female) wore smartphone devices to record episodes from their daily lives for multiple weeks, each night indicating the personally-salient attributes of each episode. Participants then relived their experiences in an fMRI scanner cued by images from their own lives. Representational Similarity Analysis revealed a broad network, including parts of the DMN, that represented personal semantics during autobiographical reminiscence. Within this network, activity in the right precuneus reflected more detailed representations of subjective contents during vivid relative to non-vivid, recollection. Our results suggest a more specific mechanism underlying the phenomenology of vivid autobiographical reminiscence, supported by rich subjective content representations in the precuneus, a hub of the DMN previously implicated in metacognitive evaluations during memory retrieval.

Feature-Specific Awake Reactivation in Human V1 after Visual Training

Brain activity patterns exhibited during task performance have been shown to spontaneously reemerge in the following restful awake state. Such "awake reactivation" has been observed across higher-order cortex for complex images or associations. However, it is still unclear whether the reactivation extends to primary sensory areas that encode simple stimulus features. To address this question, we trained human subjects from both sexes on a particular visual feature (Gabor orientation) and tested whether this feature will be reactivated immediately after training. We found robust reactivation in human V1 that lasted for at least 8 min after training offset. This effect was not present in higher retinotopic areas, such as V2, V3, V3A, or V4v. Further analyses suggested that the amount of awake reactivation was related to the amount of performance improvement on the visual task. These results demonstrate that awake reactivation extends beyond higher-order areas and also occurs in early sensory cortex.SIGNIFICANCE STATEMENT How do we acquire new memories and skills? New information is known to be consolidated during offline periods of rest. Recent studies suggest that a critical process during this period of consolidation is the spontaneous reactivation of previously experienced patterns of neural activity. However, research in humans has mostly examined such reactivation processes in higher-order cortex. Here we show that awake reactivation occurs even in the primary visual cortex V1 and that this reactivation is related to the amount of behavioral learning. These results pinpoint awake reactivation as a process that likely occurs across the entire human brain and could play an integral role in memory consolidation.

Decomposing Parietal Memory Reactivation to Predict Consequences of Remembering

Memory retrieval can strengthen, but also distort memories. Parietal cortex is a candidate region involved in retrieval-induced memory changes as it reflects retrieval success and represents retrieved content. Here, we conducted an fMRI experiment to test whether different forms of parietal reactivation predict distinct consequences of retrieval. Subjects studied associations between words and pictures of faces, scenes, or objects, and then repeatedly retrieved half of the pictures, reporting the vividness of the retrieved pictures (“retrieval practice”). On the following day, subjects completed a recognition memory test for individual pictures. Critically, the test included lures highly similar to studied pictures. Behaviorally, retrieval practice increased both hit and false alarm (FA) rates to similar lures, confirming a causal influence of retrieval on subsequent memory. Using pattern similarity analyses, we measured two different levels of reactivation during retrieval practice: generic “category-level” reactivation and idiosyncratic “item-level” reactivation. Vivid remembering during retrieval practice was associated with stronger category- and item-level reactivation in parietal cortex. However, these measures differentially predicted subsequent recognition memory performance: whereas higher category-level reactivation tended to predict FAs to lures, item-level reactivation predicted correct rejections. These findings indicate that parietal reactivation can be decomposed to tease apart distinct consequences of memory retrieval.

Time-resolved neural reinstatement and pattern separation during memory decisions in human hippocampus

Mnemonic decision-making has long been hypothesized to rely on hippocampal dynamics that bias memory processing toward the formation of new memories or the retrieval of old ones. Successful memory encoding may be best optimized by pattern separation, whereby two highly similar experiences can be represented by underlying neural populations in an orthogonal manner. By contrast, successful memory retrieval is thought to be supported by a recovery of the same neural pattern laid down during encoding. Here we examined how hippocampal pattern completion and separation emerge over time during memory decisions. We measured electrocorticography activity in the human hippocampus and posterior occipitotemporal cortex (OTC) while participants performed continuous recognition of items that were new, repeated (old), or highly similar to a prior item (similar). During retrieval decisions of old items, both regions exhibited significant reinstatement of multivariate high-frequency activity (HFA) associated with encoding. Further, the extent of reinstatement of encoding patterns during retrieval was correlated with the strength (HFA power) of hippocampal encoding. Evidence for encoding pattern reinstatement was also seen in OTC on trials requiring fine-grained discrimination of similar items. By contrast, hippocampal activity showed evidence for pattern separation during these trials. Together, these results underscore the critical role of the hippocampus in supporting both reinstatement of overlapping information and separation of similar events.