Adding Meaning to Memories: How Parietal Cortex Combines Semantic Content with Episodic Experience

Neuroimaging studies of human memory have consistently found that univariate responses in parietal cortex track episodic experience with stimuli (whether stimuli are 'old' or 'new'). More recently, pattern-based fMRI studies have shown that parietal cortex also carries information about the semantic content of remembered experiences. However, it is not well understood how memory-based and content-based signals are integrated within parietal cortex. Here, in humans (males and females), we used voxel-wise encoding models and a recognition memory task to predict the fMRI activity patterns evoked by complex natural scene images based on (1) the episodic history and (2) the semantic content of each image. Models were generated and compared across distinct subregions of parietal cortex and for occipitotemporal cortex. We show that parietal and occipitotemporal regions each encode memory and content information, but they differ in how they combine this information. Among parietal subregions, angular gyrus was characterized by robust and overlapping effects of memory and content. Moreover, subject-specific semantic tuning functions revealed that successful recognition shifted the amplitude of tuning functions in angular gyrus but did not change the selectivity of tuning. In other words, effects of memory and content were additive in angular gyrus. This pattern of data contrasted with occipitotemporal cortex where memory and content effects were interactive: memory effects were preferentially expressed by voxels tuned to the content of a remembered image. Collectively, these findings provide unique insight into how parietal cortex combines information about episodic memory and semantic content.SIGNIFICANCE STATEMENT Neuroimaging studies of human memory have identified multiple brain regions that not only carry information about "whether" a visual stimulus is successfully recognized but also "what" the content of that stimulus includes. However, a fundamental and open question concerns how the brain integrates these two types of information (memory and content). Here, using a powerful combination of fMRI analysis methods, we show that parietal cortex, particularly the angular gyrus, robustly combines memory- and content-related information, but these two forms of information are represented via additive, independent signals. In contrast, memory effects in high-level visual cortex critically depend on (and interact with) content representations. Together, these findings reveal multiple and distinct ways in which the brain combines memory- and content-related information.

Mapping multidimensional content representations to neural and behavioral expressions of episodic memory

Human neuroimaging studies have shown that the contents of episodic memories are represented in distributed patterns of neural activity. However, these studies have mostly been limited to decoding simple, unidimensional properties of stimuli. Semantic encoding models, in contrast, offer a means for characterizing the rich, multidimensional information that comprises episodic memories. Here, we extensively sampled four human fMRI subjects to build semantic encoding models and then applied these models to reconstruct content from natural scene images as they were viewed and recalled from memory. First, we found that multidimensional semantic information was successfully reconstructed from activity patterns across visual and lateral parietal cortices, both when viewing scenes and when recalling them from memory. Second, whereas visual cortical reconstructions were much more accurate when images were viewed versus recalled from memory, lateral parietal reconstructions were comparably accurate across visual perception and memory. Third, by applying natural language processing methods to verbal recall data, we showed that fMRI-based reconstructions reliably matched subjects' verbal descriptions of their memories. In fact, reconstructions from ventral temporal cortex more closely matched subjects' own verbal recall than other subjects' verbal recall of the same images. Fourth, encoding models reliably transferred across subjects: memories were successfully reconstructed using encoding models trained on data from entirely independent subjects. Together, these findings provide evidence for successful reconstructions of multidimensional and idiosyncratic memory representations and highlight the differential sensitivity of visual cortical and lateral parietal regions to information derived from the external visual environment versus internally-generated memories.

Perception and memory retrieval states are reflected in distributed patterns of background functional connectivity

The same visual input can serve as the target of perception or as a trigger for memory retrieval depending on whether cognitive processing is externally oriented (perception) or internally oriented (memory retrieval). While numerous human neuroimaging studies have characterized how visual stimuli are differentially processed during perception versus memory retrieval, perception and memory retrieval may also be associated with distinct neural states that are independent of stimulus-evoked neural activity. Here, we combined human fMRI with full correlation matrix analysis (FCMA) to reveal potential differences in "background" functional connectivity across perception and memory retrieval states. We found that perception and retrieval states could be discriminated with high accuracy based on patterns of connectivity across (1) the control network, (2) the default mode network (DMN), and (3) retrosplenial cortex (RSC). In particular, clusters in the control network increased connectivity with each other during the perception state, whereas clusters in the DMN were more strongly coupled during the retrieval state. Interestingly, RSC switched its coupling between networks as the cognitive state shifted from retrieval to perception. Finally, we show that background connectivity (1) was fully independent from stimulus-related variance in the signal and, further, (2) captured distinct aspects of cognitive states compared to traditional classification of stimulus-evoked responses. Together, our results reveal that perception and memory retrieval are associated with sustained cognitive states that manifest as distinct patterns of connectivity among large-scale brain networks.

Re-expression of CA1 and entorhinal activity patterns preserves temporal context memory at long timescales

Converging, cross-species evidence indicates that memory for time is supported by hippocampal area CA1 and entorhinal cortex. However, limited evidence characterizes how these regions preserve temporal memories over long timescales (e.g., months). At long timescales, memoranda may be encountered in multiple temporal contexts, potentially creating interference. Here, using 7T fMRI, we measured CA1 and entorhinal activity patterns as human participants viewed thousands of natural scene images distributed, and repeated, across many months. We show that memory for an image's original temporal context was predicted by the degree to which CA1/entorhinal activity patterns from the first encounter with an image were re-expressed during re-encounters occurring minutes to months later. Critically, temporal memory signals were dissociable from predictors of recognition confidence, which were carried by distinct medial temporal lobe expressions. These findings suggest that CA1 and entorhinal cortex preserve temporal memories across long timescales by coding for and reinstating temporal context information.

Abrupt hippocampal remapping signals resolution of memory interference

Remapping refers to a decorrelation of hippocampal representations of similar spatial environments. While it has been speculated that remapping may contribute to the resolution of episodic memory interference in humans, direct evidence is surprisingly limited. We tested this idea using high-resolution, pattern-based fMRI analyses. Here we show that activity patterns in human CA3/dentate gyrus exhibit an abrupt, temporally-specific decorrelation of highly similar memory representations that is precisely coupled with behavioral expressions of successful learning. The magnitude of this learning-related decorrelation was predicted by the amount of pattern overlap during initial stages of learning, with greater initial overlap leading to stronger decorrelation. Finally, we show that remapped activity patterns carry relatively more information about learned episodic associations compared to competing associations, further validating the learning-related significance of remapping. Collectively, these findings establish a critical link between hippocampal remapping and episodic memory interference and provide insight into why remapping occurs.

Adaptive Repulsion of Long-Term Memory Representations Is Triggered by Event Similarity

We tested whether similarity between events triggers adaptive biases in how those events are remembered. We generated pairs of competing objects that were identical except in color and varied the degree of color similarity for the competing objects. Subjects (N = 123 across four experiments) repeatedly studied and were tested on associations between each of these objects and corresponding faces. As expected, high color similarity between competing objects created memory interference for object-face associations. Strikingly, high color similarity also resulted in a systematic bias in how the objects themselves were remembered: Competing objects with highly similar colors were remembered as being further apart (in color space) than they actually were. This repulsion of color memories increased with learning and served a clear adaptive purpose: Greater repulsion was associated with lower associative-memory interference. These findings reveal that similarity between events triggers adaptive-memory distortions that minimize interference.

Adaptive Memory Distortions Are Predicted by Feature Representations in Parietal Cortex

Similarity between memories is a primary cause of interference and forgetting. Exaggerating subtle differences between memories is therefore a potential mechanism for reducing interference. Here, we report a human fMRI study (n = 29, 19 female) that tested whether behavioral and neural expressions of memories are adaptively distorted to reduce interference. Participants learned and repeatedly retrieved object images, some of which were identical except for subtle color differences. Behavioral measures of color memory revealed exaggeration of differences between similar objects. Importantly, greater memory exaggeration was associated with lower memory interference. fMRI pattern analyses revealed that color information in parietal cortex was stronger during memory recall when color information was critical for discriminating competing memories. Moreover, greater representational distance between competing memories in parietal cortex predicted greater color memory exaggeration and lower memory interference. Together, these findings reveal that competition between memories induces adaptive, feature-specific distortions in parietal representations and corresponding behavioral expressions.SIGNIFICANCE STATEMENT Similarity between memories is a primary cause of interference and forgetting. Here, we show that, when remembering highly similar objects, subtle differences in the features of these objects are exaggerated in memory to reduce interference. These memory distortions are reflected in, and predicted by, overlap of activity patterns in lateral parietal cortex. These findings provide unique insight into how memory interference is resolved and specifically implicate lateral parietal cortex in representing feature-specific memory distortions.

Transforming the Concept of Memory Reactivation

Reactivation refers to the phenomenon wherein patterns of neural activity expressed during perceptual experience are re-expressed at a later time, a putative neural marker of memory. Reactivation of perceptual content has been observed across many cortical areas and correlates with objective and subjective expressions of memory in humans. However, because reactivation emphasizes similarities between perceptual and memory-based representations, it obscures differences in how perceptual events and memories are represented. Here, we highlight recent evidence of systematic differences in how (and where) perceptual events and memories are represented in the brain. We argue that neural representations of memories are best thought of as spatially transformed versions of perceptual representations. We consider why spatial transformations occur and identify critical questions for future research.

When the Memory System Gets Ahead of Itself

Humans are adept at learning and exploiting statistical regularities to predict future events from current experience. A recent paper by Sherman and Turk-Browne demonstrates that statistical regularities bias the hippocampus toward representing future states over current experience and reduce the degree to which current experience is encoded into memory.

Variability in the analysis of a single neuroimaging dataset by many teams

Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2-5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.

Decoding the tradeoff between encoding and retrieval to predict memory for overlapping events

When new events overlap with past events, there is a natural tradeoff between encoding the new event and retrieving the past event. Given the ubiquity of overlap among memories, this tradeoff between memory encoding and retrieval is of central importance to computational models of episodic memory (O'Reilly & McClelland 1994; Hasselmo 2005). However, prior studies have not directly linked neural markers of encoding/retrieval tradeoffs to behavioral measures of how overlapping events are remembered. Here, by decoding patterns of scalp electroencephalography (EEG) from male and female human subjects, we show that tradeoffs between encoding and retrieval states are reflected in distributed patterns of neural activity and, critically, these neural tradeoffs predict how overlapping events will later be remembered. Namely, new events that overlapped with past events were more likely to be subsequently remembered if neural patterns were biased toward a memory encoding state-or, conversely, away from a retrieval state. Additionally, we show that neural markers of encoding vs. retrieval states are surprisingly independent from previously-described EEG predictors of subsequent memory. Instead, we demonstrate that previously-described EEG predictors of subsequent memory are better explained by task engagement than by memory encoding, per se. Collectively, our findings provide important insight into how the memory system balances memory encoding and retrieval states and, more generally, into the neural mechanisms that support successful memory formation.

Decomposing Parietal Memory Reactivation to Predict Consequences of Remembering

Memory retrieval can strengthen, but also distort memories. Parietal cortex is a candidate region involved in retrieval-induced memory changes as it reflects retrieval success and represents retrieved content. Here, we conducted an fMRI experiment to test whether different forms of parietal reactivation predict distinct consequences of retrieval. Subjects studied associations between words and pictures of faces, scenes, or objects, and then repeatedly retrieved half of the pictures, reporting the vividness of the retrieved pictures (“retrieval practice”). On the following day, subjects completed a recognition memory test for individual pictures. Critically, the test included lures highly similar to studied pictures. Behaviorally, retrieval practice increased both hit and false alarm (FA) rates to similar lures, confirming a causal influence of retrieval on subsequent memory. Using pattern similarity analyses, we measured two different levels of reactivation during retrieval practice: generic “category-level” reactivation and idiosyncratic “item-level” reactivation. Vivid remembering during retrieval practice was associated with stronger category- and item-level reactivation in parietal cortex. However, these measures differentially predicted subsequent recognition memory performance: whereas higher category-level reactivation tended to predict FAs to lures, item-level reactivation predicted correct rejections. These findings indicate that parietal reactivation can be decomposed to tease apart distinct consequences of memory retrieval.

Overlap among Spatial Memories Triggers Repulsion of Hippocampal Representations

Across the domains of spatial navigation and episodic memory, the hippocampus is thought to play a critical role in disambiguating (pattern separating) representations of overlapping events. However, it is not fully understood how and why hippocampal patterns become separated. Here, we test the idea that event overlap triggers a "repulsion" among hippocampal representations that develops over the course of learning. Using a naturalistic route-learning paradigm and spatiotemporal pattern analysis of human fMRI data, we found that hippocampal representations of overlapping routes gradually diverged with learning to the point that they became less similar than representations of non-overlapping events. In other words, the hippocampus not only disambiguated overlapping events but formed representations that "reversed" the objective similarity among routes. This finding, which was selective to the hippocampus, is not predicted by standard theoretical accounts of pattern separation. Critically, because the overlapping route stimuli that we used ultimately diverged (so that each route contained overlapping and non-overlapping segments), we were able to test whether the reversal effect was selective to the overlapping segments. Indeed, once overlapping routes diverged (eliminating spatial and visual similarity), hippocampal representations paradoxically became relatively more similar. Finally, using a novel analysis approach, we show that the degree to which individual hippocampal voxels were initially shared across route representations was predictive of the magnitude of learning-related separation. Collectively, these findings indicate that event overlap triggers a repulsion of hippocampal representations-a finding that provides critical mechanistic insight into how and why hippocampal representations become separated.

Lower Parietal Encoding Activation Is Associated with Sharper Information and Better Memory

Mean fMRI activation in ventral posterior parietal cortex (vPPC) during memory encoding often negatively predicts successful remembering. A popular interpretation of this phenomenon is that vPPC reflects "off-task" processing. However, recent fMRI studies considering distributed patterns of activity suggest that vPPC actively represents encoded material. Here, we assessed the relationships between pattern-based content representations in vPPC, mean activation in vPPC, and subsequent remembering. We analyzed data from two fMRI experiments where subjects studied then recalled word-face or word-scene associations. For each encoding trial, we measured 1) mean univariate activation within vPPC and 2) the strength of face/scene information as indexed by pattern analysis. Mean activation in vPPC negatively predicted subsequent remembering, but the strength of pattern-based information in the same vPPC voxels positively predicted later memory. Indeed, univariate amplitude averaged across vPPC voxels negatively correlated with pattern-based information strength. This dissociation reflected a tendency for univariate reductions to maximally occur in voxels that were not strongly tuned for the category of encoded stimuli. These results indicate that vPPC activity patterns reflect the content and quality of memory encoding and constitute a striking example of lower univariate activity corresponding to stronger pattern-based information.

Predicting the integration of overlapping memories by decoding mnemonic processing states during learning

The hippocampal memory system is thought to alternate between two opposing processing states: encoding and retrieval. When present experience overlaps with past experience, this creates a potential tradeoff between encoding the present and retrieving the past. This tradeoff may be resolved by memory integration-that is, by forming a mnemonic representation that links present experience with overlapping past experience. Here, we used fMRI decoding analyses to predict when - and establish how - past and present experiences become integrated in memory. In an initial experiment, we alternately instructed subjects to adopt encoding, retrieval or integration states during overlapping learning. We then trained across-subject pattern classifiers to 'read out' the instructed processing states from fMRI activity patterns. We show that an integration state was clearly dissociable from encoding or retrieval states. Moreover, trial-by-trial fluctuations in decoded evidence for an integration state during learning reliably predicted behavioral expressions of successful memory integration. Strikingly, the decoding algorithm also successfully predicted specific instances of spontaneous memory integration in an entirely independent sample of subjects for whom processing state instructions were not administered. Finally, we show that medial prefrontal cortex and hippocampus differentially contribute to encoding, retrieval, and integration states: whereas hippocampus signals the tradeoff between encoding vs. retrieval states, medial prefrontal cortex actively represents past experience in relation to new learning.

Neural portraits of perception: reconstructing face images from evoked brain activity

Recent neuroimaging advances have allowed visual experience to be reconstructed from patterns of brain activity. While neural reconstructions have ranged in complexity, they have relied almost exclusively on retinotopic mappings between visual input and activity in early visual cortex. However, subjective perceptual information is tied more closely to higher-level cortical regions that have not yet been used as the primary basis for neural reconstructions. Furthermore, no reconstruction studies to date have reported reconstructions of face images, which activate a highly distributed cortical network. Thus, we investigated (a) whether individual face images could be accurately reconstructed from distributed patterns of neural activity, and (b) whether this could be achieved even when excluding activity within occipital cortex. Our approach involved four steps. (1) Principal component analysis (PCA) was used to identify components that efficiently represented a set of training faces. (2) The identified components were then mapped, using a machine learning algorithm, to fMRI activity collected during viewing of the training faces. (3) Based on activity elicited by a new set of test faces, the algorithm predicted associated component scores. (4) Finally, these scores were transformed into reconstructed images. Using both objective and subjective validation measures, we show that our methods yield strikingly accurate neural reconstructions of faces even when excluding occipital cortex. This methodology not only represents a novel and promising approach for investigating face perception, but also suggests avenues for reconstructing 'offline' visual experiences-including dreams, memories, and imagination-which are chiefly represented in higher-level cortical areas.

Attending to the present when remembering the past

In this issue of Neuron, Guerin et al. (2012) provide novel evidence that distinct parietal mechanisms for attention and memory compete when past experiences are compared to current perceptual input. While dorsal parietal cortex supports attention to perceptual stimuli, high attentional demands suppress ventral parietal regions important for veridical remembering.

Multi-voxel patterns of visual category representation during episodic encoding are predictive of subsequent memory

Successful encoding of episodic memories is thought to depend on contributions from prefrontal and temporal lobe structures. Neural processes that contribute to successful encoding have been extensively explored through univariate analyses of neuroimaging data that compare mean activity levels elicited during the encoding of events that are subsequently remembered vs. those subsequently forgotten. Here, we applied pattern classification to fMRI data to assess the degree to which distributed patterns of activity within prefrontal and temporal lobe structures elicited during the encoding of word-image pairs were diagnostic of the visual category (Face or Scene) of the encoded image. We then assessed whether representation of category information was predictive of subsequent memory. Classification analyses indicated that temporal lobe structures contained information robustly diagnostic of visual category. Information in prefrontal cortex was less diagnostic of visual category, but was nonetheless associated with highly reliable classifier-based evidence for category representation. Critically, trials associated with greater classifier-based estimates of category representation in temporal and prefrontal regions were associated with a higher probability of subsequent remembering. Finally, consideration of trial-by-trial variance in classifier-based measures of category representation revealed positive correlations between prefrontal and temporal lobe representations, with the strength of these correlations varying as a function of the category of image being encoded. Together, these results indicate that multi-voxel representations of encoded information can provide unique insights into how visual experiences are transformed into episodic memories.

Resistance to forgetting associated with hippocampus-mediated reactivation during new learning

A primary reason we forget past experiences is because we acquire new memories in the interim. While the hippocampus is thought to play a primary role in acquiring and retaining memories for the past, there is little evidence linking neural operations during new learning to the forgetting (or remembering) of earlier events. Here we present novel evidence that, during the encoding of new memories, responses within the human hippocampus are predictive of the retention of memories for previously experienced, overlapping events. This brain-behavior relationship is evident in neural responses to individual events and in differences across individuals. We illustrate that the hippocampus accomplishes this function by reactivating older memories as new memories are formed—in this case, reactivating neural responses that represent monetary rewards associated with older memories. These data reveal a fundamental mechanism through which the hippocampus tempers the forgetting of older memories as newer memories are acquired.

Decreased demands on cognitive control reveal the neural processing benefits of forgetting

Remembering often requires the selection of goal-relevant memories in the face of competition from irrelevant memories. Although there is a cost of selecting target memories over competing memories (increased forgetting of the competing memories), here we report neural evidence for the adaptive benefits of forgetting--namely, reduced demands on cognitive control during future acts of remembering. Functional magnetic resonance imaging during selective retrieval showed that repeated retrieval of target memories was accompanied by dynamic reductions in the engagement of functionally coupled cognitive control mechanisms that detect (anterior cingulate cortex) and resolve (dorsolateral and ventrolateral prefrontal cortex) mnemonic competition. Strikingly, regression analyses revealed that this prefrontal disengagement tracked the extent to which competing memories were forgotten; greater forgetting of competing memories was associated with a greater decline in demands on prefrontal cortex during target remembering. These findings indicate that, although forgetting can be frustrating, memory might be adaptive because forgetting confers neural processing benefits.