Decoding the tradeoff between encoding and retrieval to predict memory for overlapping events

Benefits of spaced learning are predicted by re-encoding of past experience in ventromedial prefrontal cortex

More than a century of research shows that spaced learning improves long-term memory. Yet, there remains debate concerning why. A major limitation to resolving theoretical debates is the lack of evidence for how neural representations change as a function of spacing. Here, leveraging a massive-scale 7T human fMRI dataset, we tracked neural representations and behavioral expressions of memory as participants viewed thousands of natural scene images that repeated at lags ranging from seconds to many months. We show that spaced learning increases the similarity of human ventromedial prefrontal cortex representations across stimulus encounters and, critically, these increases parallel and predict the behavioral benefits of spacing. Additionally, we show that these spacing benefits critically depend on remembering and, in turn, 're-encoding' past experience. Collectively, our findings provide fundamental insight into how spaced learning influences neural representations and why spacing is beneficial.

The persistence of memory: prior memory responses modulate behavior and brain state engagement

Memory brain states may influence how we experience an event. Memory encoding and retrieval constitute neurally dissociable brain states that individuals can selectively engage based on top-down goals. To the extent that memory states linger in time - as suggested by prior behavioral work - memory states may influence not only the current experience, but also subsequent stimuli and judgments. Thus lingering memory states may have broad influences on cognition, yet this account has not been directly tested utilizing neural measures of memory states. Here we address this gap by testing the hypothesis that memory brain states are modulated by memory judgments, and that these brain states persist for several hundred milliseconds. We recorded scalp electroencephalography (EEG) while participants completed a recognition memory task. We used an independently validated multivariate mnemonic state classifier to assess memory state engagement. We replicate prior behavioral findings; however, our neural findings run counter to the predictions made on the basis of the behavioral data. Surprisingly, we find that prior responses modulate current memory state engagement on the basis of response congruency. That is, we find strong engagement of the retrieval state on incongruent trials - when a target is preceded by a correct rejection of a lure and when a lure is preceded by successful recognition of a target. These findings indicate that cortical brain states are influenced by prior judgments and suggest that a non-mnemonic, internal attention state may be recruited to in the face of changing demands in a dynamic environment.

EEG decoders track memory dynamics

Encoding- and retrieval-related neural activity jointly determine mnemonic success. We ask whether electroencephalographic activity can reliably predict encoding and retrieval success on individual trials. Each of 98 participants performed a delayed recall task on 576 lists across 24 experimental sessions. Logistic regression classifiers trained on spectral features measured immediately preceding spoken recall of individual words successfully predict whether or not those words belonged to the target list. Classifiers trained on features measured during word encoding also reliably predict whether those words will be subsequently recalled and further predict the temporal and semantic organization of the recalled items. These findings link neural variability predictive of successful memory with item-to-context binding, a key cognitive process thought to underlie episodic memory function.

Top-down task goals induce the retrieval state

Engaging the retrieval state (Tulving, 1983) impacts processing and behavior (Long & Kuhl, 2019, 2021; Smith, Moore, & Long, 2022), but the extent to which top-down factors - explicit instructions and goals - vs. bottom-up factors - stimulus properties such as repetition and similarity - jointly or independently induce the retrieval state is unclear. Identifying the impact of bottom-up and top-down factors on retrieval state engagement is critical for understanding how control of task-relevant vs. task-irrelevant brain states influence cognition. We conducted between-subjects recognition memory tasks on male and female human participants in which we varied test phase goals. We recorded scalp electroencephalography and used an independently validated mnemonic state classifier (Long, 2023) to measure retrieval state engagement as a function of top-down task goals (recognize old vs. detect new items) and bottom-up stimulus repetition (hits vs. correct rejections). We find that whereas the retrieval state is engaged for hits regardless of top-down goals, the retrieval state is only engaged during correct rejections when the top-down goal is to recognize old items. Furthermore, retrieval state engagement is greater for low compared to high confidence hits when the task goal is to recognize old items. Together, these results suggest that top-down demands to recognize old items induce the retrieval state independent from bottom-up factors, potentially reflecting the recruitment of internal attention to enable access of a stored representation.

Significance Statement

Both top-down goals and automatic bottom-up influences may lead us into a retrieval brain state - a whole-brain pattern of activity that supports our ability to remember the past. Here we tested the extent to which top-down vs. bottom-up factors independently influence the retrieval state by manipulating participants' goals and stimulus repetition during a memory test. We find that in response to the top-down goal to recognize old items, the retrieval state is engaged for both old and new probes, suggesting that top-down and bottom-up factors independently engage the retrieval state. Our interpretation is that top-down demands recruit internal attention in service of the attempt to access a stored representation.

Semantic associations restore neural encoding mechanisms

Lapses in attention can negatively impact later memory of an experience. Attention and encoding resources are thought to decline as more experiences are encountered in succession, accounting for the primacy effect in which memory is better for items encountered early compared to late in a study list. However, accessing prior knowledge during study can facilitate subsequent memory, suggesting a potential avenue to counteract this decline. Here, we investigated the extent to which semantic associations-shared meaning between experiences-can counteract declines in encoding resources. Our hypothesis is that semantic associations restore neural encoding mechanisms, which in turn improves memory. We recorded scalp electroencephalography (EEG) while male and female human participants performed a delayed free recall task. Half of the items from late in each study list were semantically associated with an item presented earlier in the list. We find that semantic associations improve memory specifically for late list items and selectively modulate the neural signals engaged during the study of late list items. Relative to other recalled items, late list items that are subsequently semantically clustered-recalled consecutively with their semantic associate-elicit increased high-frequency activity and decreased low-frequency activity, a hallmark of successful encoding. Our findings demonstrate that semantic associations restore neural encoding mechanisms and improve later memory. More broadly, these findings suggest that prior knowledge modulates the orientation of attention to influence encoding mechanisms.

Spatiotemporal Dynamics of Memory Encoding and Memory Retrieval States

Memory encoding and memory retrieval are neurally distinct brain states that can be differentiated on the basis of cortical network activity. However, it is unclear whether sustained engagement of one network or fluctuations between multiple networks give rise to these memory states. The spatiotemporal dynamics of memory states may have important implications for memory behavior and cognition; however, measuring temporally resolved signals of cortical networks poses a challenge. Here, we recorded scalp electroencephalography from participants performing a mnemonic state task in which they were biased toward memory encoding or retrieval. We performed a microstate analysis to measure the temporal dynamics of cortical networks throughout this mnemonic state task. We find that Microstate E, a putative analog of the default mode network, shows temporally sustained dissociations between memory encoding and retrieval, with greater engagement during retrieve compared with encode trials. We further show that decreased engagement of Microstate E is a general property of encoding, rather than a reflection of retrieval suppression. Thus, memory success, as well as cognition more broadly, may be influenced by the ability to engage or disengage Microstate E in a goal-dependent manner.

Memory and attention: A double dissociation between memory encoding and memory retrieval

Research has found substantial negative effects of divided attention (DA) during encoding but less substantial effects when attention is divided during retrieval, an asymmetry which has been interpreted as indicating that different control processes or forms of attention are involved in encoding and retrieval (e.g., Chun & Johnson, 2011; Craik, Govoni, Naveh-Benjamin, & Anderson, 1996; Long, Kuhl, & Chun, 2018). The extant evidence, however, is not strong support for qualitative differences and might simply indicate differential sensitivity. The present experiments document a stronger, double dissociation by focusing on the Attentional Boost Effect (ABE) - a phenomenon in which the detection of targets in a secondary task enhances encoding of co-occurring stimuli. The dual-task interaction account proposes that the classical negative effects produced by dual-task interference are offset by a transient increase in externally-directed attention brought about by target detection. Since externally-directed attention is less valuable for retrieval processes, the ABE should result in a net negative effect when applied in the test phase because the dual-task interference would no longer be offset by the externally-directed boost occurring during target trials. Experiments 1, 2 and 4 confirmed the predictions by showing that test words paired with target stimuli were recognized significantly worse than test words paired with distractor stimuli. In contrast, Experiments 3 and 4 replicated the usual positive effects of the ABE with respect to encoding. We discuss these findings in light of recent theoretical proposals suggesting that encoding and retrieval processes are subserved by different forms of attention (external [perceptual] vs. internal [reflective]). Implications for the Transfer-Appropriate-Processing view of memory are also illustrated.

Perception and memory retrieval states are reflected in distributed patterns of background functional connectivity

The same visual input can serve as the target of perception or as a trigger for memory retrieval depending on whether cognitive processing is externally oriented (perception) or internally oriented (memory retrieval). While numerous human neuroimaging studies have characterized how visual stimuli are differentially processed during perception versus memory retrieval, perception and memory retrieval may also be associated with distinct neural states that are independent of stimulus-evoked neural activity. Here, we combined human fMRI with full correlation matrix analysis (FCMA) to reveal potential differences in "background" functional connectivity across perception and memory retrieval states. We found that perception and retrieval states could be discriminated with high accuracy based on patterns of connectivity across (1) the control network, (2) the default mode network (DMN), and (3) retrosplenial cortex (RSC). In particular, clusters in the control network increased connectivity with each other during the perception state, whereas clusters in the DMN were more strongly coupled during the retrieval state. Interestingly, RSC switched its coupling between networks as the cognitive state shifted from retrieval to perception. Finally, we show that background connectivity (1) was fully independent from stimulus-related variance in the signal and, further, (2) captured distinct aspects of cognitive states compared to traditional classification of stimulus-evoked responses. Together, our results reveal that perception and memory retrieval are associated with sustained cognitive states that manifest as distinct patterns of connectivity among large-scale brain networks.

The intersection of the retrieval state and internal attention

Large-scale brain states or distributed patterns of brain activity modulate downstream processing and behavior. Sustained attention and memory retrieval states impact subsequent memory, yet how these states relate to one another is unclear. I hypothesize that internal attention is a central process of the retrieval state. The alternative is that the retrieval state specifically reflects a controlled, episodic retrieval mode, engaged only when intentionally accessing events situated within a spatiotemporal context. To test my hypothesis, I developed a mnemonic state classifier independently trained to measure retrieval state evidence and applied this classifier to a spatial attention task. I find that retrieval state evidence increases during delay and response intervals when participants are maintaining spatial information. Critically, retrieval state evidence is positively related to the amount of maintained spatial location information and predicts target detection reaction times. Together, these findings support the hypothesis that internal attention is a central process of the retrieval state.

Temporal Dynamics of Competition between Statistical Learning and Episodic Memory in Intracranial Recordings of Human Visual Cortex

The function of long-term memory is not just to reminisce about the past, but also to make predictions that help us behave appropriately and efficiently in the future. This predictive function of memory provides a new perspective on the classic question from memory research of why we remember some things but not others. If prediction is a key outcome of memory, then the extent to which an item generates a prediction signifies that this information already exists in memory and need not be encoded. We tested this principle using human intracranial EEG as a time-resolved method to quantify prediction in visual cortex during a statistical learning task and link the strength of these predictions to subsequent episodic memory behavior. Epilepsy patients of both sexes viewed rapid streams of scenes, some of which contained regularities that allowed the category of the next scene to be predicted. We verified that statistical learning occurred using neural frequency tagging and measured category prediction with multivariate pattern analysis. Although neural prediction was robust overall, this was driven entirely by predictive items that were subsequently forgotten. Such interference provides a mechanism by which prediction can regulate memory formation to prioritize encoding of information that could help learn new predictive relationships.SIGNIFICANCE STATEMENT When faced with a new experience, we are rarely at a loss for what to do. Rather, because many aspects of the world are stable over time, we rely on past experiences to generate expectations that guide behavior. Here we show that these expectations during a new experience come at the expense of memory for that experience. From intracranial recordings of visual cortex, we decoded what humans expected to see next in a series of photographs based on patterns of neural activity. Photographs that generated strong neural expectations were more likely to be forgotten in a later behavioral memory test. Prioritizing the storage of experiences that currently lead to weak expectations could help improve these expectations in future encounters.

Hippocampal convergence during anticipatory midbrain activation promotes subsequent memory formation

The hippocampus has been a focus of memory research since H.M's surgery abolished his ability to form new memories, yet its mechanistic role in memory remains debated. Here, we identify a candidate memory mechanism: an anticipatory hippocampal "convergence state", observed while awaiting valuable information, and which predicts subsequent learning. During fMRI, participants viewed trivia questions eliciting high or low curiosity, followed seconds later by its answer. We reasoned that encoding success requires a confluence of conditions, so that hippocampal states more conducive to memory formation should converge in state space. To operationalize convergence of neural states, we quantified the typicality of multivoxel patterns in the medial temporal lobes during anticipation and encoding of trivia answers. We found that the typicality of anticipatory hippocampal patterns increased during high curiosity. Crucially, anticipatory hippocampal pattern typicality increased with dopaminergic midbrain activation and uniquely accounted for the association between midbrain activation and subsequent recall. We propose that hippocampal convergence states may complete a cascade from motivation and midbrain activation to memory enhancement, and may be a general predictor of memory formation.

Cortical Representations of Visual Stimuli Shift Locations with Changes in Memory States

Episodic memory retrieval is thought to rely on reactivation of the same content-sensitive neural activity patterns initially expressed during memory encoding.^1-6 Yet there are emerging examples of content representations expressed in different brain regions during encoding versus retrieval.^7-14 Although these differences have been observed by comparing encoding and retrieval tasks that differ in terms of perceptual experience and cognitive demands, there are many real-world contexts-e.g., meeting a new colleague who reminds you of an old acquaintance-where the memory system might be intrinsically biased either toward encoding (the new colleague) or retrieval (the old acquaintance).¹⁵¹⁶ Here, we test whether intrinsic memory states, independent of task demands, determine the cortical location of content representations. In a human fMRI study, subjects (n = 33) viewed object images and were instructed to either encode the current object or retrieve a similar object from memory. Using pattern classifiers, we show that biases toward encoding versus retrieval were reflected in large-scale attentional networks.^17-19 Critically, memory states decoded from these networks-even when entirely independent from task instructions-predicted shifts of object representations from visual cortex (encoding) to ventral parietal cortex (retrieval). Finally, visual versus ventral parietal cortices exhibited differential connectivity with the hippocampus during memory encoding versus retrieval, consistent with the idea that the hippocampus mediates cortical shifts in content representations. Collectively, these findings demonstrate that intrinsic biases toward memory encoding versus retrieval determine the specific cortical locations that express content information.

When the Memory System Gets Ahead of Itself

Humans are adept at learning and exploiting statistical regularities to predict future events from current experience. A recent paper by Sherman and Turk-Browne demonstrates that statistical regularities bias the hippocampus toward representing future states over current experience and reduce the degree to which current experience is encoded into memory.