Perceptual and Semantic Representations at Encoding Contribute to True and False Recognition of Objects

Discrimination of semantically similar verbal memory traces is affected in healthy aging

Mnemonic discrimination of highly similar memory traces is affected in healthy aging via changes in hippocampal pattern separation-i.e., the ability of the hippocampus to orthogonalize highly similar neural inputs. The decline of this process leads to a loss of episodic specificity. Because previous studies have almost exclusively tested mnemonic discrimination of visuospatial stimuli (e.g., objects or scenes), less is known about age-related effects on the episodic specificity of semantically similar traces. To address this gap, we designed a task to assess mnemonic discrimination of verbal stimuli as a function of semantic similarity based on word embeddings. Forty young (Mage = 21.7 years) and 40 old adults (Mage = 69.8 years) first incidentally encoded adjective-noun phrases, then performed a surprise recognition test involving exactly repeated and highly similar lure phrases. We found that increasing semantic similarity negatively affected mnemonic discrimination in both age groups, and that compared to young adults, older adults showed worse discrimination at medium levels of semantic similarity. These results indicate that episodic specificity of semantically similar memory traces is affected in aging via less efficient mnemonic operations and strengthen the notion that mnemonic discrimination is a modality-independent process supporting memory specificity across representational domains.

Representational formats of human memory traces

Neural representations are internal brain states that constitute the brain's model of the external world or some of its features. In the presence of sensory input, a representation may reflect various properties of this input. When perceptual information is no longer available, the brain can still activate representations of previously experienced episodes due to the formation of memory traces. In this review, we aim at characterizing the nature of neural memory representations and how they can be assessed with cognitive neuroscience methods, mainly focusing on neuroimaging. We discuss how multivariate analysis techniques such as representational similarity analysis (RSA) and deep neural networks (DNNs) can be leveraged to gain insights into the structure of neural representations and their different representational formats. We provide several examples of recent studies which demonstrate that we are able to not only measure memory representations using RSA but are also able to investigate their multiple formats using DNNs. We demonstrate that in addition to slow generalization during consolidation, memory representations are subject to semantization already during short-term memory, by revealing a shift from visual to semantic format. In addition to perceptual and conceptual formats, we describe the impact of affective evaluations as an additional dimension of episodic memories. Overall, these studies illustrate how the analysis of neural representations may help us gain a deeper understanding of the nature of human memory.

Representation of event and object concepts in ventral anterior temporal lobe and angular gyrus

Semantic knowledge includes understanding of objects and their features and also understanding of the characteristics of events. The hub-and-spoke theory holds that these conceptual representations rely on multiple information sources that are integrated in a central hub in the ventral anterior temporal lobes. The dual-hub theory expands this framework with the claim that the ventral anterior temporal lobe hub is specialized for object representation, while a second hub in angular gyrus is specialized for event representation. To test these ideas, we used representational similarity analysis, univariate and psychophysiological interaction analyses of fMRI data collected while participants processed object and event concepts (e.g. "an apple," "a wedding") presented as images and written words. Representational similarity analysis showed that angular gyrus encoded event concept similarity more than object similarity, although the left angular gyrus also encoded object similarity. Bilateral ventral anterior temporal lobes encoded both object and event concept structure, and left ventral anterior temporal lobe exhibited stronger coding for events. Psychophysiological interaction analysis revealed greater connectivity between left ventral anterior temporal lobe and right pMTG, and between right angular gyrus and bilateral ITG and middle occipital gyrus, for event concepts compared to object concepts. These findings support the specialization of angular gyrus for event semantics, though with some involvement in object coding, but do not support ventral anterior temporal lobe specialization for object concepts.

Hippocampal Functions Modulate Transfer-Appropriate Cortical Representations Supporting Subsequent Memory

The hippocampus plays a central role as a coordinate system or index of information stored in neocortical loci. Nonetheless, it remains unclear how hippocampal processes integrate with cortical information to facilitate successful memory encoding. Thus, the goal of the current study was to identify specific hippocampal-cortical interactions that support object encoding. We collected fMRI data while 19 human participants (7 female and 12 male) encoded images of real-world objects and tested their memory for object concepts and image exemplars (i.e., conceptual and perceptual memory). Representational similarity analysis revealed robust representations of visual and semantic information in canonical visual (e.g., occipital cortex) and semantic (e.g., angular gyrus) regions in the cortex, but not in the hippocampus. Critically, hippocampal functions modulated the mnemonic impact of cortical representations that are most pertinent to future memory demands, or transfer-appropriate representations Subsequent perceptual memory was best predicted by the strength of visual representations in ventromedial occipital cortex in coordination with hippocampal activity and pattern information during encoding. In parallel, subsequent conceptual memory was best predicted by the strength of semantic representations in left inferior frontal gyrus and angular gyrus in coordination with either hippocampal activity or semantic representational strength during encoding. We found no evidence for transfer-incongruent hippocampal-cortical interactions supporting subsequent memory (i.e., no hippocampal interactions with cortical visual/semantic representations supported conceptual/perceptual memory). Collectively, these results suggest that diverse hippocampal functions flexibly modulate cortical representations of object properties to satisfy distinct future memory demands.Significance Statement The hippocampus is theorized to index pieces of information stored throughout the cortex to support episodic memory. Yet how hippocampal processes integrate with cortical representation of stimulus information remains unclear. Using fMRI, we examined various forms of hippocampal-cortical interactions during object encoding in relation to subsequent performance on conceptual and perceptual memory tests. Our results revealed novel hippocampal-cortical interactions that utilize semantic and visual representations in transfer-appropriate manners: conceptual memory supported by hippocampal modulation of frontoparietal semantic representations, and perceptual memory supported by hippocampal modulation of occipital visual representations. These findings provide important insights into the neural mechanisms underlying the formation of information-rich episodic memory and underscore the value of studying the flexible interplay between brain regions for complex cognition.

Recurrent connectivity supports higher-level visual and semantic object representations in the brain

Visual object recognition has been traditionally conceptualised as a predominantly feedforward process through the ventral visual pathway. While feedforward artificial neural networks (ANNs) can achieve human-level classification on some image-labelling tasks, it's unclear whether computational models of vision alone can accurately capture the evolving spatiotemporal neural dynamics. Here, we probe these dynamics using a combination of representational similarity and connectivity analyses of fMRI and MEG data recorded during the recognition of familiar, unambiguous objects. Modelling the visual and semantic properties of our stimuli using an artificial neural network as well as a semantic feature model, we find that unique aspects of the neural architecture and connectivity dynamics relate to visual and semantic object properties. Critically, we show that recurrent processing between the anterior and posterior ventral temporal cortex relates to higher-level visual properties prior to semantic object properties, in addition to semantic-related feedback from the frontal lobe to the ventral temporal lobe between 250 and 500 ms after stimulus onset. These results demonstrate the distinct contributions made by semantic object properties in explaining neural activity and connectivity, highlighting it as a core part of object recognition not fully accounted for by current biologically inspired neural networks.

Effects of Aging on Successful Object Encoding: Enhanced Semantic Representations Compensate for Impaired Visual Representations

Although episodic memory and visual processing decline substantially with healthy aging, semantic knowledge is generally spared. There is evidence that older adults' spared semantic knowledge can support episodic memory. Here, we used functional magnetic resonance imaging (fMRI) combined with representational similarity analyses (RSAs) to examine how novel visual and preexisting semantic representations at encoding predict subjective memory vividness at retrieval. Eighteen young and seventeen older adults (female and male participants) encoded images of objects during fMRI scanning and recalled these images while rating the vividness of their memories. After scanning, participants discriminated between studied images and similar lures. RSA based on a deep convolutional neural network and normative concept feature data were used to link patterns of neural activity during encoding to visual and semantic representations. Relative to young adults, the specificity of activation patterns for visual features was reduced in older adults, consistent with dedifferentiation. However, the specificity of activation patterns for semantic features was enhanced in older adults, consistent with hyperdifferentiation. Despite dedifferentiation, visual representations in early visual cortex (EVC) predicted high memory vividness in both age groups. In contrast, semantic representations in lingual gyrus (LG) and fusiform gyrus (FG) were associated with high memory vividness only in the older adults. Intriguingly, data suggests that older adults with lower specificity of visual representations in combination with higher specificity of semantic representations tended to rate their memories as more vivid. Our findings suggest that memory vividness in aging relies more on semantic representations over anterior regions, potentially compensating for age-related dedifferentiation of visual information in posterior regions.SIGNIFICANCE STATEMENT Normal aging is associated with impaired memory for events while semantic knowledge might even improve. We investigated the effects of aging on the specificity of visual and semantic information in the brain when viewing common objects and how this information enables subsequent memory vividness for these objects. Using functional magnetic resonance imaging (fMRI) combined with modeling of the stimuli we found that visual information was represented with less specificity in older than young adults while still supporting memory vividness. In contrast semantic information supported memory vividness only in older adults and especially in those individuals that had the lowest specificity of visual information. These findings provide evidence for a spared semantic memory system increasingly recruited to compensate for degraded visual representations in older age.

Anterior Hippocampal Engagement during Memory Formation Predicts Subsequent False Recognition of Similar Experiences

People better remember experiences when they orient to meaning over surface-level perceptual features. Such an orientation-related memory boost has been associated with engagement of both hippocampus (HPC) and neocortex during encoding. However, less is known about the neural mechanisms by which a cognitive orientation toward meaning might also promote memory errors, with one open question being whether the HPC-a region traditionally implicated in precise memory formation-also contributes to behavioral imprecision. We used fMRI to characterize encoding-phase signatures as people oriented toward the meaning (story) versus perceptual style (artist) of storybook-style illustrations and then linked them to subsequent true and false memories. We found that story and artist orientation tasks were each associated with both unique univariate profiles and consistent neural states defined using multivoxel patterns. Linking these neural signatures to behavior, we found that greater medial pFC activation and alignment of neural patterns to the story (but not artist) state was related to subsequent memory success on a trial-by-trial basis. Moreover, among successfully remembered experiences, greater anterior HPC engagement at encoding was associated with a higher likelihood of related false recognitions, consistent with the encoding of broad traces in this region. Interestingly, these effects did not reliably vary by cued orientation. These results suggest that, irrespective of the cued encoding orientation, neocortical and hippocampal mechanisms associated with orienting to meaning (story) over perceptual (artist) features may support memory, with the formation of generalizable memories being a specialty of anterior HPC.

Dynamic changes in neural representations underlie the repetition effect on false memory

Restudying word lists (e.g., dream, awake, and bed) strengthens true memory of the studied words and reduces false memory for unstudied but semantically related lures (e.g., sleep). Yet, the neural mechanisms involved in this repetition effect on false memory remain unclear. Possible mechanisms involve item-specific and semantic neural representations at encoding, and the memory strength between encoding and retrieval. This study first replicated the behavioral results (Exp. 1) and then investigated various neural mechanisms by using slow event-related functional magnetic resonance imaging (fMRI) and representational similarity analysis (Exp. 2). Behavioral results confirmed that restudy improved true memory and reduced false memory. The fMRI results showed that restudy induced item-specific neural representations at encoding in the left occipital pole, but reduced neural overlap between semantic representations at encoding in the left temporal pole. Individual differences in these two encoding neural mechanisms were correlated with the behavioral measure of false memory, with greater restudy-induced representational changes at encoding (item-specific neural representations and reduced neural overlap between semantic representations) being associated with lower false memory. Moreover, restudy enhanced the memory strength between encoding and retrieval in the visuoparietal cortex but reduced it in the frontal cortex. These findings suggest that dynamic changes in neural representations underlie the repetition effect on false memory, supporting a dual-coding neural framework.