Semantic congruence affects hippocampal response to repetition of visual associations

ObScene database: Semantic congruency norms for 898 pairs of object-scene pictures

Research on the interaction between object and scene processing has a long history in the fields of perception and visual memory. Most databases have established norms for pictures where the object is embedded in the scene. In this study, we provide a diverse and controlled stimulus set comprising real-world pictures of 375 objects (e.g., suitcase), 245 scenes (e.g., airport), and 898 object-scene pairs (e.g., suitcase-airport), with object and scene presented separately. Our goal was twofold. First, to create a database of object and scene pictures, normed for the same variables to have comparable measures for both types of pictures. Second, to acquire normative data for the semantic relationships between objects and scenes presented separately, which offers more flexibility in the use of the pictures and allows disentangling the processing of the object and its context (the scene). Along three experiments, participants evaluated each object or scene picture on name agreement, familiarity, and visual complexity, and rated object-scene pairs on semantic congruency. A total of 125 septuplets of one scene and six objects (three congruent, three incongruent), and 120 triplets of one object and two scenes (in congruent and incongruent pairings) were built. In future studies, these objects and scenes can be used separately or combined, while controlling for their key features. Additionally, as object-scene pairs received semantic congruency ratings along the entire scale, researchers may select among a wide range of congruency values. ObScene is a comprehensive and ecologically valid database, useful for psychology and neuroscience studies of visual object and scene processing.

The effect of context congruency on fMRI repetition suppression for objects

The recognition of objects is strongly facilitated when they are presented in the context of other objects (Biederman, 1972). Such contexts facilitate perception and induce expectations of context-congruent objects (Trapp and Bar, 2015). The neural mechanisms underlying these facilitatory effects of context on object processing, however, are not yet fully understood. In the present study, we investigate how context-induced expectations affect subsequent object processing. We used functional magnetic resonance imaging and measured repetition suppression as a proxy for prediction error processing. Participants viewed pairs of alternating or repeated object images which were preceded by context-congruent, context-incongruent or neutral cues. We found a stronger repetition suppression in congruent as compared to incongruent or neutral cues in the object sensitive lateral occipital cortex. Interestingly, this stronger effect was driven by enhanced responses to alternating stimulus pairs in the congruent contexts, rather than by suppressed responses to repeated stimulus pairs, which emphasizes the contribution of surprise-related response enhancement for the context modulation on RS when expectations are violated. In addition, in the congruent condition, we discovered significant functional connectivity between object-responsive and frontal cortical regions, as well as between object-responsive regions and the fusiform gyrus. Our findings indicate that prediction errors, reflected in enhanced brain responses to violated contextual expectations, underlie the facilitating effect of context during object perception.

Effects of prior knowledge on brain activation and functional connectivity during memory retrieval

Previous studies have shown that the ventral medial prefrontal cortex (vmPFC) plays an important role in schema-related memory. However, there is an intensive debate to what extent the activation of subregions of the hippocampus is involved in retrieving schema-related memory. In addition, it is unclear how the functional connectivity (FC) between the vmPFC and the hippocampus, as well as the connectivity of the vmPFC with other regions, are modulated by prior knowledge (PK) during memory retrieval over time. To address these issues, participants learned paragraphs that described features of each unfamiliar word from familiar and unfamiliar categories (i.e., high and low PK conditions) 20 min, 1 day, and 1 week before the test. They then performed a recognition task to judge whether the sentences were old in the scanner. The results showed that the activation of the anterior-medial hippocampus (amHPC) cluster was stronger when the old sentences with high (vs. low) PK were correctly retrieved. The activation of the posterior hippocampus (pHPC) cluster, as well as the vmPFC, was stronger when the new sentences with high (vs. low) PK were correctly rejected (i.e., CR trials), whereas the cluster of anterior-lateral hippocampus (alHPC) showed the opposite. The FC of the vmPFC with the amHPC and perirhinal cortex/inferior temporal gyrus was stronger in the high (vs. low) PK condition, whereas the FC of the vmPFC with the alHPC, thalamus and frontal regions showed the opposite for the CR trials. This study highlighted that different brain networks, which were associated with the vmPFC, subregions of the hippocampus and cognitive control regions, were responsible for retrieving the information with high and low PK.

Leveraging Prior Knowledge to Support Short-term Memory: Exploring the Role of the Ventromedial Prefrontal Cortex

It is well established that the ventromedial prefrontal cortex (vmPFC) plays a critical role in memory consolidation and the retrieval of remote long-term memories. Recent evidence suggests that the vmPFC also supports rapid neocortical learning and consolidation over shorter timescales, particularly when novel events align with stored knowledge. One mechanism by which the vmPFC has been proposed to support this learning is by integrating congruent information into existing neocortical knowledge during memory encoding. An important outstanding question is whether the vmPFC also plays a critical role in linking congruent information with existing knowledge before storage in long-term memory. The current study investigated this question by testing whether lesions to the vmPFC disrupt the ability to leverage stored knowledge in support of short-term memory. Specifically, we investigated the visuospatial bootstrapping effect, the phenomenon whereby immediate verbal recall of visually presented stimuli is better when stimuli appear in a familiar visuospatial array that is congruent with prior knowledge compared with an unfamiliar visuospatial array. We found that the overall magnitude of the bootstrapping effect did not differ between patients with vmPFC lesions and controls. However, a reliable bootstrapping effect was not present in the patient group alone. Post hoc analysis of individual patient performance revealed that the bootstrapping effect did not differ from controls in nine patients but was reduced in two patients. Although mixed, these results suggest that vmPFC lesions do not uniformly disrupt the ability to leverage stored knowledge in support of short-term memory.

Schemas provide a scaffold for neocortical integration of new memories over time

Memory transformation is increasingly acknowledged in theoretical accounts of systems consolidation, yet how memory quality and neural representation change over time and how schemas influence this process remains unclear. We examined the behavioral quality and neural representation of schema-congruent and incongruent object-scene pairs retrieved across 10-minutes and 72-hours using fMRI. When a congruent schema was available, memory became coarser over time, aided by post-encoding coupling between the anterior hippocampus and medial prefrontal cortex (mPFC). Only schema-congruent representations were integrated in the mPFC over time, and were organized according to schematic context. In the hippocampus, pattern similarity changed across 72-hours such that the posterior hippocampus represented specific details and the anterior hippocampus represented the general context of specific memories, irrespective of congruency. Our findings suggest schemas are used as a scaffold to facilitate neocortical integration of congruent information, and illustrate evolution in hippocampal organization of detailed contextual memory over time.

Schema representations in distinct brain networks support narrative memory during encoding and retrieval

Schematic prior knowledge can scaffold the construction of event memories during perception and also provide structured cues to guide memory search during retrieval. We measured the activation of story-specific and schematic representations using fMRI while participants were presented with 16 stories and then recalled each of the narratives, and related these activations to memory for specific story details. We predicted that schema representations in medial prefrontal cortex (mPFC) would be correlated with successful recall of story details. In keeping with this prediction, an anterior mPFC region showed a significant correlation between activation of schema representations at encoding and subsequent behavioral recall performance; however, this mPFC region was not implicated in schema representation during retrieval. More generally, our analyses revealed largely distinct brain networks at encoding and retrieval in which schema activation was related to successful recall. These results provide new insight into when and where event knowledge can support narrative memory.

Fractionating the episodic buffer

The episodic buffer is a putative component of working memory proposed to account for several short-term memory functions, including unexpectedly preserved immediate prose recall by amnesic patients. Over the course of time, this component has increasingly become associated with binding functions. Considering recent findings regarding the performance of both memory-impaired and healthy individuals on the range of tasks purported to require the contribution of the episodic buffer, we suggest that it should be fractionated into two functional systems. One is a schematic store instantiated in brain areas responsible for conceptual and schema representations, which is likely to be hippocampus-independent, and preserved in the face of amnesia. In contrast, short-term maintenance of novel associative binding is likely to require the contribution of the hippocampus and may therefore not be functionally dissociable from long-term memory.

The distinct and overlapping brain networks supporting semantic and spatial constructive scene processing

Scene imagery features prominently when we recall autobiographical memories, imagine the future and navigate around in the world. Consequently, in this study we sought to better understand how scene representations are supported by the brain. Processing scenes involves a variety of cognitive processes that in the real world are highly interactive. Here, however, our goal was to separate semantic and spatial constructive scene processes in order to identify the brain areas that were distinct to each process, those they had in common, and the connectivity between regions. To this end, participants searched for either semantic or spatial constructive impossibilities in scenes during functional MRI. We focussed our analyses on only those scenes that were possible, thus removing any error detection that would evoke reactions such as surprise or novelty. Importantly, we also counterbalanced possible scenes across participants, enabling us to examine brain activity and connectivity for the same possible scene images under two different conditions. We found that participants adopted different cognitive strategies, which were reflected in distinct oculomotor behaviour, for each condition. These were in turn associated with increased engagement of lateral temporal and parietal cortices for semantic scene processing, the hippocampus for spatial constructive scene processing, and increased activation of the ventromedial prefrontal cortex (vmPFC) that was common to both. Connectivity analyses showed that the vmPFC switched between semantic and spatial constructive brain networks depending on the task at hand. These findings further highlight the well-known semantic functions of lateral temporal areas, while providing additional support for the previously-asserted contribution of the hippocampus to scene construction, and recent suggestions that the vmPFC may play a key role in orchestrating scene processing.

The contributions of the left hippocampus and bilateral inferior parietal lobule to form-meaning associative learning

Existing studies have identified crucial roles for the hippocampus and a distributed set of cortical regions (e.g., the inferior parietal cortex) in learning novel words. Nevertheless, researchers have not clearly determined how the hippocampus and cortical regions dynamically interact during novel word learning, especially during form-meaning associative learning. As a method to address this question, we used an online learning paradigm and representational similarity analysis to explore the contributions of the hippocampus and neocortex to form-meaning associative learning. Twenty-nine native Chinese college students were recruited to learn 30 form-meaning pairs, which were repeated 7 times during fMRI scan. Form-meaning associative learning elicited activations in a wide neural network including regions required for word processing (i.e., the bilateral inferior frontal gyrus and the occipitotemporal cortex), regions required for encoding (i.e., the bilateral parahippocampus and hippocampus), and regions required for cognitive control (i.e., the anterior cingulate cortex and dorsolateral prefrontal cortex). More importantly, our study revealed the differential roles of the left hippocampus and bilateral inferior parietal lobule (IPL) in form-meaning associative learning. Specifically, higher pattern similarity in the bilateral IPL in the early learning phase (repetitions 1 to 3) was related to better learning performance, while higher pattern similarity in the left hippocampus in the late learning phase (repetitions 5 to 7) was associated with better learning performance. These findings indicate that the hippocampus and cortical regions (e.g., the IPL) contribute to form-meaning learning in different stages.

A multisensory perspective on object memory

Traditional studies of memory and object recognition involved objects presented within a single sensory modality (i.e., purely visual or purely auditory objects). However, in naturalistic settings, objects are often evaluated and processed in a multisensory manner. This begets the question of how object representations that combine information from the different senses are created and utilised by memory functions. Here we review research that has demonstrated that a single multisensory exposure can influence memory for both visual and auditory objects. In an old/new object discrimination task, objects that were presented initially with a task-irrelevant stimulus in another sense were better remembered compared to stimuli presented alone, most notably when the two stimuli were semantically congruent. The brain discriminates between these two types of object representations within the first 100ms post-stimulus onset, indicating early "tagging" of objects/events by the brain based on the nature of their initial presentation context. Interestingly, the specific brain networks supporting the improved object recognition vary based on a variety of factors, including the effectiveness of the initial multisensory presentation and the sense that is task-relevant. We specify the requisite conditions for multisensory contexts to improve object discrimination following single exposures, and the individual differences that exist with respect to these improvements. Our results shed light onto how memory operates on the multisensory nature of object representations as well as how the brain stores and retrieves memories of objects.

Neurobiology of Schemas and Schema-Mediated Memory

Schemas are superordinate knowledge structures that reflect abstracted commonalities across multiple experiences, exerting powerful influences over how events are perceived, interpreted, and remembered. Activated schema templates modulate early perceptual processing, as they get populated with specific informational instances (schema instantiation). Instantiated schemas, in turn, can enhance or distort mnemonic processing from the outset (at encoding), impact offline memory transformation and accelerate neocortical integration. Recent studies demonstrate distinctive neurobiological processes underlying schema-related learning. Interactions between the ventromedial prefrontal cortex (vmPFC), hippocampus, angular gyrus (AG), and unimodal associative cortices support context-relevant schema instantiation and schema mnemonic effects. The vmPFC and hippocampus may compete (as suggested by some models) or synchronize (as suggested by others) to optimize schema-related learning depending on the specific operationalization of schema memory. This highlights the need for more precise definitions of memory schemas.

Semantic Congruency Improves Recognition Memory Performance for Both Audiovisual and Visual Stimuli

Audiovisual semantic congruency during memory encoding has been shown to facilitate later recognition memory performance. However, it is still unclear whether this improvement is due to multisensory semantic congruency or just semantic congruencyper se. We investigated whether dual visual encoding facilitates recognition memory in the same way as audiovisual encoding. The participants memorized auditory or visual stimuli paired with a semantically congruent, incongruent or non-semantic stimulus in the same modality or in the other modality during encoding. Subsequent recognition memory performance was better when the stimulus was initially paired with a semantically congruent stimulus than when it was paired with a non-semantic stimulus. This congruency effect was observed with both audiovisual and dual visual stimuli. The present results indicate that not only multisensory but also unisensory semantically congruent stimuli can improve memory performance. Thus, the semantic congruency effect is not solely a multisensory phenomenon, as has been suggested previously.