A cow on the prairie vs. a cow on the street: long-term consequences of semantic conflict on episodic encoding

Schema-driven prediction effects on episodic memory across the lifespan

The predictive processing framework posits that one of the main functions of the brain is to anticipate the incoming information. Internal models facilitate interactions with the world by predicting future states against which actual evidence is compared. The difference between predicted and actual states, the prediction error (PE), signals novel information. However, how PE affects cognitive processing downstream is not fully understood: one such aspect pertains to how PE influences episodic memories, and whether those effect on memory differ across the lifespan. We examine the relationship between PE and episodic memory in children, young and older adults. We use a novel paradigm whereby rich visual narratives are used to build action schemas that enable probing different mnemonic aspects. To create different levels of PE, we manipulate the story endings to be either expected, neutral or unexpected with respect to the unfolded action. We show that (i) expected endings are better encoded than neutral endings and (ii) unexpected endings improve the encoding of mismatching events and other aspects of the narrative. These effects are differentially modulated across the lifespan with PE-driven encoding being more prominent in children and young adults and with schema integration playing a larger role on memory encoding in older adults. These results highlight the role of predictions by enriching past experiences and informing future anticipations.This article is part of the theme issue 'Elements of episodic memory: lessons from 40 years of research'.

From silos to synergy: Integrating approaches to investigate the role of prior knowledge and expectations on episodic memory

Significant progress in the investigation of how prior knowledge influences episodic memory has been made using three sometimes isolated (but not mutually exclusive) approaches: strictly adult behavioral investigations, computational models, and investigations into the development of the system. Here we point out that these approaches are complementary, each approach informs and is informed by the other. Thus, a natural next step for research is to combine all three approaches to further our understanding of the role of prior knowledge in episodic memory. Here we use studies of memory for expectation-congruent and incongruent information from each of these often disparate approaches to illustrate how combining approaches can be used to test and revise theories from the other. This domain is particularly advantageous because it highlights important features of more general memory processes, further differentiates models of memory, and can shed light on developmental change in the memory system. We then present a case study to illustrate the progress that can be made from integrating all three approaches and highlight the need for more endeavors in this vein. As a first step, we also propose a new computational model of memory that takes into account behavioral and developmental factors that can influence prior knowledge and episodic memory interactions. This integrated approach has great potential for offering novel insights into the relationship between prior knowledge and episodic memory, and cognition more broadly.

Are older adults susceptible to visual distraction when targets and distractors are spatially separated?

Older adults show preserved memory for previously distracting information due to reduced inhibitory control. In some previous studies, targets and distractors overlap both temporally and spatially. We investigated whether age differences in attentional orienting and disengagement affect recognition memory when targets and distractors are spatially separated at encoding. In Experiments 1 and 2, eye movements were recorded while participants completed an incidental encoding task under covert (i.e., restricted viewing) and overt (i.e., free-viewing) conditions, respectively. The encoding task consisted of pairs of target and distractor item-color stimuli presented in separate visual hemifields. Prior to stimulus onset, a central cue indicated the location of the upcoming target. Participants were subsequently tested on their recognition of the items, their location, and the associated color. In Experiment 3, targets were validly cued on 75% of the encoding trials; on invalid trials, participants had to disengage their attention from the distractor and reorient to the target. Associative memory for colors was reduced among older adults across all experiments, though their location memory was only reduced in Experiment 1. In Experiment 2, older and younger adults directed a similar proportion of fixations toward targets and distractors. Explicit recognition of distractors did not differ between age groups in any of the experiments. However, older adults were slower to correctly recognize distractors than false alarm to novel items in Experiment 2, suggesting some implicit memory for distraction. Together, these results demonstrate that older adults may only be vulnerable to encoding visual distraction when viewing behavior is unconstrained.

Attentional attenuation (rather than attentional boost) through task switching leads to a selective long-term memory decline

Allocating attention determines what we remember later. Attentional demands vary in a task-switching paradigm, with greater demands for switch than for repeat trials. This also results in lower subsequent memory performance for switch compared to repeat trials. The main goal of the present study was to investigate the consequences of task switching after a long study-test interval and to examine the contributions of the two memory components, recollection and familiarity. In the study phase, the participants performed a task-switching procedure in which they had to switch between two classifications tasks with pictures. After a short vs. a long study-test interval of a week, the participants performed a surprise memory test for the pictures and gave remember/know judgements. The results showed that recognition memory declined after 1 week and this was mainly due to a decrease in "remember" responses. The results also showed that the task-switching effect on memory was enduring. Whereas the results of the immediate test were mixed, the results of the delayed tests showed that the task-switching effect was based on recollection, expressed in more "remember" responses for repeat than for switch trials. As recollection is more sensitive to attention manipulations than familiarity, the results align with the notion that attentional requirements at study determine what we remember, in particular after a long study-test interval.

Episodic memory processes modulate how schema knowledge is used in spatial memory decisions

Schema knowledge can dramatically affect how we encode and retrieve memories. Current models propose that schema information is combined with episodic memory at retrieval to influence memory decisions, but it is not known how the strength or type of episodic memory (i.e., unconscious memory versus familiarity versus recollection) influences the extent to which schema information is incorporated into memory decisions. To address this question, we had participants search for target objects in semantically expected (i.e., congruent) locations or in unusual (i.e., incongruent) locations within scenes. In a subsequent test, participants indicated where in each scene the target had been located previously, then provided confidence-based recognition memory judgments that indexed recollection, familiarity strength, and unconscious memory for the scenes. In both an initial online study (n = 133) and replication (n = 59), target location recall was more accurate for targets that had been located in schema-congruent rather than incongruent locations; importantly, this effect was strongest for new scenes, decreased with unconscious memory, decreased further with familiarity strength, and was eliminated entirely for recollected scenes. Moreover, when participants recollected an incongruent scene but did not correctly remember the target location, they were still biased away from congruent regions-suggesting that detrimental schema bias was suppressed in the presence of recollection even when precise target location information was not remembered. The results indicate that episodic memory modulates how schemas are used: Schema knowledge contributes to spatial memory judgments primarily when episodic memory fails to provide precise information, and recollection can override schema bias completely.

Perception of semantic relations in scenes: A registered report study of attention hold

To what extent the semantic relations present in scenes guide spatial attention automatically remains a matter of debate. Considering that spatial attention can be understood as a sequence of shifts, engagements, and disengagements, semantic relations might affect each stage of this process differently. Therefore, we investigated whether objects that violate semantic rules engage attention for longer than objects that are expected in a given context. The experiment involved a central presentation of a distractor scene that contained a semantically congruent or incongruent object, and a peripheral presentation of a small target letter. We found that incongruent scenes did not delay responses to the peripheral target, which indicates that they did not hold attention for longer than congruent scenes. Therefore, by showing that violations of semantic relations do not engage attention automatically, our study contributes to a better understanding of how attention operates in naturalistic settings.

Response-category conflict improves target memory in a flanker paradigm

Previous studies have shown that cognitive control demands and long-term memory interact in several ways. For example, trial-unique Stroop entities which consist of two perceptually distinct stimulus dimensions can enhance subsequent memory. In the present study, we investigated whether this effect generalises to a flanker paradigm. In the study phase, 60 participants had to classify target pictures which were flanked by pictures that were either congruent or incongruent to the target with regard to the response categories, thus manipulating response-category conflict. Then we assessed recognition memory. The results showed that the response-category conflict enhanced subsequent memory for incongruent targets, implying an up-regulation of top-down control that fostered memory encoding. The results demonstrate that the beneficial memory effect of a response-category conflict generalises to a flanker task.

Proportion of conflict, contingency learning, and recency effects in a Stroop task

Recent research on the relation between learning and cognitive control has assumed that conflict modulates learning, either by increasing arousal and hence improving learning in high-conflict situations, or by inducing control, and hence inhibiting the processing of distracters and their eventual association with the imperative responses. We analyse whether the amount of conflict, manipulated through the proportion of congruency in a set of Stroop inducer trials, affects learning of contingencies established on diagnostic trials composed by neutral words associated with colour responses. The results reproduced the list-wide proportion of congruency effect on the inducer trials, and showed evidence of contingency learning on the diagnostic trials, but provided no indication that this learning was modulated by the level of conflict. Specific analyses conducted to control for the impact of episodic effects on the expression of learning indicated that contingency effects were not driven by the incremental processes that could be expected by associative learning, but rather they were due to the impact of the most recent trial involving the same distracter. Accordingly, these effects disappeared when tested selectively on trials that required a non-matching response with respect to the previous occurrence of the distracter. We interpret this result in the context of the debate on how learning and memory interact with the processes of cognitive control.

Crossmodal Semantic Congruence Interacts with Object Contextual Consistency in Complex Visual Scenes to Enhance Short-Term Memory Performance

Object sounds can enhance the attentional selection and perceptual processing of semantically-related visual stimuli. However, it is currently unknown whether crossmodal semantic congruence also affects the post-perceptual stages of information processing, such as short-term memory (STM), and whether this effect is modulated by the object consistency with the background visual scene. In two experiments, participants viewed everyday visual scenes for 500 ms while listening to an object sound, which could either be semantically related to the object that served as the STM target at retrieval or not. This defined crossmodal semantically cued vs. uncued targets. The target was either in- or out-of-context with respect to the background visual scene. After a maintenance period of 2000 ms, the target was presented in isolation against a neutral background, in either the same or different spatial position as in the original scene. The participants judged the same vs. different position of the object and then provided a confidence judgment concerning the certainty of their response. The results revealed greater accuracy when judging the spatial position of targets paired with a semantically congruent object sound at encoding. This crossmodal facilitatory effect was modulated by whether the target object was in- or out-of-context with respect to the background scene, with out-of-context targets reducing the facilitatory effect of object sounds. Overall, these findings suggest that the presence of the object sound at encoding facilitated the selection and processing of the semantically related visual stimuli, but this effect depends on the semantic configuration of the visual scene.

Increasing control improves further control, but it does not enhance memory for the targets in a face-word Stroop task

Recent research on the dynamics between attentional and memory processes have outlined the idea that applying control in a conflicting situation directly leads to enhanced episodic memory of the processed information. However, in spite of a small subset of studies supporting this claim, the majority of the evidence in the field seems to support the opposite pattern. In this study, we used a face-word Stroop task to enforce different control modes either from trial to trial or in an item-specific manner. Both manipulations of congruency proved to be effective in making participants' responses to conflicting stimuli more efficient over time by applying a trial-specific control mode. However, these manipulations had no impact on memory performance on a surprise recognition memory test. To our knowledge, this is the first attempt at measuring the memory consequences of the application of specific control modes at the trial level. The results reported here call for caution and possibly reconceptualization of the relationship between cognitive control and memory.

Automatic Attention Capture by Threatening, But Not by Semantically Incongruent Natural Scene Images

Visual objects are typically perceived as parts of an entire visual scene, and the scene’s context provides information crucial in the object recognition process. Fundamental insights into the mechanisms of context-object integration have come from research on semantically incongruent objects, which are defined as objects with a very low probability of occurring in a given context. However, the role of attention in processing of the context-object mismatch remains unclear, with some studies providing evidence in favor, but other against an automatic capture of attention by incongruent objects. Therefore, in the present study, 25 subjects completed a dot-probe task, in which pairs of scenes—congruent and incongruent or neutral and threatening—were presented as task-irrelevant distractors. Importantly, threatening scenes are known to robustly capture attention and thus were included in the present study to provide a context for interpretation of results regarding incongruent scenes. Using N2 posterior-contralateral ERP component as a primary measure, we revealed that threatening images indeed capture attention automatically and rapidly, but semantically incongruent scenes do not benefit from an automatic attentional selection. Thus, our results suggest that identification of the context-object mismatch is not preattentive.

Memory effects of conflict and cognitive control are processing stage-specific: evidence from pupillometry

An increasing number of studies in the conflict/control and perceptual desirable difficulty literatures show memory benefits for information in high-conflict task situations. Recent work suggests that increased conflict does not produce a task-wide encoding benefit; rather, conflict must focus high-level attention on to-be-tested information to produce an encoding benefit. We used pupil dilation measures to directly assess this stage-specific model of conflict-encoding effects. We show clear performance costs of incongruency (slower RT and larger pupil dilation) with both semantic and response distractors, but show memory benefits only with semantic conflict. Further, when participants were encouraged to focus more (eliciting greater endogenous effort and control for all trials, not just incongruent trials), we observe larger and more similar pupil responses and reduced memory differences between high versus low semantic conflict conditions. These data confirm and extend a stage-specific model of conflict-encoding effects, with converging behavioural and physiological data.

Different impact of task switching and response-category conflict on subsequent memory

The impact of cognitive control demands on long-term memory is mixed, with some conflicts leading to better, others leading to worse subsequent memory. The current study was designed to investigate how different types of cognitive control demands modulate the effects on memory. At study, participants had to switch between two classification tasks and later, free recall performance was assessed. The stimuli consisted of two interleaved words, one word had to be categorized and the other word had to be ignored. In four experiments, the congruency between target and ignored words was manipulated by changing the distractor category. This allowed us to investigate the impact of different types of conflict (i.e., task switching, perceptual load, response-category conflict, stimulus-category conflict). The results revealed that task switching impaired memory in all experiments. In Experiment 1, higher perceptual load also impaired memory. Experiments 2–4 showed that the co-activation of two words which required different responses (i.e., response-category conflict) enhanced memory performance but only when the conflict stimuli were presented in pure blocks. Overall, memory performance seems to depend on attentional policies. Withdrawing attention from target encoding results in lower memory performance. In contrast, focusing attention on the target results in enhanced memory performance.

Congruency Encoding Effects on Recognition Memory: A Stage-Specific Account of Desirable Difficulty

Recent research suggests that selectively attending to relevant stimuli while having to ignore or resist conflicting stimuli can lead to improvements in learning. While mostly discussed within a broader "desirable difficulty" framework in the memory and education literatures, some recent work has focused on more mechanistic questions of how processing conflict (e.g., from incongruent primes) might elicit increased attention and control, producing enhanced incidental encoding of high-conflict stimuli. This encoding benefit for high-control-demand or high-difficulty situations has been broadly conceptualized as a task-general property, with no strong prediction of what particular task elements should produce this effect. From stage processing models of single- and dual-task performance, we propose that memory-enhancing difficulty manipulations should strongly depend on inducing additional cognitive control at particular processing stages. Over six experiments, we show that a memory benefit is produced when increased cognitive control (via incongruency priming) focuses additional processing on the core meaning of to-be-tested stimuli at the semantic categorization stage. In contrast, incongruency priming targeted at response selection within the same task produces similar effects on initial task performance, but gives no memory benefit for high-conflict trials. We suggest that a simple model of limited-capacity and stage-specific cognitive control allocation can account for and predict where and when conflict/difficulty encoding benefits will occur, and may serve as a model for desirable difficulty effects more broadly.

Task Switching Hurts Memory Encoding

Research consistently shows that task switching slows down performance on switch compared to repeat trials, but the consequences on memory are less clear. In the present study, we investigated the impact of task switching on subsequent memory performance. Participants had to switch between two semantic classification tasks. In Experiment 1, the stimuli were univalent; in Experiment 2, the stimuli were bivalent (relevant for both tasks). The aim was to disentangle the conflicts triggered by task switching and bivalency. In both experiments, recognition memory for switch and repeat stimuli was tested subsequently. During encoding, task switching produced switch costs. Critically, subsequent memory was lower for switch compared to repeat stimuli in both experiments, and this effect was increased in Experiment 2 with bivalent material. We suggest that the requirement to switch tasks hurts the encoding of task-relevant information and thus impairs subsequent memory performance.

Selective attention effects on recognition: the roles of list context and perceptual difficulty

Two recent studies reported superior recognition memory for items that were incongruent targets than for items that were congruent targets in a prior incidental study phase (Krebs et al. in Cereb Cortex (New York, NY) 25(3):833-843, 2015; Rosner et al. in Psychol Res 79(3):411-424, 2015). The present study examined this effect further by addressing two issues. First, we examined whether this effect is sensitive to the list context in which congruent and incongruent items are presented. In Experiment 1, this issue was addressed by manipulating the relative proportions of congruent and incongruent trials in the study phase. In Experiments 2A and 2B, the same issue was examined by contrasting randomly intermixed and blocked manipulations of congruency. The results of these experiments, as well as a trial-to-trial sequence analysis, demonstrate that the recognition advantage for incongruent over congruent items is robust and remarkably insensitive to list context. Second, we examined recognition of incongruent and congruent items relative to a single word baseline condition. Incongruent (Experiment 3A) and congruent (Experiment 3B) items were both better recognized than single word items, though this effect was substantially stronger for incongruent items. These results suggest that perceptual processing difficulty, rather than interference caused by different target and distractor identities on its own, contributes to the enhanced recognition of incongruent items. Together, the results demonstrate that processes that are sensitive to perceptual processing difficulty of items but largely insensitive to list context produce heightened recognition sensitivity for incongruent targets.

Attentional influences on memory formation: A tale of a not-so-simple story

Is there a learning mechanism triggered by mere expectation violation? Is there some form of memory enhancement inherent to an event mismatching our predictions? Across seven experiments, we explore this issue by means of a validity paradigm. Although our manipulation clearly succeeded in generating an expectation and breaking it, the memory consequences of that expectation mismatch are not so obvious. We report here evidence of a null effect of expectation on memory formation. Our results (1) show that enhanced memory for unexpected events is not easily achieved and (2) call for a reevaluation of previous accounts of memory enhancements based on prediction error or difficulty of processing. Limitations of this study and possible implications for the field are discussed in detail.