Novelty Manipulations, Memory Performance, and Predictive Coding: the Role of Unexpectedness

A library for innovative category exemplars (ALICE) database: Streamlining research with printable 3D novel objects

This paper introduces A Library for Innovative Category Exemplars (ALICE) database, a resource that enhances research efficiency in cognitive and developmental studies by providing printable 3D objects representing 30 novel categories. Our research consists of three experiments to validate the novelty and complexity of the objects in ALICE. Experiment 1 assessed the novelty of objects through adult participants' subjective familiarity ratings and agreement on object naming and descriptions. The results confirm the general novelty of the objects. Experiment 2 employed multidimensional scaling (MDS) to analyze perceived similarities between objects, revealing a three-dimensional structure based solely on shape, indicative of their complexity. Experiment 3 used two clustering techniques to categorize objects: k-means clustering for creating nonoverlapping global categories, and hierarchical clustering for allowing global categories that overlap and have a hierarchical structure. Through stability tests, we verified the robustness of each clustering method and observed a moderate to good consensus between them, affirming the strength of our dual approach in effectively and accurately delineating meaningful object categories. By offering easy access to customizable novel stimuli, ALICE provides a practical solution to the challenges of creating novel physical objects for experimental purposes.

The effects of task novelty for age cohort and cognition level on memory for everyday virtual meal preparation tasks

Nonsensical information increases task novelty, which makes it difficult to rely on previous learning and provides insight into the learning of new tasks. This study investigated procedural-based action scripts in everyday memory for meal preparation tasks in virtual reality. The sample (N = 171) consisted of 3 groups determined by age and cognitive function: young adults (YA; n = 61), older adults with normal cognition (OA; n = 82), and older adults with impaired cognition (IC; n = 28). The three groups completed the Virtual Kitchen Protocol, a virtual reality-based measure of learning and memory for cooking both familiar and nonsensical meals. Results showed that YAs had a greater recall for both familiar and nonsensical meals than OAs or ICs. Additionally, novel stimuli (i.e., nonsensical meal tasks) appear to impact older adults more than young adults. Among older adults, impaired cognition was associated with lower performance on both the sensical and nonsensical meals compared to normal cognition. All three groups performed better on familiar tasks than nonsensical tasks. These results were consistent with the notion that familiarity may be of greater use than novelty in the context of procedural-based action scripts.

Contemporary neurocognitive models of memory: A descriptive comparative analysis

The great complexity involved in the study of memory has given rise to numerous hypotheses and models associated with various phenomena at different levels of analysis. This has allowed us to delve deeper in our knowledge about memory but has also made it difficult to synthesize and integrate data from different lines of research. In this context, this work presents a descriptive comparative analysis of contemporary models that address the structure and function of multiple memory systems. The main goal is to outline a panoramic view of the key elements that constitute these models in order to visualize both the current state of research and possible future directions. The elements that stand out from different levels of analysis are distributed neural networks, hierarchical organization, predictive coding, homeostasis, and evolutionary perspective.

Contextual novelty detection and novelty-related memory enhancement in amnestic mild cognitive impairment

INTRODUCTION

Though novelty processing plays a critical role in memory function, little is known about how it influences learning in memory-impaired populations, such as amnestic Mild Cognitive Impairment (aMCI).

METHODS

21 aMCI patients and 22 age- and education-matched healthy older participants performed two tasks-(i) an oddball paradigm where fractals that were often repeated (60 % of the stimuli), less frequently repeated (20 %), or novel (presented once each) were shown to assess novelty preference (longer viewing time for novel than familiar stimuli), and (ii) a Von Restorff paradigm assessing novelty-related effects on memory. Participants studied 22 lists of 10 words. Among these lists, 18 contained an isolated word different from the others by its distinctive aspect, here the font size (90-point, 120-point or 150-point against 60-point for non-isolated words). The remaining four were control lists without isolated words. After studying each list, participants freely recalled the maximum words possible.

RESULTS

For the oddball task, a group-by-stimulus type ANOVA on median viewing times revealed a significant effect of stimulus type, but not of group. Both groups spent more time on novel stimuli. For the Von Restorff task, both aMCI and healthy controls recalled the isolated words (presented in 120-point or 150-point, but not 90-point) better than others (excluding primacy and recency effects). Novelty-related memory benefit-gain factor-was computed as the difference between the recall scores for isolated and other words. A group-by-font size ANOVA on gain factors revealed no group effect, nor interaction, suggesting that aMCI patients benefited from novelty, alike controls.

CONCLUSION

Novelty preference and the boosting effect of isolation-related novelty on subsequent recall seem preserved despite impaired episodic memory in aMCI patients. This is discussed in the light of contemporary divergent theories regarding the relationship between novelty and memory, as either being independent or parts of a continuum.

Prediction error and event segmentation in episodic memory

Organizing the continuous flow of experiences into meaningful events is a crucial prerequisite for episodic memory. Prediction error and event segmentation both play important roles in supporting the genesis of meaningful mnemonic representations of events. We review theoretical contributions discussing the relationship between prediction error and event segmentation, as well as literature on episodic memory related to prediction error and event segmentation. We discuss the extent of overlap of mechanisms underlying memory emergence through prediction error and event segmentation, with a specific focus on attention and working memory. Finally, we identify areas in research that are currently developing and suggest future directions. We provide an overview of mechanisms underlying memory formation through predictions, violations of predictions, and event segmentation.

Long-term, multi-event surprise correlates with enhanced autobiographical memory

Neurobiological and psychological models of learning emphasize the importance of prediction errors (surprises) for memory formation. This relationship has been shown for individual momentary surprising events; however, it is less clear whether surprise that unfolds across multiple events and timescales is also linked with better memory of those events. We asked basketball fans about their most positive and negative autobiographical memories of individual plays, games and seasons, allowing surprise measurements spanning seconds, hours and months. We used advanced analytics on National Basketball Association play-by-play data and betting odds spanning 17 seasons, more than 22,000 games and more than 5.6 million plays to compute and align the estimated surprise value of each memory. We found that surprising events were associated with better recall of positive memories on the scale of seconds and months and negative memories across all three timescales. Game and season memories could not be explained by surprise at shorter timescales, suggesting that long-term, multi-event surprise correlates with memory. These results expand notions of surprise in models of learning and reinforce its relevance in real-world domains.

Brainstem fMRI signaling of surprise across different types of deviant stimuli

Detection of deviant stimuli is crucial to orient and adapt our behavior. Previous work shows that deviant stimuli elicit phasic activation of the locus coeruleus (LC), which releases noradrenaline and controls central arousal. However, it is unclear whether the detection of behaviorally relevant deviant stimuli selectively triggers LC responses or other neuromodulatory systems (dopamine, serotonin, and acetylcholine). We combine human functional MRI (fMRI) recordings optimized for brainstem imaging with pupillometry to perform a mapping of deviant-related responses in subcortical structures. Participants have to detect deviant items in a "local-global" paradigm that distinguishes between deviance based on the stimulus probability and the sequence structure. fMRI responses to deviant stimuli are distributed in many cortical areas. Both types of deviance elicit responses in the pupil, LC, and other neuromodulatory systems. Our results reveal that the detection of task-relevant deviant items recruits the same multiple subcortical systems across computationally different types of deviance.

The graded novelty encoding task: Novelty gradually improves recognition of visual stimuli under incidental learning conditions

It has been argued that novel compared to familiar stimuli are preferentially encoded into memory. Nevertheless, treating novelty as a categorical variable in experimental research is considered simplistic. We highlight the dimensional aspect of novelty and propose an experimental design that manipulates novelty continuously. We created the Graded Novelty Encoding Task (GNET), in which the difference between stimuli (i.e. novelty) is parametrically manipulated, paving the way for quantitative models of novelty processing. We designed an algorithm which generates visual stimuli by placing colored shapes in a grid. During the familiarization phase of the task, we repeatedly presented five pictures to the participants. In a subsequent incidental learning phase, participants were asked to differentiate between the "familiars" and novel images that varied in the degree of difference to the familiarized pictures (i.e. novelty). Finally, participants completed a surprise recognition memory test, where the novel stimuli from the previous phase were interspersed with distractors with similar difference characteristics. We numerically expressed the differences between the stimuli to compute a dimensional indicator of novelty and assessed whether it predicted recognition memory performance. Based on previous studies showing the beneficial effect of novelty on memory formation, we hypothesized that the more novel a given picture was, the better subsequent recognition performance participants would demonstrate. Our hypothesis was confirmed: recognition performance was higher for more novel stimuli. The GNET captures the continuous nature of novelty, and it may be useful in future studies that examine the behavioral and neurocognitive aspects of novelty processing.

A generic neural factor linking resting-state neural dynamics and the brain's response to unexpectedness in multilevel cognition

The brain's response to change is fundamental to learning and adaptation; this implies the presence of a universal neural mechanism under various contexts. We hypothesized that this mechanism manifests in neural activity patterns across low and high levels of cognition during task processing as well as in resting-state neural dynamics, because both these elements are different facets of the same dynamical system. We tested our hypothesis by (i) characterizing (a) the neural response to changes in low-level continuous information stream and unexpectedness at different cognitive levels and (b) the spontaneous neural dynamics in resting state, and (ii) examining the associations among the dynamics according to cross-individual variability (n = 200). Our results showed that the brain's response magnitude was monotonically correlated with the magnitude of information fluctuation in a low-level task, forming a simple psychophysical function; moreover, this effect was found to be associated with the brain's response to unexpectedness in high-level cognitive tasks (including language processing). These coherent multilevel neural effects in task processing were also shown to be strongly associated with resting-state neural dynamics characterized by the waxing and waning of Alpha oscillation. Taken together, our results revealed large-scale consistency between the neural dynamic system and multilevel cognition.

Solidity Meets Surprise: Cerebral and Behavioral Effects of Learning from Episodic Prediction Errors

How susceptible a memory is to later modification might depend on how stable the episode has been encoded. This stability was proposed to increase when retrieving information more (vs. less) often and in a spaced (vs. massed) practice. Using fMRI, we examined the effects of these different pre-fMRI retrieval protocols on the subsequent propensity to learn from episodic prediction errors. After encoding a set of different action stories, participants came back for two pre-fMRI retrieval sessions in which they encountered original episodes either 2 or 8 times in either a spaced or a massed retrieval protocol. One week later, we cued episodic retrieval during the fMRI session by using original or modified videos of encoded action stories. Recurrent experience of modified episodes was associated with increasing activity in the episodic memory network including hippocampal and cortical areas, when leading to false memories in a post-fMRI memory test. While this observation clearly demonstrated learning from episodic prediction errors, we found no evidence for a modulatory effect of the different retrieval protocols. As expected, the benefit of retrieving an episode more often was reflected in better memory for originally encoded episodes. In addition, frontal activity increased for episodic prediction errors when episodes had been less frequently retrieved pre-fMRI. A history of spaced versus massed retrieval was associated with increased activation throughout the episodic memory network, with no significant effect on behavioral performance. Our findings show that episodic prediction errors led to false memories. The history of different retrieval protocols was reflected in memory performance and brain responses to episodic prediction errors, but did not interact with the brain's episodic learning response.

Quantifying the narrative flow of imagined versus autobiographical stories

Lifelong experiences and learned knowledge lead to shared expectations about how common situations tend to unfold. Such knowledge of narrative event flow enables people to weave together a story. However, comparable computational tools to evaluate the flow of events in narratives are limited. We quantify the differences between autobiographical and imagined stories by introducing sequentiality, a measure of narrative flow of events, drawing probabilistic inferences from a cutting-edge large language model (GPT-3). Sequentiality captures the flow of a narrative by comparing the probability of a sentence with and without its preceding story context. We applied our measure to study thousands of diary-like stories, collected from crowdworkers, about either a recent remembered experience or an imagined story on the same topic. The results show that imagined stories have higher sequentiality than autobiographical stories and that the sequentiality of autobiographical stories increases when the memories are retold several months later. In pursuit of deeper understandings of how sequentiality measures the flow of narratives, we explore proportions of major and minor events in story sentences, as annotated by crowdworkers. We find that lower sequentiality is associated with higher proportions of major events. The methods and results highlight opportunities to use cutting-edge computational analyses, such as sequentiality, on large corpora of matched imagined and autobiographical stories to investigate the influences of memory and reasoning on language generation processes.

What Happened When? Cerebral Processing of Modified Structure and Content in Episodic Cueing

Episodic memories are not static but can change on the basis of new experiences, potentially allowing us to make valid predictions in the face of an ever-changing environment. Recent research has identified prediction errors during memory retrieval as a possible trigger for such changes. In this study, we used modified episodic cues to investigate whether different types of mnemonic prediction errors modulate brain activity and subsequent memory performance. Participants encoded episodes that consisted of short toy stories. During a subsequent fMRI session, participants were presented videos showing the original episodes, or slightly modified versions thereof. In modified videos, either the order of two subsequent action steps was changed or an object was exchanged for another. Content modifications recruited parietal, temporo-occipital, and parahippocampal areas reflecting the processing of the new object information. In contrast, structure modifications elicited activation in right dorsal premotor, posterior temporal, and parietal areas, reflecting the processing of new sequence information. In a post-fMRI memory test, the participants' tendency to accept modified episodes as originally encoded increased significantly when they had been presented modified versions already during the fMRI session. After experiencing modifications, especially those of the episodes' structure, the recognition of originally encoded episodes was impaired as well. Our study sheds light onto the neural processing of different types of episodic prediction errors and their influence on subsequent memory recall.

Expectation of irrelevant novel stimuli has no consistent effect on recognition memory

Novelty is defined as the part of an experience that is not yet represented by memory systems. Novelty has been claimed to exert various memory-enhancing effects. A pioneering study by Wittmann et al. (2007) has shown that memory formation may even benefit from the expectation of novelty. We aimed to replicate this assumed memory effect in four behavioral studies. However, our results do not support the idea that anticipated novel stimuli are more memorable than unexpected novelty. In our experiments, we systematically manipulated the novelty predicting cues to ensure that the expectations were correctly formed by the participants, however, the results showed that there was no memory enhancement for expected novel pictures in any of the examined indices, thus we could not replicate the main behavioral finding of Wittmann et al. (2007). These results call into question the original effect, and we argue that this fits more into current thinking on memory formation and brain function in general. Our results are more consistent with the view that unexpected stimuli are more likely to be retained by memory systems. Predictive coding theory suggests that unexpected stimuli are prioritized by the nervous system and this may also benefit memory processes. Novel stimuli may be unexpected and thus recognized better in some experimental setups, yet novelty and unexpectedness do not always coincide. We hope that our work can bring more consistency in the literature on novelty, as educational methods in general could also benefit from this clarification.

Mnemonic prediction errors promote detailed memories

When our experience violates our predictions, it is adaptive to update our knowledge to promote a more accurate representation of the world and facilitate future predictions. Theoretical models propose that these mnemonic prediction errors should be encoded into a distinct memory trace to prevent interference with previous, conflicting memories. We investigated this proposal by repeatedly exposing participants to pairs of sequentially presented objects (A → B), thus evoking expectations. Then, we violated participants' expectations by replacing the second object in the pairs with a novel object (A → C). The following item memory test required participants to discriminate between identical old items and similar lures, thus testing detailed and distinctive item memory representations. In two experiments, mnemonic prediction errors enhanced item memory: Participants correctly identified more old items as old when those items violated expectations during learning, compared with items that did not violate expectations. This memory enhancement for C items was only observed when participants later showed intact memory for the related A → B pairs, suggesting that strong predictions are required to facilitate memory for violations. Following up on this, a third experiment reduced prediction strength prior to violation and subsequently eliminated the memory advantage of violations. Interestingly, mnemonic prediction errors did not increase gist-based mistakes of identifying old items as similar lures or identifying similar lures as old. Enhanced item memory in the absence of gist-based mistakes suggests that violations enhanced memory for items' details, which could be mediated via distinct memory traces. Together, these results advance our knowledge of how mnemonic prediction errors promote memory formation.

A Review of Possible EEG Markers of Abstraction, Attentiveness, and Memorisation in Cyber-Physical Systems for Special Education

Cyber-physical systems (CPSs) for special education rely on effective mental and brain processing during the lesson, performed with the assistance of humanoid robots. The improved diagnostic ability of the CPS is a prerogative of the system for efficient technological support of the pedagogical process. The article focuses on the available knowledge of possible EEG markers of abstraction, attentiveness, and memorisation (in some cases combined with eye tracking) related to predicting effective mental and brain processing during the lesson. The role of processing abstraction is emphasised as the learning mechanism, which is given priority over the other mechanisms by the cognitive system. The main markers in focus are P1, N170, Novelty P3, RewP, N400, and P600. The description of the effects is accompanied by the analysis of some implications for the design of novel educational scenarios in inclusive classes.

Expectation-driven novelty effects in episodic memory

Novel and unexpected stimuli are often prioritised in memory, given their inherent salience. Nevertheless, not all forms of novelty show such an enhancement effect. Here, we discuss the role expectation plays in modulating the way novelty affects memory processes, circuits, and subsequent performance. We first review independent effects of expectation on memory, and then consider how different types of novelty are characterised by expectation. We argue that different types of novelty defined by expectation implicate differential neurotransmission in memory formation brain regions and may also result in the creation of different types of memory. Contextual novelty, which is unexpected by definition, is often associated with better recollection, supported by dopaminergic-hippocampal interactions. On the other hand, expected stimulus novelty is supported by engagement of medial temporal cortices, as well as the hippocampus, through cholinergic modulation. Furthermore, when expected stimulus novelty results in enhanced memory, it is predominantly driven by familiarity. The literature reviewed here highlights the complexity of novelty-sensitive memory systems, the distinction between types of novelty, and how they are differentially affected by expectancy.

A predictive account of how novelty influences declarative memory

A rich body of studies in the human and non-human literature has examined the question how novelty influences memory. For a variety of different stimuli, ranging from simple objects and words to vastly complex scenarios, the literature reports that novelty improves memory in some cases, but impairs memory in other cases. In recent attempts to reconcile these conflicting findings, novelty has been divided into different subtypes, such as relative versus absolute novelty, or stimulus versus contextual novelty. Nevertheless, a single overarching theory of novelty and memory has been difficult to attain, probably due to the complexities in the interactions among stimuli, environmental factors (e.g., spatial and temporal context) and level of prior knowledge (but see Duszkiewicz et al., 2019; Kafkas & Montaldi, 2018b; Schomaker & Meeter, 2015). Here we describe how a predictive coding framework might be able to shed new light on different types of novelty and how they affect declarative memory in humans. More precisely, we consider how prior expectations modulate the influence of novelty on encoding episodes into memory, e.g., in terms of surprise, and how novelty/surprise affect memory for surrounding information. By reviewing a range of behavioural findings and their possible underlying neurobiological mechanisms, we highlight where a predictive coding framework succeeds and where it appears to struggle.

Encoding-linked pupil response is modulated by expected and unexpected novelty: Implications for memory formation and neurotransmission

Whether a novel stimulus is expected or unexpected may have implications for the kind of ensuing encoding and the type of subsequent memory. Pupil response was used in the present study to explore the way expected and unexpected stimuli are encoded and whether encoding-linked pupil response is modulated by expectation. Participants first established a contingency relationship between a series of symbols and the type of stimulus (man-made or natural) that followed each one. At encoding, some of the target stimuli violated the previously established relationship (i.e., unexpected), while the majority conformed to this relationship (i.e., expected). Expectation at encoding had opposite effects on familiarity and recollection, the two types of memory that support recognition, and modulated differently the way pupil response predicted subsequent memory. Encoding of unexpected novel stimuli was associated with increased pupil dilation as a predictor of subsequent memory type and strength. In contrast, encoding of expected novel stimuli was associated with decreased pupil response (constriction), which was predictive of subsequent memory type and strength. The findings support the close link between pupil response and memory formation, but critically indicate that this is modulated by the type of novelty as defined by expectation. These novel findings have important implications for the encoding mechanisms involved when different types of novelty are detected and is proposed to indicate the operation of different neurotransmitters during memory formation.