Novelty or surprise?

Surprising sounds influence risky decision making

Adaptive behavior depends on appropriate responses to environmental uncertainty. Incidental sensory events might simply be distracting and increase errors, but alternatively can lead to stereotyped responses despite their irrelevance. To evaluate these possibilities, we test whether task-irrelevant sensory prediction errors influence risky decision making in humans across seven experiments (total n = 1600). Rare auditory sequences preceding option presentation systematically increase risk taking and decrease choice perseveration (i.e., increased tendency to switch away from previously chosen options). The risk-taking and perseveration effects are dissociable by manipulating auditory statistics: when rare sequences end on standard tones, including when rare sequences consist only of standard tones, participants are less likely to perseverate after rare sequences but not more likely to take risks. Computational modeling reveals that these effects cannot be explained by increased decision noise but can be explained by value-independent risky bias and perseveration parameters, decision biases previously linked to dopamine. Control experiments demonstrate that both surprise effects can be eliminated when tone sequences are presented in a balanced or fully predictable manner, and that surprise effects cannot be explained by erroneous beliefs. These findings suggest that incidental sounds may influence many of the decisions we make in daily life.

Dynamic adaptation to novelty in the brain is related to arousal and intelligence

How does the human brain respond to novelty? Here, we address this question using fMRI data wherein human participants watch the same movie scene four times. On the first viewing, this movie scene is novel, and on later viewings it is not. We find that brain activity is lower-dimensional in response to novelty. At a finer scale, we find that this reduction in the dimensionality of brain activity is the result of increased coupling in specific brain systems, most specifically within and between the control and dorsal attention systems. Additionally, we found that novelty induced an increase in between-subject synchronization of brain activity in the same brain systems. We also find evidence that adaptation to novelty, herein operationalized as the difference between baseline coupling and novelty-response coupling, is related to fluid intelligence. Finally, using separately collected out-of-sample data, we find that the above results may be linked to psychological arousal.

The effects of task novelty for age cohort and cognition level on memory for everyday virtual meal preparation tasks

Nonsensical information increases task novelty, which makes it difficult to rely on previous learning and provides insight into the learning of new tasks. This study investigated procedural-based action scripts in everyday memory for meal preparation tasks in virtual reality. The sample (N = 171) consisted of 3 groups determined by age and cognitive function: young adults (YA; n = 61), older adults with normal cognition (OA; n = 82), and older adults with impaired cognition (IC; n = 28). The three groups completed the Virtual Kitchen Protocol, a virtual reality-based measure of learning and memory for cooking both familiar and nonsensical meals. Results showed that YAs had a greater recall for both familiar and nonsensical meals than OAs or ICs. Additionally, novel stimuli (i.e., nonsensical meal tasks) appear to impact older adults more than young adults. Among older adults, impaired cognition was associated with lower performance on both the sensical and nonsensical meals compared to normal cognition. All three groups performed better on familiar tasks than nonsensical tasks. These results were consistent with the notion that familiarity may be of greater use than novelty in the context of procedural-based action scripts.

'Snakes and ladders' in paleoanthropology: From cognitive surprise to skillfulness a million years ago

A paradigmatic account may suffice to explain behavioral evolution in early Homo. We propose a parsimonious account that (1) could explain a particular, frequently-encountered, archeological outcome of behavior in early Homo - namely, the fashioning of a Paleolithic stone 'handaxe' - from a biological theoretic perspective informed by the free energy principle (FEP); and that (2) regards instances of the outcome as postdictive or retrodictive, circumstantial corroboration. Our proposal considers humankind evolving as a self-organizing biological ecosystem at a geological time-scale. We offer a narrative treatment of this self-organization in terms of the FEP. Specifically, we indicate how 'cognitive surprises' could underwrite an evolving propensity in early Homo to express sporadic unorthodox or anomalous behavior. This co-evolutionary propensity has left us a legacy of Paleolithic artifacts that is reminiscent of a 'snakes and ladders' board game of appearances, disappearances, and reappearances of particular archeological traces of Paleolithic behavior. When detected in the Early and Middle Pleistocene record, anthropologists and archeologists often imagine evidence of unusual or novel behavior in terms of early humankind ascending the rungs of a figurative phylogenetic 'ladder' - as if these corresponded to progressive evolution of cognitive abilities that enabled incremental achievements of increasingly innovative technical prowess, culminating in the cognitive ascendancy of Homo sapiens. The conjecture overlooks a plausible likelihood that behavior by an individual who was atypical among her conspecifics could have been disregarded in a community of Hominina (for definition see Appendix 1) that failed to recognize, imagine, or articulate potential advantages of adopting hitherto unorthodox behavior. Such failure, as well as diverse fortuitous demographic accidents, would cause exceptional personal behavior to be ignored and hence unremembered. It could disappear by a pitfall, down a 'snake', as it were, in the figurative evolutionary board game; thereby causing a discontinuity in the evolution of human behavior that presents like an evolutionary puzzle. The puzzle discomforts some paleoanthropologists trained in the natural and life sciences. They often dismiss it, explaining it away with such self-justifying conjectures as that, maybe, separate paleospecies of Homo differentially possessed different cognitive abilities, which, supposedly, could account for the presence or absence in the Pleistocene archeological record of traces of this or that behavioral outcome or skill. We argue that an alternative perspective - that inherits from the FEP and an individual's 'active inference' about its surroundings and of its own responses - affords a prosaic, deflationary, and parsimonious way to account for appearances, disappearances, and reappearances of particular behavioral outcomes and skills of early humankind.

Which intellectual activities are related to cognitive reserve? Introduction and testing a three-dimensional model

BACKGROUND

A common belief among people and some researchers is that keeping yourself mentally active may decrease the risk of dementia. Over the past years, despite widespread efforts to identify proxies for protecting cognitive reserve against age-related changes, it is still not clear what type of intellectual activity would be beneficial for cognitive reserve. To fill this gap, we propose a three-dimensional model of intellectual activity. According to this conceptual model, intellectual activities could be distinguished based on their locations in a three-dimensions space, including; (1) Activation: active vs. passive, (2) Novelty: novel vs. familiar, and (3) Productivity: productive vs. receptive. We assumed that the activities that are categorized as more active, novel, and productive could be considered as a cognitive reserve proxy.

METHODS

To test this hypothesis, a sample of 237 participants older than 50 years (Mage = 58.76 ± 6.66; 63.7% women) was recruited to take part in the study. Episodic, semantic and working memory were assessed with computerized battery tests (Sepidar) and a self-report questionnaire was used to assess intellectual activities. Activities were categorized in terms of; (1) passive, familiar, and receptive activities (radio/watching TV), (2) active, familiar, and receptive activities (solving crosswords), (3) active, novel, and receptive activities (reading), and (4) active, novel, and productive activities (writing).

RESULTS

The results indicated that writing moderates the effect of age on episodic and semantic memory. Reading only moderates the effect of age on semantic memory, and radio/watching TV and solving crosswords do not play a role in moderation analysis.

CONCLUSIONS

Our finding suggests that intellectual activities have different moderating effects on the relationships between age and memory performance. Individuals with high levels of participation in novel and productive activities over the life course are less likely to clinically demonstrate cognitive impairments. Our results support the potential benefit of the three-dimensional model to provide a better insight into the complex role of intellectual activities in cognitive reserve, particularly for older adults. Further research is needed to evaluate the efficacy and the benefits of the model.

Breaking down (and moving beyond) novelty as a trigger of curiosity

The Novelty Seeking Model (NSM) places "novelty" at center stage in characterizing the mechanisms behind curiosity. We argue that the NSM's conception of novelty is too broad, obscuring distinct constructs. More critically, the NSM underemphasizes triggers of curiosity that better unify these constructs and that have received stronger empirical support: those that signal the potential for useful learning.

Computational models of intrinsic motivation for curiosity and creativity

We link Ivancovsky et al.'s novelty-seeking model (NSM) to computational models of intrinsically motivated behavior and learning. We argue that dissociating different forms of curiosity, creativity, and memory based on the involvement of distinct intrinsic motivations (e.g., surprise and novelty) is essential to empirically test the conceptual claims of the NSM.

Neural representation in active inference: Using generative models to interact with-and understand-the lived world

This paper considers neural representation through the lens of active inference, a normative framework for understanding brain function. It delves into how living organisms employ generative models to minimize the discrepancy between predictions and observations (as scored with variational free energy). The ensuing analysis suggests that the brain learns generative models to navigate the world adaptively, not (or not solely) to understand it. Different living organisms may possess an array of generative models, spanning from those that support action-perception cycles to those that underwrite planning and imagination; namely, from explicit models that entail variables for predicting concurrent sensations, like objects, faces, or people-to action-oriented models that predict action outcomes. It then elucidates how generative models and belief dynamics might link to neural representation and the implications of different types of generative models for understanding an agent's cognitive capabilities in relation to its ecological niche. The paper concludes with open questions regarding the evolution of generative models and the development of advanced cognitive abilities-and the gradual transition from pragmatic to detached neural representations. The analysis on offer foregrounds the diverse roles that generative models play in cognitive processes and the evolution of neural representation.

Effects of exergames on student physical education learning in the context of the artificial intelligence era: a meta-analysis

Whether the application of exergames in physical education (PE) courses can significantly improve student performance in PE learning is still controversial. This review explores the promoting effect of exergames on student PE learning and the conditions in which the effect of exergames can be maximized. Based on the PICOS method, two researchers independently searched the ProQuest database, EBSCO database, Web of Science (WoS) database, PubMed database, Chinese National Knowledge Infrastructure (CNKI) database, Wanfang database, and VIP database, evaluated the literature quality using the Cochrane system evaluation manual, and performed a meta-analysis of the included literature. A total of 16 randomized controlled trials involving 2962 subjects were included in this study. The meta-analysis showed that exergames effectively improved student performance in PE learning (SMD = 0.45, 95% CI: 0.27-0.63, P < 0.00001). Subgroup analysis indicated that better results could be achieved when exergames were introduced in small kindergarten classes and continued for 1-2 months.

Belief-consistent information is most shared despite being the least surprising

In the classical information theoretic framework, information "value" is proportional to how novel/surprising the information is. Recent work building on such notions claimed that false news spreads faster than truth online because false news is more novel and therefore surprising. However, another determinant of surprise, semantic meaning (e.g., information's consistency or inconsistency with prior beliefs), should also influence value and sharing. Examining sharing behavior on Twitter, we observed separate relations of novelty and belief consistency with sharing. Though surprise could not be assessed in those studies, belief consistency should relate to less surprise, suggesting the relevance of semantic meaning beyond novelty. In two controlled experiments, belief-consistent (vs. belief-inconsistent) information was shared more despite consistent information being the least surprising. Manipulated novelty did not predict sharing or surprise. Thus, classical information theoretic predictions regarding perceived value and sharing would benefit from considering semantic meaning in contexts where people hold pre-existing beliefs.

Curiosity: primate neural circuits for novelty and information seeking

For many years, neuroscientists have investigated the behavioural, computational and neurobiological mechanisms that support value-based decisions, revealing how humans and animals make choices to obtain rewards. However, many decisions are influenced by factors other than the value of physical rewards or second-order reinforcers (such as money). For instance, animals (including humans) frequently explore novel objects that have no intrinsic value solely because they are novel and they exhibit the desire to gain information to reduce their uncertainties about the future, even if this information cannot lead to reward or assist them in accomplishing upcoming tasks. In this Review, I discuss how circuits in the primate brain responsible for detecting, predicting and assessing novelty and uncertainty regulate behaviour and give rise to these behavioural components of curiosity. I also briefly discuss how curiosity-related behaviours arise during postnatal development and point out some important reasons for the persistence of curiosity across generations.

Fast adaptation to rule switching using neuronal surprise

In humans and animals, surprise is a physiological reaction to an unexpected event, but how surprise can be linked to plausible models of neuronal activity is an open problem. We propose a self-supervised spiking neural network model where a surprise signal is extracted from an increase in neural activity after an imbalance of excitation and inhibition. The surprise signal modulates synaptic plasticity via a three-factor learning rule which increases plasticity at moments of surprise. The surprise signal remains small when transitions between sensory events follow a previously learned rule but increases immediately after rule switching. In a spiking network with several modules, previously learned rules are protected against overwriting, as long as the number of modules is larger than the total number of rules-making a step towards solving the stability-plasticity dilemma in neuroscience. Our model relates the subjective notion of surprise to specific predictions on the circuit level.

Order and change in art: towards an active inference account of aesthetic experience

How to account for the power that art holds over us? Why do artworks touch us deeply, consoling, transforming or invigorating us in the process? In this paper, we argue that an answer to this question might emerge from a fecund framework in cognitive science known as predictive processing (a.k.a. active inference). We unpack how this approach connects sense-making and aesthetic experiences through the idea of an 'epistemic arc', consisting of three parts (curiosity, epistemic action and aha experiences), which we cast as aspects of active inference. We then show how epistemic arcs are built and sustained by artworks to provide us with those satisfying experiences that we tend to call 'aesthetic'. Next, we defuse two key objections to this approach; namely, that it places undue emphasis on the cognitive component of our aesthetic encounters-at the expense of affective aspects-and on closure and uncertainty minimization (order)-at the expense of openness and lingering uncertainty (change). We show that the approach offers crucial resources to account for the open-ended, free and playful behaviour inherent in aesthetic experiences. The upshot is a promising but deflationary approach, both philosophically informed and psychologically sound, that opens new empirical avenues for understanding our aesthetic encounters. This article is part of the theme issue 'Art, aesthetics and predictive processing: theoretical and empirical perspectives'.

Neural encoding of musical expectations in a non-human primate

The appreciation of music is a universal trait of humankind.1,2,3 Evidence supporting this notion includes the ubiquity of music across cultures4,5,6,7 and the natural predisposition toward music that humans display early in development.8,9,10 Are we musical animals because of species-specific predispositions? This question cannot be answered by relying on cross-cultural or developmental studies alone, as these cannot rule out enculturation.11 Instead, it calls for cross-species experiments testing whether homologous neural mechanisms underlying music perception are present in non-human primates. We present music to two rhesus monkeys, reared without musical exposure, while recording electroencephalography (EEG) and pupillometry. Monkeys exhibit higher engagement and neural encoding of expectations based on the previously seeded musical context when passively listening to real music as opposed to shuffled controls. We then compare human and monkey neural responses to the same stimuli and find a species-dependent contribution of two fundamental musical features-pitch and timing12-in generating expectations: while timing- and pitch-based expectations13 are similarly weighted in humans, monkeys rely on timing rather than pitch. Together, these results shed light on the phylogeny of music perception. They highlight monkeys' capacity for processing temporal structures beyond plain acoustic processing, and they identify a species-dependent contribution of time- and pitch-related features to the neural encoding of musical expectations.

Sensory Reinforced Corticostriatal Plasticity

BACKGROUND

Regional changes in corticostriatal transmission induced by phasic dopaminergic signals are an essential feature of the neural network responsible for instrumental reinforcement during discovery of an action. However, the timing of signals that are thought to contribute to the induction of corticostriatal plasticity is difficult to reconcile within the framework of behavioural reinforcement learning, because the reinforcer is normally delayed relative to the selection and execution of causally-related actions.

OBJECTIVE

While recent studies have started to address the relevance of delayed reinforcement signals and their impact on corticostriatal processing, our objective was to establish a model in which a sensory reinforcer triggers appropriately delayed reinforcement signals relayed to the striatum via intact neuronal pathways and to investigate the effects on corticostriatal plasticity.

METHODS

We measured corticostriatal plasticity with electrophysiological recordings using a light flash as a natural sensory reinforcer, and pharmacological manipulations were applied in an in vivo anesthetized rat model preparation.

RESULTS

We demonstrate that the spiking of striatal neurons evoked by single-pulse stimulation of the motor cortex can be potentiated by a natural sensory reinforcer, operating through intact afferent pathways, with signal timing approximating that required for behavioural reinforcement. The pharmacological blockade of dopamine receptors attenuated the observed potentiation of corticostriatal neurotransmission.

CONCLUSION

This novel in vivo model of corticostriatal plasticity offers a behaviourally relevant framework to address the physiological, anatomical, cellular, and molecular bases of instrumental reinforcement learning.

Path integrals, particular kinds, and strange things

This paper describes a path integral formulation of the free energy principle. The ensuing account expresses the paths or trajectories that a particle takes as it evolves over time. The main results are a method or principle of least action that can be used to emulate the behaviour of particles in open exchange with their external milieu. Particles are defined by a particular partition, in which internal states are individuated from external states by active and sensory blanket states. The variational principle at hand allows one to interpret internal dynamics-of certain kinds of particles-as inferring external states that are hidden behind blanket states. We consider different kinds of particles, and to what extent they can be imbued with an elementary form of inference or sentience. Specifically, we consider the distinction between dissipative and conservative particles, inert and active particles and, finally, ordinary and strange particles. Strange particles can be described as inferring their own actions, endowing them with apparent autonomy or agency. In short-of the kinds of particles afforded by a particular partition-strange kinds may be apt for describing sentient behaviour.

Are curiosity and situational interest different? Exploring distinct antecedents and consequences

BACKGROUND

A controversy over the distinction between curiosity and situational interest has recently resurfaced. Nonetheless, empirical research comparing the two is noticeably lacking.

AIMS

We attempted to fill this gap and provide much-needed evidence of the distinction between curiosity and situational interest by examining the antecedents and consequences of the two constructs.

METHODS

We assessed enjoyment, novelty, uncertainty and surprise as potential antecedents and information seeking, individual interest, career intention and achievement as potential outcomes of curiosity and situational interest among 219 Korean sixth graders in the domain of science.

RESULTS

Of the hypothesized antecedents, enjoyment during science class related most strongly to students' situational interest in science, whereas novelty in science class related most strongly to students' science curiosity. Uncertainty and surprise in science class related to only science curiosity and not situational interest in science. Among the outcomes considered, situational interest in science related to only students' individual interest in science. In comparison, science curiosity related significantly to all science outcomes measured in this study. Science curiosity also significantly mediated the relationships between the antecedents and outcomes in science.

CONCLUSIONS

Together, these results support the distinction between curiosity and situational interest and suggest different ways to promote each motivation construct depending on desired outcomes in the science classroom.

Curiosity-driven exploration: foundations in neuroscience and computational modeling

Curiosity refers to the intrinsic desire of humans and animals to explore the unknown, even when there is no apparent reason to do so. Thus far, no single, widely accepted definition or framework for curiosity has emerged, but there is growing consensus that curious behavior is not goal-directed but related to seeking or reacting to information. In this review, we take a phenomenological approach and group behavioral and neurophysiological studies which meet these criteria into three categories according to the type of information seeking observed. We then review recent computational models of curiosity from the field of machine learning and discuss how they enable integrating different types of information seeking into one theoretical framework. Combinations of behavioral and neurophysiological studies along with computational modeling will be instrumental in demystifying the notion of curiosity.

CHL1 depletion affects dopamine receptor D2-dependent modulation of mouse behavior

Introduction

The dopaminergic system plays a key role in the appropriate functioning of the central nervous system, where it is essential for emotional balance, arousal, reward, and motor control. The cell adhesion molecule close homolog of L1 (CHL1) contributes to dopaminergic system development, and CHL1 and the dopamine receptor D2 (D2R) are associated with mental disorders like schizophrenia, addiction, autism spectrum disorder and depression.

Methods

Here, we investigated how the interplay between CHL1 and D2R affects the behavior of young adult male and female wild-type (CHL+/+) and CHL1-deficient (CHL1-/-) mice, when D2R agonist quinpirole and antagonist sulpiride are applied.

Results

Low doses of quinpirole (0.02 mg/kg body weight) induced hypolocomotion of CHL1+/+ and CHL1-/- males and females, but led to a delayed response in CHL1-/- mice. Sulpiride (1 mg/kg body weight) affected locomotion of CHL1-/- females and social interaction of CHL1+/+ females as well as social interactions of CHL1-/- and CHL1+/+ males. Quinpirole increased novelty-seeking behavior of CHL1-/- males compared to CHL1+/+ males. Vehicle-treated CHL1-/- males and females showed enhanced working memory and reduced stress-related behavior.

Discussion

We propose that CHL1 regulates D2R-dependent functions in vivo. Deficiency of CHL1 leads to abnormal locomotor activity and emotionality, and to sex-dependent behavioral differences.

The technology, opportunities, and challenges of Synthetic Biological Intelligence

Integrating neural cultures developed through synthetic biology methods with digital computing has enabled the early development of Synthetic Biological Intelligence (SBI). Recently, key studies have emphasized the advantages of biological neural systems in some information processing tasks. However, neither the technology behind this early development, nor the potential ethical opportunities or challenges, have been explored in detail yet. Here, we review the key aspects that facilitate the development of SBI and explore potential applications. Considering these foreseeable use cases, various ethical implications are proposed. Ultimately, this work aims to provide a robust framework to structure ethical considerations to ensure that SBI technology can be both researched and applied responsibly.

Opposite effects of emotion and event segmentation on temporal order memory and object-context binding

Our daily lives unfold continuously, yet our memories are organised into distinct events, situated in a specific context of space and time, and chunked when this context changes (at event boundaries). Previous research showed that this process, termed event segmentation, enhances object-context binding but impairs temporal order memory. Physiologically, peaks in pupil dilation index event segmentation, similar to emotion-induced bursts of autonomic arousal. Emotional arousal also modulates object-context binding and temporal order memory. Yet, these two critical factors have not been systematically studied together. To address this gap, we ran a behavioural experiment using a paradigm validated to study event segmentation and extended it with emotion manipulation. During encoding, we sequentially presented greyscale objects embedded in coloured frames (colour changes defining events), with a neutral or aversive sound. During retrieval, we tested participants' memory of temporal order memory and object-colour binding. We found opposite effects of emotion and event segmentation on episodic memory. While event segmentation enhanced object-context binding, emotion impaired it. On the contrary, event segmentation impaired temporal order memory, but emotion enhanced it. These findings increase our understanding of episodic memory organisation in laboratory settings, and potentially in real life with perceptual changes and emotion fluctuations constantly interacting.

Surprise and novelty in the brain

Notions of surprise and novelty have been used in various experimental and theoretical studies across multiple brain areas and species. However, 'surprise' and 'novelty' refer to different quantities in different studies, which raises concerns about whether these studies indeed relate to the same functionalities and mechanisms in the brain. Here, we address these concerns through a systematic investigation of how different aspects of surprise and novelty relate to different brain functions and physiological signals. We review recent classifications of definitions proposed for surprise and novelty along with links to experimental observations. We show that computational modeling and quantifiable definitions enable novel interpretations of previous findings and form a foundation for future theoretical and experimental studies.

An active inference model of hierarchical action understanding, learning and imitation

We advance a novel active inference model of the cognitive processing that underlies the acquisition of a hierarchical action repertoire and its use for observation, understanding and imitation. We illustrate the model in four simulations of a tennis learner who observes a teacher performing tennis shots, forms hierarchical representations of the observed actions, and imitates them. Our simulations show that the agent's oculomotor activity implements an active information sampling strategy that permits inferring the kinematic aspects of the observed movement, which lie at the lowest level of the action hierarchy. In turn, this low-level kinematic inference supports higher-level inferences about deeper aspects of the observed actions: proximal goals and intentions. Finally, the inferred action representations can steer imitative responses, but interfere with the execution of different actions. Our simulations show that hierarchical active inference provides a unified account of action observation, understanding, learning and imitation and helps explain the neurobiological underpinnings of visuomotor cognition, including the multiple routes for action understanding in the dorsal and ventral streams and mirror mechanisms.

Prior knowledge about events depicted in scenes decreases oculomotor exploration

The visual input that the eyes receive usually contains temporally continuous information about unfolding events. Therefore, humans can accumulate knowledge about their current environment. Typical studies on scene perception, however, involve presenting multiple unrelated images and thereby render this accumulation unnecessary. Our study, instead, facilitated it and explored its effects. Specifically, we investigated how recently-accumulated prior knowledge affects gaze behavior. Participants viewed sequences of static film frames that contained several 'context frames' followed by a 'critical frame'. The context frames showed either events from which the situation depicted in the critical frame naturally followed, or events unrelated to this situation. Therefore, participants viewed identical critical frames while possessing prior knowledge that was either relevant or irrelevant to the frames' content. In the former case, participants' gaze behavior was slightly more exploratory, as revealed by seven gaze characteristics we analyzed. This result demonstrates that recently-gained prior knowledge reduces exploratory eye movements.

P3-like signatures of temporal predictions: a computational EEG study

Many cognitive processes, ranging from perception to action, depend on the ability to predict the timing of forthcoming events. Yet, how the brain uses predictive models in the temporal domain is still an unsolved question. In previous work, we began to explore the neural correlates of temporal predictions by using a computational approach in which an ideal Bayesian observer learned the temporal probabilities of target onsets in a simple reaction time task. Because the task was specifically designed to disambiguate updating of predictive models and surprise, changes in temporal probabilities were explicitly cued. However, in the real world, we are usually incidentally exposed to changes in the statistics of the environment. Here, we thus aimed to further investigate the electroencephalographic (EEG) correlates of Bayesian belief updating and surprise associated with incidental learning of temporal probabilities. In line with our previous EEG study, results showed distinct P3-like modulations for updating and surprise. While surprise was indexed by an early fronto-central P3-like modulation, updating was associated with a later and more posterior P3 modulation. Moreover, updating was associated with a P2-like potential at centro-parietal electrodes, likely capturing integration processes between prior beliefs and likelihood of the observed event. These findings support previous evidence of trial-by-trial variability of P3 amplitudes as an index of dissociable inferential processes. Coupled with our previous findings, the present study strongly bolsters the view of the P3 as a key brain signature of temporal Bayesian inference. Data and scripts are shared on OSF: osf.io/sdy8j/.

The graded novelty encoding task: Novelty gradually improves recognition of visual stimuli under incidental learning conditions

It has been argued that novel compared to familiar stimuli are preferentially encoded into memory. Nevertheless, treating novelty as a categorical variable in experimental research is considered simplistic. We highlight the dimensional aspect of novelty and propose an experimental design that manipulates novelty continuously. We created the Graded Novelty Encoding Task (GNET), in which the difference between stimuli (i.e. novelty) is parametrically manipulated, paving the way for quantitative models of novelty processing. We designed an algorithm which generates visual stimuli by placing colored shapes in a grid. During the familiarization phase of the task, we repeatedly presented five pictures to the participants. In a subsequent incidental learning phase, participants were asked to differentiate between the "familiars" and novel images that varied in the degree of difference to the familiarized pictures (i.e. novelty). Finally, participants completed a surprise recognition memory test, where the novel stimuli from the previous phase were interspersed with distractors with similar difference characteristics. We numerically expressed the differences between the stimuli to compute a dimensional indicator of novelty and assessed whether it predicted recognition memory performance. Based on previous studies showing the beneficial effect of novelty on memory formation, we hypothesized that the more novel a given picture was, the better subsequent recognition performance participants would demonstrate. Our hypothesis was confirmed: recognition performance was higher for more novel stimuli. The GNET captures the continuous nature of novelty, and it may be useful in future studies that examine the behavioral and neurocognitive aspects of novelty processing.

VASE: Variational Assorted Surprise Exploration for Reinforcement Learning

Exploration in environments with continuous control and sparse rewards remains a key challenge in reinforcement learning (RL). One of the approaches to encourage more systematic and efficient exploration relies on surprise as an intrinsic reward for the agent. We introduce a new definition of surprise and its RL implementation named variational assorted surprise exploration (VASE). VASE uses a Bayesian neural network as a model of the environment dynamics and is trained using variational inference, alternately updating the accuracy of the agent's model and policy. Our experiments show that in continuous control sparse reward environments, VASE outperforms other surprise-based exploration techniques.

Bayesian surprise shapes neural responses in somatosensory cortical circuits

Numerous psychophysical studies show that Bayesian inference governs sensory decision-making; however, the specific neural circuitry underlying this probabilistic mechanism remains unclear. We record extracellular neural activity along the somatosensory pathway of mice while delivering sensory stimulation paradigms designed to isolate the response to the surprise generated by Bayesian inference. Our results demonstrate that laminar cortical circuits in early sensory areas encode Bayesian surprise. Systematic sensitivity to surprise is not identified in the somatosensory thalamus, rather emerging in the primary (S1) and secondary (S2) somatosensory cortices. Multiunit spiking activity and evoked potentials in layer 6 of these regions exhibit the highest sensitivity to surprise. Gamma power in S1 layer 2/3 exhibits an NMDAR-dependent scaling with surprise, as does alpha power in layers 2/3 and 6 of S2. These results show a precise spatiotemporal neural representation of Bayesian surprise and suggest that Bayesian inference is a fundamental component of cortical processing.

An Information-Theoretic Perspective on Intrinsic Motivation in Reinforcement Learning: A Survey

The reinforcement learning (RL) research area is very active, with an important number of new contributions, especially considering the emergent field of deep RL (DRL). However, a number of scientific and technical challenges still need to be resolved, among which we acknowledge the ability to abstract actions or the difficulty to explore the environment in sparse-reward settings which can be addressed by intrinsic motivation (IM). We propose to survey these research works through a new taxonomy based on information theory: we computationally revisit the notions of surprise, novelty, and skill-learning. This allows us to identify advantages and disadvantages of methods and exhibit current outlooks of research. Our analysis suggests that novelty and surprise can assist the building of a hierarchy of transferable skills which abstracts dynamics and makes the exploration process more robust.

What message features influence the intention to share misinformation about COVID-19 on social media? The role of efficacy and novelty

Given the amount of misinformation being circulated on social media during the COVID-19 pandemic and its potential threat to public health, it is imperative to investigate ways to hinder its transmission. To this end, this study aimed to identify message features that may contribute to misinformation sharing on social media. Based on the theory of social sharing of emotion and the extant research on message credibility, this study examined if emotions and message credibility serve as mechanisms through which novelty and efficacy of misinformation influence sharing intention. An online experiment concerning COVID-19 misinformation was conducted by employing a 2 (novelty conditions: high vs. low) × 2 (efficacy conditions: high vs. low) between-subjects design using a national quota sample in South Korea (N = 1,012). The findings suggested that, contrary to the expectation, the overall effects of novelty on sharing intention were negative. The specific mechanisms played significant and unique roles in different directions: novelty increased sharing intention by evoking surprise, while also exerting a negative influence on sharing intention through an increase in negative emotions and a decrease in positive emotions and message credibility. Consistent with the expectation, efficacy exhibited positive total effects on sharing intention, which was explained by higher levels of (self- and response-) efficacy of protective action increasing positive emotions and message credibility but decreasing negative emotions. The implications and limitations of the study are discussed.

Computational psychiatry: from synapses to sentience

This review considers computational psychiatry from a particular viewpoint: namely, a commitment to explaining psychopathology in terms of pathophysiology. It rests on the notion of a generative model as underwriting (i) sentient processing in the brain, and (ii) the scientific process in psychiatry. The story starts with a view of the brain-from cognitive and computational neuroscience-as an organ of inference and prediction. This offers a formal description of neuronal message passing, distributed processing and belief propagation in neuronal networks; and how certain kinds of dysconnection lead to aberrant belief updating and false inference. The dysconnections in question can be read as a pernicious synaptopathy that fits comfortably with formal notions of how we-or our brains-encode uncertainty or its complement, precision. It then considers how the ensuing process theories are tested empirically, with an emphasis on the computational modelling of neuronal circuits and synaptic gain control that mediates attentional set, active inference, learning and planning. The opportunities afforded by this sort of modelling are considered in light of in silico experiments; namely, computational neuropsychology, computational phenotyping and the promises of a computational nosology for psychiatry. The resulting survey of computational approaches is not scholarly or exhaustive. Rather, its aim is to review a theoretical narrative that is emerging across subdisciplines within psychiatry and empirical scales of investigation. These range from epilepsy research to neurodegenerative disorders; from post-traumatic stress disorder to the management of chronic pain, from schizophrenia to functional medical symptoms.

Dynamics of personality: The Zurich model of motivation revived, extended, and applied to personality

Personality researchers are increasingly interested in the dynamics of personality, that is, the proximal causal mechanisms underlying personality and behavior. Here, we review the Zurich Model of Social Motivation concerning its potential to explain central aspects of personality. It is a cybernetic model that provides a nomothetic structure of the causal relationships among needs for security, arousal, and power, and uses them to explain an individual's approach-avoidance or "proximity-distance" behavior. We review core features of the model and extend them by adding features based on recent behavioral and neuroscientific evidence. We close by discussing the model considering contemporary issues in personality science such as the dynamics of personality, five-factor personality traits and states, and personality growth.

Rat's response to a novelty and increased complexity of the environment resulting from the introduction of movable vs. stationary objects in the free exploration test

Most animals, including rats, show a preference for more complex environments. This is demonstrated particularly well when complexity increases due to the addition of new elements to the environment. The aim of the study was to investigate the reaction to novelty, understood as a change in environmental properties that involve both changes in complexity and controllability. Controllability may allow for dealing with challenges of an environment of low predictability in a way that the animal's own activity reduces the uncertainty of environmental events. In our study, the animals underwent a spontaneous exploration test in low-stress conditions. After a period of habituation to the experimental arena, additional stationary (increased complexity) and/or movable (increased complexity and controllability) tunnels were introduced, and the reaction of the rats to the novel objects was measured. The results of the study confirmed that an increase in the complexity of the environment through the addition of objects triggers a more intensive exploratory activity in rats. However, an increased spatial complexity combined with the movability of the novel objects seems to result in increased caution towards the novelty after an initial inspection of the changed objects. It suggests that the complexity of the novelty may trigger both neophilia and neophobia depending on the level of the predictability of the novel environment and that the movability of newly introduced objects is not independent of other parameters of the environment.

An intrinsically motivated learning algorithm based on Bayesian surprise for cognitive radar in autonomous vehicles

Introduction

This paper proposes a Bayesian surprise learning algorithm that internally motivates the cognitive radar to estimate a target's state (i.e., velocity, distance) from noisy measurements and make decisions to reduce the estimation error gradually. The work exhibits how the sensor learns from experiences, anticipates future responses, and adjusts its waveform parameters to achieve informative measurements based on the Bayesian surprise.

Methods

For a simple vehicle-following scenario where the radar measurements are generated from linear Gaussian state-space models, the article adopts the Kalman filter to carry out state estimation. According to the information within the filter's estimate, the sensor intrinsically assigns a surprise-based reward value to the immediate past action and updates the value-to-go function. Through a series of hypothetical steps, the cognitive radar considers the impact of future transmissions for a prescribed set of waveforms–available from the sensor profile library–to improve the estimation process.

Results and discussion

Numerous experiments investigate the performance of the proposed design for various surprise-based reward expressions. The robustness of the proposed method is compared to the state-of-the-art for practical and risky driving situations. Results show that the reward functions inspired by estimation credibility measures outperform their competitors when one-step planning is considered. Simulation results also indicate that multiple-step planning does not necessarily lead to lower error, particularly when the environment changes abruptly.

Solidity Meets Surprise: Cerebral and Behavioral Effects of Learning from Episodic Prediction Errors

How susceptible a memory is to later modification might depend on how stable the episode has been encoded. This stability was proposed to increase when retrieving information more (vs. less) often and in a spaced (vs. massed) practice. Using fMRI, we examined the effects of these different pre-fMRI retrieval protocols on the subsequent propensity to learn from episodic prediction errors. After encoding a set of different action stories, participants came back for two pre-fMRI retrieval sessions in which they encountered original episodes either 2 or 8 times in either a spaced or a massed retrieval protocol. One week later, we cued episodic retrieval during the fMRI session by using original or modified videos of encoded action stories. Recurrent experience of modified episodes was associated with increasing activity in the episodic memory network including hippocampal and cortical areas, when leading to false memories in a post-fMRI memory test. While this observation clearly demonstrated learning from episodic prediction errors, we found no evidence for a modulatory effect of the different retrieval protocols. As expected, the benefit of retrieving an episode more often was reflected in better memory for originally encoded episodes. In addition, frontal activity increased for episodic prediction errors when episodes had been less frequently retrieved pre-fMRI. A history of spaced versus massed retrieval was associated with increased activation throughout the episodic memory network, with no significant effect on behavioral performance. Our findings show that episodic prediction errors led to false memories. The history of different retrieval protocols was reflected in memory performance and brain responses to episodic prediction errors, but did not interact with the brain's episodic learning response.

Striatal dopamine explains novelty-induced behavioral dynamics and individual variability in threat prediction

Animals both explore and avoid novel objects in the environment, but the neural mechanisms that underlie these behaviors and their dynamics remain uncharacterized. Here, we used multi-point tracking (DeepLabCut) and behavioral segmentation (MoSeq) to characterize the behavior of mice freely interacting with a novel object. Novelty elicits a characteristic sequence of behavior, starting with investigatory approach and culminating in object engagement or avoidance. Dopamine in the tail of the striatum (TS) suppresses engagement, and dopamine responses were predictive of individual variability in behavior. Behavioral dynamics and individual variability are explained by a reinforcement-learning (RL) model of threat prediction in which behavior arises from a novelty-induced initial threat prediction (akin to "shaping bonus") and a threat prediction that is learned through dopamine-mediated threat prediction errors. These results uncover an algorithmic similarity between reward- and threat-related dopamine sub-systems.

Oversampled and undersolved: Depressive rumination from an active inference perspective

Rumination is a widely recognized cognitive deviation in depression. Despite the recognition, researchers have struggled to explain why patients cannot disengage from the process, although it depresses their mood and fails to lead to effective problem-solving. We rethink rumination as repetitive but unsuccessful problem-solving attempts. Appealing to an active inference account, we suggest that adaptive problem-solving is based on the generation, evaluation, and performance of candidate policies that increase an organism's knowledge of its environment. We argue that the problem-solving process is distorted during rumination. Specifically, rumination is understood as engaging in excessive yet unsuccessful oversampling of policy candidates that do not resolve uncertainty. Because candidates are sampled from policies that were selected in states resembling one's current state, "bad" starting points (e.g., depressed mood, physical inactivity) make the problem-solving process vulnerable for generating a ruminative "halting problem". This problem leads to high opportunity costs, learned helplessness and diminished overt behavior. Besides reviewing evidence for the conceptual paths of this model, we discuss its neurophysiological correlates and point towards clinical implications.

In the Body’s Eye: The computational anatomy of interoceptive inference

A growing body of evidence highlights the intricate linkage of exteroceptive perception to the rhythmic activity of the visceral body. In parallel, interoceptive inference theories of affective perception and self-consciousness are on the rise in cognitive science. However, thus far no formal theory has emerged to integrate these twin domains; instead, most extant work is conceptual in nature. Here, we introduce a formal model of cardiac active inference, which explains how ascending cardiac signals entrain exteroceptive sensory perception and uncertainty. Through simulated psychophysics, we reproduce the defensive startle reflex and commonly reported effects linking the cardiac cycle to affective behaviour. We further show that simulated ‘interoceptive lesions’ blunt affective expectations, induce psychosomatic hallucinations, and exacerbate biases in perceptual uncertainty. Through synthetic heart-rate variability analyses, we illustrate how the balance of arousal-priors and visceral prediction errors produces idiosyncratic patterns of physiological reactivity. Our model thus offers a roadmap for computationally phenotyping disordered brain-body interaction.

Understanding interactions between the brain and the body has become a topic of increased interest in computational neuroscience and psychiatry. A particular question here concerns how visceral, homeostatic rhythms such as the heart beat influence sensory, affective, and cognitive processing. To better understand these and other oscillatory brain-body interactions, we here introduce a novel computational model of interoceptive inference in which a synthetic agent’s perceptual beliefs are coupled to the rhythm of the heart. Our model both helps to explain emerging empirical data indicating that perceptual inference depends upon beat-to-beat heart rhythms, and can be used to better quantify intra- and inter-individual differences in heart-brain coupling. Using proof-of-principle simulations, we demonstrate how future empirical works could utilize our model to better understand and stratify disorders of interoception and brain-body interaction.

A neuro-inspired computational model of life-long learning and catastrophic interference, mimicking hippocampus novelty-based dopamine modulation and lateral inhibitory plasticity

The human brain has a remarkable lifelong learning capability to acquire new experiences while retaining previously acquired information. Several hypotheses have been proposed to explain this capability, but the underlying mechanisms are still unclear. Here, we propose a neuro-inspired firing-rate computational model involving the hippocampus and surrounding areas, that encompasses two key mechanisms possibly underlying this capability. The first is based on signals encoded by the neuromodulator dopamine, which is released by novel stimuli and enhances plasticity only when needed. The second is based on a homeostatic plasticity mechanism that involves the lateral inhibitory connections of the pyramidal neurons of the hippocampus. These mechanisms tend to protect neurons that have already been heavily employed in encoding previous experiences. The model was tested with images from the MNIST machine learning dataset, and with more naturalistic images, for its ability to mitigate catastrophic interference in lifelong learning. The results show that the proposed biologically grounded mechanisms can effectively enhance the learning of new stimuli while protecting previously acquired knowledge. The proposed mechanisms could be investigated in future empirical animal experiments and inspire machine learning models.

Dopamine activity on the perceptual salience for recognition memory

To survive, animals must recognize relevant stimuli and distinguish them from inconspicuous information. Usually, the properties of the stimuli, such as intensity, duration, frequency, and novelty, among others, determine the salience of the stimulus. However, previously learned experiences also facilitate the perception and processing of information to establish their salience. Here, we propose “perceptual salience” to define how memory mediates the integration of inconspicuous stimuli into a relevant memory trace without apparently altering the recognition of the physical attributes or valence, enabling the detection of stimuli changes in future encounters. The sense of familiarity is essential for successful recognition memory; in general, familiarization allows the transition of labeling a stimulus from the novel (salient) to the familiar (non-salient). The novel object recognition (NOR) and object location recognition (OLRM) memory paradigms represent experimental models of recognition memory that allow us to study the neurobiological mechanisms involved in episodic memory. The catecholaminergic system has been of vital interest due to its role in several aspects of recognition memory. This review will discuss the evidence that indicates changes in dopaminergic activity during exposure to novel objects or places, promoting the consolidation and persistence of memory. We will discuss the relationship between dopaminergic activity and perceptual salience of stimuli enabling learning and consolidation processes necessary for the novel-familiar transition. Finally, we will describe the effect of dopaminergic deregulation observed in some pathologies and its impact on recognition memory.

Complexification of eukaryote phenotype: Adaptive immuno-cognitive systems as unique Gödelian block chain distributed ledger

The digitization of inheritable information in the genome has been called the 'algorithmic take-over of biology'. The McClintock discovery that viral software based transposable elements that conduct cut-paste (transposon) and copy-paste (retrotransposon) operations are needed for genomic evolvability underscores the truism that only software can change software and also that viral hacking by internal and external bio-malware is the Achilles heel of genomic digital systems. There was a paradigm shift in genomic information processing with the Adaptive Immune System (AIS) 500 mya followed by the Mirror Neuron System (MNS), latterly mostly in primate brains, which reaches its apogee in human social cognition. The AIS and MNS involve distinctive Gödelian features of self-reference (Self-Ref) and offline virtual self-representation (Self-Rep) for complex self-other interaction with prodigious open-ended capacity for anticipative malware detection and novelty production within a unique blockchain distributed ledger (BCDL). The role of self-referential information processing, often considered to be central to the sentient self with origins in the immune system 'Thymic self', is shown to be part of the Gödel logic behind a generator-selector framework at a molecular level, which exerts stringent selection criteria to maintain genomic BCDL. The latter manifests digital and decentralized record keeping where no internal or external bio-malware can compromise the immutability of the life's building blocks and no novel blocks can be added that is not consistent with extant blocks. This is demonstrated with regard to somatic hypermutation with novel anti-body production in the face of external non-self antigen attacks.

Attention- versus significance-driven memory formation: Taxonomy, neural substrates, and meta-analyses

Functional neuroimaging data on episodic memory formation have expanded rapidly over the last 30 years, which raises the need for an integrative framework. This study proposes a taxonomy of episodic memory formation to address this need. At the broadest level, the taxonomy distinguishes between attention-driven vs. significance-driven memory formation. The three subtypes of attention-driven memory formation are selection-, fluctuation-, and level-related. The three subtypes of significance-driven memory formation are novelty-, emotion-, and reward-related. Meta-analytic data indicated that attention-driven memory formation affects the functioning of the extra-medial temporal lobe more strongly than the medial temporal lobe (MTL) regions. In contrast, significance-driven memory formation affects the functioning of the MTL more strongly than the extra-MTL regions. This study proposed a model in which attention has a stronger impact on the formation of neocortical traces than hippocampus/MTL traces, whereas significance has a stronger impact on the formation of hippocampus/MTL traces than neocortical traces. Overall, the taxonomy and model provide an integrative framework in which to place diverse encoding-related findings into a proper perspective.

The limited reach of surprise: Evidence against effects of surprise on memory for preceding elements of an event

When reflecting on the past, some of our strongest memories are for experiences that took us by surprise. Extensive research has backed this intuition that we are more likely to remember surprising moments than mundane ones. But what about the moments leading up to the surprise? Are we more likely to remember those as well? While surprise is a well-established modulator of memory, it is unknown whether memory for the entire event will be enhanced, or only for the surprising occurrence itself. We developed a novel paradigm utilising stop-motion films, depicting of a sequence of narrative events, in which specific occurrences could be replaced with surprising ones, while keeping the rest of the film unaltered. Using this design, we tested whether surprise exerts retroactive effects on memory, and specifically whether any potential effect would be confined to elements in the same event as the surprising occurrence. In a large cohort of participants (n = 340), we found strong evidence that surprise did not retroactively modulate memory, neither when participants were tested immediately after study nor when they were tested 24 hours later. We suggest two possible accounts for these findings: (1) that the components of an event are encoded as independent episodic elements (not as a cohesive unit), or (2) that surprise segments experience, sectioning off the preceding elements as a separate event.

Novelty Knows No Boundaries: Why a Proper Investigation of Novelty Effects Within SHRI Should Begin by Addressing the Scientific Plurality of the Field

Research on psychological novelty effects within the fields of Social Robotics and Human-Robot Interaction (together: SHRI) so far has failed to gather the momentum it deserves. With the aid of exemplary descriptions of how psychological novelty is currently approached and researched across (certain main regions of) the larger scientific landscape, I argue that the treatment of novelty effects within the multidisciplinary SHRI reflects larger circumstances of fragmentation and heterogeneity in novelty research in general. I further propose that while the concept of novelty may currently function as a Boundary Object between the contributing domains of SHRI, a properly integrated, interdisciplinary concept of novelty is needed in order to capture and investigate the scope and scale of novelty effects within research on social human-robot interaction. Building on research on the New Ontological Category Hypothesis and related studies, I argue that the novelty of social robots can be understood as radical to the extent that their comprehension requires revisions of traditional core categories of being. In order to investigate the sui generis effects of such novelty, which should not be narrowly understood as mere “noise” in the data, it is paramount that the field of SHRI begin by working out a shared, integrative framework of psychological novelty and novelty effects.

A Bayesian Surprise Approach in Designing Cognitive Radar for Autonomous Driving

This article proposes the Bayesian surprise as the main methodology that drives the cognitive radar to estimate a target’s future state (i.e., velocity, distance) from noisy measurements and execute a decision to minimize the estimation error over time. The research aims to demonstrate whether the cognitive radar as an autonomous system can modify its internal model (i.e., waveform parameters) to gain consecutive informative measurements based on the Bayesian surprise. By assuming that the radar measurements are constructed from linear Gaussian state-space models, the paper applies Kalman filtering to perform state estimation for a simple vehicle-following scenario. According to the filter’s estimate, the sensor measures the contribution of prospective waveforms—which are available from the sensor profile library—to state estimation and selects the one that maximizes the expectation of Bayesian surprise. Numerous experiments examine the estimation performance of the proposed cognitive radar for single-target tracking in practical highway and urban driving environments. The robustness of the proposed method is compared to the state-of-the-art for various error measures. Results indicate that the Bayesian surprise outperforms its competitors with respect to the mean square relative error when one-step and multiple-step planning is considered.

What Happened When? Cerebral Processing of Modified Structure and Content in Episodic Cueing

Episodic memories are not static but can change on the basis of new experiences, potentially allowing us to make valid predictions in the face of an ever-changing environment. Recent research has identified prediction errors during memory retrieval as a possible trigger for such changes. In this study, we used modified episodic cues to investigate whether different types of mnemonic prediction errors modulate brain activity and subsequent memory performance. Participants encoded episodes that consisted of short toy stories. During a subsequent fMRI session, participants were presented videos showing the original episodes, or slightly modified versions thereof. In modified videos, either the order of two subsequent action steps was changed or an object was exchanged for another. Content modifications recruited parietal, temporo-occipital, and parahippocampal areas reflecting the processing of the new object information. In contrast, structure modifications elicited activation in right dorsal premotor, posterior temporal, and parietal areas, reflecting the processing of new sequence information. In a post-fMRI memory test, the participants' tendency to accept modified episodes as originally encoded increased significantly when they had been presented modified versions already during the fMRI session. After experiencing modifications, especially those of the episodes' structure, the recognition of originally encoded episodes was impaired as well. Our study sheds light onto the neural processing of different types of episodic prediction errors and their influence on subsequent memory recall.