Boundaries Shape Cognitive Representations of Spaces and Events

Large language models can segment narrative events similarly to humans

Humans perceive discrete events such as "restaurant visits" and "train rides" in their continuous experience. One important prerequisite for studying human event perception is the ability of researchers to quantify when one event ends and another begins. Typically, this information is derived by aggregating behavioral annotations from several observers. Here, we present an alternative computational approach where event boundaries are derived using a large language model, GPT-3, instead of using human annotations. We demonstrate that GPT-3 can segment continuous narrative text into events. GPT-3-annotated events are significantly correlated with human event annotations. Furthermore, these GPT-derived annotations achieve a good approximation of the "consensus" solution (obtained by averaging across human annotations); the boundaries identified by GPT-3 are closer to the consensus, on average, than boundaries identified by individual human annotators. This finding suggests that GPT-3 provides a feasible solution for automated event annotations, and it demonstrates a further parallel between human cognition and prediction in large language models. In the future, GPT-3 may thereby help to elucidate the principles underlying human event perception.

Goal Shifts Structure Memories and Prioritize Event-defining Information in Memory

Every day, we encounter far more information than we could possibly remember. Thus, our memory systems must organize and prioritize the details from an experience that can adaptively guide the storage and retrieval of specific episodic events. Prior work has shown that shifts in internal goal states can function as event boundaries, chunking experiences into distinct and memorable episodes. In addition, at short delays, memory for contextual information at boundaries has been shown to be enhanced compared with items within each event. However, it remains unclear if these memory enhancements are limited to features that signal a meaningful transition between events. To determine how changes in dynamic goal states influence the organization and content of long-term memory, we designed a 2-day experiment in which participants viewed a series of black-and-white objects surrounded by a color border on a two-by-two grid. The location of the object on the grid determined which of two tasks participants performed on a given trial. To examine if distinct types of goal shifts modulate the effects of event segmentation, we changed the border color, the task, or both after every four items in a sequence. We found that goal shifts influenced temporal memory in a manner consistent with the formation of distinct events. However, for subjective memory representations in particular, these effects differed by the type of event boundary. Furthermore, to examine if goal shifts lead to the prioritization of goal-relevant features in longer lasting memories, we tested source memory for each object's color and grid location both immediately and after a 24-hr delay. On the immediate test, boundaries enhanced the memory for all concurrent source features compared with nonboundary items, but only if those boundaries involved a goal shift. In contrast, after a delay, the source memory was selectively enhanced for the feature relevant to the goal shift. These findings suggest that goals can adaptively structure memories by prioritizing contextual features that define a unique episode in memory.

Causal and Chronological Relationships Predict Memory Organization for Nonlinear Narratives

While recounting an experience, one can employ multiple strategies to transition from one part to the next. For instance, if the event was learned out of linear order, one can recall events according to the time they were learned (temporal), similar events (semantic), events occurring nearby in time (chronological), or events produced by the current event (causal). To disentangle the importance of these factors, we had participants watch the nonlinear narrative, Memento, under different task instructions and presentation orders. For each scene of the film, we also separately computed semantic and causal networks. We then contrasted the evidence for temporal, semantic, chronological, or causal strategies during recall. Critically, there was stronger evidence for the causal and chronological strategies than semantic or temporal strategies. Moreover, the causal and chronological strategies outperformed the temporal one even when we asked participants to recall the film in the presented order, underscoring the fundamental nature of causal structure in scaffolding understanding and organizing recall. Nevertheless, time still marginally predicted recall transitions, suggesting it operates as a weak signal in the presence of more salient forms of structure. In addition, semantic and causal network properties predicted scene memorability, including a stronger role for incoming causes to an event than its outgoing effects. In summary, these findings highlight the importance of accounting for complex, causal networks in knowledge building and memory.

Turns around periodic spatial boundaries facilitate increasing event segmentation over time

Event segmentation is a neurocognitive process bridging perception and episodic memory. To our knowledge, almost all segmentation work is framed towards humans, yet evolutionarily conserved mechanisms in event cognition exist across species. Here, we addressed segmentation in a way that is applicable to humans and non-human animals, inspired by research in rats; specifically, the fragmentation of grid-cell spatial representations following the insertion of boundaries into an environment (forming a corridor maze). Participants indicated when they felt a meaningful unit of activity ended and another began, while watching an agent traverse from a first-person perspective. A virtual corridor maze (experiment 1) and two other mazes were used (experiment 2), with participants viewing/segmenting the same stimuli twice. We found that people segmented more during turns relative to corridors, with elevated segmentation occurring in discrete moments around turns. Interestingly, we also found that boundaries of the corridor maze facilitated an increase in segmentation within and across viewings. These results suggest that segmentation can be driven by recognized repeating activity that can become more meaningful over time, highlighting an important link between event segmentation and pattern separation that is relevant to many species in their formation of episodic-(like) memory.

The influence of event similarity on the detailed recall of autobiographical memories

Memories for life events are thought to be organised based on their relationships with one another, affecting the order in which events are recalled such that similar events tend to be recalled together. However, less is known about how detailed recall for a given event is affected by its associations to other events. Here, we used a cued autobiographical memory recall task where participants verbally recalled events corresponding to personal photographs. Importantly, we characterised the temporal, spatial, and semantic associations between each event to assess how similarity between adjacently cued events affected detailed recall. We found that participants provided more non-episodic details for cued events when the preceding event was both semantically similar and either temporally or spatially dissimilar. However, similarity along time, space, or semantics between adjacent events did not affect the episodic details recalled. We interpret this by considering organisation at the level of a life narrative, rather than individual events. When recalling a stream of personal events, we may feel obligated to justify seeming discrepancies between adjacent events that are semantically similar, yet simultaneously temporally or spatially dissimilar - to do so, we provide additional supplementary detail to help maintain global coherence across the events in our lives.

Behavioural interference at event boundaries reduces long-term memory performance in the virtual water maze task without affecting working memory performance

Narrative episodic memory of movie clips can be retroactively impaired by presenting unrelated stimuli coinciding with event boundaries. This effect has been linked with rapid hippocampal processes triggered by the offset of the event, that are alternatively related either to memory consolidation or with working memory processes. Here we tested whether this effect extended to spatial memory, the temporal specificity and extent of the interference, and its effect on working- vs long-term memory. In three computerized adaptations of the Morris Water Maze, participants learned the location of an invisible target over three trials each. A second spatial navigation task was presented either immediately after finding the target, after a 10-s delay, or no second task was presented (control condition). A recall session, in which participants indicated the learned target location with 10 'pin-drop' trials for each condition, was performed after a 1-h or a 24-h break. Spatial memory was measured by the mean distance between pins and the true location. Results indicated that the immediate presentation of the second task led to worse memory performance, for both break durations, compared to the delayed condition. There was no difference in performance between the delayed presentation and the control condition. Despite this long-term memory effect, we found no difference in the rate of performance improvement during the learning session, indicating no effect of the second task on working memory. Our findings are in line with a rapid process, linked to the offset of an event, that is involved in the early stages of memory consolidation.

The neuroscience of active learning and direct instruction

Throughout the educational system, students experiencing active learning pedagogy perform better and fail less than those taught through direct instruction. Can this be ascribed to differences in learning from a neuroscientific perspective? This review examines mechanistic, neuroscientific evidence that might explain differences in cognitive engagement contributing to learning outcomes between these instructional approaches. In classrooms, direct instruction comprehensively describes academic content, while active learning provides structured opportunities for learners to explore, apply, and manipulate content. Synaptic plasticity and its modulation by arousal or novelty are central to all learning and both approaches. As a form of social learning, direct instruction relies upon working memory. The reinforcement learning circuit, associated agency, curiosity, and peer-to-peer social interactions combine to enhance motivation, improve retention, and build higher-order-thinking skills in active learning environments. When working memory becomes overwhelmed, additionally engaging the reinforcement learning circuit improves retention, providing an explanation for the benefits of active learning. This analysis provides a mechanistic examination of how emerging neuroscience principles might inform pedagogical choices at all educational levels.

Impoverished details with preserved gist in remote and recent spatial memory following hippocampal and fornix lesions

INTRODUCTION

Cognitive Map Theory predicts that the hippocampus (HPC) plays a specialized, time-invariant role in supporting allocentric spatial memory, while Standard Consolidation Theory makes the competing prediction that the HPC plays a time-limited role, with more remote memories gaining independence of HPC function. These theories, however, are largely informed by the results of laboratory-based tests that are unlikely to simulate the demands of representing real-world environments in humans. Validation of these theories is further limited by an overall focus on spatial memory of newly encountered environments and on individuals with extensive lesions to the HPC and to surrounding medial temporal lobe (MTL) regions. The current study incorporates naturalistic tests of spatial memory based on recently and remotely encountered environments navigated by individuals with lesions to the HPC/MTL or that are limited to the HPC's major output, the fornix.

METHODS

Four participants with bilateral HPC/MTL and/or fornix lesions drew sketch maps of recently and remotely experienced neighbourhoods and houses. Tests of the appearance, distances, and routes between landmarks from the same real-world environments were also administered. Performance on the tasks was compared to that of control participants closely matched in terms of exposure to the same neighbourhoods and home environments as well as to actual maps.

RESULTS

The performance of individuals with fornix/MTL lesions was found to be largely comparable to that of controls on objective tests of spatial memory, other than one case who was impaired on remote and recent conditions for several tasks. The nature of deficits in recent and remote spatial memory were further revealed on house floorplan drawings, which contained spatial distortions, room/structure transpositions, and omissions, and on neighbourhood sketch maps, which were intact in terms of overall layout but sparse in details such as landmarks.

CONCLUSION

Lab-based tests of spatial memory of newly learned environments are unlikely to fully capture patterns of spared and impaired representations of real-world environments (e.g., peripheral features, configurations). Naturalistic tasks, including generative drawing tasks, indicate that contrary to Cognitive Map Theory, neither HPC nor MTL are critical for allocentric gross representations of large-scale environments. Conversely, the HPC appears critical for representing detailed spatial information of local naturalistic environments and environmental objects regardless of the age of the memory, contrary to Standard Consolidation Theory.

Prediction error and event segmentation in episodic memory

Organizing the continuous flow of experiences into meaningful events is a crucial prerequisite for episodic memory. Prediction error and event segmentation both play important roles in supporting the genesis of meaningful mnemonic representations of events. We review theoretical contributions discussing the relationship between prediction error and event segmentation, as well as literature on episodic memory related to prediction error and event segmentation. We discuss the extent of overlap of mechanisms underlying memory emergence through prediction error and event segmentation, with a specific focus on attention and working memory. Finally, we identify areas in research that are currently developing and suggest future directions. We provide an overview of mechanisms underlying memory formation through predictions, violations of predictions, and event segmentation.

Emotional arousal lingers in time to bind discrete episodes in memory

Temporal stability and change in neutral contexts can transform continuous experiences into distinct and memorable events. However, less is known about how shifting emotional states influence these memory processes, despite ample evidence that emotion impacts non-temporal aspects of memory. Here, we examined if emotional stimuli influence temporal memory for recent event sequences. Participants encoded lists of neutral images while listening to auditory tones. At regular intervals within each list, participants heard emotional positive, negative, or neutral sounds, which served as "emotional event boundaries" that divided each sequence into discrete events. Temporal order memory was tested for neutral item pairs that either spanned an emotional sound or were encountered within the same auditory event. Encountering a highly arousing event boundary led to faster response times for items encoded within the next event. Critically, we found that highly arousing sounds had different effects on binding ongoing versus ensuing sequential representations in memory. Specifically, highly arousing sounds were significantly more likely to enhance temporal order memory for ensuing information compared to information that spanned those boundaries, especially for boundaries with negative valence. These findings suggest that within aversive emotional contexts, fluctuations in arousal help shape the temporal organisation of events in memory.

Generalization of cognitive maps across space and time

Prominent theories posit that associative memory structures, known as cognitive maps, support flexible generalization of knowledge across cognitive domains. Here, we evince a representational account of cognitive map flexibility by quantifying how spatial knowledge formed one day was used predictively in a temporal sequence task 24 hours later, biasing both behavior and neural response. Participants learned novel object locations in distinct virtual environments. After learning, hippocampus and ventromedial prefrontal cortex (vmPFC) represented a cognitive map, wherein neural patterns became more similar for same-environment objects and more discriminable for different-environment objects. Twenty-four hours later, participants rated their preference for objects from spatial learning; objects were presented in sequential triplets from either the same or different environments. We found that preference response times were slower when participants transitioned between same- and different-environment triplets. Furthermore, hippocampal spatial map coherence tracked behavioral slowing at the implicit sequence transitions. At transitions, predictive reinstatement of virtual environments decreased in anterior parahippocampal cortex. In the absence of such predictive reinstatement after sequence transitions, hippocampus and vmPFC responses increased, accompanied by hippocampal-vmPFC functional decoupling that predicted individuals' behavioral slowing after a transition. Collectively, these findings reveal how expectations derived from spatial experience generalize to support temporal prediction.

Goal-oriented representations in the human hippocampus during planning and navigation

Recent work in cognitive and systems neuroscience has suggested that the hippocampus might support planning, imagination, and navigation by forming cognitive maps that capture the abstract structure of physical spaces, tasks, and situations. Navigation involves disambiguating similar contexts, and the planning and execution of a sequence of decisions to reach a goal. Here, we examine hippocampal activity patterns in humans during a goal-directed navigation task to investigate how contextual and goal information are incorporated in the construction and execution of navigational plans. During planning, hippocampal pattern similarity is enhanced across routes that share a context and a goal. During navigation, we observe prospective activation in the hippocampus that reflects the retrieval of pattern information related to a key-decision point. These results suggest that, rather than simply representing overlapping associations or state transitions, hippocampal activity patterns are shaped by context and goals.

Evidence That Event Boundaries Are Access Points for Memory Retrieval

When recalling memories, we often scan information-rich continuous episodes, for example, to find our keys. How does our brain access and search through those memories? We suggest that high-level structure, marked by event boundaries, guides us through this process: In our computational model, memory scanning is sped up by skipping ahead to the next event boundary upon reaching a decision threshold. In adult Mechanical Turk workers from the United States, we used a movie (normed for event boundaries; Study 1, N = 203) to prompt memory scanning of movie segments for answers (Study 2, N = 298) and mental simulation (Study 3, N = 100) of these segments. Confirming model predictions, we found that memory-scanning times varied as a function of the number of event boundaries within a segment and the distance of the search target to the previous boundary (the key diagnostic parameter). Mental simulation times were also described by a skipping process with a higher skipping threshold than memory scanning. These findings identify event boundaries as access points to memory.

From cognitive maps to spatial schemas

A schema refers to a structured body of prior knowledge that captures common patterns across related experiences. Schemas have been studied separately in the realms of episodic memory and spatial navigation across different species and have been grounded in theories of memory consolidation, but there has been little attempt to integrate our understanding across domains, particularly in humans. We propose that experiences during navigation with many similarly structured environments give rise to the formation of spatial schemas (for example, the expected layout of modern cities) that share properties with but are distinct from cognitive maps (for example, the memory of a modern city) and event schemas (such as expected events in a modern city) at both cognitive and neural levels. We describe earlier theoretical frameworks and empirical findings relevant to spatial schemas, along with more targeted investigations of spatial schemas in human and non-human animals. Consideration of architecture and urban analytics, including the influence of scale and regionalization, on different properties of spatial schemas may provide a powerful approach to advance our understanding of spatial schemas.

Entorhinal grid-like codes and time-locked network dynamics track others navigating through space

Navigating through crowded, dynamically changing environments requires the ability to keep track of other individuals. Grid cells in the entorhinal cortex are a central component of self-related navigation but whether they also track others' movement is unclear. Here, we propose that entorhinal grid-like codes make an essential contribution to socio-spatial navigation. Sixty human participants underwent functional magnetic resonance imaging (fMRI) while observing and re-tracing different paths of a demonstrator that navigated a virtual reality environment. Results revealed that grid-like codes in the entorhinal cortex tracked the other individual navigating through space. The activity of grid-like codes was time-locked to increases in co-activation and entorhinal-cortical connectivity that included the striatum, the hippocampus, parahippocampal and right posterior parietal cortices. Surprisingly, the grid-related effects during observation were stronger the worse participants performed when subsequently re-tracing the demonstrator's paths. Our findings suggests that network dynamics time-locked to entorhinal grid-cell-related activity might serve to distribute information about the location of others throughout the brain.

Image of a City through Big Data Analytics: Colombo from the Lens of Geo-Coded Social Media Data

The image of a city represents the sum of beliefs, ideas, and impressions that people have of that city. Mostly, city images are assessed through direct or indirect interviews and cognitive mapping exercises. Such methods consume more time and effort and are limited to a small number of people. However, recently, people tend to use social media to express their thoughts and experiences of a place. Taking this into consideration, this paper attempts to explore city images through social media big data, considering Colombo, Sri Lanka, as the testbed. The aim of the study is to examine the image of a city through Lynchian elements—i.e., landmarks, paths, nodes, edges, and districts—by using community sentiments expressed and images posted on social media platforms. For that, this study conducted various analyses—i.e., descriptive, image processing, sentiment, popularity, and geo-coded social media analyses. The study findings revealed that: (a) the community sentiments toward the same landmarks, paths, nodes, edges, and districts change over time; (b) decisions related to locating landmarks, paths, nodes, edges, and districts have a significant impact on community cognition in perceiving cities; and (c) geo-coded social media data analytics is an invaluable approach to capture the image of a city. The study informs urban authorities in their placemaking efforts by introducing a novel methodological approach to capture an image of a city.

Multisensory synchrony of contextual boundaries affects temporal order memory, but not encoding or recognition

We memorize our daily life experiences, which are often multisensory in nature, by segmenting them into distinct event models, in accordance with perceived contextual or situational changes. However, very little is known about how multisensory boundaries affect segmentation, as most studies have focused on unisensory (visual or audio) segmentation. In three experiments, we investigated the effect of multisensory boundaries on segmentation in memory and perception. In Experiment 1, participants encoded lists of pictures while audio and visual contexts changed synchronously or asynchronously. After each list, we tested recognition and temporal associative memory for pictures that were encoded in the same audio-visual context or that crossed a synchronous or an asynchronous multisensory change. We found no effect of multisensory synchrony for recognition memory: synchronous and asynchronous changes similarly impaired recognition for pictures encoded at those changes, compared to pictures encoded further away from those changes. Multisensory synchrony did affect temporal associative memory, which was worse for pictures encoded at synchronous than at asynchronous changes. Follow up experiments showed that this effect was not due to the higher dimensionality of multisensory over unisensory contexts (Experiment 2), nor that it was due to the temporal unpredictability of contextual changes inherent to Experiment 1 (Experiment 3). We argue that participants formed situational expectations through multisensory synchronicity, such that synchronous multisensory changes deviated more strongly from those expectations than asynchronous changes. We discuss our findings in light of supportive and conflicting findings of uni- and multi-sensory segmentation.

Navigational roots of spatial and temporal memory structure

Our minds are constantly in transit, from the present to the past to the future, across places we have and have not directly experienced. Nevertheless, memories of our mental time travel are not organized continuously and are adaptively chunked into contexts and episodes. In this paper, I will review evidence that suggests that spatial boundary representations play a critical role in providing structure to both our spatial and temporal memories. I will illustrate the intimate connection between hippocampal spatial mapping and temporal sequencing of episodic memory to propose that high-level cognitive processes like mental time travel and conceptual mapping are rooted in basic navigational mechanisms that we humans and nonhuman animals share. Our neuroscientific understanding of hippocampal function across species may provide new insight into the origins of even the most uniquely human cognitive abilities.

Grounding the Attentional Boost Effect in Events and the Efficient Brain

Attention and memory for everyday experiences vary over time, wherein some moments are better attended and subsequently better remembered than others. These effects have been demonstrated in naturalistic viewing tasks with complex and relatively uncontrolled stimuli, as well as in more controlled laboratory tasks with simpler stimuli. For example, in the attentional boost effect (ABE), participants perform two tasks at once: memorizing a series of briefly presented stimuli (e.g., pictures of outdoor scenes) for a later memory test, and responding to other concurrently presented cues that meet pre-defined criteria (e.g., participants press a button for a blue target square and do nothing for a red distractor square). However, rather than increasing dual-task interference, attending to a target cue boosts, rather than impairs, subsequent memory for concurrently presented information. In this review we describe current data on the extent and limitations of the attentional boost effect and whether it may be related to activity in the locus coeruleus neuromodulatory system. We suggest that insight into the mechanisms that produce the attentional boost effect may be found in recent advances in the locus coeruleus literature and from understanding of how the neurocognitive system handles stability and change in everyday events. We consequently propose updates to an early account of the attentional boost effect, the dual-task interaction model, to better ground it in what is currently known about event cognition and the role that the LC plays in regulating brain states.

A generalized cortical activity pattern at internally generated mental context boundaries during unguided narrative recall

Current theory and empirical studies suggest that humans segment continuous experiences into events based on the mismatch between predicted and actual sensory inputs; detection of these 'event boundaries' evokes transient neural responses. However, boundaries can also occur at transitions between internal mental states, without relevant external input changes. To what extent do such 'internal boundaries' share neural response properties with externally driven boundaries? We conducted an fMRI experiment where subjects watched a series of short movies and then verbally recalled the movies, unprompted, in the order of their choosing. During recall, transitions between movies thus constituted major boundaries between internal mental contexts, generated purely by subjects' unguided thoughts. Following the offset of each recalled movie, we observed stereotyped spatial activation patterns in the default mode network, especially the posterior medial cortex, consistent across different movie contents and even across the different tasks of movie watching and recall. Surprisingly, the between-movie boundary patterns did not resemble patterns at boundaries between events within a movie. Thus, major transitions between mental contexts elicit neural phenomena shared across internal and external modes and distinct from within-context event boundary detection, potentially reflecting a cognitive state related to the flushing and reconfiguration of situation models.

Effects of spatial boundaries on episodic memory development

Children's spatial mapping starts out particularly sensitive to 3D wall-like boundaries and develops over early childhood to flexibly include other boundary types. This study investigated whether spatial boundaries influence children's episodic memory, as in adults, and whether this effect is modulated by boundary type. Eighty-one Korean children (34 girls, 36-84 months old) re-enacted a sequence of three discrete hiding events within a space containing one of three boundaries: 3D wall, aligned objects, or 2D line. Children's memory of events occurring on one side of the boundary developed earlier than those that crossed the boundary. At first, this interaction only applied to the 3D wall and extended to other boundary types with age, suggesting that children's changing spatial representations influence their episodic memory development.

Caught in the ACTS: Defining Abstract Cognitive Task Sequences as an Independent Process

Cognitive neuroscience currently conflates the study of serial responses (e.g., delay match to sample/nonsample, n-back) with the study of sequential operations. In this essay, our goal is to define and disentangle the latter, termed abstract cognitive task sequences (ACTS). Existing literatures address tasks requiring serial events, including procedural learning of implicit motor responses, statistical learning of predictive relationships, and judgments of attributes. These findings do not describe the behavior and underlying mechanism required to succeed at remembering to evaluate color, then shape; or to multiply, then add. A new literature is needed to characterize these sorts of second-order cognitive demands of studying a sequence of operations. Our second goal is to characterize gaps in knowledge related to ACTS that merit further investigation. In the following sections, we define more precisely what we mean by ACTS and suggest research questions' further investigation would be positioned to address.

Neurophysiological Evidence for Cognitive Map Formation during Sequence Learning

Humans deftly parse statistics from sequences. Some theories posit that humans learn these statistics by forming cognitive maps, or underlying representations of the latent space which links items in the sequence. Here, an item in the sequence is a node, and the probability of transitioning between two items is an edge. Sequences can then be generated from walks through the latent space, with different spaces giving rise to different sequence statistics. Individual or group differences in sequence learning can be modeled by changing the time scale over which estimates of transition probabilities are built, or in other words, by changing the amount of temporal discounting. Latent space models with temporal discounting bear a resemblance to models of navigation through Euclidean spaces. However, few explicit links have been made between predictions from Euclidean spatial navigation and neural activity during human sequence learning. Here, we use a combination of behavioral modeling and intracranial encephalography (iEEG) recordings to investigate how neural activity might support the formation of space-like cognitive maps through temporal discounting during sequence learning. Specifically, we acquire human reaction times from a sequential reaction time task, to which we fit a model that formulates the amount of temporal discounting as a single free parameter. From the parameter, we calculate each individual's estimate of the latent space. We find that neural activity reflects these estimates mostly in the temporal lobe, including areas involved in spatial navigation. Similar to spatial navigation, we find that low-dimensional representations of neural activity allow for easy separation of important features, such as modules, in the latent space. Lastly, we take advantage of the high temporal resolution of iEEG data to determine the time scale on which latent spaces are learned. We find that learning typically happens within the first 500 trials, and is modulated by the underlying latent space and the amount of temporal discounting characteristic of each participant. Ultimately, this work provides important links between behavioral models of sequence learning and neural activity during the same behavior, and contextualizes these results within a broader framework of domain general cognitive maps.

Homing tasks and distance matching tasks reveal different types of perceptual variables associated with perceiving self-motion during over-ground locomotion

Self-motion perception refers to the ability to perceive how the body is moving through the environment. Perception of self-motion has been shown to depend upon the locomotor action patterns used to move the body through the environment. Two separate lines of enquiry have led to the establishment of two distinct theories regarding this effect. One theory has proposed that distances travelled during locomotion are perceived via higher order perceptual variables detected by the haptic perceptual system. This theory proposes that two higher order haptic perceptual variables exist, and that the implication of one of these variables depends upon the type of gait pattern that is used. A second theory proposes that self-motion is perceived via a higher order perceptual variable termed multimodally specified energy expenditure (MSEE). This theory proposes that the effect of locomotor actions patterns upon self-motion perception is related to changes in the metabolic cost of locomotion per unit of perceptually specified traversed distance. Here, we test the hypothesis that the development of these distinct theories is the result of different choices in methodology. The theory of gait type has been developed based largely on the results of homing tasks, whereas the effect of MSEE has been developed based on the results of distance matching tasks. Here we test the hypothesis that the seemly innocuous change in experimental design from using a homing task to using a distance matching task changes the type of perceptual variables implicated in self-motion perception. To test this hypothesis, we closely replicated a recent study of the effect of gait type in all details bar one-we investigated a distance matching task rather than a homing task. As hypothesized, this change yielded results consistent with the predictions of MSEE, and distinct from gait type. We further show that, unlike the effect of gait type, the effect of MSEE is unaffected by the availability of vision. In sum, our findings support the existence of two distinct types of higher order perceptual variables in self-motion perception. We discuss the roles of these two types of perceptual variables in supporting effective human wayfinding.

Predicting speech from a cortical hierarchy of event-based time scales

How do predictions in the brain incorporate the temporal unfolding of context in our natural environment? We here provide evidence for a neural coding scheme that sparsely updates contextual representations at the boundary of events. This yields a hierarchical, multilayered organization of predictive language comprehension. Training artificial neural networks to predict the next word in a story at five stacked time scales and then using model-based functional magnetic resonance imaging, we observe an event-based “surprisal hierarchy” evolving along a temporoparietal pathway. Along this hierarchy, surprisal at any given time scale gated bottom-up and top-down connectivity to neighboring time scales. In contrast, surprisal derived from continuously updated context influenced temporoparietal activity only at short time scales. Representing context in the form of increasingly coarse events constitutes a network architecture for making predictions that is both computationally efficient and contextually diverse.

Predicting speech from a cortical hierarchy of event-based time scales

How do predictions in the brain incorporate the temporal unfolding of context in our natural environment? We here provide evidence for a neural coding scheme that sparsely updates contextual representations at the boundary of events. This yields a hierarchical, multilayered organization of predictive language comprehension. Training artificial neural networks to predict the next word in a story at five stacked time scales and then using model-based functional magnetic resonance imaging, we observe an event-based “surprisal hierarchy” evolving along a temporoparietal pathway. Along this hierarchy, surprisal at any given time scale gated bottom-up and top-down connectivity to neighboring time scales. In contrast, surprisal derived from continuously updated context influenced temporoparietal activity only at short time scales. Representing context in the form of increasingly coarse events constitutes a network architecture for making predictions that is both computationally efficient and contextually diverse.

The Hippocampal Horizon: Constructing and Segmenting Experience for Episodic Memory

How do we recollect specific events that have occurred during continuous ongoing experience? There is converging evidence from non-human animals that spatially modulated cellular activity of the hippocampal formation supports the construction of ongoing events. On the other hand, recent human oriented event cognition models have outlined that our experience is segmented into discrete units, and that such segmentation can operate on shorter or longer timescales. Here, we describe a unification of how these dynamic physiological mechanisms of the hippocampus relate to ongoing externally and internally driven event segmentation, facilitating the demarcation of specific moments during experience. Our cross-species interdisciplinary approach offers a novel perspective in the way we construct and remember specific events, leading to the generation of many new hypotheses for future research.

The human brain uses spatial schemas to represent segmented environments

Humans and animals use cognitive maps to represent the spatial structure of the environment. Although these maps are typically conceptualized as extending in an equipotential manner across known space, psychological evidence suggests that people mentally segment complex environments into subspaces. To understand the neurocognitive mechanisms behind this operation, we familiarized participants with a virtual courtyard that was divided into two halves by a river; we then used behavioral testing and fMRI to understand how spatial locations were encoded within this environment. Participants' spatial judgments and multivoxel activation patterns were affected by the division of the courtyard, indicating that the presence of a boundary can induce mental segmentation even when all parts of the environment are co-visible. In the hippocampus and occipital place area (OPA), the segmented organization of the environment manifested in schematic spatial codes that represented geometrically equivalent locations in the two subspaces as similar. In the retrosplenial complex (RSC), responses were more consistent with an integrated spatial map. These results demonstrate that people use both local spatial schemas and integrated spatial maps to represent segmented environment. We hypothesize that schematization may serve as a general mechanism for organizing complex knowledge structures in terms of their component elements.

Replay in minds and machines

Experience-related brain activity patterns reactivate during sleep, wakeful rest, and brief pauses from active behavior. In parallel, machine learning research has found that experience replay can lead to substantial performance improvements in artificial agents. Together, these lines of research suggest replay has a variety of computational benefits for decision-making and learning. Here, we provide an overview of putative computational functions of replay as suggested by machine learning and neuroscientific research. We show that replay can lead to faster learning, less forgetting, reorganization or augmentation of experiences, and support planning and generalization. In addition, we highlight the benefits of reactivating abstracted internal representations rather than veridical memories, and discuss how replay could provide a mechanism to build internal representations that improve learning and decision-making.

Evolution, Emotion, and Episodic Engagement

Although rodent research provides important insights into neural correlates of human psychology, new cortical areas, connections, and cognitive abilities emerged during primate evolution, including human evolution. Comparison of human brains with those of nonhuman primates reveals two aspects of human brain evolution particularly relevant to emotional disorders: expansion of homotypical association areas and expansion of the hippocampus. Two uniquely human cognitive capacities link these phylogenetic developments with emotion: a subjective sense of participating in and reexperiencing remembered events and a limitless capacity to imagine details of future events. These abilities provided evolving humans with selective advantages, but they also created proclivities for emotional problems. The first capacity evokes the "reliving" of past events in the "here-and-now," accompanied by emotional responses that occurred during memory encoding. It contributes to risk for stress-related syndromes, such as posttraumatic stress disorder. The second capacity, an ability to imagine future events without temporal limitations, facilitates flexible, goal-related behavior by drawing on and creating a uniquely rich array of mental representations. It promotes goal achievement and reduces errors, but the mental construction of future events also contributes to developmental aspects of anxiety and mood disorders. With maturation of homotypical association areas, the concrete concerns of childhood expand to encompass the abstract apprehensions of adolescence and adulthood. These cognitive capacities and their dysfunction are amenable to a research agenda that melds experimental therapeutic interventions, cognitive neuropsychology, and developmental psychology in both humans and nonhuman primates.

Geometric models reveal behavioural and neural signatures of transforming experiences into memories

How do we preserve and distort our ongoing experiences when encoding them into episodic memories? The mental contexts in which we interpret experiences are often person-specific, even when the experiences themselves are shared. Here we develop a geometric framework for mathematically characterizing the subjective conceptual content of dynamic naturalistic experiences. We model experiences and memories as trajectories through word-embedding spaces whose coordinates reflect the universe of thoughts under consideration. Memory encoding can then be modelled as geometrically preserving or distorting the 'shape' of the original experience. We applied our approach to data collected as participants watched and verbally recounted a television episode while undergoing functional neuroimaging. Participants' recountings preserved coarse spatial properties (essential narrative elements) but not fine spatial scale (low-level) details of the episode's trajectory. We also identified networks of brain structures sensitive to these trajectory shapes.

The Continuity of Context: A Role for the Hippocampus

Tracking moment-to-moment change in input and detecting change sufficient to require altering behavior is crucial to survival. Here, we discuss how the brain evaluates change over time, focusing on the hippocampus and its role in tracking context. We leverage the anatomy and physiology of the hippocampal longitudinal axis, re-entrant loops, and amorphous networks to account for stimulus equivalence and the updating of an organism's sense of its context. Place cells have a central role in tracking contextual continuities and discontinuities across multiple scales, a capacity beyond current models of pattern separation and completion. This perspective highlights the critical role of the hippocampus in both spatial cognition and episodic memory: tracking change and detecting boundaries separating one context, or episode, from another.

Conspecific Presence Improves Episodic-Like Memory in Rats

A number of studies have provided evidence that animals, including rats, remember past episodes. However, few experiments have addressed episodic-like memory from a social perspective. In the present study, we evaluated Wistar rats in the WWWhen/ELM task as single setups and in dyads, applying a long retention interval. We also investigated behaviors that could subserve the emergence of this type of memory. We found that only rats tested in the social setting were able to recollect an integrated episodic-like memory that lasted 24 h. Additionally, rats in dyads presented higher levels of exploration during the task. When exposed to the testing environment, the dyads exhibited affiliative behavior toward each other and presented fewer anxiety-like responses. Our findings indicate that the presence of a conspecific could act as a facilitating factor in memory evaluations based on spontaneous exploration of objects and provide empirical support for applying more naturalistic settings in investigations of episodic-like memory in rats.

Structuring Knowledge with Cognitive Maps and Cognitive Graphs

Humans and animals use mental representations of the spatial structure of the world to navigate. The classical view is that these representations take the form of Euclidean cognitive maps, but alternative theories suggest that they are cognitive graphs consisting of locations connected by paths. We review evidence suggesting that both map-like and graph-like representations exist in the mind/brain that rely on partially overlapping neural systems. Maps and graphs can operate simultaneously or separately, and they may be applied to both spatial and nonspatial knowledge. By providing structural frameworks for complex information, cognitive maps and cognitive graphs may provide fundamental organizing schemata that allow us to navigate in physical, social, and conceptual spaces.

Perception of urban subdivisions in pedestrian movement simulation

The perception of urban subdivisions, deriving from regionalisation processes and the identification of separating elements (barriers), has proven to dynamically shape peoples’ cognitive representations of space and route choice behaviour in cities. However, existing Agent-Based Models (ABMs) for pedestrian simulation have not accounted for these particular cognitive mapping processes. The aim of this paper is to explore the behaviour of pedestrian agents endowed with knowledge about urban subdivisions. Drawing from literature in spatial cognition, we adapted a region-based route choice model, which contemplates a high- and a local planning level, and advanced a barrier-based route choice model, wherein the influence of separating elements is manipulated. Finally, we combined these two approaches in a region-barrier based model. The patterns emerging from the movement of agents employing such approaches were examined in the city centres of London and Paris. The introduction of regions in the routing mechanisms reduced the unbalanced concentration of agents across the street network brought up by the widely employed least cumulative angular change model (-.08 Gini coefficient). The inclusion of barriers further raised the dispersal of the agents through secondary roads, while leading agents to walk along waterfronts and across parks; it also yielded a more regular usage of pedestrian roads. Moreover, the region- and the region-barrier based routes showed deviation ratio values from the road distance shortest path (region-based: 1.18 London, 1.16 Paris, region-barrier based: 1.43 London, 1.33 Paris) consistent with empirical observations from pedestrian behaviour research. A further evaluation of the model with macro-level observational data may enhance the understanding of pedestrian dynamics and help tuning the interplay amongst urban salient elements at the agent level. Yet, we consider the movement flows arising from our current implementation insightful for assessing the distribution of pedestrians and testing possible interventions for the design of legible and walkable spaces.

An Important Step toward Understanding the Role of Body-based Cues on Human Spatial Memory for Large-Scale Environments

Moving our body through space is fundamental to human navigation; however, technical and physical limitations have hindered our ability to study the role of these body-based cues experimentally. We recently designed an experiment using novel immersive virtual-reality technology, which allowed us to tightly control the availability of body-based cues to determine how these cues influence human spatial memory [Huffman, D. J., & Ekstrom, A. D. A modality-independent network underlies the retrieval of large-scale spatial environments in the human brain. Neuron, 104, 611-622, 2019]. Our analysis of behavior and fMRI data revealed a similar pattern of results across a range of body-based cues conditions, thus suggesting that participants likely relied primarily on vision to form and retrieve abstract, holistic representations of the large-scale environments in our experiment. We ended our paper by discussing a number of caveats and future directions for research on the role of body-based cues in human spatial memory. Here, we reiterate and expand on this discussion, and we use a commentary in this issue by A. Steel, C. E. Robertson, and J. S. Taube (Current promises and limitations of combined virtual reality and functional magnetic resonance imaging research in humans: A commentary on Huffman and Ekstrom (2019). Journal of Cognitive Neuroscience, 2020) as a helpful discussion point regarding some of the questions that we think will be the most interesting in the coming years. We highlight the exciting possibility of taking a more naturalistic approach to study the behavior, cognition, and neuroscience of navigation. Moreover, we share the hope that researchers who study navigation in humans and nonhuman animals will synergize to provide more rapid advancements in our understanding of cognition and the brain.

Culture influences how people divide continuous sensory experience into events

Everyday experience is divided into meaningful events as a part of human perception. Current accounts of this process, known as event segmentation, focus on how characteristics of the experience (e.g., situation changes) influence segmentation. However, characteristics of the viewers themselves have been largely neglected. We test whether one such viewer characteristic, their cultural background, impacts online event segmentation. Culture could impact event segmentation (1) by emphasizing different aspects of experiences as being important for comprehension, memory, and communication, and (2) by providing different exemplars of how everyday activities are performed, which objects are likely to be used, and how scenes are laid out. Indian and US viewers (N = 152) identified events in everyday activities (e.g., making coffee) recorded in Indian and US settings. Consistent with their cultural preference for analytical processing, US viewers segmented the activities into more events than did Indian viewers. Furthermore, event boundaries identified by US viewers were more strongly associated with visual changes, whereas boundaries identified by Indian viewers were more strongly associated with goal changes. There was no evidence that familiarity with an activity impacted segmentation. Thus, culture impacts event perception by altering the types of information people prioritize when dividing experience into meaningful events.

Reward prediction errors create event boundaries in memory

We remember when things change. Particularly salient are experiences where there is a change in rewards, eliciting reward prediction errors (RPEs). How do RPEs influence our memory of those experiences? One idea is that this signal directly enhances the encoding of memory. Another, not mutually exclusive, idea is that the RPE signals a deeper change in the environment, leading to the mnemonic separation of subsequent experiences from what came before, thereby creating a new latent context and a more separate memory trace. We tested this in four experiments where participants learned to predict rewards associated with a series of trial-unique images. High-magnitude RPEs indicated a change in the underlying distribution of rewards. To test whether these large RPEs created a new latent context, we first assessed recognition priming for sequential pairs that included a high-RPE event or not (Exp. 1: n = 27 & Exp. 2: n = 83). We found evidence of recognition priming for the high-RPE event, indicating that the high-RPE event is bound to its predecessor in memory. Given that high-RPE events are themselves preferentially remembered (Rouhani, Norman, & Niv, 2018), we next tested whether there was an event boundary across a high-RPE event (i.e., excluding the high-RPE event itself; Exp. 3: n = 85). Here, sequential pairs across a high RPE no longer showed recognition priming whereas pairs within the same latent reward state did, providing initial evidence for an RPE-modulated event boundary. We then investigated whether RPE event boundaries disrupt temporal memory by asking participants to order and estimate the distance between two events that had either included a high-RPE event between them or not (Exp. 4). We found (n = 49) and replicated (n = 77) worse sequence memory for events across a high RPE. In line with our recognition priming results, we did not find sequence memory to be impaired between the high-RPE event and its predecessor, but instead found worse sequence memory for pairs across a high-RPE event. Moreover, greater distance between events at encoding led to better sequence memory for events across a low-RPE event, but not a high-RPE event, suggesting separate mechanisms for the temporal ordering of events within versus across a latent reward context. Altogether, these findings demonstrate that high-RPE events are both more strongly encoded, show intact links with their predecessor, and act as event boundaries that interrupt the sequential integration of events. We captured these effects in a variant of the Context Maintenance and Retrieval model (CMR; Polyn, Norman, & Kahana, 2009), modified to incorporate RPEs into the encoding process.

Dynamic representations in networked neural systems

A group of neurons can generate patterns of activity that represent information about stimuli; subsequently, the group can transform and transmit activity patterns across synapses to spatially distributed areas. Recent studies in neuroscience have begun to independently address the two components of information processing: the representation of stimuli in neural activity and the transmission of information in networks that model neural interactions. Yet only recently are studies seeking to link these two types of approaches. Here we briefly review the two separate bodies of literature; we then review the recent strides made to address this gap. We continue with a discussion of how patterns of activity evolve from one representation to another, forming dynamic representations that unfold on the underlying network. Our goal is to offer a holistic framework for understanding and describing neural information representation and transmission while revealing exciting frontiers for future research.

Mechanisms of memory: An intermediate level of analysis and organization

Research in the last five years has made great strides toward mechanistic explanations of how the brain enables memory. This progress builds upon decades of research from two complementary strands: a Levels of Analysis approach and a Levels of Organization approach. We review how research in cognitive psychology and cognitive neuroscience under these two approaches has recently converged on mechanistic, brain-based theories, couched at the optimal level for explaining cognitive phenomena - the intermediate level. Furthermore, novel empirical and data analysis techniques are now providing ways to test these theories' predictions, a crucial step in unraveling the mechanisms of memory.

For humans navigating without vision, navigation depends upon the layout of mechanically contacted ground surfaces

Navigation can be haptically guided. In specific, tissue deformations arising from both limb motions during locomotion (i.e., gait patterns) and mechanical interactions between the limbs and the environment can convey information, detected by the haptic perceptual system, about how the body is moving relative to the environment. Here, we test hypotheses concerning the properties of mechanically contacted environments relevant to navigation of this kind. We studied blindfolded participants implicitly learning to perceive their location within environments that were physically encountered via walking on, stepping on, and probing ground surfaces with a cane. Environments were straight-line paths with elevated sections where the path either narrowed or remained the same width. We formed hypotheses concerning how these two environments would affect spatial updating and reorientation processes. In the constant pathwidth environment, homing task accuracy was higher and a manipulation of the elevated surface, to be either unchanged or (unbeknown to participants) shortened, biased the performance. This was consistent with our hypothesis of a metric recalibration scaled to elevated surface extent. In the narrowing pathwidth environment, elevated surface shortening did not bias performance. This supported our hypothesis of positional recalibration resulting from contact with the leading edge of the elevated surface. We discuss why certain environmental properties, such as path-narrowing, have significance for how one becomes implicitly oriented the surrounding environment.

Zooming In and Out on One's Life: Autobiographical Representations at Multiple Time Scales

The ability to decouple from the present environment and explore other times is a central feature of the human mind. Research in cognitive psychology and neuroscience has shown that the personal past and future is represented at multiple timescales and levels of resolution, from broad lifetime periods that span years to short-time slices of experience that span seconds. Here, I review this evidence and propose a theoretical framework for understanding mental time travel as the capacity to flexibly navigate hierarchical layers of autobiographical representations. On this view, past and future thoughts rely on two main systems-event simulation and autobiographical knowledge-that allow us to represent experiential contents that are decoupled from sensory input and to place these on a personal timeline scaffolded from conceptual knowledge of the content and structure of our life. The neural basis of this cognitive architecture is discussed, emphasizing the possible role of the medial pFC in integrating layers of autobiographical representations in the service of mental time travel.

The prevalence and importance of statistical learning in human cognition and behavior

Statistical learning, the ability to extract regularities from the environment over time, has become a topic of burgeoning interest. Its influence on behavior, spanning infancy to adulthood, has been demonstrated across a range of tasks, both those labeled as tests of statistical learning and those from other learning domains that predated statistical learning research or that are not typically considered in the context of that literature. Given this pervasive role in human cognition, statistical learning has the potential to reconcile seemingly distinct learning phenomena and may be an under-appreciated but important contributor to a wide range of human behaviors that are studied as unrelated processes, such as episodic memory and spatial navigation.

The dynamics of memory retrieval for internal mentation

Daily life situations often require people to remember internal mentation, such as their future plans or interpretations of events. Little is known, however, about the principles that govern memory for thoughts experienced during real-world events. In particular, it remains unknown whether factors that structure the retrieval of external stimuli also apply to thought recall, and whether some thought features affect their accessibility in memory. To examine these questions, we asked participants to undertake a walk on a university campus while wearing a lifelogging camera. They then received unexpected recall tasks about the thoughts they experienced during the walk, rated the phenomenological features of retrieved thoughts, and indicated the moment when they were experienced. Results showed that thought retrieval demonstrates primacy, recency, and temporal contiguity effects, and is also influenced by event boundaries. In addition, thoughts that involved planning and that were recurrent during the walk were more accessible in memory. Together, these results shed new light on the principles that govern memory for internal mentation and suggest that at least partially similar processes structure the retrieval of thoughts and stimuli from the external environment.