Memory as Perception of the Past: Compressed Time inMind and Brain

Minute-scale periodicity of neuronal firing in the human entorhinal cortex

Grid cells in the entorhinal cortex demonstrate spatially periodic firing, thought to provide a spatial map on behaviorally relevant length scales. Whether such periodicity exists for behaviorally relevant time scales in the human brain remains unclear. We investigate neuronal firing during a temporally continuous experience by presenting 14 neurosurgical patients with a video while recording neuronal activity from multiple brain regions. We report on neurons that modulate their activity in a periodic manner across different time scales-from seconds to many minutes, most prevalently in the entorhinal cortex. These neurons remap their dominant periodicity to shorter time scales during a subsequent recognition memory task. When the video is presented at two different speeds, a significant percentage of these temporally periodic cells (TPCs) maintain their time scales, suggesting a degree of invariance. The TPCs' temporal periodicity might complement the spatial periodicity of grid cells and together provide scalable spatiotemporal metrics for human experience.

Scanning a compressed ordered representation of the future

Several authors have suggested a deep symmetry between the psychological processes that underlie our ability to remember the past and make predictions about the future. The judgment of recency (JOR) task measures temporal order judgments for the past by presenting pairs of probe stimuli; participants choose the probe that was presented more recently. We performed a short-term relative JOR task and introduced a novel judgment of imminence (JOI) task to study temporal order judgments for the future. In the JOR task, participants were presented with a sequence of stimuli and asked to choose which of two probe stimuli was presented closer to the present. In the JOI task, participants were trained on a probabilistic sequence. After training, the sequence was interrupted with probe stimuli. Participants were asked to choose which of two probe stimuli was expected to be presented closer to the present. Replicating prior work on JOR, we found that RT results supported a backward self-terminating search model operating on a temporally organized representation of the past. We also showed that RT distributions are consistent with this model and that the temporally organized representation is compressed. Critically, results for the JOI task probing expectations of the future suggest a forward self-terminating search model operating on a temporally organized representation of the future. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

The tie that binds: temporal coding and adaptive emotion

Emotions are temporally dynamic, but the persistence of emotions outside of their appropriate temporal context is detrimental to health and well-being. Yet, precisely how temporal coding and emotional processing interact remains unclear. Recently unveiled temporal context representations in the hippocampus, entorhinal cortex (EC), and prefrontal cortex (PFC) support memory for what happened when. Here, we discuss how these neural temporal representations may interact with densely interconnected amygdala circuitry to shape emotional functioning. We propose a neuroanatomically informed framework suggesting that high-fidelity temporal representations linked to dynamic experiences promote emotion regulation and adaptive emotional memories. Then, we discuss how newly-identified synaptic and molecular features of amygdala-hippocampal projections suggest that intense, amygdala-dependent emotional responses may distort temporal-coding mechanisms. We conclude by identifying key avenues for future research.

Generalized Simultaneous Localization and Mapping (G-SLAM) as unification framework for natural and artificial intelligences: towards reverse engineering the hippocampal/entorhinal system and principles of high-level cognition

Simultaneous localization and mapping (SLAM) represents a fundamental problem for autonomous embodied systems, for which the hippocampal/entorhinal system (H/E-S) has been optimized over the course of evolution. We have developed a biologically-inspired SLAM architecture based on latent variable generative modeling within the Free Energy Principle and Active Inference (FEP-AI) framework, which affords flexible navigation and planning in mobile robots. We have primarily focused on attempting to reverse engineer H/E-S "design" properties, but here we consider ways in which SLAM principles from robotics may help us better understand nervous systems and emergent minds. After reviewing LatentSLAM and notable features of this control architecture, we consider how the H/E-S may realize these functional properties not only for physical navigation, but also with respect to high-level cognition understood as generalized simultaneous localization and mapping (G-SLAM). We focus on loop-closure, graph-relaxation, and node duplication as particularly impactful architectural features, suggesting these computational phenomena may contribute to understanding cognitive insight (as proto-causal-inference), accommodation (as integration into existing schemas), and assimilation (as category formation). All these operations can similarly be describable in terms of structure/category learning on multiple levels of abstraction. However, here we adopt an ecological rationality perspective, framing H/E-S functions as orchestrating SLAM processes within both concrete and abstract hypothesis spaces. In this navigation/search process, adaptive cognitive equilibration between assimilation and accommodation involves balancing tradeoffs between exploration and exploitation; this dynamic equilibrium may be near optimally realized in FEP-AI, wherein control systems governed by expected free energy objective functions naturally balance model simplicity and accuracy. With respect to structure learning, such a balance would involve constructing models and categories that are neither too inclusive nor exclusive. We propose these (generalized) SLAM phenomena may represent some of the most impactful sources of variation in cognition both within and between individuals, suggesting that modulators of H/E-S functioning may potentially illuminate their adaptive significances as fundamental cybernetic control parameters. Finally, we discuss how understanding H/E-S contributions to G-SLAM may provide a unifying framework for high-level cognition and its potential realization in artificial intelligences.

Spatialization of Time in the Entorhinal-Hippocampal System

The functional role of the entorhinal-hippocampal system has been a long withstanding mystery. One key theory that has become most popular is that the entorhinal-hippocampal system represents space to facilitate navigation in one's surroundings. In this Perspective article, I introduce a novel idea that undermines the inherent uniqueness of spatial information in favor of time driving entorhinal-hippocampal activity. Specifically, by spatializing events that occur in succession (i.e., across time), the entorhinal-hippocampal system is critical for all types of cognitive representations. I back up this argument with empirical evidence that hints at a role for the entorhinal-hippocampal system in non-spatial representation, and computational models of the logarithmic compression of time in the brain.

Temporal self-compression: Behavioral and neural evidence that past and future selves are compressed as they move away from the present

For centuries, great thinkers have struggled to understand how people represent a personal identity that changes over time. Insight may come from a basic principle of perception: as objects become distant, they also become less discriminable or “compressed.” In Studies 1–3, we demonstrate that people’s ratings of their own personality become increasingly less differentiated as they consider more distant past and future selves. In Study 4, we found neural evidence that the brain compresses self-representations with time as well. When we peer out a window, objects close to us are in clear view, whereas distant objects are hard to tell apart. We provide evidence that self-perception may operate similarly, with the nuance of distant selves increasingly harder to perceive.

A basic principle of perception is that as objects increase in distance from an observer, they also become logarithmically compressed in perception (i.e., not differentiated from one another), making them hard to distinguish. Could this basic principle apply to perhaps our most meaningful mental representation: our own sense of self? Here, we report four studies that suggest selves are increasingly non-discriminable with temporal distance from the present as well. In Studies 1 through 3, participants made trait ratings across various time points in the past and future. We found that participants compressed their past and future selves, relative to their present self. This effect was preferential to the self and could not be explained by the alternative possibility that individuals simply perceive arbitrary self-change with time irrespective of temporal distance. In Study 4, we tested for neural evidence of temporal self-compression by having participants complete trait ratings across time points while undergoing functional MRI. Representational similarity analysis was used to determine whether neural self-representations are compressed with temporal distance as well. We found evidence of temporal self-compression in areas of the default network, including medial prefrontal cortex and posterior cingulate cortex. Specifically, neural pattern similarity between self-representations was logarithmically compressed with temporal distance. Taken together, these findings reveal a “temporal self-compression” effect, with temporal selves becoming increasingly non-discriminable with distance from the present.

The diminishing precision of memory for time

Knowing when an event took place can provide several benefits to episodic memory, such as distinguishing among multiple traces, learning sequences of events, and guiding a search strategy. As a tool for understanding memory, time is particularly appealing given its ever-changing quality, the constant possibility to associate it with encoded events, and the ease with which it can be targeted at retrieval. Whereas studies of episodic retrieval typically employ categorical and probabilistic measures of retrieval success, characterizing a continuous feature such as time warrants measures particularly sensitive to the fidelity, or precision, of retrieved information. Here, we adapt a paradigm for assessing the fine-grained precision of retrieval to understand the nature of judging the time at which a memory was encoded. Subjects studied a series of pictures and then undertook a test in which they placed each picture, as precisely as possible, along a continuous time line representing the study list. Based on mixture-modeling analyses of the test response errors, the primary results were that temporal judgments were less accurate with passing time, and this change was due to diminished precision as opposed to an increased rate of guessing. Moreover, although we observed a negligible influence of guessing, subjects exhibited a clear effect of bias that favored recent responses. Together, in contrast to numerous studies of memory for other continuous features (e.g., color and location), our findings demonstrate a novel pattern of decision factors, suggesting that the retrieval of time might highlight distinct attributes of episodic memory.

Collective Strategy Condensation: When Envy Splits Societies

Human societies are characterized by three constituent features, besides others. (A) Options, as for jobs and societal positions, differ with respect to their associated monetary and non-monetary payoffs. (B) Competition leads to reduced payoffs when individuals compete for the same option as others. (C) People care about how they are doing relatively to others. The latter trait -the propensity to compare one's own success with that of others- expresses itself as envy. It is shown that the combination of (A)-(C) leads to spontaneous class stratification. Societies of agents split endogenously into two social classes, an upper and a lower class, when envy becomes relevant. A comprehensive analysis of the Nash equilibria characterizing a basic reference game is presented. Class separation is due to the condensation of the strategies of lower-class agents, which play an identical mixed strategy. Upper-class agents do not condense, following individualist pure strategies. The model and results are size-consistent, holding for arbitrary large numbers of agents and options. Analytic results are confirmed by extensive numerical simulations. An analogy to interacting confined classical particles is discussed.

Time as the fourth dimension in the hippocampus

Experiences of animal and human beings are structured by the continuity of space and time coupled with the uni-directionality of time. In addition to its pivotal position in spatial processing and navigation, the hippocampal system also plays a central, multiform role in several types of temporal processing. These include timing and sequence learning, at scales ranging from meso-scales of seconds to macro-scales of minutes, hours, days and beyond, encompassing the classical functions of short term memory, working memory, long term memory, and episodic memories (comprised of information about when, what, and where). This review article highlights the principal findings and behavioral contexts of experiments in rats showing: 1) timing: tracking time during delays by hippocampal 'time cells' and during free behavior by hippocampal-afferent lateral entorhinal cortex ramping cells; 2) 'online' sequence processing: activity coding sequences of events during active behavior; 3) 'off-line' sequence replay: during quiescence or sleep, orderly reactivation of neuronal assemblies coding awake sequences. Studies in humans show neurophysiological correlates of episodic memory comparable to awake replay. Neural mechanisms are discussed, including ion channel properties, plateau and ramping potentials, oscillations of excitation and inhibition of population activity, bursts of high amplitude discharges (sharp wave ripples), as well as short and long term synaptic modifications among and within cell assemblies. Specifically conceived neural network models will suggest processes supporting the emergence of scalar properties (Weber's law), and include different classes of feedforward and recurrent network models, with intrinsic hippocampal coding for 'transitions' (sequencing of events or places).

Sequence Memory in the Hippocampal-Entorhinal Region

Episodic memories are constructed from sequences of events. When recalling such a memory, we not only recall individual events, but we also retrieve information about how the sequence of events unfolded. Here, we focus on the role of the hippocampal-entorhinal region in processing and remembering sequences of events, which are thought to be stored in relational networks. We summarize evidence that temporal relations are a central organizational principle for memories in the hippocampus. Importantly, we incorporate novel insights from recent studies about the role of the adjacent entorhinal cortex in sequence memory. In rodents, the lateral entorhinal subregion carries temporal information during ongoing behavior. The human homologue is recruited during memory recall where its representations reflect the temporal relationships between events encountered in a sequence. We further introduce the idea that the hippocampal-entorhinal region might enable temporal scaling of sequence representations. Flexible changes of sequence progression speed could underlie the traversal of episodic memories and mental simulations at different paces. In conclusion, we describe how the entorhinal cortex and hippocampus contribute to remembering event sequences-a core component of episodic memory.

Self-induced class stratification in competitive societies of agents: Nash stability in the presence of envy

Envy, the inclination to compare rewards, can be expected to unfold when inequalities in terms of pay-off differences are generated in competitive societies. It is shown that increasing levels of envy lead inevitably to a self-induced separation into a lower and an upper class. Class stratification is Nash stable and strict, with members of the same class receiving identical rewards. Upper-class agents play exclusively pure strategies, all lower-class agents the same mixed strategy. The fraction of upper-class agents decreases progressively with larger levels of envy, until a single upper-class agent is left. Numerical simulations and a complete analytic treatment of a basic reference model, the shopping trouble model, are presented. The properties of the class-stratified society are universal and only indirectly controllable through the underlying utility function, which implies that class-stratified societies are intrinsically resistant to political control. Implications for human societies are discussed. It is pointed out that the repercussions of envy are amplified when societies become increasingly competitive.

Repetition attenuates the influence of recency on recognition memory: Behavioral and electrophysiological evidence

Studies of recognition memory often demonstrate a recency effect on behavioral performance, whereby response times (RTs) are faster for stimuli that were previously presented recently as opposed to more remotely in the past. One account of this relationship between performance and presentation lag posits that memories are accessed by serially searching backward in time, such that RT indicates the self-terminating moment of such a process. Here, we investigated the conditions under which this serial search gives way to more efficient means of retrieving memories. Event-related potentials (ERPs) were recorded during a continuous recognition task, in which subjects made binary old/new judgments to stimuli that were each presented up to four times across a range of lags. Stimulus repetition and shorter presentation lag both gave rise to speeded RTs, consistent with previous findings, and we novelly extend these effects to a robust latency measure of the left parietal ERP correlate of retrieval success. Importantly, the relationship between repetition and recency was further elucidated, such that repetition attenuated lag-related differences that were initially present in both the behavioral and neural latency data. These findings are consistent with the idea that an effortful search through recent memory can quickly be abandoned in favor of relying on more efficient "time-independent" cognitive processes or neural signals.

Mechanisms of neural organization and rhythmogenesis during hippocampal and cortical ripples

Neural activity during ripples has attracted great theoretical and experimental attention over the last three decades. Perhaps one reason for such interest is that ripples occur during quiet waking moments and during sleep, times when we reflect and dream about what has just occurred and what we expect to happen next. The hope is that understanding such 'offline' activity may yield insights into reflection, planning, and the purposes of sleep. This review focuses on the mechanisms by which neurons organize during these high-frequency events. In studying ripples, broader principles have emerged that relate intrinsic neural properties, network topology and synaptic plasticity in controlling neural activity. Ripples, therefore, serve as an excellent model for studying how properties of a neural network relate to neural dynamics. This article is part of the Theo Murphy meeting issue 'Memory reactivation: replaying events past, present and future'.

Generation of Scale-Invariant Sequential Activity in Linear Recurrent Networks

Sequential neural activity has been observed in many parts of the brain and has been proposed as a neural mechanism for memory. The natural world expresses temporal relationships at a wide range of scales. Because we cannot know the relevant scales a priori, it is desirable that memory, and thus the generated sequences, is scale invariant. Although recurrent neural network models have been proposed as a mechanism for generating sequences, the requirements for scale-invariant sequences are not known. This letter reports the constraints that enable a linear recurrent neural network model to generate scale-invariant sequential activity. A straightforward eigendecomposition analysis results in two independent conditions that are required for scale invariance for connectivity matrices with real, distinct eigenvalues. First, the eigenvalues of the network must be geometrically spaced. Second, the eigenvectors must be related to one another via translation. These constraints are easily generalizable for matrices that have complex and distinct eigenvalues. Analogous albeit less compact constraints hold for matrices with degenerate eigenvalues. These constraints, along with considerations on initial conditions, provide a general recipe to build linear recurrent neural networks that support scale-invariant sequential activity.

The what, where and how of delay activity

Working memory is characterized by neural activity that persists during the retention interval of delay tasks. Despite the ubiquity of this delay activity across tasks, species and experimental techniques, our understanding of this phenomenon remains incomplete. Although initially there was a narrow focus on sustained activation in a small number of brain regions, methodological and analytical advances have allowed researchers to uncover previously unobserved forms of delay activity various parts of the brain. In light of these new findings, this Review reconsiders what delay activity is, where in the brain it is found, what roles it serves and how it may be generated.

Mapping sequence structure in the human lateral entorhinal cortex

Remembering event sequences is central to episodic memory and presumably supported by the hippocampal-entorhinal region. We previously demonstrated that the hippocampus maps spatial and temporal distances between events encountered along a route through a virtual city (Deuker et al., 2016), but the content of entorhinal mnemonic representations remains unclear. Here, we demonstrate that multi-voxel representations in the anterior-lateral entorhinal cortex (alEC) - the human homologue of the rodent lateral entorhinal cortex - specifically reflect the temporal event structure after learning. Holistic representations of the sequence structure related to memory recall and the timeline of events could be reconstructed from entorhinal multi-voxel patterns. Our findings demonstrate representations of temporal structure in the alEC; dovetailing with temporal information carried by population signals in the lateral entorhinal cortex of navigating rodents and alEC activations during temporal memory retrieval. Our results provide novel evidence for the role of the alEC in representing time for episodic memory.

Five decades of US, UK, German and Dutch music charts show that cultural processes are accelerating

Analysing the timeline of US, UK, German and Dutch music charts, we find that the evolution of album lifetimes and of the size of weekly rank changes provide evidence for an acceleration of cultural processes. For most of the past five decades, number one albums needed more than a month to climb to the top, nowadays an album is in contrast top ranked either from the start, or not at all. Over the last three decades, the number of top-listed albums increased as a consequence from roughly a dozen per year, to about 40. The distribution of album lifetimes evolved during the last decades from a log-normal distribution to a power law, a profound change. Presenting an information-theoretical approach to human activities, we suggest that the fading relevance of personal time horizons may be causing this phenomenon. Furthermore, we find that sales and airplay- based charts differ statistically and that the inclusion of streaming affects chart diversity adversely. We point out in addition that opinion dynamics may accelerate not only in cultural domains, as found here, but also in other settings, in particular in politics, where it could have far reaching consequences.

Testing Models of Human Declarative Memory at the Single-Neuron Level

Deciphering the mechanisms of declarative memory is a major goal of neuroscience. While much theoretical progress has been made, it has proven difficult to experimentally verify key predictions of some foundational models of memory. Recently, single-neuron recordings in human patients have started to provide direct experimental verification of some theories, including mnemonic evidence accumulation, balance-of-evidence for confidence judgments, sparse coding, contextual reinstatement, and the ventral tegmental area (VTA)-hippocampus loop model. Here, we summarize the cell types that have been described in the medial temporal lobe and posterior parietal cortex, discuss their properties, and reflect on how these findings inform theoretical work. This body of work exemplifies the scientific power of a synergistic combination of modeling and human single-neuron recordings to advance cognitive neuroscience.

A neural microcircuit model for a scalable scale-invariant representation of time

Scale-invariant timing has been observed in a wide range of behavioral experiments. The firing properties of recently described time cells provide a possible neural substrate for scale-invariant behavior. Earlier neural circuit models do not produce scale-invariant neural sequences. In this article, we present a biologically detailed network model based on an earlier mathematical algorithm. The simulations incorporate exponentially decaying persistent firing maintained by the calcium-activated nonspecific (CAN) cationic current and a network structure given by the inverse Laplace transform to generate time cells with scale-invariant firing rates. This model provides the first biologically detailed neural circuit for generating scale-invariant time cells. The circuit that implements the inverse Laplace transform merely consists of off-center/on-surround receptive fields. Critically, rescaling temporal sequences can be accomplished simply via cortical gain control (changing the slope of the f-I curve).

Precise time-matching in chimpanzee allogrooming does not occur after a short delay

Allogrooming is a key aspect of chimpanzee sociality and many studies have investigated the role of reciprocity in a biological market. One theoretical form of reciprocity is time-matching, where payback consists of an equal duration of effort (e.g. twenty seconds of grooming repaid with twenty seconds of grooming). Here, we report a study of allogrooming in a group of twenty-six captive chimpanzees (Chester Zoo, UK), based on more than 150 hours of data. For analysis, we introduce a methodological innovation called the "Delta scale", which unidimensionally measures the accuracy of time-matching according to the extent of delay after the cessation of grooming. Delta is positive when reciprocation occurs after any non-zero delay (e.g. A grooms B and then B grooms A after a five second break) and it is negative when reciprocation begins whilst the original grooming has not yet ceased. Using a generalized linear mixed-method, we found evidence for time-matched reciprocation. However, this was true only for immediate reciprocation (Delta less than zero). If there was a temporal break in grooming between two members of a dyad, then there was no evidence that chimpanzees were using new bouts to retroactively correct for time-matching imbalances from previous bouts. Our results have implications for some of the cognitive constraints that differentiate real-life reciprocation from abstract theoretical models. Furthermore, we suggest that some apparent patterns of time-matched reciprocity may arise merely due to the law of large numbers, and we introduce a statistical test which takes this into account when aggregating grooming durations over a window of time.

Discrete Sequential Information Coding: Heteroclinic Cognitive Dynamics

Discrete sequential information coding is a key mechanism that transforms complex cognitive brain activity into a low-dimensional dynamical process based on the sequential switching among finite numbers of patterns. The storage size of the corresponding process is large because of the permutation capacity as a function of control signals in ensembles of these patterns. Extracting low-dimensional functional dynamics from multiple large-scale neural populations is a central problem both in neuro- and cognitive- sciences. Experimental results in the last decade represent a solid base for the creation of low-dimensional models of different cognitive functions and allow moving toward a dynamical theory of consciousness. We discuss here a methodology to build simple kinetic equations that can be the mathematical skeleton of this theory. Models of the corresponding discrete information processing can be designed using the following dynamical principles: (i) clusterization of the neural activity in space and time and formation of information patterns; (ii) robustness of the sequential dynamics based on heteroclinic chains of metastable clusters; and (iii) sensitivity of such sequential dynamics to intrinsic and external informational signals. We analyze sequential discrete coding based on winnerless competition low-frequency dynamics. Under such dynamics, entrainment, and heteroclinic coordination leads to a large variety of coding regimes that are invariant in time.

Is working memory stored along a logarithmic timeline? Converging evidence from neuroscience, behavior and models

A growing body of evidence suggests that short-term memory does not only store the identity of recently experienced stimuli, but also information about when they were presented. This representation of 'what' happened 'when' constitutes a neural timeline of recent past. Behavioral results suggest that people can sequentially access memories for the recent past, as if they were stored along a timeline to which attention is sequentially directed. In the short-term judgment of recency (JOR) task, the time to choose between two probe items depends on the recency of the more recent probe but not on the recency of the more remote probe. This pattern of results suggests a backward self-terminating search model. We review recent neural evidence from the macaque lateral prefrontal cortex (lPFC) (Tiganj, Cromer, Roy, Miller, & Howard, in press) and behavioral evidence from human JOR task (Singh & Howard, 2017) bearing on this question. Notably, both lines of evidence suggest that the timeline is logarithmically compressed as predicted by Weber-Fechner scaling. Taken together, these findings provide an integrative perspective on temporal organization and neural underpinnings of short-term memory.

Human Episodic Memory Retrieval Is Accompanied by a Neural Contiguity Effect

Cognitive psychologists have long hypothesized that experiences are encoded in a temporal context that changes gradually over time. When an episodic memory is retrieved, the state of context is recovered-a jump back in time. We recorded from single units in the medial temporal lobe of epilepsy patients performing an item recognition task. The population vector changed gradually over minutes during presentation of the list. When a probe from the list was remembered with high confidence, the population vector reinstated the temporal context of the original presentation of that probe during study, a neural contiguity effect that provides a possible mechanism for behavioral contiguity effects. This pattern was only observed for well remembered probes; old probes that were not well remembered showed an anti-contiguity effect. These results constitute the first direct evidence that recovery of an episodic memory in humans is associated with retrieval of a gradually changing state of temporal context, a neural "jump back in time" that parallels the act of remembering.SIGNIFICANCE STATEMENT Episodic memory is the ability to relive a specific experience from one's life. For decades, researchers have hypothesized that, unlike other forms of memory that can be described as simple associations between stimuli, episodic memory depends on the recovery of a neural representation of spatiotemporal context. During study of a sequence of stimuli, the brain state of epilepsy patients changed slowly over at least a minute. When the participant remembered a particular event from the list, this gradually changing state was recovered. This provides direct confirmation of the prediction from computational models of episodic memory. The resolution of this point means that the study of episodic memory can focus on the mechanisms by which this representation of spatiotemporal context is maintained and sometimes recovered.