Long-term coding of personal and universal associations underlying the memory web in the human brain

Human hippocampal and entorhinal neurons encode the temporal structure of experience

Extracting the underlying temporal structure of experience is a fundamental aspect of learning and memory that allows us to predict what is likely to happen next. Current knowledge about the neural underpinnings of this cognitive process in humans stems from functional neuroimaging research1-5. As these methods lack direct access to the neuronal level, it remains unknown how this process is computed by neurons in the human brain. Here we record from single neurons in individuals who have been implanted with intracranial electrodes for clinical reasons, and show that human hippocampal and entorhinal neurons gradually modify their activity to encode the temporal structure of a complex image presentation sequence. This representation was formed rapidly, without providing specific instructions to the participants, and persisted when the prescribed experience was no longer present. Furthermore, the structure recovered from the population activity of hippocampal-entorhinal neurons closely resembled the structural graph defining the sequence, but at the same time, also reflected the probability of upcoming stimuli. Finally, learning of the sequence graph was related to spontaneous, time-compressed replay of individual neurons' activity corresponding to previously experienced graph trajectories. These findings demonstrate that neurons in the hippocampus and entorhinal cortex integrate the 'what' and 'when' information to extract durable and predictive representations of the temporal structure of human experience.

A theory of hippocampal function: New developments

We develop further here the only quantitative theory of the storage of information in the hippocampal episodic memory system and its recall back to the neocortex. The theory is upgraded to account for a revolution in understanding of spatial representations in the primate, including human, hippocampus, that go beyond the place where the individual is located, to the location being viewed in a scene. This is fundamental to much primate episodic memory and navigation: functions supported in humans by pathways that build 'where' spatial view representations by feature combinations in a ventromedial visual cortical stream, separate from those for 'what' object and face information to the inferior temporal visual cortex, and for reward information from the orbitofrontal cortex. Key new computational developments include the capacity of the CA3 attractor network for storing whole charts of space; how the correlations inherent in self-organizing continuous spatial representations impact the storage capacity; how the CA3 network can combine continuous spatial and discrete object and reward representations; the roles of the rewards that reach the hippocampus in the later consolidation into long-term memory in part via cholinergic pathways from the orbitofrontal cortex; and new ways of analysing neocortical information storage using Potts networks.

Neural mechanisms of face familiarity and learning in the human amygdala and hippocampus

Recognizing familiar faces and learning new faces play an important role in social cognition. However, the underlying neural computational mechanisms remain unclear. Here, we record from single neurons in the human amygdala and hippocampus and find a greater neuronal representational distance between pairs of familiar faces than unfamiliar faces, suggesting that neural representations for familiar faces are more distinct. Representational distance increases with exposures to the same identity, suggesting that neural face representations are sharpened with learning and familiarization. Furthermore, representational distance is positively correlated with visual dissimilarity between faces, and exposure to visually similar faces increases representational distance, thus sharpening neural representations. Finally, we construct a computational model that demonstrates an increase in the representational distance of artificial units with training. Together, our results suggest that the neuronal population geometry, quantified by the representational distance, encodes face familiarity, similarity, and learning, forming the basis of face recognition and memory.

A Computational Framework for Memory Engrams

Memory engrams in mice brains are potentially related to groups of concept cells in human brains. A single concept cell in human hippocampus responds, for example, not only to different images of the same object or person but also to its name written down in characters. Importantly, a single mental concept (object or person) is represented by several concept cells and each concept cell can respond to more than one concept. Computational work shows how mental concepts can be embedded in recurrent artificial neural networks as memory engrams and how neurons that are shared between different engrams can lead to associations between concepts. Therefore, observations at the level of neurons can be linked to cognitive notions of memory recall and association chains between memory items.

A dynamic attractor network model of memory formation, reinforcement and forgetting

Empirical evidence shows that memories that are frequently revisited are easy to recall, and that familiar items involve larger hippocampal representations than less familiar ones. In line with these observations, here we develop a modelling approach to provide a mechanistic understanding of how hippocampal neural assemblies evolve differently, depending on the frequency of presentation of the stimuli. For this, we added an online Hebbian learning rule, background firing activity, neural adaptation and heterosynaptic plasticity to a rate attractor network model, thus creating dynamic memory representations that can persist, increase or fade according to the frequency of presentation of the corresponding memory patterns. Specifically, we show that a dynamic interplay between Hebbian learning and background firing activity can explain the relationship between the memory assembly sizes and their frequency of stimulation. Frequently stimulated assemblies increase their size independently from each other (i.e. creating orthogonal representations that do not share neurons, thus avoiding interference). Importantly, connections between neurons of assemblies that are not further stimulated become labile so that these neurons can be recruited by other assemblies, providing a neuronal mechanism of forgetting.

Encoding identity in the marmoset

Hippocampal cells integrate multisensory input to represent the identity of others.

Modeling Biological Face Recognition with Deep Convolutional Neural Networks

Deep convolutional neural networks (DCNNs) have become the state-of-the-art computational models of biological object recognition. Their remarkable success has helped vision science break new ground, and recent efforts have started to transfer this achievement to research on biological face recognition. In this regard, face detection can be investigated by comparing face-selective biological neurons and brain areas to artificial neurons and model layers. Similarly, face identification can be examined by comparing in vivo and in silico multidimensional "face spaces." In this review, we summarize the first studies that use DCNNs to model biological face recognition. On the basis of a broad spectrum of behavioral and computational evidence, we conclude that DCNNs are useful models that closely resemble the general hierarchical organization of face recognition in the ventral visual pathway and the core face network. In two exemplary spotlights, we emphasize the unique scientific contributions of these models. First, studies on face detection in DCNNs indicate that elementary face selectivity emerges automatically through feedforward processing even in the absence of visual experience. Second, studies on face identification in DCNNs suggest that identity-specific experience and generative mechanisms facilitate this particular challenge. Taken together, as this novel modeling approach enables close control of predisposition (i.e., architecture) and experience (i.e., training data), it may be suited to inform long-standing debates on the substrates of biological face recognition.

Single neurons in the human medial temporal lobe flexibly shift representations across spatial and memory tasks

Investigations into how individual neurons encode behavioral variables of interest have revealed specific representations in single neurons, such as place and object cells, as well as a wide range of cells with conjunctive encodings or mixed selectivity. However, as most experiments examine neural activity within individual tasks, it is currently unclear if and how neural representations change across different task contexts. Within this discussion, the medial temporal lobe is particularly salient, as it is known to be important for multiple behaviors including spatial navigation and memory, however the relationship between these functions is currently unclear. Here, to investigate how representations in single neurons vary across different task contexts in the medial temporal lobe, we collected and analyzed single-neuron activity from human participants as they completed a paired-task session consisting of a passive-viewing visual working memory and a spatial navigation and memory task. Five patients contributed 22 paired-task sessions, which were spike sorted together to allow for the same putative single neurons to be compared between the different tasks. Within each task, we replicated concept-related activations in the working memory task, as well as target-location and serial-position responsive cells in the navigation task. When comparing neuronal activity between tasks, we first established that a significant number of neurons maintained the same kind of representation, responding to stimuli presentations across tasks. Further, we found cells that changed the nature of their representation across tasks, including a significant number of cells that were stimulus responsive in the working memory task that responded to serial position in the spatial task. Overall, our results support a flexible encoding of multiple, distinct aspects of different tasks by single neurons in the human medial temporal lobe, whereby some individual neurons change the nature of their feature coding between task contexts.

Hippocampal spatial view cells for memory and navigation, and their underlying connectivity in humans

Hippocampal and parahippocampal gyrus spatial view neurons in primates respond to the spatial location being looked at. The representation is allocentric, in that the responses are to locations "out there" in the world, and are relatively invariant with respect to retinal position, eye position, head direction, and the place where the individual is located. The underlying connectivity in humans is from ventromedial visual cortical regions to the parahippocampal scene area, leading to the theory that spatial view cells are formed by combinations of overlapping feature inputs self-organized based on their closeness in space. Thus, although spatial view cells represent "where" for episodic memory and navigation, they are formed by ventral visual stream feature inputs in the parahippocampal gyrus in what is the parahippocampal scene area. A second "where" driver of spatial view cells are parietal inputs, which it is proposed provide the idiothetic update for spatial view cells, used for memory recall and navigation when the spatial view details are obscured. Inferior temporal object "what" inputs and orbitofrontal cortex reward inputs connect to the human hippocampal system, and in macaques can be associated in the hippocampus with spatial view cell "where" representations to implement episodic memory. Hippocampal spatial view cells also provide a basis for navigation to a series of viewed landmarks, with the orbitofrontal cortex reward inputs to the hippocampus providing the goals for navigation, which can then be implemented by hippocampal connectivity in humans to parietal cortex regions involved in visuomotor actions in space. The presence of foveate vision and the highly developed temporal lobe for object and scene processing in primates including humans provide a basis for hippocampal spatial view cells to be key to understanding episodic memory in the primate and human hippocampus, and the roles of this system in primate including human navigation.

Using multi-task experiments to test principles of hippocampal function

Investigations of hippocampal functions have revealed a dizzying array of findings, from lesion-based behavioral deficits, to a diverse range of characterized neural activations, to computational models of putative functionality. Across these findings, there remains an ongoing debate about the core function of the hippocampus and the generality of its representation. Researchers have debated whether the hippocampus's primary role relates to the representation of space, the neural basis of (episodic) memory, or some more general computation that generalizes across various cognitive domains. Within these different perspectives, there is much debate about the nature of feature encodings. Here, we suggest that in order to evaluate hippocampal responses-investigating, for example, whether neuronal representations are narrowly targeted to particular tasks or if they subserve domain-general purposes-a promising research strategy may be the use of multi-task experiments, or more generally switching between multiple task contexts while recording from the same neurons in a given session. We argue that this strategy-when combined with explicitly defined theoretical motivations that guide experiment design-could be a fruitful approach to better understand how hippocampal representations support different behaviors. In doing so, we briefly review key open questions in the field, as exemplified by articles in this special issue, as well as previous work using multi-task experiments, and extrapolate to consider how this strategy could be further applied to probe fundamental questions about hippocampal function.

Single-neuron mechanisms of neural adaptation in the human temporal lobe

A central function of the human brain is to adapt to new situations based on past experience. Adaptation is reflected behaviorally by shorter reaction times to repeating or similar stimuli, and neurophysiologically by reduced neural activity in bulk-tissue measurements with fMRI or EEG. Several potential single-neuron mechanisms have been hypothesized to cause this reduction of activity at the macroscopic level. We here explore these mechanisms using an adaptation paradigm with visual stimuli bearing abstract semantic similarity. We recorded intracranial EEG (iEEG) simultaneously with spiking activity of single neurons in the medial temporal lobes of 25 neurosurgical patients. Recording from 4917 single neurons, we demonstrate that reduced event-related potentials in the macroscopic iEEG signal are associated with a sharpening of single-neuron tuning curves in the amygdala, but with an overall reduction of single-neuron activity in the hippocampus, entorhinal cortex, and parahippocampal cortex, consistent with fatiguing in these areas.

An integrative view of human hippocampal function: Differences with other species and capacity considerations

We describe an integrative model that encodes associations between related concepts in the human hippocampal formation, constituting the skeleton of episodic memories. The model, based on partially overlapping assemblies of "concept cells," contrast markedly with the well-established notion of pattern separation, which relies on conjunctive, context dependent single neuron responses, instead of the invariant, context independent responses found in the human hippocampus. We argue that the model of partially overlapping assemblies is better suited to cope with memory capacity limitations, that the finding of different types of neurons and functions in this area is due to a flexible and temporary use of the extraordinary machinery of the hippocampus to deal with the task at hand, and that only information that is relevant and frequently revisited will consolidate into long-term hippocampal representations, using partially overlapping assemblies. Finally, we propose that concept cells are uniquely human and that they may constitute the neuronal underpinnings of cognitive abilities that are much further developed in humans compared to other species.

Single neurons in the human medial temporal lobe flexibly shift representations across spatial and memory tasks

Investigations into how individual neurons encode behavioral variables of interest have revealed specific representations in single neurons, such as place and object cells, as well as a wide range of cells with conjunctive encodings or mixed selectivity. However, as most experiments examine neural activity within individual tasks, it is currently unclear if and how neural representations change across different task contexts. Within this discussion, the medial temporal lobe is particularly salient, as it is known to be important for multiple behaviors including spatial navigation and memory, however the relationship between these functions is currently unclear. Here, to investigate how representations in single neurons vary across different task contexts in the MTL, we collected and analyzed single-neuron activity from human participants as they completed a paired-task session consisting of a passive-viewing visual working memory and a spatial navigation and memory task. Five patients contributed 22 paired-task sessions, which were spike sorted together to allow for the same putative single neurons to be compared between the different tasks. Within each task, we replicated concept-related activations in the working memory task, as well as target-location and serial-position responsive cells in the navigation task. When comparing neuronal activity between tasks, we first established that a significant number of neurons maintained the same kind of representation, responding to stimuli presentations across tasks. Further, we found cells that changed the nature of their representation across tasks, including a significant number of cells that were stimulus responsive in the working memory task that responded to serial position in the spatial task. Overall, our results support a flexible encoding of multiple, distinct aspects of different tasks by single neurons in the human MTL, whereby some individual neurons change the nature of their feature coding between task contexts.

Organization and Priming of Long-term Memory Representations with Two-phase Plasticity

Background / Introduction

In recurrent neural networks in the brain, memories are represented by so-called Hebbian cell assemblies. Such assemblies are groups of neurons with particularly strong synaptic connections formed by synaptic plasticity and consolidated by synaptic tagging and capture (STC). To link these synaptic mechanisms to long-term memory on the level of cognition and behavior, their functional implications on the level of neural networks have to be understood.

Methods

We employ a biologically detailed recurrent network of spiking neurons featuring synaptic plasticity and STC to model the learning and consolidation of long-term memory representations. Using this, we investigate the effects of different organizational paradigms, and of priming stimulation, on the functionality of multiple memory representations. We quantify these effects by the spontaneous activation of memory representations driven by background noise.

Results

We find that the learning order of the memory representations significantly biases the likelihood of activation towards more recently learned representations, and that hub-like overlap structure counters this effect. We identify long-term depression as the mechanism underlying these findings. Finally, we demonstrate that STC has functional consequences for the interaction of long-term memory representations: 1. intermediate consolidation in between learning the individual representations strongly alters the previously described effects, and 2. STC enables the priming of a long-term memory representation on a timescale of minutes to hours.

Conclusion

Our findings show how synaptic and neuronal mechanisms can provide an explanatory basis for known cognitive effects.

A human single-neuron dataset for face perception

The human amygdala and hippocampus have long been associated with face perception. Here, we present a dataset of single-neuron activity in the human amygdala and hippocampus during face perception. We recorded 2082 neurons from the human amygdala and hippocampus when neurosurgical patients with intractable epilepsy performed a one-back task using natural face stimuli, which mimics natural face perception. Specifically, our data include (1) single-neuron activity from the amygdala (996 neurons) and hippocampus (1086 neurons), (2) eye movements (gaze position and pupil), (3) psychological assessment of the patients, and (4) social trait judgment ratings from a subset of patients and a large sample of participants from the general population. Together, our comprehensive dataset with a large population of neurons can facilitate multifaceted investigation of face perception with the highest spatial and temporal resolution currently available in humans.

Face identity coding in the deep neural network and primate brain

A central challenge in face perception research is to understand how neurons encode face identities. This challenge has not been met largely due to the lack of simultaneous access to the entire face processing neural network and the lack of a comprehensive multifaceted model capable of characterizing a large number of facial features. Here, we addressed this challenge by conducting in silico experiments using a pre-trained face recognition deep neural network (DNN) with a diverse array of stimuli. We identified a subset of DNN units selective to face identities, and these identity-selective units demonstrated generalized discriminability to novel faces. Visualization and manipulation of the network revealed the importance of identity-selective units in face recognition. Importantly, using our monkey and human single-neuron recordings, we directly compared the response of artificial units with real primate neurons to the same stimuli and found that artificial units shared a similar representation of facial features as primate neurons. We also observed a region-based feature coding mechanism in DNN units as in human neurons. Together, by directly linking between artificial and primate neural systems, our results shed light on how the primate brain performs face recognition tasks.

A Simplified Plasticity Model Based on Synaptic Tagging and Capture Theory: Simplified STC

The formation and consolidation of memory play a vital role for survival in an ever-changing environment. In the brain, the change and stabilization of potentiated and depressed synapses are the neural basis of memory formation and maintenance. These changes can be induced by rather short stimuli (only a few seconds or even less) but should then be stable for months or years. Recently, the neural mechanism of conversion from rapid change during the early phase of synaptic plasticity into a stable memory trace in the late phase of synaptic plasticity is more and more clear at the protein and molecular levels, among which synaptic tagging and capture (STC) theory is one of the most popular theories. According to the STC theory, the change and stabilization of synaptic efficiency mainly depend on three processes related to calcium concentration, including synaptic tagging, synthesis of plasticity-related product (PRP), and the capture of PRP by tagged synapse. Based on the STC theory, several computational models are proposed. However, these models hardly take simplicity and biological interpretability into account simultaneously. Here, we propose a simplified STC (SM-STC) model to address this issue. In the SM-STC model, the concentration of calcium ion in each neuronal compartment and synapse is first calculated, and then the tag state of synapse and PRP are updated, and the coupling effect of tagged synapse and PRP is further considered to determine the plasticity state of the synapse, either potentiation or depression. We simulated the Schaffer collaterals pathway of the hippocampus targeting a multicompartment CA1 neuron for several hours of biological time. The results show that the SM-STC model can produce a broad range of experimental phenomena known in the physiological experiments, including long-term potentiation induced by high-frequency stimuli, long-term depression induced by low-frequency stimuli, and cross-capture with two stimuli separated by a delay. Thus, the SM-STC model proposed in this study provides an effective learning rule for brain-like computation on the premise of ensuring biological plausibility and computational efficiency.

Neurons as will and representation

Memory recollections and voluntary actions are often perceived as spontaneously generated irrespective of external stimuli. Although products of our neurons, they are only rarely accessible in humans at the neuronal level. Here I review insights gleaned from unique neurosurgical opportunities to record and stimulate single-neuron activity in people who can declare their thoughts, memories and wishes. I discuss evidence that the subjective experience of human recollection and that of voluntary action arise from the activity of two internal neuronal generators, the former from medial temporal lobe reactivation and the latter from frontoparietal preactivation. I characterize properties of these generators and their interaction, enabling flexible recruitment of memory-based choices for action as well as recruitment of action-based plans for the representation of conceptual knowledge in memories. Both internal generators operate on surprisingly explicit but different neuronal codes, which appear to arise with distinct single-neuron activity, often observed before participants' reports of conscious awareness. I discuss prediction of behaviour based on these codes, and the potential for their modulation. The prospects of editing human memories and volitions by enhancement, inception or deletion of specific, selected content raise therapeutic possibilities and ethical concerns.

Lacking Social Support is Associated With Structural Divergences in Hippocampus-Default Network Co-Variation Patterns

Elaborate social interaction is a pivotal asset of the human species. The complexity of people’s social lives may constitute the dominating factor in the vibrancy of many individuals’ environment. The neural substrates linked to social cognition thus appear especially susceptible when people endure periods of social isolation: here, we zoom in on the systematic inter-relationships between two such neural substrates, the allocortical hippocampus (HC) and the neocortical default network (DN). Previous human social neuroscience studies have focused on the DN, while HC subfields have been studied in most detail in rodents and monkeys. To bring into contact these two separate research streams, we directly quantified how DN subregions are coherently co-expressed with specific HC subfields in the context of social isolation. A two-pronged decomposition of structural brain scans from ∼40 000 UK Biobank participants linked lack of social support to mostly lateral subregions in the DN patterns. This lateral DN association co-occurred with HC patterns that implicated especially subiculum, presubiculum, CA2, CA3 and dentate gyrus. Overall, the subregion divergences within spatially overlapping signatures of HC–DN co-variation followed a clear segregation into the left and right brain hemispheres. Separable regimes of structural HC–DN co-variation also showed distinct associations with the genetic predisposition for lacking social support at the population level.

Task Modulation of Single-Neuron Activity in the Human Amygdala and Hippocampus

The human amygdala and hippocampus are critically involved in various processes in face perception. However, it remains unclear how task demands or evaluative contexts modulate processes underlying face perception. In this study, we employed two task instructions when participants viewed the same faces and recorded single-neuron activity from the human amygdala and hippocampus. We comprehensively analyzed task modulation for three key aspects of face processing and we found that neurons in the amygdala and hippocampus (1) encoded high-level social traits such as perceived facial trustworthiness and dominance and this response was modulated by task instructions; (2) encoded low-level facial features and demonstrated region-based feature coding, which was not modulated by task instructions; and (3) encoded fixations on salient face parts such as the eyes and mouth, which was not modulated by task instructions. Together, our results provide a comprehensive survey of task modulation of neural processes underlying face perception at the single-neuron level in the human amygdala and hippocampus.

When shared concept cells support associations: Theory of overlapping memory engrams

Assemblies of neurons, called concepts cells, encode acquired concepts in human Medial Temporal Lobe. Those concept cells that are shared between two assemblies have been hypothesized to encode associations between concepts. Here we test this hypothesis in a computational model of attractor neural networks. We find that for concepts encoded in sparse neural assemblies there is a minimal fraction c_min of neurons shared between assemblies below which associations cannot be reliably implemented; and a maximal fraction c_max of shared neurons above which single concepts can no longer be retrieved. In the presence of a periodically modulated background signal, such as hippocampal oscillations, recall takes the form of association chains reminiscent of those postulated by theories of free recall of words. Predictions of an iterative overlap-generating model match experimental data on the number of concepts to which a neuron responds.

Experimental evidence suggests that associations between concepts are encoded in the hippocampus by cells shared between neuronal assemblies (“overlap” of concepts). What is the necessary overlap that ensures a reliable encoding of associations? Under which conditions can associations induce a simultaneous or a chain-like activation of concepts? Our theoretical model shows that the ideal overlap presents a tradeoff: the overlap should be larger than a minimum value in order to reliably encode associations, but lower than a maximum value to prevent loss of individual memories. Our theory explains experimental data from human Medial Temporal Lobe and provides a mechanism for chain-like recall in presence of inhibition, while still allowing for simultaneous recall if inhibition is weak.

Retrospective memory integration accompanies reconfiguration of neural cell assemblies

Memory is a dynamic process that is based on and can be altered by experiences. Integrating memories of multiple experiences (memory integration) is the basis of flexible and complex decision-making. However, the mechanism of memory integration in neural networks of the brain remains poorly understood. In this study, we built a recurrent spiking network model and investigated the neural mechanism of memory integration before a decision is made (retrospective memory integration) at the neural circuit level. Our simulations suggest that retrospective memory integration accompanies reconfiguration of neural cell assemblies. Additionally, partially blocking neural network plasticity leads to failure of memory integration. These findings can potentially guide the experimental investigation of the neural mechanism of retrospective memory integration and serve as the basis for developing new artificial intelligence algorithms.

Concept neurons in the human medial temporal lobe flexibly represent abstract relations between concepts

Concept neurons in the medial temporal lobe respond to semantic features of presented stimuli. Analyzing 61 concept neurons recorded from twelve patients who underwent surgery to treat epilepsy, we show that firing patterns of concept neurons encode relations between concepts during a picture comparison task. Thirty-three of these responded to non-preferred stimuli with a delayed but well-defined onset whenever the task required a comparison to a response-eliciting concept, but not otherwise. Supporting recent theories of working memory, concept neurons increased firing whenever attention was directed towards this concept and could be reactivated after complete activity silence. Population cross-correlations of pairs of concept neurons exhibited order-dependent asymmetric peaks specifically when their response-eliciting concepts were to be compared. Our data are consistent with synaptic mechanisms that support reinstatement of concepts and their relations after activity silence, flexibly induced through task-specific sequential activation. This way arbitrary contents of experience could become interconnected in both working and long-term memory.

It is unclear how distinct concepts are processed in the brain. Here, the authors recorded from concept cells in human subjects with epilepsy and found that a subset of concept cells responded to non-preferred concepts if those non-preferred concepts required comparison to a preferred concept.

Person knowledge shapes face identity perception

Recognition of others' identity through facial features is essential in life. Using both correlational and experimental approaches, we examined how person knowledge biases the perception of others' facial identity. When a participant believed any two individuals were more similar in personality, their faces were perceived to be correspondingly more similar (assessed via mousetracking, Study 1). Further, participants' facial representations of target individuals that were believed to have a more similar personality were found to have a greater physical resemblance (assessed via reverse-correlation, Studies 2 and 3). Finally, when participants learned about novel individuals who had a more similar personality, their faces were visually represented more similarly (Study 4). Together, the findings show that the perception of facial identity is driven not only by facial features but also the person knowledge we have learned about others, biasing it toward alternate identities despite the fact that those identities lack any physical resemblance.

STDP Forms Associations between Memory Traces in Networks of Spiking Neurons

Memory traces and associations between them are fundamental for cognitive brain function. Neuron recordings suggest that distributed assemblies of neurons in the brain serve as memory traces for spatial information, real-world items, and concepts. However, there is conflicting evidence regarding neural codes for associated memory traces. Some studies suggest the emergence of overlaps between assemblies during an association, while others suggest that the assemblies themselves remain largely unchanged and new assemblies emerge as neural codes for associated memory items. Here we study the emergence of neural codes for associated memory items in a generic computational model of recurrent networks of spiking neurons with a data-constrained rule for spike-timing-dependent plasticity. The model depends critically on 2 parameters, which control the excitability of neurons and the scale of initial synaptic weights. By modifying these 2 parameters, the model can reproduce both experimental data from the human brain on the fast formation of associations through emergent overlaps between assemblies, and rodent data where new neurons are recruited to encode the associated memories. Hence, our findings suggest that the brain can use both of these 2 neural codes for associations, and dynamically switch between them during consolidation.

No consolidation without representation: Correspondence between neural and psychological representations in recent and remote memory

Memory systems consolidation is often conceived as the linear, time-dependent, neurobiological shift of memory from hippocampal-cortical to cortico-cortical dependency. We argue that contrary to this unidirectional view of memory reorganization, information about events may be retained in multiple forms (e.g., event-specific sensory-near episodic memory, event-specific gist information, event-general schematic information, or abstract semantic memory). These representations can all form at the time of the event and may continue to coexist for long durations. Their relative strength, composition, and dominance of expression change with time and experience, with task demands, and through their dynamic interaction with one another. These different psychological mnemonic representations depend on distinct functional and structural neurobiological substrates such that there is a neural-psychological representation correspondence (NPRC) among them. We discuss how the dynamics of psychological memory representations are reflected in multiple levels of neurobiological markers and their interactions. By this view, there are only variations of synaptic consolidation and memory dynamics without assuming a distinct systems consolidation process.

Concept formation as a computational cognitive process

Hippocampus and entorhinal cortex form cognitive maps that represent relations among memories within a multidimensional space. While these relational maps have long been proposed to contribute to episodic memory, recent work suggests that they also support concept formation by representing relevant features for discriminating among related concepts. Cognitive maps may be refined by medial prefrontal cortex, which selects dimensions to represent based on their behavioral relevance. Hippocampal pattern completion, which is critical for retrieval of episodic memories, may also contribute to generalization of existing concepts to new exemplars. Navigation within hippocampal cognitive maps, which is guided by grid coding in entorhinal cortex, may contribute to imagination through recombination of event elements or concept features.

How Human Single-Neuron Recordings Can Help Us Understand Cognition: Insights from Memory Studies

Understanding human cognition is a key goal of contemporary neuroscience. Due to the complexity of the human brain, animal studies and noninvasive techniques, however valuable, are incapable of providing us with a full understanding of human cognition. In the light of existing cognitive theories, we describe findings obtained thanks to human single-neuron recordings, including the discovery of concept cells and novelty-dependent cells, or activity patterns behind working memory, such as persistent activity. We propose future directions for studies using human single-neuron recordings and we discuss possible opportunities of investigating pathological brain.

Semantic Knowledge of Famous People and Places Is Represented in Hippocampus and Distinct Cortical Networks

Studies have found that anterior temporal lobe (ATL) is critical for detailed knowledge of object categories, suggesting that it has an important role in semantic memory. However, in addition to information about entities, such as people and objects, semantic memory also encompasses information about places. We tested predictions stemming from the PMAT model, which proposes there are distinct systems that support different kinds of semantic knowledge: an anterior temporal (AT) network, which represents information about entities; and a posterior medial (PM) network, which represents information about places. We used representational similarity analysis to test for activation of semantic features when human participants viewed pictures of famous people and places, while controlling for visual similarity. We used machine learning techniques to quantify the semantic similarity of items based on encyclopedic knowledge in the Wikipedia page for each item and found that these similarity models accurately predict human similarity judgments. We found that regions within the AT network, including ATL and inferior frontal gyrus, represented detailed semantic knowledge of people. In contrast, semantic knowledge of places was represented within PM network areas, including precuneus, posterior cingulate cortex, angular gyrus, and parahippocampal cortex. Finally, we found that hippocampus, which has been proposed to serve as an interface between the AT and PM networks, represented fine-grained semantic similarity for both individual people and places. Our results provide evidence that semantic knowledge of people and places is represented separately in AT and PM areas, whereas hippocampus represents semantic knowledge of both categories.SIGNIFICANCE STATEMENT Humans acquire detailed semantic knowledge about people (e.g., their occupation and personality) and places (e.g., their cultural or historical significance). While research has demonstrated that brain regions preferentially respond to pictures of people and places, less is known about whether these regions preferentially represent semantic knowledge about specific people and places. We used machine learning techniques to develop a model of semantic similarity based on information available from Wikipedia, validating the model against similarity ratings from human participants. Using our computational model, we found that semantic knowledge about people and places is represented in distinct anterior temporal and posterior medial brain networks, respectively. We further found that hippocampus, an important memory center, represented semantic knowledge for both types of stimuli.

The Architecture of Human Memory: Insights from Human Single-Neuron Recordings

Deciphering the mechanisms of human memory is a central goal of neuroscience, both from the point of view of the fundamental biology of memory and for its translational relevance. Here, we review some contributions that recordings from neurons in humans implanted with electrodes for clinical purposes have made toward this goal. Recordings from the medial temporal lobe, including the hippocampus, reveal the existence of two classes of cells: those encoding highly selective and invariant representations of abstract concepts, and memory-selective cells whose activity is related to familiarity and episodic retrieval. Insights derived from observing these cells in behaving humans include that semantic representations are activated before episodic representations, that memory content and memory strength are segregated, and that the activity of both types of cells is related to subjective awareness as expected from a substrate for declarative memory. Visually selective cells can remain persistently active for several seconds, thereby revealing a cellular substrate for working memory in humans. An overarching insight is that the neural code of human memory is interpretable at the single-neuron level. Jointly, intracranial recording studies are starting to reveal aspects of the building blocks of human memory at the single-cell level. This work establishes a bridge to cellular-level work in animals on the one hand, and the extensive literature on noninvasive imaging in humans on the other hand. More broadly, this work is a step toward a detailed mechanistic understanding of human memory that is needed to develop therapies for human memory disorders.

Brain computation by assemblies of neurons

Our expanding understanding of the brain at the level of neurons and synapses, and the level of cognitive phenomena such as language, leaves a formidable gap between these two scales. Here we introduce a computational system which promises to bridge this gap: the Assembly Calculus. It encompasses operations on assemblies of neurons, such as project, associate, and merge, which appear to be implicated in cognitive phenomena, and can be shown, analytically as well as through simulations, to be plausibly realizable at the level of neurons and synapses. We demonstrate the reach of this system by proposing a brain architecture for syntactic processing in the production of language, compatible with recent experimental results.

Assemblies are large populations of neurons believed to imprint memories, concepts, words, and other cognitive information. We identify a repertoire of operations on assemblies. These operations correspond to properties of assemblies observed in experiments, and can be shown, analytically and through simulations, to be realizable by generic, randomly connected populations of neurons with Hebbian plasticity and inhibition. Assemblies and their operations constitute a computational model of the brain which we call the Assembly Calculus, occupying a level of detail intermediate between the level of spiking neurons and synapses and that of the whole brain. The resulting computational system can be shown, under assumptions, to be, in principle, capable of carrying out arbitrary computations. We hypothesize that something like it may underlie higher human cognitive functions such as reasoning, planning, and language. In particular, we propose a plausible brain architecture based on assemblies for implementing the syntactic processing of language in cortex, which is consistent with recent experimental results.

Single Neuron Coding of Identity in the Human Hippocampal Formation

Experimental findings show the ubiquitous presence of graded responses and tuning curves in the neocortex, particularly in visual areas [1-15]. Among these, inferotemporal-cortex (IT) neurons respond to complex visual stimuli, but differences in the neurons' responses can be used to distinguish the stimuli eliciting the responses [8, 9, 16-18]. The IT projects directly to the medial temporal lobe (MTL) [19], where neurons respond selectively to different pictures of specific persons and even to their written and spoken names [20-22]. However, it is not clear whether this is done through a graded coding, as in the neocortex, or a truly invariant code, in which the response-eliciting stimuli cannot be distinguished from each other. To address this issue, we recorded single neurons during the repeated presentation of different stimuli (pictures and written and spoken names) corresponding to the same persons. Using statistical tests and a decoding approach, we found that only in a minority of cases can the different pictures of a given person be distinguished from the neurons' responses and that in a larger proportion of cases, the responses to the pictures were different to the ones to the written and spoken names. We argue that MTL neurons tend to lack a representation of sensory features (particularly within a sensory modality), which can be advantageous for the memory function attributed to this area [23-25], and that a full representation of memories is given by a combination of mostly invariant coding in the MTL with a representation of sensory features in the neocortex.

Modeling of Brain-Like Concept Coding with Adulthood Neurogenesis in the Dentate Gyrus

Mammalian brains respond to new concepts via a type of neural coding termed “concept coding.” During concept coding, the dentate gyrus (DG) plays a vital role in pattern separation and pattern integration of concepts because it is a brain region with substantial neurogenesis in adult mammals. Although concept coding properties of the brain have been extensively studied by experimental work, modeling of the process to guide both further experimental studies and applications such as natural language processing is scarce. To model brain-like concept coding, we built a spiking neural network inspired by adulthood neurogenesis in the DG. Our model suggests that neurogenesis may facilitate integration of closely related concepts and separation of less relevant concepts. Such pattern agrees with the previous experimental observations in classification tasks and place cells in the hippocampus. Therefore, our simulation provides insight for future experimental studies on the neural coding difference between perception and cognition. By presenting 14 contexts each containing 4 concepts to the network, we found that neural responses of the DG changed dynamically as the context repetition time increased and were eventually consistent with the category organization of humans. Thus, our work provides a new framework of word representation for the construction of brain-like knowledge map.

Representation of abstract semantic knowledge in populations of human single neurons in the medial temporal lobe

Sensory experience elicits complex activity patterns throughout the neocortex. Projections from the neocortex converge onto the medial temporal lobe (MTL), in which distributed neocortical firing patterns are distilled into sparse representations. The precise nature of these neuronal representations is still unknown. Here, we show that population activity patterns in the MTL are governed by high levels of semantic abstraction. We recorded human single-unit activity in the MTL (4,917 units, 25 patients) while subjects viewed 100 images grouped into 10 semantic categories of 10 exemplars each. High levels of semantic abstraction were indicated by representational similarity analyses (RSAs) of patterns elicited by individual stimuli. Moreover, pattern classifiers trained to decode semantic categories generalised successfully to unseen exemplars, and classifiers trained to decode exemplar identity more often confused exemplars of the same versus different categories. Semantic abstraction and generalisation may thus be key to efficiently distill the essence of an experience into sparse representations in the human MTL. Although semantic abstraction is efficient and may facilitate generalisation of knowledge to novel situations, it comes at the cost of a loss of detail and may be central to the generation of false memories.

Single-neuron representations of stimuli in the human medial temporal lobe at the population level are governed by highly abstract semantic principles, but the attendant efficiency and potential for generalization comes at the cost of confusion between related stimuli.

What is the neuronal code for sensory experience in the human medial temporal lobe (MTL)? Single-cell electrophysiology in the awake human brain during chronic, invasive epilepsy monitoring has previously revealed the existence of so-called concept cells. These cells have been found to increase their firing rate in response to, for example, the famous tennis player ‘Roger Federer’, whether his name is spoken by a computer voice or a picture of him is presented on a computer screen. These neurons thus seem to encode the semantic content of a stimulus, regardless of the sensory modality through which it is delivered. Previous work has predominantly focused on individual neurons that were selected based on their strong response to a particular stimulus using rather conservative statistical criteria. Those studies stressed that concept cells encode a single, concrete concept in an all-or-nothing fashion. Here, we analysed the neuronal code on the level of the entire population of neurons without any preselection. We conducted representational similarity analyses (RSAs) and pattern classification analyses of firing patterns evoked by visual stimuli (for example, a picture of an apple) that could be grouped into semantic categories on multiple levels of abstraction (‘fruit’, ‘food’, ‘natural things’). We found that neuronal activation patterns contain information on higher levels of categorical abstraction rather than just the level of individual exemplars. On the one hand, the neuronal code in the human MTL thus seems well suited to generalise semantic knowledge to new situations; on the other hand, it could also be responsible for the generation of false memories.

Testing Models of Human Declarative Memory at the Single-Neuron Level

Deciphering the mechanisms of declarative memory is a major goal of neuroscience. While much theoretical progress has been made, it has proven difficult to experimentally verify key predictions of some foundational models of memory. Recently, single-neuron recordings in human patients have started to provide direct experimental verification of some theories, including mnemonic evidence accumulation, balance-of-evidence for confidence judgments, sparse coding, contextual reinstatement, and the ventral tegmental area (VTA)-hippocampus loop model. Here, we summarize the cell types that have been described in the medial temporal lobe and posterior parietal cortex, discuss their properties, and reflect on how these findings inform theoretical work. This body of work exemplifies the scientific power of a synergistic combination of modeling and human single-neuron recordings to advance cognitive neuroscience.

Encoding of long-term associations through neural unitization in the human medial temporal lobe

Besides decades of research showing the role of the medial temporal lobe (MTL) in memory and the encoding of associations, the neural substrates underlying these functions remain unknown. We identified single neurons in the human MTL that responded to multiple and, in most cases, associated stimuli. We observed that most of these neurons exhibit no differences in their spike and local field potential (LFP) activity associated with the individual response-eliciting stimuli. In addition, LFP responses in the theta band preceded single neuron responses by ~70 ms, with the single trial phase providing fine tuning of the spike response onset. We postulate that the finding of similar neuronal responses to associated items provides a simple and flexible way of encoding memories in the human MTL, increasing the effective capacity for memory storage and successful retrieval.

In this work, the authors recorded single neurons and field potentials from the human medial temporal lobe (MTL) and show indistinguishable responses to associated stimuli. This coding mechanism provides a simple and flexible way of encoding memories in the human MTL.

A novel and fully automatic spike-sorting implementation with variable number of features

The most widely used spike-sorting algorithms are semiautomatic in practice, requiring manual tuning of the automatic solution to achieve good performance. In this work, we propose a new fully automatic spike-sorting algorithm that can capture multiple clusters of different sizes and densities. In addition, we introduce an improved feature selection method, by using a variable number of wavelet coefficients, based on the degree of non-Gaussianity of their distributions. We evaluated the performance of the proposed algorithm with real and simulated data. With real data from single-channel recordings, in ~95% of the cases the new algorithm replicated, in an unsupervised way, the solutions obtained by expert sorters, who manually optimized the solution of a previous semiautomatic algorithm. This was done while maintaining a low number of false positives. With simulated data from single-channel and tetrode recordings, the new algorithm was able to correctly detect many more neurons compared with previous implementations and also compared with recently introduced algorithms, while significantly reducing the number of false positives. In addition, the proposed algorithm showed good performance when tested with real tetrode recordings. NEW & NOTEWORTHY We propose a new fully automatic spike-sorting algorithm, including several steps that allow the selection of multiple clusters of different sizes and densities. Moreover, it defines the dimensionality of the feature space in an unsupervised way. We evaluated the performance of the algorithm with real and simulated data, from both single-channel and tetrode recordings. The proposed algorithm was able to outperform manual sorting from experts and other recent unsupervised algorithms.

Doctor, Teacher, and Stethoscope: Neural Representation of Different Types of Semantic Relations

Concepts can be related in many ways. They can belong to the same taxonomic category (e.g., "doctor" and "teacher," both in the category of people) or be associated with the same event context (e.g., "doctor" and "stethoscope," both associated with medical scenarios). How are these two major types of semantic relations coded in the brain? We constructed stimuli from three taxonomic categories (people, manmade objects, and locations) and three thematic categories (school, medicine, and sports) and investigated the neural representations of these two dimensions using representational similarity analyses in human participants (10 men and nine women). In specific regions of interest, the left anterior temporal lobe (ATL) and the left temporoparietal junction (TPJ), we found that, whereas both areas had significant effects of taxonomic information, the taxonomic relations had stronger effects in the ATL than in the TPJ ("doctor" and "teacher" closer in ATL neural activity), with the reverse being true for thematic relations ("doctor" and "stethoscope" closer in TPJ neural activity). A whole-brain searchlight analysis revealed that widely distributed regions, mainly in the left hemisphere, represented the taxonomic dimension. Interestingly, the significant effects of the thematic relations were only observed after the taxonomic differences were controlled for in the left TPJ, the right superior lateral occipital cortex, and other frontal, temporal, and parietal regions. In summary, taxonomic grouping is a primary organizational dimension across distributed brain regions, with thematic grouping further embedded within such taxonomic structures.SIGNIFICANCE STATEMENT How are concepts organized in the brain? It is well established that concepts belonging to the same taxonomic categories (e.g., "doctor" and "teacher") share neural representations in specific brain regions. How concepts are associated in other manners (e.g., "doctor" and "stethoscope," which are thematically related) remains poorly understood. We used representational similarity analyses to unravel the neural representations of these different types of semantic relations by testing the same set of words that could be differently grouped by taxonomic categories or by thematic categories. We found that widely distributed brain areas primarily represented taxonomic categories, with the thematic categories further embedded within the taxonomic structure.

An online adaptive screening procedure for selective neuronal responses

BACKGROUND

A common problem in neurophysiology is to identify stimuli that elicit neuronal responses in a given brain region. Particularly in situations where electrode positions are fixed, this can be a time-consuming task that requires presentation of a large number of stimuli. Such a screening for response-eliciting stimuli is employed, e.g., as a standard procedure to identify 'concept cells' in the human medial temporal lobe.

NEW METHOD

Our new method evaluates neuronal responses to stimuli online during a screening session, which allows us to successively exclude stimuli that do not evoke a response. Using this method, we can screen a larger number of stimuli which in turn increases the chances of finding responsive neurons and renders time-consuming offline analysis unnecessary.

RESULTS

Our method enabled us to present 30% more stimuli in the same period of time with additional presentations of the most promising candidate stimuli. Our online method ran smoothly on a standard computer and network.

COMPARISON WITH AN EXISTING METHOD

To analyze how our online screening procedure performs in comparison to an established offline method, we used the Wave_Clus software package. We did not observe any major drawbacks in our method, but a much higher efficiency and analysis speed.

CONCLUSIONS

By transitioning from a traditional offline screening procedure to our new online method, we substantially increased the number of visual stimuli presented in a given time period. This allows to identify more response-eliciting stimuli, which forms the basis to better address a great number of questions in cognitive neuroscience.

How Do We Recognize a Face?

How individual faces are encoded by neurons in high-level visual areas has been a subject of active debate. An influential model is that neurons encode specific faces. However, Chang and Tsao conclusively show that, instead, these neurons encode features along specific axes, which explains why they were previously found to respond to apparently different faces.

Human Cortical Neurons in the Anterior Temporal Lobe Reinstate Spiking Activity during Verbal Memory Retrieval

When we recall an experience, we rely upon the associations that we formed during the experience, such as those among objects, time, and place [1]. These associations are better remembered when they are familiar and draw upon generalized knowledge, suggesting that we use semantic memory in the service of episodic memory [2, 3]. Moreover, converging evidence suggests that episodic memory retrieval involves the reinstatement of neural activity that was present when we first experienced the event. Therefore, we hypothesized that retrieving associations should also reinstate the neural activity responsible for semantic processing. Indeed, previous studies have suggested that verbal memory retrieval leads to the reinstatement of activity across regions of the brain that include the distributed semantic processing network [4-6], but it is unknown whether and how individual neurons in the human cortex participate in the reinstatement of semantic representations. Recent advances using high-density microelectrode arrays (MEAs) have allowed clinicians to record from populations of neurons in the human cortex [7, 8]. Here we used MEAs to record neuronal spiking activity in the human middle temporal gyrus (MTG), a cortical region supporting the semantic representation of words [9-11], as participants performed a verbal paired-associates task. We provide novel evidence that population spiking activity in the MTG forms distinct representations of semantic concepts and that these representations are reinstated during the retrieval of those words.

Plasticity of hippocampal memories in humans

The human hippocampus is a brain region that supports episodic and spatial memory. Recent experiments have drawn on animal research and computational modelling to reveal how the unique computations and representations of the hippocampus support episodic and spatial memory. Invasive electrophysiological recordings and non-invasive functional brain imaging have provided evidence for the rapid formation of hippocampal representations, as well as the ability of the hippocampus to both pattern-separate and pattern-complete input from the neocortex. Further, recent evidence has shown that hippocampal representations are in constant flux, undergoing a continual process of strengthening, weakening and altering. This research offers a glimpse into the highly plastic and flexible nature of the human hippocampal system in relation to episodic memory.