A Hierarchy of Associations in Hippocampo-Cortical Systems: Cognitive Maps and Navigation Strategies

Time as the fourth dimension in the hippocampus

Experiences of animal and human beings are structured by the continuity of space and time coupled with the uni-directionality of time. In addition to its pivotal position in spatial processing and navigation, the hippocampal system also plays a central, multiform role in several types of temporal processing. These include timing and sequence learning, at scales ranging from meso-scales of seconds to macro-scales of minutes, hours, days and beyond, encompassing the classical functions of short term memory, working memory, long term memory, and episodic memories (comprised of information about when, what, and where). This review article highlights the principal findings and behavioral contexts of experiments in rats showing: 1) timing: tracking time during delays by hippocampal 'time cells' and during free behavior by hippocampal-afferent lateral entorhinal cortex ramping cells; 2) 'online' sequence processing: activity coding sequences of events during active behavior; 3) 'off-line' sequence replay: during quiescence or sleep, orderly reactivation of neuronal assemblies coding awake sequences. Studies in humans show neurophysiological correlates of episodic memory comparable to awake replay. Neural mechanisms are discussed, including ion channel properties, plateau and ramping potentials, oscillations of excitation and inhibition of population activity, bursts of high amplitude discharges (sharp wave ripples), as well as short and long term synaptic modifications among and within cell assemblies. Specifically conceived neural network models will suggest processes supporting the emergence of scalar properties (Weber's law), and include different classes of feedforward and recurrent network models, with intrinsic hippocampal coding for 'transitions' (sequencing of events or places).

Biologically-Inspired Computational Neural Mechanism for Human Action/activity Recognition: A Review

Theoretical neuroscience investigation shows valuable information on the mechanism for recognizing the biological movements in the mammalian visual system. This involves many different fields of researches such as psychological, neurophysiology, neuro-psychological, computer vision, and artificial intelligence (AI). The research on these areas provided massive information and plausible computational models. Here, a review on this subject is presented. This paper describes different perspective to look at this task including action perception, computational and knowledge based modeling, psychological, and neuroscience approaches.

Merging information in the entorhinal cortex: what can we learn from robotics experiments and modeling?

Place recognition is a complex process involving idiothetic and allothetic information. In mammals, evidence suggests that visual information stemming from the temporal and parietal cortical areas (‘what’ and ‘where’ information) is merged at the level of the entorhinal cortex (EC) to build a compact code of a place. Local views extracted from specific feature points can provide information important for view cells (in primates) and place cells (in rodents) even when the environment changes dramatically. Robotics experiments using conjunctive cells merging ‘what’ and ‘where’ information related to different local views show their important role for obtaining place cells with strong generalization capabilities. This convergence of information may also explain the formation of grid cells in the medial EC if we suppose that: (1) path integration information is computed outside the EC, (2) this information is compressed at the level of the EC owing to projection (which follows a modulo principle) of cortical activities associated with discretized vector fields representing angles and/or path integration, and (3) conjunctive cells merge the projections of different modalities to build grid cell activities. Applying modulo projection to visual information allows an interesting compression of information and could explain more recent results on grid cells related to visual exploration. In conclusion, the EC could be dedicated to the build-up of a robust yet compact code of cortical activity whereas the hippocampus proper recognizes these complex codes and learns to predict the transition from one state to another.

Neurorobotics-A Thriving Community and a Promising Pathway Toward Intelligent Cognitive Robots

Neurorobots are robots whose control has been modeled after some aspect of the brain. Since the brain is so closely coupled to the body and situated in the environment, Neurorobots can be a powerful tool for studying neural function in a holistic fashion. It may also be a means to develop autonomous systems that have some level of biological intelligence. The present article provides my perspective on this field, points out some of the landmark events, and discusses its future potential.

Dendrites, deep learning, and sequences in the hippocampus

The hippocampus places us both in time and space. It does so over remarkably large spans: milliseconds to years, and centimeters to kilometers. This works for sensory representations, for memory, and for behavioral context. How does it fit in such wide ranges of time and space scales, and keep order among the many dimensions of stimulus context? A key organizing principle for a wide sweep of scales and stimulus dimensions is that of order in time, or sequences. Sequences of neuronal activity are ubiquitous in sensory processing, in motor control, in planning actions, and in memory. Against this strong evidence for the phenomenon, there are currently more models than definite experiments about how the brain generates ordered activity. The flip side of sequence generation is discrimination. Discrimination of sequences has been extensively studied at the behavioral, systems, and modeling level, but again physiological mechanisms are fewer. It is against this backdrop that I discuss two recent developments in neural sequence computation, that at face value share little beyond the label "neural." These are dendritic sequence discrimination, and deep learning. One derives from channel physiology and molecular signaling, the other from applied neural network theory - apparently extreme ends of the spectrum of neural circuit detail. I suggest that each of these topics has deep lessons about the possible mechanisms, scales, and capabilities of hippocampal sequence computation.

Hippocampus activation related to 'real-time' processing of visuospatial change

The delay associated with cerebral processing time implies a lack of real-time representation of changes in the observed environment. To bridge this gap for motor actions in a dynamical environment, the brain uses predictions of the most plausible future reality based on previously provided information. To optimise these predictions, adjustments to actual experiences are necessary. This requires a perceptual memory buffer. In our study we gained more insight how the brain treats (real-time) information by comparing cerebral activations related to judging past-, present- and future locations of a moving ball, respectively. Eighteen healthy subjects made these estimations while fMRI data was obtained. All three conditions evoked bilateral dorsal-parietal and premotor activations, while judgment of the location of the ball at the moment of judgment showed increased bilateral posterior hippocampus activation relative to making both future and past judgments at the one-second time-sale. Since the condition of such 'real-time' judgments implied undistracted observation of the ball's actual movements, the associated hippocampal activation is consistent with the concept that the hippocampus participates in a top-down exerted sensory gating mechanism. In this way, it may play a role in novelty (saliency) detection.

"Taller and Shorter": Human 3-D Spatial Memory Distorts Familiar Multilevel Buildings

Animal experiments report contradictory findings on the presence of a behavioural and neuronal anisotropy exhibited in vertical and horizontal capabilities of spatial orientation and navigation. We performed a pointing experiment in humans on the imagined 3-D direction of the location of various invisible goals that were distributed horizontally and vertically in a familiar multilevel hospital building. The 21 participants were employees who had worked for years in this building. The hypothesis was that comparison of the experimentally determined directions and the true directions would reveal systematic inaccuracy or dimensional anisotropy of the localizations. The study provides first evidence that the internal representation of a familiar multilevel building was distorted compared to the dimensions of the true building: vertically 215% taller and horizontally 51% shorter. This was not only demonstrated in the mathematical reconstruction of the mental model based on the analysis of the pointing experiments but also by the participants’ drawings of the front view and the ground plan of the building. Thus, in the mental model both planes were altered in different directions: compressed for the horizontal floor plane and stretched for the vertical column plane. This could be related to human anisotropic behavioural performance of horizontal and vertical navigation in such buildings.

From grid cells and visual place cells to multimodal place cell: a new robotic architecture

In the present study, a new architecture for the generation of grid cells (GC) was implemented on a real robot. In order to test this model a simple place cell (PC) model merging visual PC activity and GC was developed. GC were first built from a simple "several to one" projection (similar to a modulo operation) performed on a neural field coding for path integration (PI). Robotics experiments raised several practical and theoretical issues. To limit the important angular drift of PI, head direction information was introduced in addition to the robot proprioceptive signal coming from the wheel rotation. Next, a simple associative learning between visual place cells and the neural field coding for the PI has been used to recalibrate the PI and to limit its drift. Finally, the parameters controlling the shape of the PC built from the GC have been studied. Increasing the number of GC obviously improves the shape of the resulting place field. Yet, other parameters such as the discretization factor of PI or the lateral interactions between GC can have an important impact on the place field quality and avoid the need of a very large number of GC. In conclusion, our results show our GC model based on the compression of PI is congruent with neurobiological studies made on rodent. GC firing patterns can be the result of a modulo transformation of PI information. We argue that such a transformation may be a general property of the connectivity from the cortex to the entorhinal cortex. Our model predicts that the effect of similar transformations on other kinds of sensory information (visual, tactile, auditory, etc…) in the entorhinal cortex should be observed. Consequently, a given EC cell should react to non-contiguous input configurations in non-spatial conditions according to the projection from its different inputs.

Reconceiving the hippocampal map as a topological template

The role of the hippocampus in spatial cognition is incontrovertible yet controversial. Place cells, initially thought to be location-specifiers, turn out to respond promiscuously to a wide range of stimuli. Here we test the idea, which we have recently demonstrated in a computational model, that the hippocampal place cells may ultimately be interested in a space's topological qualities (its connectivity) more than its geometry (distances and angles); such higher-order functioning would be more consistent with other known hippocampal functions. We recorded place cell activity in rats exploring morphing linear tracks that allowed us to dissociate the geometry of the track from its topology. The resulting place fields preserved the relative sequence of places visited along the track but did not vary with the metrical features of the track or the direction of the rat's movement. These results suggest a reinterpretation of previous studies and new directions for future experiments.

From self-assessment to frustration, a small step toward autonomy in robotic navigation

Autonomy and self-improvement capabilities are still challenging in the fields of robotics and machine learning. Allowing a robot to autonomously navigate in wide and unknown environments not only requires a repertoire of robust strategies to cope with miscellaneous situations, but also needs mechanisms of self-assessment for guiding learning and for monitoring strategies. Monitoring strategies requires feedbacks on the behavior's quality, from a given fitness system in order to take correct decisions. In this work, we focus on how a second-order controller can be used to (1) manage behaviors according to the situation and (2) seek for human interactions to improve skills. Following an incremental and constructivist approach, we present a generic neural architecture, based on an on-line novelty detection algorithm that may be able to self-evaluate any sensory-motor strategies. This architecture learns contingencies between sensations and actions, giving the expected sensation from the previous perception. Prediction error, coming from surprising events, provides a measure of the quality of the underlying sensory-motor contingencies. We show how a simple second-order controller (emotional system) based on the prediction progress allows the system to regulate its behavior to solve complex navigation tasks and also succeeds in asking for help if it detects dead-lock situations. We propose that this model could be a key structure toward self-assessment and autonomy. We made several experiments that can account for such properties for two different strategies (road following and place cells based navigation) in different situations.

The hippocampo-cortical loop: spatio-temporal learning and goal-oriented planning in navigation

We present a neural network model where the spatial and temporal components of a task are merged and learned in the hippocampus as chains of associations between sensory events. The prefrontal cortex integrates this information to build a cognitive map representing the environment. The cognitive map can be used after latent learning to select optimal actions to fulfill the goals of the animal. A simulation of the architecture is made and applied to learning and solving tasks that involve both spatial and temporal knowledge. We show how this model can be used to solve the continuous place navigation task, where a rat has to navigate to an unmarked goal and wait for 2 seconds without moving to receive a reward. The results emphasize the role of the hippocampus for both spatial and timing prediction, and the prefrontal cortex in the learning of goals related to the task.

Integrating cortico-limbic-basal ganglia architectures for learning model-based and model-free navigation strategies

Behavior in spatial navigation is often organized into map-based (place-driven) vs. map-free (cue-driven) strategies; behavior in operant conditioning research is often organized into goal-directed vs. habitual strategies. Here we attempt to unify the two. We review one powerful theory for distinct forms of learning during instrumental conditioning, namely model-based (maintaining a representation of the world) and model-free (reacting to immediate stimuli) learning algorithms. We extend these lines of argument to propose an alternative taxonomy for spatial navigation, showing how various previously identified strategies can be distinguished as “model-based” or “model-free” depending on the usage of information and not on the type of information (e.g., cue vs. place). We argue that identifying “model-free” learning with dorsolateral striatum and “model-based” learning with dorsomedial striatum could reconcile numerous conflicting results in the spatial navigation literature. From this perspective, we further propose that the ventral striatum plays key roles in the model-building process. We propose that the core of the ventral striatum is positioned to learn the probability of action selection for every transition between states of the world. We further review suggestions that the ventral striatal core and shell are positioned to act as “critics” contributing to the computation of a reward prediction error for model-free and model-based systems, respectively.

Modeling the contributions of Basal ganglia and Hippocampus to spatial navigation using reinforcement learning

A computational neural model that describes the competing roles of Basal Ganglia and Hippocampus in spatial navigation is presented. Model performance is evaluated on a simulated Morris water maze explored by a model rat. Cue-based and place-based navigational strategies, thought to be subserved by the Basal ganglia and Hippocampus respectively, are described. In cue-based navigation, the model rat learns to directly head towards a visible target, while in place-based navigation the target position is represented in terms of spatial context provided by an array of poles placed around the pool. Learning is formulated within the framework of Reinforcement Learning, with the nigrostriatal dopamine signal playing the role of Temporal Difference Error. Navigation inherently involves two apparently contradictory movements: goal oriented movements vs. random, wandering movements. The model hypothesizes that while the goal-directedness is determined by the gradient in Value function, randomness is driven by the complex activity of the SubThalamic Nucleus (STN)-Globus Pallidus externa (GPe) system. Each navigational system is associated with a Critic, prescribing actions that maximize value gradients for the corresponding system. In the integrated system, that incorporates both cue-based and place-based forms of navigation, navigation at a given position is determined by the system whose value function is greater at that position. The proposed model describes the experimental results of [1], a lesion-study that investigates the competition between cue-based and place-based navigational systems. The present study also examines impaired navigational performance under Parkinsonian-like conditions. The integrated navigational system, operated under dopamine-deficient conditions, exhibits increased escape latency as was observed in experimental literature describing MPTP model rats navigating a water maze.

Spatial learning and action planning in a prefrontal cortical network model

The interplay between hippocampus and prefrontal cortex (PFC) is fundamental to spatial cognition. Complementing hippocampal place coding, prefrontal representations provide more abstract and hierarchically organized memories suitable for decision making. We model a prefrontal network mediating distributed information processing for spatial learning and action planning. Specific connectivity and synaptic adaptation principles shape the recurrent dynamics of the network arranged in cortical minicolumns. We show how the PFC columnar organization is suitable for learning sparse topological-metrical representations from redundant hippocampal inputs. The recurrent nature of the network supports multilevel spatial processing, allowing structural features of the environment to be encoded. An activation diffusion mechanism spreads the neural activity through the column population leading to trajectory planning. The model provides a functional framework for interpreting the activity of PFC neurons recorded during navigation tasks. We illustrate the link from single unit activity to behavioral responses. The results suggest plausible neural mechanisms subserving the cognitive “insight” capability originally attributed to rodents by Tolman & Honzik. Our time course analysis of neural responses shows how the interaction between hippocampus and PFC can yield the encoding of manifold information pertinent to spatial planning, including prospective coding and distance-to-goal correlates.

We study spatial cognition, a high-level brain function based upon the ability to elaborate mental representations of the environment supporting goal-oriented navigation. Spatial cognition involves parallel information processing across a distributed network of interrelated brain regions. Depending on the complexity of the spatial navigation task, different neural circuits may be primarily involved, corresponding to different behavioral strategies. Navigation planning, one of the most flexible strategies, is based on the ability to prospectively evaluate alternative sequences of actions in order to infer optimal trajectories to a goal. The hippocampal formation and the prefrontal cortex are two neural substrates likely involved in navigation planning. We adopt a computational modeling approach to show how the interactions between these two brain areas may lead to learning of topological representations suitable to mediate action planning. Our model suggests plausible neural mechanisms subserving the cognitive spatial capabilities attributed to rodents. We provide a functional framework for interpreting the activity of prefrontal and hippocampal neurons recorded during navigation tasks. Akin to integrative neuroscience approaches, we illustrate the link from single unit activity to behavioral responses while solving spatial learning tasks.

Virtual Neurorobotics (VNR) to Accelerate Development of Plausible Neuromorphic Brain Architectures

Traditional research in artificial intelligence and machine learning has viewed the brain as a specially adapted information-processing system. More recently the field of social robotics has been advanced to capture the important dynamics of human cognition and interaction. An overarching societal goal of this research is to incorporate the resultant knowledge about intelligence into technology for prosthetic, assistive, security, and decision support applications. However, despite many decades of investment in learning and classification systems, this paradigm has yet to yield truly “intelligent” systems. For this reason, many investigators are now attempting to incorporate more realistic neuromorphic properties into machine learning systems, encouraged by over two decades of neuroscience research that has provided parameters that characterize the brain's interdependent genomic, proteomic, metabolomic, anatomic, and electrophysiological networks. Given the complexity of neural systems, developing tenable models to capture the essence of natural intelligence for real-time application requires that we discriminate features underlying information processing and intrinsic motivation from those reflecting biological constraints (such as maintaining structural integrity and transporting metabolic products). We propose herein a conceptual framework and an iterative method of virtual neurorobotics (VNR) intended to rapidly forward-engineer and test progressively more complex putative neuromorphic brain prototypes for their ability to support intrinsically intelligent, intentional interaction with humans. The VNR system is based on the viewpoint that a truly intelligent system must be driven by emotion rather than programmed tasking, incorporating intrinsic motivation and intentionality. We report pilot results of a closed-loop, real-time interactive VNR system with a spiking neural brain, and provide a video demonstration as online supplemental material.

Framework and implications of virtual neurorobotics

Despite decades of societal investment in artificial learning systems, truly "intelligent" systems have yet to be realized. These traditional models are based on input-output pattern optimization and/or cognitive production rule modeling. One response has been social robotics, using the interaction of human and robot to capture important cognitive dynamics such as cooperation and emotion; to date, these systems still incorporate traditional learning algorithms. More recently, investigators are focusing on the core assumptions of the brain "algorithm" itself-trying to replicate uniquely "neuromorphic" dynamics such as action potential spiking and synaptic learning. Only now are large-scale neuromorphic models becoming feasible, due to the availability of powerful supercomputers and an expanding supply of parameters derived from research into the brain's interdependent electrophysiological, metabolomic and genomic networks. Personal computer technology has also led to the acceptance of computer-generated humanoid images, or "avatars", to represent intelligent actors in virtual realities. In a recent paper, we proposed a method of virtual neurorobotics (VNR) in which the approaches above (social-emotional robotics, neuromorphic brain architectures, and virtual reality projection) are hybridized to rapidly forward-engineer and develop increasingly complex, intrinsically intelligent systems. In this paper, we synthesize our research and related work in the field and provide a framework for VNR, with wider implications for research and practical applications.

Modeling the role of working memory and episodic memory in behavioral tasks

The mechanisms of goal-directed behavior have been studied using reinforcement learning theory, but these theoretical techniques have not often been used to address the role of memory systems in performing behavioral tasks. This work addresses this shortcoming by providing a way in which working memory (WM) and episodic memory may be included in the reinforcement learning framework, then simulating the successful acquisition and performance of six behavioral tasks, drawn from or inspired by the rat experimental literature, that require WM or episodic memory for correct performance. With no delay imposed during the tasks, simulations with WM can solve all of the tasks at above the chance level. When a delay is imposed, simulations with both episodic memory and WM can solve all of the tasks except a disambiguation of odor sequences task.

A model of grid cells involving extra hippocampal path integration, and the hippocampal loop

In this paper, we present a model for the generation of grid cells and the emergence of place cells from multimodal input to the entorhinal cortex (EC). In this model, grid cell activity in the dorsocaudal medial entorhinal cortex (dMEC) [28] results from the operation of a long-distance path integration system located outside the hippocampal formation, presumably in retrosplenial and/or parietal cortex. If the connections between these structures and dMEC are organized as a modulo N operator, the resulting activity of dMEC neurons is a grid cell pattern. Furthermore, a robust high-resolution positional code can be built from a small set of different grid cells if the modulo factors are relatively prime. On the other hand, broad visual place cell activity in the MEC can result from the integration of visual information depending on the view-field of the visual input. The merging of entorhinal visual place cell information and grid cell information in the EC and/or in the dentate gyrus (DG) allows the building of precise and robust "place cells" (e.g., whose activity is maintained if light is suppressed for a short duration). Our model supports our previous proposition that hippocampal "place cell" activity code transitions between two successive states ("transition cells") rather than mere current locations. Furthermore, we discuss the possibility that the hippocampal loop participates in the emergence of grid cell activity but is not sufficient by itself. Finally, path integration at a short time scale (which is reset from one place to the next) would be merged in the subiculum with CA3/CA1 "transition cells" [22] to provide a robust feedback about current action to the deep layer of the entorhinal cortex in order to predict the recognition of the new animal location.

Neurobiologically inspired mobile robot navigation and planning

After a short review of biologically inspired navigation architectures, mainly relying on modeling the hippocampal anatomy, or at least some of its functions, we present a navigation and planning model for mobile robots. This architecture is based on a model of the hippocampal and prefrontal interactions. In particular, the system relies on the definition of a new cell type "transition cells" that encompasses traditional "place cells".

Decoding human brain activity during real-world experiences

The human brain evolved to function and survive in a highly stimulating, complex and fast-changing world. Attempting to ascertain the neural substrates of operating in naturalistic contexts represents a huge challenge. Recently, however, researchers have begun to use several innovative analysis methods to interrogate functional magnetic resonance imaging (fMRI) data collected during dynamic naturalistic tasks. Central to these new developments is the inventive approach taken to segregating neural activity linked to specific events within the overall continuous stream of complex stimulation. In this review, we discuss the recent literature, detailing the key studies and their methods. These analytical techniques can be applied in a wide range of cognitive domains and, thus, offer exciting new opportunities for gaining insights into the brain bases of thoughts and behaviours in the real-world setting where they normally occur.

Retrospective and prospective responses arising in a modeled hippocampus during maze navigation by a brain-based device

Recent recordings of place field activity in rodent hippocampus have revealed correlates of current, recent past, and imminent future events in spatial memory tasks. To analyze these properties, we used a brain-based device, Darwin XI, that incorporated a detailed model of medial temporal structures shaped by experience-dependent synaptic activity. Darwin XI was tested on a plus maze in which it approached a goal arm from different start arms. In the task, a journey corresponded to the route from a particular starting point to a particular goal. During maze navigation, the device developed place-dependent responses in its simulated hippocampus. Journey-dependent place fields, whose activity differed in different journeys through the same maze arm, were found in the recordings of simulated CA1 neuronal units. We also found an approximately equal number of journey-independent place fields. The journey-dependent responses were either retrospective, where activity was present in the goal arm, or prospective, where activity was present in the start arm. Detailed analysis of network dynamics of the neural simulation during behavior revealed that many different neural pathways could stimulate any single CA1 unit. That analysis also revealed that place activity was driven more by hippocampal and entorhinal cortical influences than by sensory cortical input. Moreover, journey-dependent activity was driven more strongly by hippocampal influence than journey-independent activity.

Thoughts, behaviour, and brain dynamics during navigation in the real world

How does the human brain allow us to interact with and navigate through a constantly changing world? Whilst controlled experiments using functional brain imaging can give insightful snapshots of neuronal responses to relatively simplified stimuli, they cannot hope to mirror the challenges faced by the brain in the real world. However, trying to study the brain mechanisms supporting daily living represents a huge challenge. By combining functional neuroimaging, an accurate interactive virtual simulation of a bustling central London (UK), and a novel means of 'reading' participants' thoughts whilst they moved around the city, we ascertained the online neural correlates underpinning navigation in this real-world context. A complex choreography of neural dynamics was revealed comprising focal and distributed, transient and sustained brain activity. Our results provide new insights into the specific roles of individual brain areas, in particular the hippocampus, retrosplenial, and frontal cortices, as well as offering clues about how functional specialisations operate within dynamic brain systems.

Robustness of Visual Place Cells in Dynamic Indoor and Outdoor Environment

In this paper, a model of visual place cells (PCs) based on precise neurobiological data is presented. The robustness of the model in real indoor and outdoor environments is tested. Results show that the interplay between neurobiological modelling and robotic experiments can promote the understanding of the neural structures and the achievement of robust robot navigation algorithms. Short Term Memory (STM), soft competition and sparse coding are important for both landmark identification and computation of PC activities. The extension of the paradigm to outdoor environments has confirmed the robustness of the vision-based model and pointed to improvements in order to further foster its performance.