The traveling salesrat: insights into the dynamics of efficient spatial navigation in the rodent

Contribution of the hippocampus to performance on the traveling salesperson problem in rats

The Traveling Salesman Problem (TSP) is an optimization problem in which the subject attempts to find the shortest possible route that passes through a set of fixed locations exactly once. The TSP is used in cognitive and behavioral research to study problem solving and spatial navigation. While the TSP has been studied in some depth from this perspective, the biological mechanisms underlying the behavior have not yet been explored. The hippocampus is a structure in the brain that is known to be involved in tasks that require spatial memory. Because the TSP requires spatial problem solving, we designed the current study to determine whether the hippocampus is required to find efficient solutions to the TSP, and if so, what role the hippocampus serves. Rats were pretrained on the TSP, which involved learning to retrieve bait from targets in a variety of spatial configurations. Matched for performance, rats were then divided into two groups, receiving either a hippocampal lesion or a control sham surgery. After recovering from surgery, the rats were tested on eight new configurations. A variety of behavioral measures were recorded, including distance travelled, number of revisits, memory span, and latency. The results showed that the sham group outperformed the lesion group on most of these measures. Based on the behavioral data and histological tissue analysis of each group, we determined that the hippocampus is involved in successful performance in the TSP, particularly regarding memory for which targets have already been visited.

Real-time sensory-motor integration of hippocampal place cell replay and prefrontal sequence learning in simulated and physical rat robots for novel path optimization

An open problem in the cognitive dimensions of navigation concerns how previous exploratory experience is reorganized in order to allow the creation of novel efficient navigation trajectories. This behavior is revealed in the "traveling salesrat problem" (TSP) when rats discover the shortest path linking baited food wells after a few exploratory traversals. We have recently published a model of navigation sequence learning, where sharp wave ripple replay of hippocampal place cells transmit "snippets" of the recent trajectories that the animal has explored to the prefrontal cortex (PFC) (Cazin et al. in PLoS Comput Biol 15:e1006624, 2019). PFC is modeled as a recurrent reservoir network that is able to assemble these snippets into the efficient sequence (trajectory of spatial locations coded by place cell activation). The model of hippocampal replay generates a distribution of snippets as a function of their proximity to a reward, thus implementing a form of spatial credit assignment that solves the TSP task. The integrative PFC reservoir reconstructs the efficient TSP sequence based on exposure to this distribution of snippets that favors paths that are most proximal to rewards. While this demonstrates the theoretical feasibility of the PFC-HIPP interaction, the integration of such a dynamic system into a real-time sensory-motor system remains a challenge. In the current research, we test the hypothesis that the PFC reservoir model can operate in a real-time sensory-motor loop. Thus, the main goal of the paper is to validate the model in simulated and real robot scenarios. Place cell activation encoding the current position of the simulated and physical rat robot feeds the PFC reservoir which generates the successor place cell activation that represents the next step in the reproduced sequence in the readout. This is input to the robot, which advances to the coded location and then generates de novo the current place cell activation. This allows demonstration of the crucial role of embodiment. If the spatial code readout from PFC is played back directly into PFC, error can accumulate, and the system can diverge from desired trajectories. This required a spatial filter to decode the PFC code to a location and then recode a new place cell code for that location. In the robot, the place cell vector output of PFC is used to physically displace the robot and then generate a new place cell coded input to the PFC, replacing part of the software recoding procedure that was required otherwise. We demonstrate how this integrated sensory-motor system can learn simple navigation sequences and then, importantly, how it can synthesize novel efficient sequences based on prior experience, as previously demonstrated (Cazin et al. 2019). This contributes to the understanding of hippocampal replay in novel navigation sequence formation and the important role of embodiment.

Reservoir computing model of prefrontal cortex creates novel combinations of previous navigation sequences from hippocampal place-cell replay with spatial reward propagation

As rats learn to search for multiple sources of food or water in a complex environment, they generate increasingly efficient trajectories between reward sites. Such spatial navigation capacity involves the replay of hippocampal place-cells during awake states, generating small sequences of spatially related place-cell activity that we call “snippets”. These snippets occur primarily during sharp-wave-ripples (SWRs). Here we focus on the role of such replay events, as the animal is learning a traveling salesperson task (TSP) across multiple trials. We hypothesize that snippet replay generates synthetic data that can substantially expand and restructure the experience available and make learning more optimal. We developed a model of snippet generation that is modulated by reward, propagated in the forward and reverse directions. This implements a form of spatial credit assignment for reinforcement learning. We use a biologically motivated computational framework known as ‘reservoir computing’ to model prefrontal cortex (PFC) in sequence learning, in which large pools of prewired neural elements process information dynamically through reverberations. This PFC model consolidates snippets into larger spatial sequences that may be later recalled by subsets of the original sequences. Our simulation experiments provide neurophysiological explanations for two pertinent observations related to navigation. Reward modulation allows the system to reject non-optimal segments of experienced trajectories, and reverse replay allows the system to “learn” trajectories that it has not physically experienced, both of which significantly contribute to the TSP behavior.

As rats search for multiple sources of food in a complex environment, they generate increasingly efficient trajectories between reward sites, across multiple trials. This spatial navigation optimization behavior can be measured in the laboratory using a traveling salesperson task (TSP). This likely involves the coordinated replay of place-cell “snippets” between successive trials. We hypothesize that “snippets” can be used by the prefrontal cortex (PFC) to implement a form of reward-modulated reinforcement learning. Our simulation experiments provide neurophysiological explanations for two pertinent observations related to navigation. Reward modulation allows the system to reject non-optimal segments of experienced trajectories, and reverse replay allows the system to “learn” trajectories that it has not physically experienced, both of which significantly contribute to the TSP behavior.

A new rat-compatible robotic framework for spatial navigation behavioral experiments

BACKGROUND

Understanding the neural substrate of information encoding and processing requires a precise control of the animal's behavior. Most of what has been learned from the rodent navigational system results from relatively simple tasks in which the movements of the animal is controlled by corridors or walkways, passive movements, treadmills or virtual reality environments. While a lot has been and continues to be learned from these types of experiments, recent evidence has shown that such artificial constraints may have significant consequences on the functioning of the neural circuits of spatial navigation.

NEW METHODS

We present a novel and alternative approach for effectively controlling the precise direction and speed of movement of the animal in an ethologically realistic environment, using a small robot (Sphero).

RESULTS

We describe the robotic framework and demonstrate its use in replicating pre-programmed or rat-recorded paths. We show that the robot can control the movement of a rat in order to produce specific trajectories and speeds. We demonstrate that the robot can be used to aid the rat in learning a spatial memory task in a large and complex environment. We show that dorsal hippocampal CA1 place cells do not remap when the rat is following the robot. Comparison with existing method(s): Our framework only involves positive motivation and has been tested together with wireless electrophysiology in large and complex environments.

CONCLUSIONS

Our robotic framework can be used to design novel tasks and experiments in which electrophysiological recordings would be largely devoid of maze or task-dependent artifacts.