A computational model of the hippocampus that represents environmental structure and goal location, and guides movement

Learning with sparse reward in a gap junction network inspired by the insect mushroom body

Animals can learn in real-life scenarios where rewards are often only available when a goal is achieved. This 'distal' or 'sparse' reward problem remains a challenge for conventional reinforcement learning algorithms. Here we investigate an algorithm for learning in such scenarios, inspired by the possibility that axo-axonal gap junction connections, observed in neural circuits with parallel fibres such as the insect mushroom body, could form a resistive network. In such a network, an active node represents the task state, connections between nodes represent state transitions and their connection to actions, and current flow to a target state can guide decision making. Building on evidence that gap junction weights are adaptive, we propose that experience of a task can modulate the connections to form a graph encoding the task structure. We demonstrate that the approach can be used for efficient reinforcement learning under sparse rewards, and discuss whether it is plausible as an account of the insect mushroom body.

The functional role of sequentially neuromodulated synaptic plasticity in behavioural learning

To survive, animals have to quickly modify their behaviour when the reward changes. The internal representations responsible for this are updated through synaptic weight changes, mediated by certain neuromodulators conveying feedback from the environment. In previous experiments, we discovered a form of hippocampal Spike-Timing-Dependent-Plasticity (STDP) that is sequentially modulated by acetylcholine and dopamine. Acetylcholine facilitates synaptic depression, while dopamine retroactively converts the depression into potentiation. When these experimental findings were implemented as a learning rule in a computational model, our simulations showed that cholinergic-facilitated depression is important for reversal learning. In the present study, we tested the model's prediction by optogenetically inactivating cholinergic neurons in mice during a hippocampus-dependent spatial learning task with changing rewards. We found that reversal learning, but not initial place learning, was impaired, verifying our computational prediction that acetylcholine-modulated plasticity promotes the unlearning of old reward locations. Further, differences in neuromodulator concentrations in the model captured mouse-by-mouse performance variability in the optogenetic experiments. Our line of work sheds light on how neuromodulators enable the learning of new contingencies.

The formation and use of hierarchical cognitive maps in the brain: A neural network model

Many researchers have tried to model how environmental knowledge is learned by the brain and used in the form of cognitive maps. However, previous work was limited in various important ways: there was little consensus on how these cognitive maps were formed and represented, the planning mechanism was inherently limited to performing relatively simple tasks, and there was little consideration of how these mechanisms would scale up. This paper makes several significant advances. Firstly, the planning mechanism used by the majority of previous work propagates a decaying signal through the network to create a gradient that points towards the goal. However, this decaying signal limited the scale and complexity of tasks that can be solved in this manner. Here we propose several ways in which a network can can self-organize a novel planning mechanism that does not require decaying activity. We also extend this model with a hierarchical planning mechanism: a layer of cells that identify frequently-used sequences of actions and reuse them to significantly increase the efficiency of planning. We speculate that our results may explain the apparent ability of humans and animals to perform model-based planning on both small and large scales without a noticeable loss of efficiency.

The hippocampus and entorhinal cortex encode the path and Euclidean distances to goals during navigation

Background

Despite decades of research on spatial memory, we know surprisingly little about how the brain guides navigation to goals. While some models argue that vectors are represented for navigational guidance, other models postulate that the future path is computed. Although the hippocampal formation has been implicated in processing spatial goal information, it remains unclear whether this region processes path- or vector-related information.

Results

We report neuroimaging data collected from subjects navigating London’s Soho district; these data reveal that both the path distance and the Euclidean distance to the goal are encoded by the medial temporal lobe during navigation. While activity in the posterior hippocampus was sensitive to the distance along the path, activity in the entorhinal cortex was correlated with the Euclidean distance component of a vector to the goal. During travel periods, posterior hippocampal activity increased as the path to the goal became longer, but at decision points, activity in this region increased as the path to the goal became closer and more direct. Importantly, sensitivity to the distance was abolished in these brain areas when travel was guided by external cues.

Conclusions

The results indicate that the hippocampal formation contains representations of both the Euclidean distance and the path distance to goals during navigation. These findings argue that the hippocampal formation houses a flexible guidance system that changes how it represents distance to the goal depending on the fluctuating demands of navigation.

•

The hippocampus represents both the path and the Euclidean distances to goals

•

Entorhinal activity reflects the change in the Euclidean distance when the goal is set

•

The posterior hippocampus represents the future path at different stages en route

•

Significant correlations are abolished when travel is guided by external cues