Latent learning, shortcuts and detours: a computational model

The detour problem in a stochastic environment: Tolman revisited

We designed a grid world task to study human planning and re-planning behavior in an unknown stochastic environment. In our grid world, participants were asked to travel from a random starting point to a random goal position while maximizing their reward. Because they were not familiar with the environment, they needed to learn its characteristics from experience to plan optimally. Later in the task, we randomly blocked the optimal path to investigate whether and how people adjust their original plans to find a detour. To this end, we developed and compared 12 different models. These models were different on how they learned and represented the environment and how they planned to catch the goal. The majority of our participants were able to plan optimally. We also showed that people were capable of revising their plans when an unexpected event occurred. The result from the model comparison showed that the model-based reinforcement learning approach provided the best account for the data and outperformed heuristics in explaining the behavioral data in the re-planning trials.

Goal-directed decision making as probabilistic inference: a computational framework and potential neural correlates

Recent work has given rise to the view that reward-based decision making is governed by two key controllers: a habit system, which stores stimulus-response associations shaped by past reward, and a goal-oriented system that selects actions based on their anticipated outcomes. The current literature provides a rich body of computational theory addressing habit formation, centering on temporal-difference learning mechanisms. Less progress has been made toward formalizing the processes involved in goal-directed decision making. We draw on recent work in cognitive neuroscience, animal conditioning, cognitive and developmental psychology, and machine learning to outline a new theory of goal-directed decision making. Our basic proposal is that the brain, within an identifiable network of cortical and subcortical structures, implements a probabilistic generative model of reward, and that goal-directed decision making is effected through Bayesian inversion of this model. We present a set of simulations implementing the account, which address benchmark behavioral and neuroscientific findings, and give rise to a set of testable predictions. We also discuss the relationship between the proposed framework and other models of decision making, including recent models of perceptual choice, to which our theory bears a direct connection.

A computational model of the hippocampus that represents environmental structure and goal location, and guides movement

Hippocampal place cells (PCs) are believed to represent environmental structure. However, it is unclear how and which brain regions represent goals and guide movements. Recently, another type of cells that fire around a goal was found in rat hippocampus (we designate these cells as goal place cells, GPCs). This suggests that the hippocampus is also involved in goal representation. Assuming that the activities of GPCs depend on the distance to a goal, we propose an adaptive navigation model. By monitoring the population activity of GPCs, the model navigates to shorten the distance to the goal. To achieve the distance-dependent activities of GPCs, plastic connections are assumed between PCs and GPCs, which are modified depending on two reward-triggered activities: activity propagation through PC-PC network representing the topological environmental structure, and the activity of GPCs with different durations. The former activity propagation is regarded as a computational interpretation of "reverse replay" phenomenon found in rat hippocampus. Simulation results confirm that after reaching a goal only once, the model can navigate to the goal along almost the shortest path from arbitrary places in the environment. This indicates that the hippocampus might play a primary role in the representation of not only the environmental structure but also the goal, in addition to guiding the movement. This navigation strategy using the population activity of GPCs is equivalent to the taxis strategy, the simplest and most basic for biological systems. Our model is unique because this simple strategy allows the model to follow the shortest path in the topological map of the environment.

Stochastic modelling of animal movement

Modern animal movement modelling derives from two traditions. Lagrangian models, based on random walk behaviour, are useful for multi-step trajectories of single animals. Continuous Eulerian models describe expected behaviour, averaged over stochastic realizations, and are usefully applied to ensembles of individuals. We illustrate three modern research arenas. (i) Models of home-range formation describe the process of an animal 'settling down', accomplished by including one or more focal points that attract the animal's movements. (ii) Memory-based models are used to predict how accumulated experience translates into biased movement choices, employing reinforced random walk behaviour, with previous visitation increasing or decreasing the probability of repetition. (iii) Lévy movement involves a step-length distribution that is over-dispersed, relative to standard probability distributions, and adaptive in exploring new environments or searching for rare targets. Each of these modelling arenas implies more detail in the movement pattern than general models of movement can accommodate, but realistic empiric evaluation of their predictions requires dense locational data, both in time and space, only available with modern GPS telemetry.

Hierarchical cognitive maps

We describe a computational model of spatial navigation based on experimental studies conducted with human participants. The model builds and uses a hierarchical cognitive map of a large environment. Computer simulations show that the model correctly describes experimental results including hierarchical organization of space and distance estimation. Furthermore, the model predicts that reaction time for distance estimation varies nonlinearly with distance.