Mice in a labyrinth show rapid learning, sudden insight, and efficient exploration

A vast space of compact strategies for effective decisions

Inference-based decision-making, which underlies a broad range of behavioral tasks, is typically studied using a small number of handcrafted models. We instead enumerate a complete ensemble of strategies that could be used to effectively, but not necessarily optimally, solve a dynamic foraging task. Each strategy is expressed as a behavioral "program" that uses a limited number of internal states to specify actions conditioned on past observations. We show that the ensemble of strategies is enormous-comprising a quarter million programs with up to five internal states-but can nevertheless be understood in terms of algorithmic "mutations" that alter the structure of individual programs. We devise embedding algorithms that reveal how mutations away from a Bayesian-like strategy can diversify behavior while preserving performance, and we construct a compositional description to link low-dimensional changes in algorithmic structure with high-dimensional changes in behavior. Together, this work provides an alternative approach for understanding individual variability in behavior across animals and tasks.

Behavioral strategy shapes activation of the Vip-Sst disinhibitory circuit in visual cortex

In complex environments, animals can adopt diverse strategies to find rewards. How distinct strategies differentially engage brain circuits is not well understood. Here, we investigate this question, focusing on the cortical Vip-Sst disinhibitory circuit between vasoactive intestinal peptide-postive (Vip) interneurons and somatostatin-positive (Sst) interneurons. We characterize the behavioral strategies used by mice during a visual change detection task. Using a dynamic logistic regression model, we find that individual mice use mixtures of a visual comparison strategy and a statistical timing strategy. Separately, mice also have periods of task engagement and disengagement. Two-photon calcium imaging shows large strategy-dependent differences in neural activity in excitatory, Sst inhibitory, and Vip inhibitory cells in response to both image changes and image omissions. In contrast, task engagement has limited effects on neural population activity. We find that the diversity of neural correlates of strategy can be understood parsimoniously as the increased activation of the Vip-Sst disinhibitory circuit during the visual comparison strategy, which facilitates task-appropriate responses.

From innate to instructed: A new look at perceptual decision-making

Understanding how subjects perceive sensory stimuli in their environment and use this information to guide appropriate actions is a major challenge in neuroscience. To study perceptual decision-making in animals, researchers use tasks that either probe spontaneous responses to stimuli (often described as "naturalistic") or train animals to associate stimuli with experimenter-defined responses. Spontaneous decisions rely on animals' pre-existing knowledge, while trained tasks offer greater versatility, albeit often at the cost of extensive training. Here, we review emerging approaches to investigate perceptual decision-making using both spontaneous and trained behaviors, highlighting their strengths and limitations. Additionally, we propose how trained decision-making tasks could be improved to achieve faster learning and a more generalizable understanding of task rules.

Performance errors during rodent learning reflect a dynamic choice strategy

Humans, even as infants, use cognitive strategies, such as exploration and hypothesis testing, to learn about causal interactions in the environment. In animal learning studies, however, it is challenging to disentangle higher-order behavioral strategies from errors arising from imperfect task knowledge or inherent biases. Here, we trained head-fixed mice on a wheel-based auditory two-choice task and exploited the intra- and inter-animal variability to understand the drivers of errors during learning. During learning, performance errors are dominated by a choice bias, which, despite appearing maladaptive, reflects a dynamic strategy. Early in learning, mice develop an internal model of the task contingencies such that violating their expectation of reward on correct trials (by using short blocks of non-rewarded "probe" trials) leads to an abrupt shift in strategy. During the probe block, mice behave more accurately with less bias, thereby using their learned stimulus-action knowledge to test whether the outcome contingencies have changed. Despite having this knowledge, mice continued to exhibit a strong choice bias during reinforced trials. This choice bias operates on a timescale of tens to hundreds of trials with a dynamic structure, shifting between left, right, and unbiased epochs. Biased epochs also coincided with faster motor kinematics. Although bias decreased across learning, expert mice continued to exhibit short bouts of biased choices interspersed with longer bouts of unbiased choices and higher performance. These findings collectively suggest that during learning, rodents actively probe their environment in a structured manner to refine their decision-making and maintain long-term flexibility.

Magnetic voluntary head-fixation in transgenic rats enables lifespan imaging of hippocampal neurons

The precise neural mechanisms within the brain that contribute to the remarkable lifetime persistence of memory are not fully understood. Two-photon calcium imaging allows the activity of individual cells to be followed across long periods, but conventional approaches require head-fixation, which limits the type of behavior that can be studied. We present a magnetic voluntary head-fixation system that provides stable optical access to the brain during complex behavior. Compared to previous systems that used mechanical restraint, there are no moving parts and animals can engage and disengage entirely at will. This system is failsafe, easy for animals to use and reliable enough to allow long-term experiments to be routinely performed. Animals completed hundreds of trials per session of an odor discrimination task that required 2-4 s fixations. Together with a reflectance fluorescence collection scheme that increases two-photon signal and a transgenic Thy1-GCaMP6f rat line, we are able to reliably image the cellular activity in the hippocampus during behavior over long periods (median 6 months), allowing us track the same neurons over a large fraction of animals' lives (up to 19 months).

Rodent maze studies: from following simple rules to complex map learning

More than 100 years since the first maze designed for rodent research, researchers now have the choice of a variety of mazes that come in many different shapes and sizes. Still old designs get modified and new designs are introduced to fit new research questions. Yet, which maze is the most optimal to use or which training paradigm should be applied, remains up for debate. In this review, we not only provide a historical overview of maze designs and usages in rodent learning and memory research, but also discuss the possible navigational strategies the animals can use to solve each maze. Furthermore, we summarize the different phases of learning that take place when a maze is used as the experimental task. At last, we delve into how training and maze design can affect what the rodents are actually learning in a spatial task.

Learning to learn: Single session acquisition of new rules by freely moving mice

Learning from examples and adapting to new circumstances are fundamental attributes of human cognition. However, it is unclear what conditions allow for fast and successful learning, especially in nonhuman subjects. To determine how rapidly freely moving mice can learn a new discrimination criterion (DC), we design a two-alternative forced-choice visual discrimination paradigm in which the DCs governing the task can change between sessions. We find that experienced animals can learn a new DC after being exposed to only five training and three testing trials. The propensity for single session learning improves over time and is accurately predicted based on animal experience and criterion difficulty. After establishing the procedural learning of a paradigm, mice continuously improve their performance in new circumstances. Thus, mice learn to learn.

Characterizing the dynamic learning process: Implications of a quantitative analysis

Understanding the neural mechanisms involved in learning processes is crucial for unraveling the complexities of behavior and cognition. Sudden change from the untrained level to the fully-learned level is a pivotal feature of instrumental learning. However, the concept of change point and suitable methods to conveniently analyze the characteristics of sudden change in groups remain elusive, which might hinder a fuller understanding of the neural mechanism underlying dynamic leaning process. In the current study, we investigated the learning processes of mice that were trained in an aversive instrumental learning task, and introduced a novel strategy to analyze behavioral variations in instrumental learning, leading to improved clarity on the concept of sudden change and enabling comprehensive group analysis. By applying this novel strategy, we examined the effects of cocaine and a cannabinoid receptor agonist on instrumental learning. Intriguingly, our analysis revealed significant differences in timing and occurrence of sudden changes that were previously overlooked using traditional analysis. Overall, our research advances understanding of behavioral variation during instrumental learning and the interplay between learning behaviors and neurotransmitter systems, contributing to a deeper comprehension of learning processes and informing future investigations and therapeutic interventions.

Endotaxis: A neuromorphic algorithm for mapping, goal-learning, navigation, and patrolling

An animal entering a new environment typically faces three challenges: explore the space for resources, memorize their locations, and navigate towards those targets as needed. Here we propose a neural algorithm that can solve all these problems and operates reliably in diverse and complex environments. At its core, the mechanism makes use of a behavioral module common to all motile animals, namely the ability to follow an odor to its source. We show how the brain can learn to generate internal "virtual odors" that guide the animal to any location of interest. This endotaxis algorithm can be implemented with a simple 3-layer neural circuit using only biologically realistic structures and learning rules. Several neural components of this scheme are found in brains from insects to humans. Nature may have evolved a general mechanism for search and navigation on the ancient backbone of chemotaxis.

Sound-seeking before and after hearing loss in mice

How we move our bodies affects how we perceive sound. For instance, we can explore an environment to seek out the source of a sound and we can use head movements to compensate for hearing loss. How we do this is not well understood because many auditory experiments are designed to limit head and body movements. To study the role of movement in hearing, we developed a behavioral task called sound-seeking that rewarded mice for tracking down an ongoing sound source. Over the course of learning, mice more efficiently navigated to the sound. We then asked how auditory behavior was affected by hearing loss induced by surgical removal of the malleus from the middle ear. An innate behavior, the auditory startle response, was abolished by bilateral hearing loss and unaffected by unilateral hearing loss. Similarly, performance on the sound-seeking task drastically declined after bilateral hearing loss and did not recover. In striking contrast, mice with unilateral hearing loss were only transiently impaired on sound-seeking; over a recovery period of about a week, they regained high levels of performance, increasingly reliant on a different spatial sampling strategy. Thus, even in the face of permanent unilateral damage to the peripheral auditory system, mice recover their ability to perform a naturalistic sound-seeking task. This paradigm provides an opportunity to examine how body movement enables better hearing and resilient adaptation to sensory deprivation.

Visually guided and context-dependent spatial navigation in the translucent fish Danionella cerebrum

Danionella cerebrum (DC) is a promising vertebrate animal model for systems neuroscience due to its small adult brain volume and inherent optical transparency, but the scope of their cognitive abilities remains an area of active research. In this work, we established a behavioral paradigm to study visual spatial navigation in DC and investigate their navigational capabilities and strategies. We initially observed that adult DC exhibit strong negative phototaxis in groups but less so as individuals. Using their dark preference as a motivator, we designed a spatial navigation task inspired by the Morris water maze. Through a series of environmental cue manipulations, we found that DC utilize visual cues to anticipate a reward location and found evidence for landmark-based navigational strategies wherein DC could use both proximal and distal visual cues. When subsets of proximal visual cues were occluded, DC were capable of using distant contextual visual information to solve the task, providing evidence for allocentric spatial navigation. Without proximal visual cues, DC tended to seek out a direct line of sight with at least one distal visual cue while maintaining a positional bias toward the reward location. In total, our behavioral results suggest that DC can be used to study the neural mechanisms underlying spatial navigation with cellular resolution imaging across an adult vertebrate brain.

A mild impairment in reversal learning in a bowl-digging substrate deterministic task but not other cognitive tests in the Dlg2+/- rat model of genetic risk for psychiatric disorder

Variations in the Dlg2 gene have been linked to increased risk for psychiatric disorders, including schizophrenia, autism spectrum disorders, intellectual disability, bipolar disorder, attention deficit hyperactivity disorder, and pubertal disorders. Recent studies have reported disrupted brain circuit function and behaviour in models of Dlg2 knockout and haploinsufficiency. Specifically, deficits in hippocampal synaptic plasticity were found in heterozygous Dlg2+/- rats suggesting impacts on hippocampal dependent learning and cognitive flexibility. Here, we tested these predicted effects with a behavioural characterisation of the heterozygous Dlg2+/- rat model. Dlg2+/- rats exhibited a specific, mild impairment in reversal learning in a substrate deterministic bowl-digging reversal learning task. The performance of Dlg2+/- rats in other bowl digging task, visual discrimination and reversal, novel object preference, novel location preference, spontaneous alternation, modified progressive ratio, and novelty-suppressed feeding test were not impaired. These findings suggest that despite altered brain circuit function, behaviour across different domains is relatively intact in Dlg2+/- rats, with the deficits being specific to only one test of cognitive flexibility. The specific behavioural phenotype seen in this Dlg2+/- model may capture features of the clinical presentation associated with variation in the Dlg2 gene.

Mixtures of strategies underlie rodent behavior during reversal learning

In reversal learning tasks, the behavior of humans and animals is often assumed to be uniform within single experimental sessions to facilitate data analysis and model fitting. However, behavior of agents can display substantial variability in single experimental sessions, as they execute different blocks of trials with different transition dynamics. Here, we observed that in a deterministic reversal learning task, mice display noisy and sub-optimal choice transitions even at the expert stages of learning. We investigated two sources of the sub-optimality in the behavior. First, we found that mice exhibit a high lapse rate during task execution, as they reverted to unrewarded directions after choice transitions. Second, we unexpectedly found that a majority of mice did not execute a uniform strategy, but rather mixed between several behavioral modes with different transition dynamics. We quantified the use of such mixtures with a state-space model, block Hidden Markov Model (block HMM), to dissociate the mixtures of dynamic choice transitions in individual blocks of trials. Additionally, we found that blockHMM transition modes in rodent behavior can be accounted for by two different types of behavioral algorithms, model-free or inference-based learning, that might be used to solve the task. Combining these approaches, we found that mice used a mixture of both exploratory, model-free strategies and deterministic, inference-based behavior in the task, explaining their overall noisy choice sequences. Together, our combined computational approach highlights intrinsic sources of noise in rodent reversal learning behavior and provides a richer description of behavior than conventional techniques, while uncovering the hidden states that underlie the block-by-block transitions.

A unified open-source platform for multimodal neural recording and perturbation during naturalistic behavior

Behavioral neuroscience faces two conflicting demands: long-duration recordings from large neural populations and unimpeded animal behavior. To meet this challenge, we developed ONIX, an open-source data acquisition system with high data throughput (2GB/sec) and low closed-loop latencies (<1ms) that uses a novel 0.3 mm thin tether to minimize behavioral impact. Head position and rotation are tracked in 3D and used to drive active commutation without torque measurements. ONIX can acquire from combinations of passive electrodes, Neuropixels probes, head-mounted microscopes, cameras, 3D-trackers, and other data sources. We used ONIX to perform uninterrupted, long (~7 hours) neural recordings in mice as they traversed complex 3-dimensional terrain. ONIX allowed exploration with similar mobility as non-implanted animals, in contrast to conventional tethered systems which restricted movement. By combining long recordings with full mobility, our technology will enable new progress on questions that require high-quality neural recordings during ethologically grounded behaviors.

A map of spatial navigation for neuroscience

Spatial navigation has received much attention from neuroscientists, leading to the identification of key brain areas and the discovery of numerous spatially selective cells. Despite this progress, our understanding of how the pieces fit together to drive behavior is generally lacking. We argue that this is partly caused by insufficient communication between behavioral and neuroscientific researchers. This has led the latter to under-appreciate the relevance and complexity of spatial behavior, and to focus too narrowly on characterizing neural representations of space-disconnected from the computations these representations are meant to enable. We therefore propose a taxonomy of navigation processes in mammals that can serve as a common framework for structuring and facilitating interdisciplinary research in the field. Using the taxonomy as a guide, we review behavioral and neural studies of spatial navigation. In doing so, we validate the taxonomy and showcase its usefulness in identifying potential issues with common experimental approaches, designing experiments that adequately target particular behaviors, correctly interpreting neural activity, and pointing to new avenues of research.

An automated, low-latency environment for studying the neural basis of behavior in freely moving rats

BACKGROUND

Behavior consists of the interaction between an organism and its environment, and is controlled by the brain. Brain activity varies at sub-second time scales, but behavioral measures are usually coarse (often consisting of only binary trial outcomes).

RESULTS

To overcome this mismatch, we developed the Rat Interactive Foraging Facility (RIFF): a programmable interactive arena for freely moving rats with multiple feeding areas, multiple sound sources, high-resolution behavioral tracking, and simultaneous electrophysiological recordings. The paper provides detailed information about the construction of the RIFF and the software used to control it. To illustrate the flexibility of the RIFF, we describe two complex tasks implemented in the RIFF, a foraging task and a sound localization task. Rats quickly learned to obtain rewards in both tasks. Neurons in the auditory cortex as well as neurons in the auditory field in the posterior insula had sound-driven activity during behavior. Remarkably, neurons in both structures also showed sensitivity to non-auditory parameters such as location in the arena and head-to-body angle.

CONCLUSIONS

The RIFF provides insights into the cognitive capabilities and learning mechanisms of rats and opens the way to a better understanding of how brains control behavior. The ability to do so depends crucially on the combination of wireless electrophysiology and detailed behavioral documentation available in the RIFF.

Few-shot learning: temporal scaling in behavioral and dopaminergic learning

How do we learn associations in the world (e.g., between cues and rewards)? Cue-reward associative learning is controlled in the brain by mesolimbic dopamine1-4. It is widely believed that dopamine drives such learning by conveying a reward prediction error (RPE) in accordance with temporal difference reinforcement learning (TDRL) algorithms5. TDRL implementations are "trial-based": learning progresses sequentially across individual cue-outcome experiences. Accordingly, a foundational assumption-often considered a mere truism-is that the more cue-reward pairings one experiences, the more one learns this association. Here, we disprove this assumption, thereby falsifying a foundational principle of trial-based learning algorithms. Specifically, when a group of head-fixed mice received ten times fewer experiences over the same total time as another, a single experience produced as much learning as ten experiences in the other group. This quantitative scaling also holds for mesolimbic dopaminergic learning, with the increase in learning rate being so high that the group with fewer experiences exhibits dopaminergic learning in as few as four cue-reward experiences and behavioral learning in nine. An algorithm implementing reward-triggered retrospective learning explains these findings. The temporal scaling and few-shot learning observed here fundamentally changes our understanding of the neural algorithms of associative learning.

A computational model of learning flexible navigation in a maze by layout-conforming replay of place cells

Recent experimental observations have shown that the reactivation of hippocampal place cells (PC) during sleep or wakeful immobility depicts trajectories that can go around barriers and can flexibly adapt to a changing maze layout. However, existing computational models of replay fall short of generating such layout-conforming replay, restricting their usage to simple environments, like linear tracks or open fields. In this paper, we propose a computational model that generates layout-conforming replay and explains how such replay drives the learning of flexible navigation in a maze. First, we propose a Hebbian-like rule to learn the inter-PC synaptic strength during exploration. Then we use a continuous attractor network (CAN) with feedback inhibition to model the interaction among place cells and hippocampal interneurons. The activity bump of place cells drifts along paths in the maze, which models layout-conforming replay. During replay in sleep, the synaptic strengths from place cells to striatal medium spiny neurons (MSN) are learned by a novel dopamine-modulated three-factor rule to store place-reward associations. During goal-directed navigation, the CAN periodically generates replay trajectories from the animal's location for path planning, and the trajectory leading to a maximal MSN activity is followed by the animal. We have implemented our model into a high-fidelity virtual rat in the MuJoCo physics simulator. Extensive experiments have demonstrated that its superior flexibility during navigation in a maze is due to a continuous re-learning of inter-PC and PC-MSN synaptic strength.

Mental navigation and the neural mechanisms of insight

How do new ideas come about? The central hypothesis presented here states that insights might happen during mental navigation and correspond to rapid plasticity at the cellular level. We highlight the differences between neocortical and hippocampal mechanisms of insight. We argue that the suddenness of insight can be related to the sudden emergence of place fields in the hippocampus. According to our hypothesis, insights are supported by a state of mind-wandering that can be tied to the process of combining knowledge pieces during sharp-wave ripples (SWRs). Our framework connects the dots between research on creativity, mental navigation, and specific synaptic plasticity mechanisms in the hippocampus.

Humans account for cognitive costs when finding shortcuts: An information-theoretic analysis of navigation

When faced with navigating back somewhere we have been before we might either retrace our steps or seek a shorter path. Both choices have costs. Here, we ask whether it is possible to characterize formally the choice of navigational plans as a bounded rational process that trades off the quality of the plan (e.g., its length) and the cognitive cost required to find and implement it. We analyze the navigation strategies of two groups of people that are firstly trained to follow a "default policy" taking a route in a virtual maze and then asked to navigate to various known goal destinations, either in the way they want ("Go To Goal") or by taking novel shortcuts ("Take Shortcut"). We address these wayfinding problems using InfoRL: an information-theoretic approach that formalizes the cognitive cost of devising a navigational plan, as the informational cost to deviate from a well-learned route (the "default policy"). In InfoRL, optimality refers to finding the best trade-off between route length and the amount of control information required to find it. We report five main findings. First, the navigational strategies automatically identified by InfoRL correspond closely to different routes (optimal or suboptimal) in the virtual reality map, which were annotated by hand in previous research. Second, people deliberate more in places where the value of investing cognitive resources (i.e., relevant goal information) is greater. Third, compared to the group of people who receive the "Go To Goal" instruction, those who receive the "Take Shortcut" instruction find shorter but less optimal solutions, reflecting the intrinsic difficulty of finding optimal shortcuts. Fourth, those who receive the "Go To Goal" instruction modulate flexibly their cognitive resources, depending on the benefits of finding the shortcut. Finally, we found a surprising amount of variability in the choice of navigational strategies and resource investment across participants. Taken together, these results illustrate the benefits of using InfoRL to address navigational planning problems from a bounded rational perspective.

A reinforcement-based mechanism for discontinuous learning

Problem-solving and reasoning involve mental exploration and navigation in sparse relational spaces. A physical analogue is spatial navigation in structured environments such as a network of burrows. Recent experiments with mice navigating a labyrinth show a sharp discontinuity during learning, corresponding to a distinct moment of "sudden insight" when mice figure out long, direct paths to the goal. This discontinuity is seemingly at odds with reinforcement learning (RL), which involves a gradual build-up of a value signal during learning. Here, we show that biologically plausible RL rules combined with persistent exploration generically exhibit discontinuous learning. In tree-like structured environments, positive feedback from learning on behavior generates a "reinforcement wave" with a steep profile. The discontinuity occurs when the wave reaches the starting point. By examining the nonlinear dynamics of reinforcement propagation, we establish a quantitative relationship between the learning rule, the agent's exploration biases, and learning speed. Predictions explain existing data and motivate specific experiments to isolate the phenomenon. Additionally, we characterize the exact learning dynamics of various RL rules for a complex sequential task.

ΔFosB accumulation in hippocampal granule cells drives cFos pattern separation during spatial learning

Mice display signs of fear when neurons that express cFos during fear conditioning are artificially reactivated. This finding gave rise to the notion that cFos marks neurons that encode specific memories. Here we show that cFos expression patterns in the mouse dentate gyrus (DG) change dramatically from day to day in a water maze spatial learning paradigm, regardless of training level. Optogenetic inhibition of neurons that expressed cFos on the first training day affected performance days later, suggesting that these neurons continue to be important for spatial memory recall. The mechanism preventing repeated cFos expression in DG granule cells involves accumulation of ΔFosB, a long-lived splice variant of FosB. CA1 neurons, in contrast, repeatedly expressed cFos. Thus, cFos-expressing granule cells may encode new features being added to the internal representation during the last training session. This form of timestamping is thought to be required for the formation of episodic memories.

Behavior is movement only but how to interpret it? Problems and pitfalls in translational neuroscience-a 40-year experience

Translational research in behavioral neuroscience seeks causes and remedies for human mental health problems in animals, following leads imposed by clinical research in psychiatry. This endeavor faces several problems because scientists must read and interpret animal movements to represent human perceptions, mood, and memory processes. Yet, it is still not known how mammalian brains bundle all these processes into a highly compressed motor output in the brain stem and spinal cord, but without that knowledge, translational research remains aimless. Based on some four decades of experience in the field, the article identifies sources of interpretation problems and illustrates typical translational pitfalls. (1) The sensory world of mice is different. Smell, hearing, and tactile whisker sensations dominate in rodents, while visual input is comparatively small. In humans, the relations are reversed. (2) Mouse and human brains are equated inappropriately: the association cortex makes up a large portion of the human neocortex, while it is relatively small in rodents. The predominant associative cortex in rodents is the hippocampus itself, orchestrating chiefly inputs from secondary sensorimotor areas and generating species-typical motor patterns that are not easily reconciled with putative human hippocampal functions. (3) Translational interpretation of studies of memory or emotionality often neglects the ecology of mice, an extremely small species surviving by freezing or flight reactions that do not need much cognitive processing. (4) Further misinterpretations arise from confounding neuronal properties with system properties, and from rigid mechanistic thinking unaware that many experimentally induced changes in the brain do partially reflect unpredictable compensatory plasticity. (5) Based on observing hippocampal lesion effects in mice indoors and outdoors, the article offers a simplistic general model of hippocampal functions in relation to hypothalamic input and output, placing hypothalamus and the supraspinal motor system at the top of a cerebral hierarchy. (6) Many translational problems could be avoided by inclusion of simple species-typical behaviors as end-points comparable to human cognitive or executive processing, and to rely more on artificial intelligence for recognizing patterns not classifiable by traditional psychological concepts.

Predictive maps in rats and humans for spatial navigation

Much of our understanding of navigation comes from the study of individual species, often with specific tasks tailored to those species. Here, we provide a novel experimental and analytic framework integrating across humans, rats, and simulated reinforcement learning (RL) agents to interrogate the dynamics of behavior during spatial navigation. We developed a novel open-field navigation task ("Tartarus maze") requiring dynamic adaptation (shortcuts and detours) to frequently changing obstructions on the path to a hidden goal. Humans and rats were remarkably similar in their trajectories. Both species showed the greatest similarity to RL agents utilizing a "successor representation," which creates a predictive map. Humans also displayed trajectory features similar to model-based RL agents, which implemented an optimal tree-search planning procedure. Our results help refine models seeking to explain mammalian navigation in dynamic environments and highlight the utility of modeling the behavior of different species to uncover the shared mechanisms that support behavior.

Shared and specialized coding across posterior cortical areas for dynamic navigation decisions

Animals adaptively integrate sensation, planning, and action to navigate toward goal locations in ever-changing environments, but the functional organization of cortex supporting these processes remains unclear. We characterized encoding in approximately 90,000 neurons across the mouse posterior cortex during a virtual navigation task with rule switching. The encoding of task and behavioral variables was highly distributed across cortical areas but differed in magnitude, resulting in three spatial gradients for visual cue, spatial position plus dynamics of choice formation, and locomotion, with peaks respectively in visual, retrosplenial, and parietal cortices. Surprisingly, the conjunctive encoding of these variables in single neurons was similar throughout the posterior cortex, creating high-dimensional representations in all areas instead of revealing computations specialized for each area. We propose that, for guiding navigation decisions, the posterior cortex operates in parallel rather than hierarchically, and collectively generates a state representation of the behavior and environment, with each area specialized in handling distinct information modalities.

Learning, fast and slow

Animals can learn efficiently from a single experience and change their future behavior in response. However, in other instances, animals learn very slowly, requiring thousands of experiences. Here, I survey tasks involving fast and slow learning and consider some hypotheses for what differentiates the underlying neural mechanisms. It has been proposed that fast learning relies on neural representations that favor efficient Hebbian modification of synapses. These efficient representations may be encoded in the genome, resulting in a repertoire of fast learning that differs across species. Alternatively, the required neural representations may be acquired from experience through a slow process of unsupervised learning from the environment.

Innate heuristics and fast learning support escape route selection in mice

When faced with imminent danger, animals must rapidly take defensive actions to reach safety. Mice can react to threatening stimuli in ∼250 milliseconds¹ and, in simple environments, use spatial memory to quickly escape to shelter.^2,3 Natural habitats, however, often offer multiple routes to safety that animals must identify and choose from.⁴ This is challenging because although rodents can learn to navigate complex mazes,^5,6 learning the value of different routes through trial and error during escape could be deadly. Here, we investigated how mice learn to choose between different escape routes. Using environments with paths to shelter of varying length and geometry, we find that mice prefer options that minimize path distance and angle relative to the shelter. This strategy is already present during the first threat encounter and after only ∼10 minutes of exploration in a novel environment, indicating that route selection does not require experience of escaping. Instead, an innate heuristic assigns survival value to each path after rapidly learning the spatial environment. This route selection process is flexible and allows quick adaptation to arenas with dynamic geometries. Computational modeling shows that model-based reinforcement learning agents replicate the observed behavior in environments where the shelter location is rewarding during exploration. These results show that mice combine fast spatial learning with innate heuristics to choose escape routes with the highest survival value. The results further suggest that integrating prior knowledge acquired through evolution with knowledge learned from experience supports adaptation to changing environments and minimizes the need for trial and error when the errors are costly.

•

Mice learn to escape via the fastest route after ∼10 minutes in a new environment

•

Escape routes are learned during exploration and do not require threat exposure

•

Mice prefer escape routes that minimize path distance and angle to shelter

•

Fast route learning can be replicated by model-based reinforcement learning agents

Mice exhibit stochastic and efficient action switching during probabilistic decision making

To obtain rewards in changing and uncertain environments, animals must adapt their behavior. We found that mouse choice and trial-to-trial switching behavior in a dynamic and probabilistic two-choice task could be modeled by equivalent theoretical, algorithmic, and descriptive models. These models capture components of evidence accumulation, choice history bias, and stochasticity in mouse behavior. Furthermore, they reveal that mice adapt their behavior in different environmental contexts by modulating their level of stickiness to their previous choice. Despite deviating from the behavior of a theoretically ideal observer, the empirical models achieve comparable levels of near-maximal reward. These results make predictions to guide interrogation of the neural mechanisms underlying flexible decision-making strategies.

In probabilistic and nonstationary environments, individuals must use internal and external cues to flexibly make decisions that lead to desirable outcomes. To gain insight into the process by which animals choose between actions, we trained mice in a task with time-varying reward probabilities. In our implementation of such a two-armed bandit task, thirsty mice use information about recent action and action–outcome histories to choose between two ports that deliver water probabilistically. Here we comprehensively modeled choice behavior in this task, including the trial-to-trial changes in port selection, i.e., action switching behavior. We find that mouse behavior is, at times, deterministic and, at others, apparently stochastic. The behavior deviates from that of a theoretically optimal agent performing Bayesian inference in a hidden Markov model (HMM). We formulate a set of models based on logistic regression, reinforcement learning, and sticky Bayesian inference that we demonstrate are mathematically equivalent and that accurately describe mouse behavior. The switching behavior of mice in the task is captured in each model by a stochastic action policy, a history-dependent representation of action value, and a tendency to repeat actions despite incoming evidence. The models parsimoniously capture behavior across different environmental conditionals by varying the stickiness parameter, and like the mice, they achieve nearly maximal reward rates. These results indicate that mouse behavior reaches near-maximal performance with reduced action switching and can be described by a set of equivalent models with a small number of relatively fixed parameters.

How learning unfolds in the brain: toward an optimization view

How do changes in the brain lead to learning? To answer this question, consider an artificial neural network (ANN), where learning proceeds by optimizing a given objective or cost function. This "optimization framework" may provide new insights into how the brain learns, as many idiosyncratic features of neural activity can be recapitulated by an ANN trained to perform the same task. Nevertheless, there are key features of how neural population activity changes throughout learning that cannot be readily explained in terms of optimization and are not typically features of ANNs. Here we detail three of these features: (1) the inflexibility of neural variability throughout learning, (2) the use of multiple learning processes even during simple tasks, and (3) the presence of large task-nonspecific activity changes. We propose that understanding the role of these features in the brain will be key to describing biological learning using an optimization framework.

Temporally delayed linear modelling (TDLM) measures replay in both animals and humans

There are rich structures in off-task neural activity which are hypothesized to reflect fundamental computations across a broad spectrum of cognitive functions. Here, we develop an analysis toolkit - temporal delayed linear modelling (TDLM) - for analysing such activity. TDLM is a domain-general method for finding neural sequences that respect a pre-specified transition graph. It combines nonlinear classification and linear temporal modelling to test for statistical regularities in sequences of task-related reactivations. TDLM is developed on the non-invasive neuroimaging data and is designed to take care of confounds and maximize sequence detection ability. Notably, as a linear framework, TDLM can be easily extended, without loss of generality, to capture rodent replay in electrophysiology, including in continuous spaces, as well as addressing second-order inference questions, for example, its temporal and spatial varying pattern. We hope TDLM will advance a deeper understanding of neural computation and promote a richer convergence between animal and human neuroscience.