Vicarious trial and error

Disrupting the medial prefrontal cortex with designer receptors exclusively activated by designer drug alters hippocampal sharp-wave ripples and their associated cognitive processes

The hippocampus and medial prefrontal cortex (mPFC) interact during a myriad of cognitive processes including decision-making and long-term memory consolidation. Exactly how the mPFC and hippocampus interact during goal-directed decision-making remains to be fully elucidated. During periods of rest, bursts of high-frequency oscillations, termed sharp-wave ripple (SWR), appear in the local field potential. Impairing SWRs on the maze or during post-learning rest can interfere with memory-guided decision-making and memory consolidation. We hypothesize that the hippocampus and mPFC bidirectionally interact during SWRs to support memory consolidation and decision-making. Rats were trained on the neuroeconomic spatial decision-making task, Restaurant Row, to make serial stay-skip decisions where the amount of effort (delay to reward) varied upon entry to each restaurant. Hippocampal cells and SWRs were recorded in rats with the mPFC transduced with inhibitory DREADDs. We found that disrupting the mPFC impaired consolidating SWRs in the hippocampus. Hippocampal SWR rates depended on the internalized value of the reward (derived from individual flavor preferences), a parameter important in decision-making, and disrupting the mPFC changed this relationship. Additionally, we found a dissociation between SWRs that occurred while rats were on the maze dependent upon whether those SWRs occurred while the rat was anticipating food reward or during post-reward consumption.

Frontal circuit specialisations for decision making

There is widespread consensus that distributed circuits across prefrontal and anterior cingulate cortex (PFC/ACC) are critical for reward-based decision making. The circuit specialisations of these areas in primates were likely shaped by their foraging niche, in which decision making is typically sequential, attention-guided and temporally extended. Here, I argue that in humans and other primates, PFC/ACC circuits are functionally specialised in two ways. First, microcircuits found across PFC/ACC are highly recurrent in nature and have synaptic properties that support persistent activity across temporally extended cognitive tasks. These properties provide the basis of a computational account of time-varying neural activity within PFC/ACC as a decision is being made. Second, the macrocircuit connections (to other brain areas) differ between distinct PFC/ACC cytoarchitectonic subregions. This variation in macrocircuit connections explains why PFC/ACC subregions make unique contributions to reward-based decision tasks and how these contributions are shaped by attention. They predict dissociable neural representations to emerge in orbitofrontal, anterior cingulate and dorsolateral prefrontal cortex during sequential attention-guided choice, as recently confirmed in neurophysiological recordings.

Rapid coordination of effective learning by the human hippocampus

Although the human hippocampus is necessary for long-term memory, controversial findings suggest that it may also support short-term memory in the service of guiding effective behaviors during learning. We tested the counterintuitive theory that the hippocampus contributes to long-term memory through remarkably short-term processing, as reflected in eye movements during scene encoding. While viewing scenes for the first time, short-term retrieval operative within the episode over only hundreds of milliseconds was indicated by a specific eye-movement pattern, which was effective in that it enhanced spatiotemporal memory formation. This viewing pattern was predicted by hippocampal theta oscillations recorded from depth electrodes and by shifts toward top-down influence of hippocampal theta on activity within visual perception and attention networks. The hippocampus thus supports short-term memory processing that coordinates behavior in the service of effective spatiotemporal learning.

Chimpanzees (Pan troglodytes) show subtle signs of uncertainty when choices are more difficult

Humans can tell when they find a task difficult. Subtle uncertainty behaviors like changes in motor speed and muscle tension precede and affect these experiences. Theories of animal metacognition likewise stress the importance of endogenous signals of uncertainty as cues that motivate metacognitive behaviors. However, while researchers have investigated second-order behaviors like information seeking and declining difficult trials in nonhuman animals, they have devoted little attention to the behaviors that express the cognitive conflict that gives rise to such behaviors in the first place. Here we explored whether three chimpanzees would, like humans, show hand wavering more when faced with more difficult choices in a touch screen transitive inference task. While accuracy was very high across all conditions, all chimpanzees wavered more frequently in trials that were objectively more difficult, demonstrating a signature behavior which accompanies experiences of difficulty in humans. This lends plausibility to the idea that feelings of uncertainty, like other emotions, can be studied in nonhuman animals. We propose to routinely assess uncertainty behaviors to inform models of procedural metacognition in nonhuman animals.

Reply to commentaries to willpower with and without effort

Twenty-six commentators from several disciplines have written on the assumption that choice is determined by comparative valuation in a common denominator of reward, the "competitive marketplace." There was no apparent disagreement that prospective rewards are discounted hyperbolically, although some found that the resulting predictions could come just as well from other models, including the interpretation of delay as risk and analysis in terms of hot versus cold valuation systems. Several novel ideas emerged.

Clone-structured graph representations enable flexible learning and vicarious evaluation of cognitive maps

Cognitive maps are mental representations of spatial and conceptual relationships in an environment, and are critical for flexible behavior. To form these abstract maps, the hippocampus has to learn to separate or merge aliased observations appropriately in different contexts in a manner that enables generalization and efficient planning. Here we propose a specific higher-order graph structure, clone-structured cognitive graph (CSCG), which forms clones of an observation for different contexts as a representation that addresses these problems. CSCGs can be learned efficiently using a probabilistic sequence model that is inherently robust to uncertainty. We show that CSCGs can explain a variety of cognitive map phenomena such as discovering spatial relations from aliased sensations, transitive inference between disjoint episodes, and formation of transferable schemas. Learning different clones for different contexts explains the emergence of splitter cells observed in maze navigation and event-specific responses in lap-running experiments. Moreover, learning and inference dynamics of CSCGs offer a coherent explanation for disparate place cell remapping phenomena. By lifting aliased observations into a hidden space, CSCGs reveal latent modularity useful for hierarchical abstraction and planning. Altogether, CSCG provides a simple unifying framework for understanding hippocampal function, and could be a pathway for forming relational abstractions in artificial intelligence.

Vicarious Trial-and-Error Is Enhanced During Deliberation in Human Virtual Navigation in a Translational Foraging Task

Foraging tasks provide valuable insights into decision-making as animals decide how to allocate limited resources (such as time). In rodents, vicarious trial-and-error (back and forth movements), or VTE, is an important behavioral measure of deliberation which is enhanced early in learning and when animals are presented with difficult decisions. Using new translational versions of a rodent foraging task (the "Movie Row" and "Candy Row"), humans navigated a virtual maze presented on standard computers to obtain rewards (either short videos or candy) offered after a variable delay. Decision latencies were longer when participants were presented with difficult offers, overrode their preferences, and when they accepted an offer after rejecting a previous offer. In these situations, humans showed VTE-like behavior, where they were more likely to pause and/or reorient one or more times before making a decision. Behavior on these tasks replicated previous results from the rodent foraging task ("Restaurant Row") and a human version lacking a navigation component ("Web-Surf") and revealed some species differences. Compared to survey measures of delay-discounting, willingness to wait for rewards in the foraging task was not related to willingness to wait for hypothetical rewards. And, smoking status (use of cigarettes or e-cigarettes) was associated with stronger discounting of hypothetical future rewards, but was not well-related to performance on the foraging tasks. In contrast, individuals with overweight or obese BMI (≥25) did not show stronger delay-discounting, but individuals with BMI ≥ 25, and especially females, showed reduced sensitivity to sunk-costs (where their decisions were less sensitive to irrecoverable investments of effort) and less deliberation when presented with difficult offers. These data indicate that VTE is a behavioral index of deliberation in humans, and further support the Movie and Candy Row as translational tools to study decision-making in humans with the potential to provide novel insights about decision-making that are relevant to public health.

The Versatile Wayfinder: Prefrontal Contributions to Spatial Navigation

The prefrontal cortex (PFC) supports decision-making, goal tracking, and planning. Spatial navigation is a behavior that taxes these cognitive processes, yet the role of the PFC in models of navigation has been largely overlooked. In humans, activity in dorsolateral PFC (dlPFC) and ventrolateral PFC (vlPFC) during detours, reveal a role in inhibition and replanning. Dorsal anterior cingulate cortex (dACC) is implicated in planning and spontaneous internally-generated changes of route. Orbitofrontal cortex (OFC) integrates representations of the environment with the value of actions, providing a 'map' of possible decisions. In rodents, medial frontal areas interact with hippocampus during spatial decisions and switching between navigation strategies. In reviewing these advances, we provide a framework for how different prefrontal regions may contribute to different stages of navigation.

Decreased investigatory head scanning during exploration in learning-impaired, aged rats

"Head scanning" is an investigatory behavior that has been linked to spatial exploration and the one-trial formation or strengthening of place cells in the hippocampus. Previous studies have demonstrated that a subset of aged rats with normal spatial learning performance show head scanning rates during a novel, local-global cue-mismatch manipulation that are similar to those of young rats. However, these aged rats demonstrated different patterns of expression of neural activity markers in brain regions associated with spatial learning, perhaps suggesting neural mechanisms that compensate for age-related brain changes. These prior studies did not investigate the head scanning properties of aged rats that had spatial learning impairments. The present study analyzed head scanning behavior in young, aged-unimpaired, and aged-impaired Long Evans rats. Aged-impaired rats performed the head scan behavior at a lower rate than the young rats. These results suggest that decreased attention to spatial landmarks may be a contributing factor to the spatial learning deficits shown by the aged-impaired rats.

Reinforcement learning approaches to hippocampus-dependent flexible spatial navigation

Humans and non-human animals show great flexibility in spatial navigation, including the ability to return to specific locations based on as few as one single experience. To study spatial navigation in the laboratory, watermaze tasks, in which rats have to find a hidden platform in a pool of cloudy water surrounded by spatial cues, have long been used. Analogous tasks have been developed for human participants using virtual environments. Spatial learning in the watermaze is facilitated by the hippocampus. In particular, rapid, one-trial, allocentric place learning, as measured in the delayed-matching-to-place variant of the watermaze task, which requires rodents to learn repeatedly new locations in a familiar environment, is hippocampal dependent. In this article, we review some computational principles, embedded within a reinforcement learning framework, that utilise hippocampal spatial representations for navigation in watermaze tasks. We consider which key elements underlie their efficacy, and discuss their limitations in accounting for hippocampus-dependent navigation, both in terms of behavioural performance (i.e. how well do they reproduce behavioural measures of rapid place learning) and neurobiological realism (i.e. how well do they map to neurobiological substrates involved in rapid place learning). We discuss how an actor-critic architecture, enabling simultaneous assessment of the value of the current location and of the optimal direction to follow, can reproduce one-trial place learning performance as shown on watermaze and virtual delayed-matching-to-place tasks by rats and humans, respectively, if complemented with map-like place representations. The contribution of actor-critic mechanisms to delayed-matching-to-place performance is consistent with neurobiological findings implicating the striatum and hippocampo-striatal interaction in delayed-matching-to-place performance, given that the striatum has been associated with actor-critic mechanisms. Moreover, we illustrate that hierarchical computations embedded within an actor-critic architecture may help to account for aspects of flexible spatial navigation. The hierarchical reinforcement learning approach separates trajectory control via a temporal-difference error from goal selection via a goal prediction error and may account for flexible, trial-specific, navigation to familiar goal locations, as required in some arm-maze place memory tasks, although it does not capture one-trial learning of new goal locations, as observed in open field, including watermaze and virtual, delayed-matching-to-place tasks. Future models of one-shot learning of new goal locations, as observed on delayed-matching-to-place tasks, should incorporate hippocampal plasticity mechanisms that integrate new goal information with allocentric place representation, as such mechanisms are supported by substantial empirical evidence.

Considerations for Pairing Cognitive Behavioral Therapies and Non-invasive Brain Stimulation: Ignore at Your Own Risk

Multimodal approaches combining cognitive behavioral therapies (CBT) with non-invasive brain stimulation (NIBS) hold promise for improving the treatment of neuropsychiatric disorders. As this is a relatively new approach, it is a critical time to identify guiding principles and methodological considerations to enhance research rigor. In the current paper, we argue for a principled approach to CBT and NIBS pairings based on synergistic activation of neural circuits and identify key considerations about CBT that may influence pairing with NIBS. Careful consideration of brain-state interactions and CBT-related nuances will increase the potential for these combinations to be positively synergistic.

Potential roles of the rodent medial prefrontal cortex in conflict resolution between multiple decision-making systems

Mammalian decision-making is mediated by the interaction of multiple, neurally and computationally separable decision systems. Having multiple systems requires a mechanism to manage conflict and converge onto the selection of singular actions. A long history of evidence has pointed to the prefrontal cortex as a central component in processing the interactions between distinct decision systems and resolving conflicts among them. In this chapter we review four theories of how that interaction might occur and identify how the medial prefrontal cortex in the rodent may be involved in each theory. We then present experimental predictions implied by the neurobiological data in the context of each theory as a starting point for future investigation of medial prefrontal cortex and decision-making.

The Missing Link Between Memory and Reinforcement Learning

Reinforcement learning systems usually assume that a value function is defined over all states (or state-action pairs) that can immediately give the value of a particular state or action. These values are used by a selection mechanism to decide which action to take. In contrast, when humans and animals make decisions, they collect evidence for different alternatives over time and take action only when sufficient evidence has been accumulated. We have previously developed a model of memory processing that includes semantic, episodic and working memory in a comprehensive architecture. Here, we describe how this memory mechanism can support decision making when the alternatives cannot be evaluated based on immediate sensory information alone. Instead we first imagine, and then evaluate a possible future that will result from choosing one of the alternatives. Here we present an extended model that can be used as a model for decision making that depends on accumulating evidence over time, whether that information comes from the sequential attention to different sensory properties or from internal simulation of the consequences of making a particular choice. We show how the new model explains both simple immediate choices, choices that depend on multiple sensory factors and complicated selections between alternatives that require forward looking simulations based on episodic and semantic memory structures. In this framework, vicarious trial and error is explained as an internal simulation that accumulates evidence for a particular choice. We argue that a system like this forms the "missing link" between more traditional ideas of semantic and episodic memory, and the associative nature of reinforcement learning.

Time as the fourth dimension in the hippocampus

Experiences of animal and human beings are structured by the continuity of space and time coupled with the uni-directionality of time. In addition to its pivotal position in spatial processing and navigation, the hippocampal system also plays a central, multiform role in several types of temporal processing. These include timing and sequence learning, at scales ranging from meso-scales of seconds to macro-scales of minutes, hours, days and beyond, encompassing the classical functions of short term memory, working memory, long term memory, and episodic memories (comprised of information about when, what, and where). This review article highlights the principal findings and behavioral contexts of experiments in rats showing: 1) timing: tracking time during delays by hippocampal 'time cells' and during free behavior by hippocampal-afferent lateral entorhinal cortex ramping cells; 2) 'online' sequence processing: activity coding sequences of events during active behavior; 3) 'off-line' sequence replay: during quiescence or sleep, orderly reactivation of neuronal assemblies coding awake sequences. Studies in humans show neurophysiological correlates of episodic memory comparable to awake replay. Neural mechanisms are discussed, including ion channel properties, plateau and ramping potentials, oscillations of excitation and inhibition of population activity, bursts of high amplitude discharges (sharp wave ripples), as well as short and long term synaptic modifications among and within cell assemblies. Specifically conceived neural network models will suggest processes supporting the emergence of scalar properties (Weber's law), and include different classes of feedforward and recurrent network models, with intrinsic hippocampal coding for 'transitions' (sequencing of events or places).

Adult neurogenesis alters response to an aversive distractor in a labyrinth maze without affecting spatial learning or memory

Adult neurogenesis has been implicated in learning and memory of complex spatial environments. However, new neurons also play a role in nonmnemonic behavior, including the stress response and attention shifting. Many commonly used spatial tasks are very simple, and unsuitable for detecting neurogenesis effects, or are aversively motivated, making it difficult to dissociate effects on spatial learning and memory from effects on stress. We have therefore created a novel complex spatial environment, the flex maze, to enable reward-mediated testing of spatial learning in a flexibly configurable labyrinth. Using a pharmacogenetic method to completely inhibit neurogenesis in adulthood, we found that rats lacking new neurons (TK rats) and wild type controls completed and remembered most mazes equally well. However, control rats were slower to complete peppermint-scented mazes than other mazes, while neurogenesis-deficient rats showed no effect of mint on maze behavior, completing these mazes significantly faster than control rats. Additional testing found that wild type and TK rats showed similar detection of, avoidance of, and glucocorticoid response to the mint odor. These results suggest that spatial learning and memory in a labyrinth task is unaffected by the loss of new neurons, but that these cells affect the ability of an aversive stimulus to distract rats from completing the maze.

An Amygdala Circuit Mediates Experience-Dependent Momentary Arrests during Exploration

Exploration of novel environments ensures survival and evolutionary fitness. It is expressed through exploratory bouts and arrests that change dynamically based on experience. Neural circuits mediating exploratory behavior should therefore integrate experience and use it to select the proper behavioral output. Using a spatial exploration assay, we uncovered an experience-dependent increase in momentary arrests in locations where animals arrested previously. Calcium imaging in freely exploring mice revealed a genetically and projection-defined neuronal ensemble in the basolateral amygdala that is active during self-paced behavioral arrests. This ensemble was recruited in an experience-dependent manner, and closed-loop optogenetic manipulation of these neurons revealed that they are sufficient and necessary to drive experience-dependent arrests during exploration. Projection-specific imaging and optogenetic experiments revealed that these arrests are effected by basolateral amygdala neurons projecting to the central amygdala, uncovering an amygdala circuit that mediates momentary arrests in familiar places but not avoidance or anxiety/fear-like behaviors.

Neural signatures underlying deliberation in human foraging decisions

Humans have a remarkable capacity to mentally project themselves far ahead in time. This ability, which entails the mental simulation of events, is thought to be fundamental to deliberative decision making, as it allows us to search through and evaluate possible choices. Many decisions that humans make are foraging decisions, in which one must decide whether an available offer is worth taking, when compared to unknown future possibilities (i.e., the background). Using a translational decision-making paradigm designed to reveal decision preferences in rats, we found that humans engaged in deliberation when making foraging decisions. A key feature of this task is that preferences (and thus, value) are revealed as a function of serial choices. Like rats, humans also took longer to respond when faced with difficult decisions near their preference boundary, which was associated with prefrontal and hippocampal activation, exemplifying cross-species parallels in deliberation. Furthermore, we found that voxels within the visual cortices encoded neural representations of the available possibilities specifically following regret-inducing experiences, in which the subject had previously rejected a good offer only to encounter a low-valued offer on the subsequent trial.

Distinct Neural Circuits Underlie Prospective and Concurrent Memory-Guided Behavior

The past is the best predictor of the future. This simple postulate belies the complex neurobiological mechanisms that facilitate an individual's use of memory to guide decisions. Previous research has shown integration of memories bias decision-making. Alternatively, memories can prospectively guide our choices. Here, we elucidate the mechanisms and timing of hippocampal (HPC), medial prefrontal cortex (mPFC), and striatal contributions during prospective memory-guided decision-making. We develop an associative learning task in which the correct choice is conditional on the preceding stimulus. Two distinct networks emerge: (1) a prospective circuit consisting of the HPC, putamen, mPFC, and other cortical regions, which exhibit increased activation preceding successful conditional decisions and (2) a concurrent circuit comprising the caudate, dorsolateral prefrontal cortex (dlPFC), and additional cortical structures that engage during the execution of correct conditional choices. Our findings demonstrate distinct neurobiological circuits through which memory prospectively biases decisions and influences choice execution.

Interindividual differences in memory system local field potential activity predict behavioral strategy on a dual-solution T-maze

Individuals can use diverse behavioral strategies to navigate their environment including hippocampal-dependent place strategies reliant upon cognitive maps and striatal-dependent response strategies reliant upon egocentric body turns. The existence of multiple memory systems appears to facilitate successful navigation across a wide range of environmental and physiological conditions. The mechanisms by which these systems interact to ultimately generate a unitary behavioral response, however, remain unclear. We trained 20 male, Sprague-Dawley rats on a dual-solution T-maze while simultaneously recording local field potentials that were targeted to the dorsolateral striatum and dorsal hippocampus. Eight rats spontaneously exhibited a place strategy while the remaining 12 rats exhibited a response strategy. Interindividual differences in behavioral strategy were associated with distinct patterns of LFP activity between the dorsolateral striatum and dorsal hippocampus. Specifically, striatal-hippocampal theta activity was in-phase in response rats and out-of-phase in place rats and response rats exhibited elevated striatal-hippocampal coherence across a wide range of frequency bands. These contrasting striatal-hippocampal activity regimes were (a) present during both maze-learning and a 30 min premaze habituation period and (b) could be used to train support vector machines to reliably predict behavioral strategy. Distinct patterns of neuronal activity across multiple memory systems, therefore, appear to bias behavioral strategy selection and thereby contribute to interindividual differences in behavior.

Uncertainty promotes information-seeking actions, but what information?

Navigating an unfamiliar city almost certainly brings out uncertainty about getting from place to place. This uncertainty, in turn, triggers information gathering. While navigational uncertainty is common, little is known about what type of information people seek when they are uncertain. The primary choices for information types with environments include landmarks (distal or local), landmark configurations (relation between two or more landmarks), and a distinct geometry, at least for some environments. Uncertainty could lead individuals to more likely seek one of these information types. Extant research informs both predictions about and empirical work exploring this question. This review covers relevant cognitive literature and then suggests empirical approaches to better understand information-seeking actions triggered by uncertainty. Notably, we propose that examining continuous navigation data can provide important insights into information seeking. Benefits of continuous data will be elaborated through one paradigm, spatial reorientation, which intentionally induces uncertainty through disorientation and cue conflict. While this and other methods have been used previously, data have primarily reflected only the final choice. Continuous behavior during a task can better reveal the cognition-action loop contributing to spatial learning and decision making.

Willpower with and without effort

Most authors who discuss willpower assume that everyone knows what it is, but our assumptions differ to such an extent that we talk past each other. We agree that willpower is the psychological function that resists temptations - variously known as impulses, addictions, or bad habits; that it operates simultaneously with temptations, without prior commitment; and that use of it is limited by its cost, commonly called effort, as well as by the person's skill at executive functioning. However, accounts are usually not clear about how motivation functions during the application of willpower, or how motivation is related to effort. Some accounts depict willpower as the perceiving or formation of motivational contingencies that outweigh the temptation, and some depict it as a continuous use of mechanisms that interfere with re-weighing the temptation. Some others now suggest that impulse control can bypass motivation altogether, although they refer to this route as habit rather than willpower.It is argued here that willpower should be recognized as either or both of two distinct functions, which can be called resolve and suppression. Resolve is based on interpretation of a current choice as a test case for a broader set of future choices, which puts at stake more than the outcome of the current choice. Suppression is inhibiting valuation of (modulating) and/or keeping attention from (filtering) immediate alternatives to a current intention. Perception of current choices as test cases for broader outcomes may result in reliable preference for these outcomes, which is experienced as an effortless habit - a successful result of resolve, not an alternative method of self-control. Some possible brain imaging correlates are reviewed.

Spatial encoding in dorsomedial prefrontal cortex and hippocampus is related during deliberation

Deliberation is thought to involve the internal simulation of the outcomes of candidate actions, the valuation of those outcomes, and the selection of the actions with the highest expected value. While it is known that deliberation involves prefrontal cortical areas, specifically the dorsomedial prefrontal cortex (dmPFC), as well as the hippocampus (HPC) and other brain regions, how these areas process prospective information and select actions is not well understood. We recorded simultaneously from ensembles in dmPFC and CA1 of dorsal HPC in rats during performance of a spatial contingency switching task, and examined the relationships between spatial and reward encoding in these two areas during deliberation at the choice point. We found that CA1 and dmPFC represented either goal locations or the current position simultaneously, but that when goal locations were encoded, HPC and dmPFC did not always represent the same goal location. Ensemble activity in dmPFC predicted when HPC would represent goal locations, but on a broad timescale on the order of seconds. Also, reward encoding in dmPFC increased during hippocampal theta cycles where CA1 ensembles represented the goal location. These results suggest that dmPFC and HPC share prospective information during deliberation, that dmPFC may influence whether HPC represents prospective information, and that information recalled about goal locations by HPC may be integrated into dmPFC reward representations on fast timescales.

Heuristics and optimal solutions to the breadth-depth dilemma

In multialternative risky choice, we are often faced with the opportunity to allocate our limited information-gathering capacity between several options before receiving feedback. In such cases, we face a natural trade-off between breadth-spreading our capacity across many options-and depth-gaining more information about a smaller number of options. Despite its broad relevance to daily life, including in many naturalistic foraging situations, the optimal strategy in the breadth-depth trade-off has not been delineated. Here, we formalize the breadth-depth dilemma through a finite-sample capacity model. We find that, if capacity is small (∼10 samples), it is optimal to draw one sample per alternative, favoring breadth. However, for larger capacities, a sharp transition is observed, and it becomes best to deeply sample a very small fraction of alternatives, which roughly decreases with the square root of capacity. Thus, ignoring most options, even when capacity is large enough to shallowly sample all of them, is a signature of optimal behavior. Our results also provide a rich casuistic for metareasoning in multialternative decisions with bounded capacity using close-to-optimal heuristics.

Stress drives deliberative tendencies by influencing vicarious trial and error in decision making

Previous studies have reported the effects of stress on decision making. However, the wide range of findings make it difficult to identify the fundamental effects of stress on decision making and, therefore, how stress affects decision making remains unknown. To investigate the influence of stress on decision making, we employed "vicarious trial and error" (VTE), which refers to a rat's behavior of orienting the head toward options at a decision point. VTE is thought to reflect mental simulation for possible options preceding a decision. We examined effects of acute restraint stress on VTE in a T-maze choice task. VTE depended on learning and past reward outcomes. Acute restraint stress before rats ran the T-maze choice task induced VTE, especially in trials with low demand of VTE, and increased the number of head orientations and time spent during each VTE. On the other hand, stress did not affect task performance (probability of advantageous choice) and patterns of behavioral choice (win-stay lose-shift, exploration-exploitation). In addition, stress activated serotonergic and noradrenergic neurons in the dorsal raphe nucleus and locus coeruleus, which are modulators of impulsivity and attentional control in decision making. These results suggest that stress in decision making drives the VTE process, which may lead to deep consideration, over-thinking, and indecisiveness.

Awake delta and theta-rhythmic hippocampal network modes during intermittent locomotor behaviors in the rat

Delta-frequency network activity is commonly associated with sleep or behavioral disengagement accompanied by a dearth of cortical spiking, but delta in awake behaving animals is not well understood. We show that hippocampal (HC) synchronization in the delta frequency band (1-4 Hz) is related to animals' locomotor behavior using detailed analyses of the HC local field potential (LFP) and simultaneous head- and body-tracking data. In contrast to running-speed modulation of the theta rhythm (6-10 Hz), delta was most prominent when animals were stationary or moving slowly, that is, when theta and fast gamma (65-120 Hz) were weak, and often developed rapidly when animals paused briefly between runs. We next combined time-frequency decomposition of the LFP with hierarchical clustering algorithms to categorize momentary estimations of the power spectral density (PSD) into putative modes of HC activity. Delta and theta power were strikingly orthogonal across spectral modes, as well as across bouts of precisely defined running and stationary behavior. Delta-band and theta-band coherences between HC recording sites were monotonically related to theta-delta ratios across modes; and whereas theta coherence between HC and medial prefrontal cortex (mPFC) increased during running, delta-band coherence between mPFC and HC increased during stationary bouts. Taken together, our findings suggest that delta-dominated network modes (and corresponding mPFC-HC couplings) represent functionally distinct circuit dynamics that are temporally and behaviorally interspersed among theta-dominated modes during navigation. As such, delta modes could play a fundamental role in coordinating encoding and retrieval mechanisms or decision-making processes at a timescale that segments event sequences within behavioral episodes. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Multiscale Computation and Dynamic Attention in Biological and Artificial Intelligence

Biological and artificial intelligence (AI) are often defined by their capacity to achieve a hierarchy of short-term and long-term goals that require incorporating information over time and space at both local and global scales. More advanced forms of this capacity involve the adaptive modulation of integration across scales, which resolve computational inefficiency and explore-exploit dilemmas at the same time. Research in neuroscience and AI have both made progress towards understanding architectures that achieve this. Insight into biological computations come from phenomena such as decision inertia, habit formation, information search, risky choices and foraging. Across these domains, the brain is equipped with mechanisms (such as the dorsal anterior cingulate and dorsolateral prefrontal cortex) that can represent and modulate across scales, both with top-down control processes and by local to global consolidation as information progresses from sensory to prefrontal areas. Paralleling these biological architectures, progress in AI is marked by innovations in dynamic multiscale modulation, moving from recurrent and convolutional neural networks-with fixed scalings-to attention, transformers, dynamic convolutions, and consciousness priors-which modulate scale to input and increase scale breadth. The use and development of these multiscale innovations in robotic agents, game AI, and natural language processing (NLP) are pushing the boundaries of AI achievements. By juxtaposing biological and artificial intelligence, the present work underscores the critical importance of multiscale processing to general intelligence, as well as highlighting innovations and differences between the future of biological and artificial intelligence.

Spatial planning with long visual range benefits escape from visual predators in complex naturalistic environments

It is uncontroversial that land animals have more elaborated cognitive abilities than their aquatic counterparts such as fish. Yet there is no apparent a-priori reason for this. A key cognitive faculty is planning. We show that in visually guided predator-prey interactions, planning provides a significant advantage, but only on land. During animal evolution, the water-to-land transition resulted in a massive increase in visual range. Simulations of behavior identify a specific type of terrestrial habitat, clustered open and closed areas (savanna-like), where the advantage of planning peaks. Our computational experiments demonstrate how this patchy terrestrial structure, in combination with enhanced visual range, can reveal and hide agents as a function of their movement and create a selective benefit for imagining, evaluating, and selecting among possible future scenarios-in short, for planning. The vertebrate invasion of land may have been an important step in their cognitive evolution.

Three brain states in the hippocampus and cortex

Contemporary brain research seeks to understand how cognition is reducible to neural activity. Crucially, much of this effort is guided by a scientific paradigm that views neural activity as essentially driven by external stimuli. In contrast, recent perspectives argue that this paradigm is by itself inadequate and that understanding patterns of activity intrinsic to the brain is needed to explain cognition. Yet, despite this critique, the stimulus-driven paradigm still dominates-possibly because a convincing alternative has not been clear. Here, we review a series of findings suggesting such an alternative. These findings indicate that neural activity in the hippocampus occurs in one of three brain states that have radically different anatomical, physiological, representational, and behavioral correlates, together implying different functional roles in cognition. This three-state framework also indicates that neural representations in the hippocampus follow a surprising pattern of organization at the timescale of ∼1 s or longer. Lastly, beyond the hippocampus, recent breakthroughs indicate three parallel states in the cortex, suggesting shared principles and brain-wide organization of intrinsic neural activity.

Representation of probabilistic outcomes during risky decision-making

Goal-directed behaviour requires prospectively retrieving and evaluating multiple possible action outcomes. While a plethora of studies suggested sequential retrieval for deterministic choice outcomes, it remains unclear whether this is also the case when integrating multiple probabilistic outcomes of the same action. We address this question by capitalising on magnetoencephalography (MEG) in humans who made choices in a risky foraging task. We train classifiers to distinguish MEG field patterns during presentation of two probabilistic outcomes (reward, loss), and then apply these to decode such patterns during deliberation. First, decoded outcome representations have a temporal structure, suggesting alternating retrieval of the outcomes. Moreover, the probability that one or the other outcome is being represented depends on loss magnitude, but not on loss probability, and it predicts the chosen action. In summary, we demonstrate decodable outcome representations during probabilistic decision-making, which are sequentially structured, depend on task features, and predict subsequent action.

Foraging in Mind

People and other animals can search for information inside their heads. Where does this ability come from, and what does it enable cognitive systems to do? In this article, we address the behavioral and cognitive similarities between search in external environments and internal environments (e.g., memory). These require both maplike representations and the means to navigate them, and the latter involves modulation between exploitation and exploration analogous to a foraging process called area-restricted search. These findings have implications for understanding a number of cognitive abilities commonly considered to be hallmarks of the human species, such as well-developed executive control and goal-directed cognition, autonoetic consciousness (i.e., self-awareness), deliberation, and free will. Moreover, this research extends our conception of what organisms may share these abilities and how they evolved.

"Chasing the first high": memory sampling in drug choice

Although vivid memories of drug experiences are prevalent within clinical contexts and addiction folklore ("chasing the first high"), little is known about the relevance of cognitive processes governing memory retrieval to substance use disorder. Drawing on recent work that identifies episodic memory's influence on decisions for reward, we propose a framework in which drug choices are biased by selective sampling of individual memories during two phases of addiction: (i) downward spiral into persistent use and (ii) relapse. Consideration of how memory retrieval influences the addiction process suggests novel treatment strategies. Rather than try to break learned associations between drug cues and drug rewards, treatment should aim to strengthen existing and/or create new associations between drug cues and drug-inconsistent rewards.

Dorsomedial prefrontal cortex and hippocampus represent strategic context even while simultaneously changing representation throughout a task session

Dorsomedial prefrontal cortex (dmPFC) and hippocampus (HPC) are thought to play complementary roles in a spatial working memory and decision-making network, where spatial information from HPC informs representations in dmPFC, and contextual information from dmPFC biases how HPC recalls that information. We recorded simultaneously from neural ensembles in rodent dmPFC and HPC as rats performed a rule-switching task, and found that ensembles in dmPFC and HPC simultaneously encoded task contingencies and other time-varying information. While ensembles in HPC transitioned to represent new contingencies at the same time as rats updated their strategies to be consistent with the new contingency, dmPFC ensembles transitioned earlier. Neural representations of other time-varying information also changed faster in dmPFC than in HPC. Our results suggest that HPC and dmPFC represent contingencies while simultaneously representing other information which changes over time, and that this contextual information is integrated into hippocampal representations more slowly than in dmPFC.

Modeling awake hippocampal reactivations with model-based bidirectional search

Hippocampal offline reactivations during reward-based learning, usually categorized as replay events, have been found to be important for performance improvement over time and for memory consolidation. Recent computational work has linked these phenomena to the need to transform reward information into state-action values for decision making and to propagate it to all relevant states of the environment. Nevertheless, it is still unclear whether an integrated reinforcement learning mechanism could account for the variety of awake hippocampal reactivations, including variety in order (forward and reverse reactivated trajectories) and variety in the location where they occur (reward site or decision-point). Here, we present a model-based bidirectional search model which accounts for a variety of hippocampal reactivations. The model combines forward trajectory sampling from current position and backward sampling through prioritized sweeping from states associated with large reward prediction errors until the two trajectories connect. This is repeated until stabilization of state-action values (convergence), which could explain why hippocampal reactivations drastically diminish when the animal's performance stabilizes. Simulations in a multiple T-maze task show that forward reactivations are prominently found at decision-points while backward reactivations are exclusively generated at reward sites. Finally, the model can generate imaginary trajectories that are not allowed to the agent during task performance. We raise some experimental predictions and implications for future studies of the role of the hippocampo-prefronto-striatal network in learning.

Rho GTPase Regulators and Effectors in Autism Spectrum Disorders: Animal Models and Insights for Therapeutics

The Rho family GTPases are small G proteins that act as molecular switches shuttling between active and inactive forms. Rho GTPases are regulated by two classes of regulatory proteins, guanine nucleotide exchange factors (GEFs) and GTPase-activating proteins (GAPs). Rho GTPases transduce the upstream signals to downstream effectors, thus regulating diverse cellular processes, such as growth, migration, adhesion, and differentiation. In particular, Rho GTPases play essential roles in regulating neuronal morphology and function. Recent evidence suggests that dysfunction of Rho GTPase signaling contributes substantially to the pathogenesis of autism spectrum disorder (ASD). It has been found that 20 genes encoding Rho GTPase regulators and effectors are listed as ASD risk genes by Simons foundation autism research initiative (SFARI). This review summarizes the clinical evidence, protein structure, and protein expression pattern of these 20 genes. Moreover, ASD-related behavioral phenotypes in animal models of these genes are reviewed, and the therapeutic approaches that show successful treatment effects in these animal models are discussed.

Deliberating trade-offs with the future

Many fundamental choices in life are intertemporal: they involve trade-offs between sooner and later outcomes. In recent years there has been a surge of interest into how people make intertemporal decisions, given that such decisions are ubiquitous in everyday life and central in domains from substance use to climate change action. While it is clear that people make decisions according to rules, intuitions and habits, they also commonly deliberate over their options, thinking through potential outcomes and reflecting on their own preferences. In this Perspective, we bring to bear recent research into the higher-order capacities that underpin deliberation-particularly those that enable people to think about the future (prospection) and their own thinking (metacognition)-to shed light on intertemporal decision-making. We show how a greater appreciation for these mechanisms of deliberation promises to advance our understanding of intertemporal decision-making and unify a wide range of otherwise disparate choice phenomena.

Constant Sub-second Cycling between Representations of Possible Futures in the Hippocampus

Cognitive faculties such as imagination, planning, and decision-making entail the ability to represent hypothetical experience. Crucially, animal behavior in natural settings implies that the brain can represent hypothetical future experience not only quickly but also constantly over time, as external events continually unfold. To determine how this is possible, we recorded neural activity in the hippocampus of rats navigating a maze with multiple spatial paths. We found neural activity encoding two possible future scenarios (two upcoming maze paths) in constant alternation at 8 Hz: one scenario per ∼125-ms cycle. Further, we found that the underlying dynamics of cycling (both inter- and intra-cycle dynamics) generalized across qualitatively different representational correlates (location and direction). Notably, cycling occurred across moving behaviors, including during running. These findings identify a general dynamic process capable of quickly and continually representing hypothetical experience, including that of multiple possible futures.

One cranium, two brains not yet introduced: Distinct but complementary views of the social brain

Social behavior is pervasive across the animal kingdom, and elucidating how the brain enables animals to respond to social contexts is of great interest and profound importance. Our understanding of 'the social brain' has been fractured as it has matured. Two drastically different conceptualizations of the social brain have emerged with relatively little awareness of each other. In this review, we briefly recount the history behind the two dominant definitions of a social brain. The divide that has emerged between these visions can, in part, be attributed to differential attention to cortical or sub-cortical regions in the brain, and differences in methodology, comparative perspectives, and emphasis on functional specificity or generality. We discuss how these factors contribute to a lack of communication between research efforts, and propose ways in which each version of the social brain can benefit from the perspectives, tools, and approaches of the other. Interface between the two characterizations of social brain networks is sure to provide essential insight into what the social brain encompasses.

Dynamics of Awake Hippocampal-Prefrontal Replay for Spatial Learning and Memory-Guided Decision Making

Spatial learning requires remembering and choosing paths to goals. Hippocampal place cells replay spatial paths during immobility in reverse and forward order, offering a potential mechanism. However, how replay supports both goal-directed learning and memory-guided decision making is unclear. We therefore continuously tracked awake replay in the same hippocampal-prefrontal ensembles throughout learning of a spatial alternation task. We found that, during pauses between behavioral trajectories, reverse and forward hippocampal replay supports an internal cognitive search of available past and future possibilities and exhibits opposing learning gradients for prediction of past and future behavioral paths, respectively. Coordinated hippocampal-prefrontal replay distinguished correct past and future paths from alternative choices, suggesting a role in recall of past paths to guide planning of future decisions for spatial working memory. Our findings reveal a learning shift from hippocampal reverse-replay-based retrospective evaluation to forward-replay-based prospective planning, with prefrontal readout of memory-guided paths for learning and decision making.

Hippocampal BDNF regulates a shift from flexible, goal-directed to habit memory system function following cocaine abstinence

The transition from recreational drug use to addiction involves pathological learning processes that support a persistent shift from flexible, goal-directed to habit behavioral control. Here, we examined the molecular mechanisms supporting altered function in hippocampal (HPC) and dorsolateral striatum (DLS) memory systems following abstinence from repeated cocaine. After 3 weeks of cocaine abstinence (experimenter- or self-administered), we tested new behavioral learning in male rats using a dual-solution maze task, which provides an unbiased approach to assess HPC- versus DLS-dependent learning strategies. Dorsal hippocampus (dHPC) and DLS brain tissues were collected after memory testing to identify transcriptional adaptations associated with cocaine-induced shifts in behavioral learning. Our results demonstrate that following prolonged cocaine abstinence rats show a bias toward the use of an inflexible, habit memory system (DLS) in lieu of a more flexible, easily updated memory system involving the HPC. This memory system bias was associated with upregulation and downregulation of brain-derived neurotrophic factor (BDNF) gene expression and transcriptionally permissive histone acetylation (acetylated histone H3, AcH3) in the DLS and dHPC, respectively. Using viral-mediated gene transfer, we overexpressed BDNF in the dHPC during cocaine abstinence and new maze learning. This manipulation restored HPC-dependent behavioral control. These findings provide a system-level understanding of altered plasticity and behavioral learning following cocaine abstinence and inform mechanisms mediating the organization of learning and memory more broadly.

Hippocampus Is What Happens while You're Busy Making Other Plans

In this issue of Neuron, Vikbladh et al. (2019) provide evidence to suggest that the human hippocampus, long known to support spatial memory, also plays a causal role in model-based planning.

Distinct recruitment of dorsomedial and dorsolateral striatum erodes with extended training

Hypotheses of striatal orchestration of behavior ascribe distinct functions to striatal subregions, with the dorsolateral striatum (DLS) especially implicated in habitual and skilled performance. Thus neural activity patterns recorded from the DLS, but not the dorsomedial striatum (DMS), should be correlated with habitual and automatized performance. Here, we recorded DMS and DLS neural activity in rats during training in a task promoting habitual lever pressing. Despite improving performance across sessions, clear changes in corresponding neural activity patterns were not evident in DMS or DLS during early training. Although DMS and DLS activity patterns were distinct during early training, their activity was similar following extended training. Finally, performance after extended training was not associated with DMS disengagement, as would be predicted from prior work. These results suggest that behavioral sequences may continue to engage both striatal regions long after initial acquisition, when skilled performance is consolidated.

Learning, memory and consolidation mechanisms for behavioral control in hierarchically organized cortico-basal ganglia systems

This article aims to provide a synthesis on the question how brain structures cooperate to accomplish hierarchically organized behaviors, characterized by low‐level, habitual routines nested in larger sequences of planned, goal‐directed behavior. The functioning of a connected set of brain structures—prefrontal cortex, hippocampus, striatum, and dopaminergic mesencephalon—is reviewed in relation to two important distinctions: (a) goal‐directed as opposed to habitual behavior and (b) model‐based and model‐free learning. Recent evidence indicates that the orbitomedial prefrontal cortices not only subserve goal‐directed behavior and model‐based learning, but also code the “landscape” (task space) of behaviorally relevant variables. While the hippocampus stands out for its role in coding and memorizing world state representations, it is argued to function in model‐based learning but is not required for coding of action–outcome contingencies, illustrating that goal‐directed behavior is not congruent with model‐based learning. While the dorsolateral and dorsomedial striatum largely conform to the dichotomy between habitual versus goal‐directed behavior, ventral striatal functions go beyond this distinction. Next, we contextualize findings on coding of reward‐prediction errors by ventral tegmental dopamine neurons to suggest a broader role of mesencephalic dopamine cells, viz. in behavioral reactivity and signaling unexpected sensory changes. We hypothesize that goal‐directed behavior is hierarchically organized in interconnected cortico‐basal ganglia loops, where a limbic‐affective prefrontal‐ventral striatal loop controls action selection in a dorsomedial prefrontal–striatal loop, which in turn regulates activity in sensorimotor‐dorsolateral striatal circuits. This structure for behavioral organization requires alignment with mechanisms for memory formation and consolidation. We propose that frontal corticothalamic circuits form a high‐level loop for memory processing that initiates and temporally organizes nested activities in lower‐level loops, including the hippocampus and the ripple‐associated replay it generates. The evidence on hierarchically organized behavior converges with that on consolidation mechanisms in suggesting a frontal‐to‐caudal directionality in processing control.