Vicarious trial and error

Dynamics of Awake Hippocampal-Prefrontal Replay for Spatial Learning and Memory-Guided Decision Making

Spatial learning requires remembering and choosing paths to goals. Hippocampal place cells replay spatial paths during immobility in reverse and forward order, offering a potential mechanism. However, how replay supports both goal-directed learning and memory-guided decision making is unclear. We therefore continuously tracked awake replay in the same hippocampal-prefrontal ensembles throughout learning of a spatial alternation task. We found that, during pauses between behavioral trajectories, reverse and forward hippocampal replay supports an internal cognitive search of available past and future possibilities and exhibits opposing learning gradients for prediction of past and future behavioral paths, respectively. Coordinated hippocampal-prefrontal replay distinguished correct past and future paths from alternative choices, suggesting a role in recall of past paths to guide planning of future decisions for spatial working memory. Our findings reveal a learning shift from hippocampal reverse-replay-based retrospective evaluation to forward-replay-based prospective planning, with prefrontal readout of memory-guided paths for learning and decision making.

Hippocampal BDNF regulates a shift from flexible, goal-directed to habit memory system function following cocaine abstinence

The transition from recreational drug use to addiction involves pathological learning processes that support a persistent shift from flexible, goal-directed to habit behavioral control. Here, we examined the molecular mechanisms supporting altered function in hippocampal (HPC) and dorsolateral striatum (DLS) memory systems following abstinence from repeated cocaine. After 3 weeks of cocaine abstinence (experimenter- or self-administered), we tested new behavioral learning in male rats using a dual-solution maze task, which provides an unbiased approach to assess HPC- versus DLS-dependent learning strategies. Dorsal hippocampus (dHPC) and DLS brain tissues were collected after memory testing to identify transcriptional adaptations associated with cocaine-induced shifts in behavioral learning. Our results demonstrate that following prolonged cocaine abstinence rats show a bias toward the use of an inflexible, habit memory system (DLS) in lieu of a more flexible, easily updated memory system involving the HPC. This memory system bias was associated with upregulation and downregulation of brain-derived neurotrophic factor (BDNF) gene expression and transcriptionally permissive histone acetylation (acetylated histone H3, AcH3) in the DLS and dHPC, respectively. Using viral-mediated gene transfer, we overexpressed BDNF in the dHPC during cocaine abstinence and new maze learning. This manipulation restored HPC-dependent behavioral control. These findings provide a system-level understanding of altered plasticity and behavioral learning following cocaine abstinence and inform mechanisms mediating the organization of learning and memory more broadly.

Hippocampus Is What Happens while You're Busy Making Other Plans

In this issue of Neuron, Vikbladh et al. (2019) provide evidence to suggest that the human hippocampus, long known to support spatial memory, also plays a causal role in model-based planning.

Distinct recruitment of dorsomedial and dorsolateral striatum erodes with extended training

Hypotheses of striatal orchestration of behavior ascribe distinct functions to striatal subregions, with the dorsolateral striatum (DLS) especially implicated in habitual and skilled performance. Thus neural activity patterns recorded from the DLS, but not the dorsomedial striatum (DMS), should be correlated with habitual and automatized performance. Here, we recorded DMS and DLS neural activity in rats during training in a task promoting habitual lever pressing. Despite improving performance across sessions, clear changes in corresponding neural activity patterns were not evident in DMS or DLS during early training. Although DMS and DLS activity patterns were distinct during early training, their activity was similar following extended training. Finally, performance after extended training was not associated with DMS disengagement, as would be predicted from prior work. These results suggest that behavioral sequences may continue to engage both striatal regions long after initial acquisition, when skilled performance is consolidated.

Learning, memory and consolidation mechanisms for behavioral control in hierarchically organized cortico-basal ganglia systems

This article aims to provide a synthesis on the question how brain structures cooperate to accomplish hierarchically organized behaviors, characterized by low‐level, habitual routines nested in larger sequences of planned, goal‐directed behavior. The functioning of a connected set of brain structures—prefrontal cortex, hippocampus, striatum, and dopaminergic mesencephalon—is reviewed in relation to two important distinctions: (a) goal‐directed as opposed to habitual behavior and (b) model‐based and model‐free learning. Recent evidence indicates that the orbitomedial prefrontal cortices not only subserve goal‐directed behavior and model‐based learning, but also code the “landscape” (task space) of behaviorally relevant variables. While the hippocampus stands out for its role in coding and memorizing world state representations, it is argued to function in model‐based learning but is not required for coding of action–outcome contingencies, illustrating that goal‐directed behavior is not congruent with model‐based learning. While the dorsolateral and dorsomedial striatum largely conform to the dichotomy between habitual versus goal‐directed behavior, ventral striatal functions go beyond this distinction. Next, we contextualize findings on coding of reward‐prediction errors by ventral tegmental dopamine neurons to suggest a broader role of mesencephalic dopamine cells, viz. in behavioral reactivity and signaling unexpected sensory changes. We hypothesize that goal‐directed behavior is hierarchically organized in interconnected cortico‐basal ganglia loops, where a limbic‐affective prefrontal‐ventral striatal loop controls action selection in a dorsomedial prefrontal–striatal loop, which in turn regulates activity in sensorimotor‐dorsolateral striatal circuits. This structure for behavioral organization requires alignment with mechanisms for memory formation and consolidation. We propose that frontal corticothalamic circuits form a high‐level loop for memory processing that initiates and temporally organizes nested activities in lower‐level loops, including the hippocampus and the ripple‐associated replay it generates. The evidence on hierarchically organized behavior converges with that on consolidation mechanisms in suggesting a frontal‐to‐caudal directionality in processing control.

Is hippocampal remapping the physiological basis for context?

In 1980, Nadel and Wilner extended Richard Hirsh's notion that the hippocampus creates environmental representations, called "contexts," suggesting that the fundamental structure of context was the spatial representation proposed by O'Keefe and Nadel's landmark book, The Hippocampus as a Cognitive Map (1978). This book, in turn, derives from the discovery that individual hippocampal neurons act as place cells, with the complete set of place cells tiling an enclosure, forming a type of spatial map. It was found that unique environments had unique place cell representations. That is, if one takes the hippocampal map of a specific environment, this representation scrambles, or "remaps" when the animal is placed in a different environment. Several authors have speculated that "maps" and "remapping" form the physiological substrates for context and context shifting. One difficulty with this definition is that it is exclusively spatial; it can only be inferred when an animal locomotes in an enclosure. There are five aims for this article. The first is to give an historical overview of context as a variable that controls behavior. The second aim is to give an historical overview of concepts of place cell maps and remapping. The third aim is to propose an updated definition of a place cell map, based on temporal rather than spatial overlaps, which adds flexibility. The fourth aim is to address the issue of whether the biological phenomenon of hippocampal remapping, is, in fact, the substrate for shifts in the psychological phenomenon of context. The final aim is speculation of how contextual representations may contribute to effective behavior.

A passive, camera-based head-tracking system for real-time, three-dimensional estimation of head position and orientation in rodents

Tracking head position and orientation in small mammals is crucial for many applications in the field of behavioral neurophysiology, from the study of spatial navigation to the investigation of active sensing and perceptual representations. Many approaches to head tracking exist, but most of them only estimate the 2D coordinates of the head over the plane where the animal navigates. Full reconstruction of the pose of the head in 3D is much more more challenging and has been achieved only in handful of studies, which employed headsets made of multiple LEDs or inertial units. However, these assemblies are rather bulky and need to be powered to operate, which prevents their application in wireless experiments and in the small enclosures often used in perceptual studies. Here we propose an alternative approach, based on passively imaging a lightweight, compact, 3D structure, painted with a pattern of black dots over a white background. By applying a cascade of feature extraction algorithms that progressively refine the detection of the dots and reconstruct their geometry, we developed a tracking method that is highly precise and accurate, as assessed through a battery of validation measurements. We show that this method can be used to study how a rat samples sensory stimuli during a perceptual discrimination task and how a hippocampal place cell represents head position over extremely small spatial scales. Given its minimal encumbrance and wireless nature, our method could be ideal for high-throughput applications, where tens of animals need to be simultaneously and continuously tracked.NEW & NOTEWORTHY Head tracking is crucial in many behavioral neurophysiology studies. Yet reconstruction of the head's pose in 3D is challenging and typically requires implanting bulky, electrically powered headsets that prevent wireless experiments and are hard to employ in operant boxes. Here we propose an alternative approach, based on passively imaging a compact, 3D dot pattern that, once implanted over the head of a rodent, allows estimating the pose of its head with high precision and accuracy.

Hippocampal Theta-Gamma Coupling Reflects State-Dependent Information Processing in Decision Making

During decision making, hippocampal activity encodes information sometimes about present and sometimes about potential future plans. The mechanisms underlying this transition remain unknown. Building on the evidence that gamma oscillations at different frequencies (low gamma [LG], 30–55 Hz; high gamma [HG], 60–90 Hz; and epsilon, 100–140 Hz) reflect inputs from different circuits, we identified how changes in those frequencies reflect different information-processing states. Using a unique noradrenergic manipulation by clonidine, which shifted both neural representations and gamma states, we found that future representations depended on gamma components. These changes were identifiable on each cycle of theta as asymmetries in the theta cycle, which arose from changes within the ratio of LG and HG power and the underlying phases of those gamma rhythms within the theta cycle. These changes in asymmetry of the theta cycle reflected changes in representations of present and future on each theta cycle.

Caching mechanisms for habit formation in Active Inference

A popular distinction in the human and animal learning literature is between deliberate (or willed) and habitual (or automatic) modes of control. Extensive evidence indicates that, after sufficient learning, living organisms develop behavioural habits that permit them saving computational resources. Furthermore, humans and other animals are able to transfer control from deliberate to habitual modes (and vice versa), trading off efficiently flexibility and parsimony - an ability that is currently unparalleled by artificial control systems. Here, we discuss a computational implementation of habit formation, and the transfer of control from deliberate to habitual modes (and vice versa) within Active Inference: a computational framework that merges aspects of cybernetic theory and of Bayesian inference. To model habit formation, we endow an Active Inference agent with a mechanism to "cache" (or memorize) policy probabilities from previous trials, and reuse them to skip - in part or in full - the inferential steps of deliberative processing. We exploit the fact that the relative quality of policies, conditioned upon hidden states, is constant over trials; provided that contingencies and prior preferences do not change. This means the only quantity that can change policy selection is the prior distribution over the initial state - where this prior is based upon the posterior beliefs from previous trials. Thus, an agent that caches the quality (or the probability) of policies can safely reuse cached values to save on cognitive and computational resources - unless contingencies change. Our simulations illustrate the computational benefits, but also the limits, of three caching schemes under Active Inference. They suggest that key aspects of habitual behaviour - such as perseveration - can be explained in terms of caching policy probabilities. Furthermore, they suggest that there may be many kinds (or stages) of habitual behaviour, each associated with a different caching scheme; for example, caching associated or not associated with contextual estimation. These schemes are more or less impervious to contextual and contingency changes.

Pathfinder: open source software for analyzing spatial navigation search strategies

Spatial navigation is a universal behavior that varies depending on goals, experience and available sensory stimuli. Spatial navigational tasks are routinely used to study learning, memory and goal-directed behavior, in both animals and humans. One popular paradigm for testing spatial memory is the Morris water maze, where subjects learn the location of a hidden platform that offers escape from a pool of water. Researchers typically express learning as a function of the latency to escape, though this reveals little about the underlying navigational strategies. Recently, a number of studies have begun to classify water maze search strategies in order to clarify the precise spatial and mnemonic functions of different brain regions, and to identify which aspects of spatial memory are disrupted in disease models. However, despite their usefulness, strategy analyses have not been widely adopted due to the lack of software to automate analyses. To address this need we developed Pathfinder, an open source application for analyzing spatial navigation behaviors. In a representative dataset, we show that Pathfinder effectively characterizes the development of highly-specific spatial search strategies as male and female mice learn a standard spatial water maze. Pathfinder can read data files from commercially- and freely-available software packages, is optimized for classifying search strategies in water maze paradigms, and can also be used to analyze 2D navigation by other species, and in other tasks, as long as timestamped xy coordinates are available. Pathfinder is simple to use, can automatically determine pool and platform geometry, generates heat maps, analyzes navigation with respect to multiple goal locations, and can be updated to accommodate future developments in spatial behavioral analyses. Given these features, Pathfinder may be a useful tool for studying how navigational strategies are regulated by the environment, depend on specific neural circuits, and are altered by pathology.

Pathfinder: open source software for analyzing spatial navigation search strategies

Spatial navigation is a universal behavior that varies depending on goals, experience and available sensory stimuli. Spatial navigational tasks are routinely used to study learning, memory and goal-directed behavior, in both animals and humans. One popular paradigm for testing spatial memory is the Morris water maze, where subjects learn the location of a hidden platform that offers escape from a pool of water. Researchers typically express learning as a function of the latency to escape, though this reveals little about the underlying navigational strategies. Recently, a number of studies have begun to classify water maze search strategies in order to clarify the precise spatial and mnemonic functions of different brain regions, and to identify which aspects of spatial memory are disrupted in disease models. However, despite their usefulness, strategy analyses have not been widely adopted due to the lack of software to automate analyses. To address this need we developed Pathfinder, an open source application for analyzing spatial navigation behaviors. In a representative dataset, we show that Pathfinder effectively characterizes the development of highly-specific spatial search strategies as male and female mice learn a standard spatial water maze. Pathfinder can read data files from commercially- and freely-available software packages, is optimized for classifying search strategies in water maze paradigms, and can also be used to analyze 2D navigation by other species, and in other tasks, as long as timestamped xy coordinates are available. Pathfinder is simple to use, can automatically determine pool and platform geometry, generates heat maps, analyzes navigation with respect to multiple goal locations, and can be updated to accommodate future developments in spatial behavioral analyses. Given these features, Pathfinder may be a useful tool for studying how navigational strategies are regulated by the environment, depend on specific neural circuits, and are altered by pathology.

The hippocampal sharp wave-ripple in memory retrieval for immediate use and consolidation

Various cognitive functions have long been known to require the hippocampus. Recently, progress has been made in identifying the hippocampal neural activity patterns that implement these functions. One such pattern is the sharp wave-ripple (SWR), an event associated with highly synchronous neural firing in the hippocampus and modulation of neural activity in distributed brain regions. Hippocampal spiking during SWRs can represent past or potential future experience, and SWR-related interventions can alter subsequent memory performance. These findings and others suggest that SWRs support both memory consolidation and memory retrieval for processes such as decision-making. In addition, studies have identified distinct types of SWR based on representational content, behavioural state and physiological features. These various findings regarding SWRs suggest that different SWR types correspond to different cognitive functions, such as retrieval and consolidation. Here, we introduce another possibility - that a single SWR may support more than one cognitive function. Taking into account classic psychological theories and recent molecular results that suggest that retrieval and consolidation share mechanisms, we propose that the SWR mediates the retrieval of stored representations that can be utilized immediately by downstream circuits in decision-making, planning, recollection and/or imagination while simultaneously initiating memory consolidation processes.

Adult neurogenesis promotes efficient, nonspecific search strategies in a spatial alternation water maze task

Goal-directed navigation requires learning strategies that are efficient and minimize costs. In some cases it may be desirable to flexibly adjust behavioral responses depending on the cues that vary from one episode to the next. In others, successful navigation might be achieved with inflexible, habit-like responses that reduce cognitive load. Adult neurogenesis is believed to contribute to the spatial processing functions of the hippocampus, particularly when behavioral flexibility is required. However, little is known about the role of neurogenesis in spatial navigation when goals are unpredictable or change repeatedly according to certain rules. We hypothesized that neurogenesis is necessary in a spatial navigation task that involves different patterns of reinforcement. Intact and neurogenesis-deficient rats were trained to escape to one of two possible platform locations in a spatial water maze. The platform either repeated in the same location for all trials in a day, alternated between two locations across trials, or randomly moved between the two locations. Neurogenesis selectively enhanced escape performance in the alternating condition, but not by improving platform choice accuracy. Instead, neurogenesis-intact rats made fewer search errors and developed an efficient habit-like strategy where they consistently swam to a preferred location. If the platform was not present, they proceeded to the other possible location. In contrast, neurogenesis-deficient rats were indecisive and navigationally less-efficient. Thus, in conditions where goals follow a predictable spatiotemporal pattern, adult neurogenesis promotes the adoption of navigation strategies that are spatially nonspecific but, nonetheless, accurate and efficient.

Validating the theoretical bases of sleep reactivation during sharp-wave ripples and their association with emotional valence

Sleep is important for memory consolidation, and an abundant literature suggests that reactivation in the hippocampus during sleep is instrumental to this process. Yet, the current interpretation of activity during sharp-waves ripples (SWRs), as replay of wake experiences, relies on hypotheses that, while widely accepted, have only recently begun to be tested directly. Moreover, this theory has been mainly studied in the context of pure spatial learning, and it is still not clear how emotional valence can fit into this conceptual framework when considering reward- or punishment-based learning. In this review, we will present recent experimental arguments validating the interpretation of sleep replay as reactivation of awake experiences and examine the evidence showing that the emotional valence is also replayed during sleep in a coordinated fashion with hippocampal SWRs. Finally, we will detail recent experiments showing that brain-computer interfaces can be used to modify the emotional valence associated with sleep replay.

Above and beyond the concrete: The diverse representational substrates of the predictive brain

In recent years, scientists have increasingly taken to investigate the predictive nature of cognition. We argue that prediction relies on abstraction, and thus theories of predictive cognition need an explicit theory of abstract representation. We propose such a theory of the abstract representational capacities that allow humans to transcend the "here-and-now." Consistent with the predictive cognition literature, we suggest that the representational substrates of the mind are built as a hierarchy, ranging from the concrete to the abstract; however, we argue that there are qualitative differences between elements along this hierarchy, generating meaningful, often unacknowledged, diversity. Echoing views from philosophy, we suggest that the representational hierarchy can be parsed into: modality-specific representations, instantiated on perceptual similarity; multimodal representations, instantiated primarily on the discovery of spatiotemporal contiguity; and categorical representations, instantiated primarily on social interaction. These elements serve as the building blocks of complex structures discussed in cognitive psychology (e.g., episodes, scripts) and are the inputs for mental representations that behave like functions, typically discussed in linguistics (i.e., predicators). We support our argument for representational diversity by explaining how the elements in our ontology are all required to account for humans' predictive cognition (e.g., in subserving logic-based prediction; in optimizing the trade-off between accurate and detailed predictions) and by examining how the neuroscientific evidence coheres with our account. In doing so, we provide a testable model of the neural bases of conceptual cognition and highlight several important implications to research on self-projection, reinforcement learning, and predictive-processing models of psychopathology.

Indicators and Criteria of Consciousness in Animals and Intelligent Machines: An Inside-Out Approach

In today's society, it becomes increasingly important to assess which non-human and non-verbal beings possess consciousness. This review article aims to delineate criteria for consciousness especially in animals, while also taking into account intelligent artifacts. First, we circumscribe what we mean with "consciousness" and describe key features of subjective experience: qualitative richness, situatedness, intentionality and interpretation, integration and the combination of dynamic and stabilizing properties. We argue that consciousness has a biological function, which is to present the subject with a multimodal, situational survey of the surrounding world and body, subserving complex decision-making and goal-directed behavior. This survey reflects the brain's capacity for internal modeling of external events underlying changes in sensory state. Next, we follow an inside-out approach: how can the features of conscious experience, correlating to mechanisms inside the brain, be logically coupled to externally observable ("outside") properties? Instead of proposing criteria that would each define a "hard" threshold for consciousness, we outline six indicators: (i) goal-directed behavior and model-based learning; (ii) anatomic and physiological substrates for generating integrative multimodal representations; (iii) psychometrics and meta-cognition; (iv) episodic memory; (v) susceptibility to illusions and multistable perception; and (vi) specific visuospatial behaviors. Rather than emphasizing a particular indicator as being decisive, we propose that the consistency amongst these indicators can serve to assess consciousness in particular species. The integration of scores on the various indicators yields an overall, graded criterion for consciousness, somewhat comparable to the Glasgow Coma Scale for unresponsive patients. When considering theoretically derived measures of consciousness, it is argued that their validity should not be assessed on the basis of a single quantifiable measure, but requires cross-examination across multiple pieces of evidence, including the indicators proposed here. Current intelligent machines, including deep learning neural networks (DLNNs) and agile robots, are not indicated to be conscious yet. Instead of assessing machine consciousness by a brief Turing-type of test, evidence for it may gradually accumulate when we study machines ethologically and across time, considering multiple behaviors that require flexibility, improvisation, spontaneous problem-solving and the situational conspectus typically associated with conscious experience.

Retrosplenial Cortical Representations of Space and Future Goal Locations Develop with Learning

Recent findings suggest that long-term spatial and contextual memories depend on the retrosplenial cortex (RSC) [1-5]. RSC damage impairs navigation in humans and rodents [6-8], and the RSC is closely interconnected with brain regions known to play a role in navigation, including the hippocampus and anterior thalamus [9, 10]. Navigation-related neural activity is seen in humans [11] and rodents, including spatially localized firing [12, 13], directional firing [12, 14, 15], and responses to navigational cues [16]. RSC neuronal activity is modulated by allocentric, egocentric, and route-centered spatial reference frames [17, 18], consistent with an RSC role in integrating different kinds of navigational information [19]. However, the relationship between RSC firing patterns and spatial memory remains largely unexplored, as previous physiology studies have not employed behavioral tasks with a clear memory demand. To address this, we trained rats on a continuous T-maze alternation task and examined RSC firing patterns throughout learning. We found that the RSC developed a distributed population-level representation of the rat's spatial location and current trajectory to the goal as the rats learned. After the rats reached peak performance, RSC firing patterns began to represent the upcoming goal location as the rats approached the choice point. These neural simulations of the goal emerged at the same time that lesions impaired alternation performance, suggesting that the RSC gradually acquired task representations that contribute to navigational decision-making.

Hippocampal Sequences During Exploration: Mechanisms and Functions

Although the hippocampus plays a critical role in spatial and episodic memories, the mechanisms underlying memory formation, stabilization, and recall for adaptive behavior remain relatively unknown. During exploration, within single cycles of the ongoing theta rhythm that dominates hippocampal local field potentials, place cells form precisely ordered sequences of activity. These neural sequences result from the integration of both external inputs conveying sensory-motor information, and intrinsic network dynamics possibly related to memory processes. Their endogenous replay during subsequent sleep is critical for memory consolidation. The present review discusses possible mechanisms and functions of hippocampal theta sequences during exploration. We present several lines of evidence suggesting that these neural sequences play a key role in information processing and support the formation of initial memory traces, and discuss potential functional distinctions between neural sequences emerging during theta vs. awake sharp-wave ripples.

Disrupting the medial prefrontal cortex alters hippocampal sequences during deliberative decision making

Current theories of deliberative decision making suggest that deliberative decisions arise from imagined simulations that require interactions between the prefrontal cortex and hippocampus. In rodent navigation experiments, hippocampal theta sequences advance from the location of the rat ahead to the subsequent goal. To examine the role of the medial prefrontal cortex (mPFC) on the hippocampus, we disrupted the mPFC with DREADDs (designer receptors exclusively activated by designer drugs). Using the Restaurant Row foraging task, we found that mPFC disruption resulted in decreased vicarious trial and error behavior, reduced the number of theta sequences, and impaired theta sequences in hippocampus. mPFC disruption led to larger changes in the initiation of the hippocampal theta sequences that represent the current location of the rat rather than to the later portions that represent the future outcomes. These data suggest that the mPFC likely provides an important component to the initiation of deliberative sequences and provides support for an episodic-future thinking, working memory interpretation of deliberation. NEW & NOTEWORTHY The medial prefrontal cortex (mPFC) and hippocampus interact during deliberative decision making. Disruption of the mPFC impaired hippocampal processes, including the local and nonlocal representations of space along each theta cycle and the initiation of hippocampal theta sequences, while sparing place cell firing characteristics and phase precession. mPFC disruption reduced the deliberative behavioral process vicarious trial and error and improved economic behaviors on this task.

The perirhinal cortex supports spatial intertemporal choice stability

In order to optimize outcomes in the face of uncertainty, one must recall past experiences and extrapolate to the future by assigning values to different choice outcomes. This behavior requires an interplay between memory and reward valuation, necessitating communication across many brain regions. At the anatomical nexus of this interplay is the perirhinal cortex (PRC). The PRC is densely connected to the amygdala and orbital frontal cortex, regions that have been implicated in reward-based decision making, as well as the hippocampus. Thus, the PRC could serve as a hub for integrating memory, reward, and prediction. The PRC's role in value-based decision making, however, has not been empirically examined. Therefore, we tested the role of the PRC in a spatial delay discounting task, which allows rats to choose between a 1-s delay for a small food reward and a variable delay for a large food reward, with the delay to the large reward increasing after choice of each large reward and decreasing after each small reward. The rat can therefore adjust the delay by consecutively choosing the same reward or stabilize the delay by alternating between sides. The latter has been shown to occur once the 'temporal cost' of the large reward is established and is a decision-making process termed 'exploitation'. When the PRC was bilaterally inactivated with the GABA(A) agonist muscimol, rats spent fewer trials successfully exploiting to maintain a fixed delay compared to the vehicle control condition. Moreover, PRC inactivation resulted in an increased number of vicarious trial and error (VTE) events at the choice point, where rats had to decide between the two rewards. These behavioral patterns suggest that the PRC is critical for maintaining stability in linking a choice to a reward outcome in the face of a variable cost.

Dopamine, Oxidative Stress and Protein-Quinone Modifications in Parkinson's and Other Neurodegenerative Diseases

Dopamine (DA) is the most important catecholamine in the brain, as it is the most abundant and the precursor of other neurotransmitters. Degeneration of nigrostriatal neurons of substantia nigra pars compacta in Parkinson's disease represents the best-studied link between DA neurotransmission and neuropathology. Catecholamines are reactive molecules that are handled through complex control and transport systems. Under normal conditions, small amounts of cytosolic DA are converted to neuromelanin in a stepwise process involving melanization of peptides and proteins. However, excessive cytosolic or extraneuronal DA can give rise to nonselective protein modifications. These reactions involve DA oxidation to quinone species and depend on the presence of redox-active transition metal ions such as iron and copper. Other oxidized DA metabolites likely participate in post-translational protein modification. Thus, protein-quinone modification is a heterogeneous process involving multiple DA-derived residues that produce structural and conformational changes of proteins and can lead to aggregation and inactivation of the modified proteins.

Habits without values

Habits form a crucial component of behavior. In recent years, key computational models have conceptualized habits as arising from model-free reinforcement learning mechanisms, which typically select between available actions based on the future value expected to result from each. Traditionally, however, habits have been understood as behaviors that can be triggered directly by a stimulus, without requiring the animal to evaluate expected outcomes. Here, we develop a computational model instantiating this traditional view, in which habits develop through the direct strengthening of recently taken actions rather than through the encoding of outcomes. We demonstrate that this model accounts for key behavioral manifestations of habits, including insensitivity to outcome devaluation and contingency degradation, as well as the effects of reinforcement schedule on the rate of habit formation. The model also explains the prevalent observation of perseveration in repeated-choice tasks as an additional behavioral manifestation of the habit system. We suggest that mapping habitual behaviors onto value-free mechanisms provides a parsimonious account of existing behavioral and neural data. This mapping may provide a new foundation for building robust and comprehensive models of the interaction of habits with other, more goal-directed types of behaviors and help to better guide research into the neural mechanisms underlying control of instrumental behavior more generally. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Hippocampal replays under the scrutiny of reinforcement learning models

Multiple in vivo studies have shown that place cells from the hippocampus replay previously experienced trajectories. These replays are commonly considered to mainly reflect memory consolidation processes. Some data, however, have highlighted a functional link between replays and reinforcement learning (RL). This theory, extensively used in machine learning, has introduced efficient algorithms and can explain various behavioral and physiological measures from different brain regions. RL algorithms could constitute a mechanistic description of replays and explain how replays can reduce the number of iterations required to explore the environment during learning. We review the main findings concerning the different hippocampal replay types and the possible associated RL models (either model-based, model-free, or hybrid model types). We conclude by tying these frameworks together. We illustrate the link between data and RL through a series of model simulations. This review, at the frontier between informatics and biology, paves the way for future work on replays.

Remembering rewarding futures: A simulation-selection model of the hippocampus

Despite tremendous progress, the neural circuit dynamics underlying hippocampal mnemonic processing remain poorly understood. We propose a new model for hippocampal function-the simulation-selection model-based on recent experimental findings and neuroecological considerations. Under this model, the mammalian hippocampus evolved to simulate and evaluate arbitrary navigation sequences. Specifically, we suggest that CA3 simulates unexperienced navigation sequences in addition to remembering experienced ones, and CA1 selects from among these CA3-generated sequences, reinforcing those that are likely to maximize reward during offline idling states. High-value sequences reinforced in CA1 may allow flexible navigation toward a potential rewarding location during subsequent navigation. We argue that the simulation-selection functions of the hippocampus have evolved in mammals mostly because of the unique navigational needs of land mammals. Our model may account for why the mammalian hippocampus has evolved not only to remember, but also to imagine episodes, and how this might be implemented in its neural circuits.

A Detour Task in Four Species of Fishes

Four species of fish (Danio rerio, Xenotoca eiseni, Carassius auratus, and Pterophyllum scalare) were tested in a detour task requiring them to temporarily abandon the view of the goal-object (a group of conspecifics) to circumvent an obstacle. Fishes were placed in the middle of a corridor, at the end of which there was an opaque wall with a small window through which the goal was visible. Midline along the corridor two symmetrical apertures allowed animals to access two compartments for each aperture. After passing the aperture, fishes showed searching behavior in the two correct compartments close to the goal, appearing able to localize it, although they had to temporarily move away from the object's view. Here we provide the first evidence that fishes can solve such a detour task and therefore seem able to represent the "permanence in existence" of objects, which continue to exist even if they are not momentarily visible.

Assembly Responses of Hippocampal CA1 Place Cells Predict Learned Behavior in Goal-Directed Spatial Tasks on the Radial Eight-Arm Maze

Hippocampus is needed for both spatial working and reference memories. Here, using a radial eight-arm maze, we examined how the combined demand on these memories influenced CA1 place cell assemblies while reference memories were partially updated. This was contrasted with control tasks requiring only working memory or the update of reference memory. Reference memory update led to the reward-directed place field shifts at newly rewarded arms and to the gradual strengthening of firing in passes between newly rewarded arms but not between those passes that included a familiar-rewarded arm. At the maze center, transient network synchronization periods preferentially replayed trajectories of the next chosen arm in reference memory tasks but the previously visited arm in the working memory task. Hence, reference memory demand was uniquely associated with a gradual, goal novelty-related reorganization of place cell assemblies and with trajectory replay that reflected the animal's decision of which arm to visit next.

Attenuating Neural Threat Expression with Imagination

Imagination is an internal simulation of real-life events and a common treatment tool for anxiety disorders; however, the neural processes by which imagination exerts behavioral control are unclear. This investigation tests whether and how imagined exposures to a threatening stimulus, conditioned in the real world, influence neural and physiological manifestations of threat. We found that imagined and real extinction are equally effective in the reduction of threat-related neural patterns and physiological responses elicited upon re-exposure to real-world threatening cues. Network connectivity during the extinction phase showed that imagined, like real, extinction engaged the ventromedial prefrontal cortex (vmPFC) as a central hub. vmPFC, primary auditory cortex, and amygdala activation during imagined and real extinction were predictive of individual differences in extinction success. The nucleus accumbens, however, predicted extinction success in the imagined extinction group alone. We conclude that deliberate imagination can attenuate reactions to threat through perceptual and associative learning mechanisms.

Retroactive and graded prioritization of memory by reward

Many decisions are based on an internal model of the world. Yet, how such a model is constructed from experience and represented in memory remains unknown. We test the hypothesis that reward shapes memory for sequences of events by retroactively prioritizing memory for objects as a function of their distance from reward. Human participants encountered neutral objects while exploring a series of mazes for reward. Across six data sets, we find that reward systematically modulates memory for neutral objects, retroactively prioritizing memory for objects closest to the reward. This effect of reward on memory emerges only after a 24-hour delay and is stronger for mazes followed by a longer rest interval, suggesting a role for post-reward replay and overnight consolidation, as predicted by neurobiological data in animals. These findings demonstrate that reward retroactively prioritizes memory along a sequential gradient, consistent with the role of memory in supporting adaptive decision-making.

Navigating Social Space

Cognitive maps are encoded in the hippocampal formation and related regions and range from the spatial to the purely conceptual. Neural mechanisms that encode information into relational structures, up to an arbitrary level of abstraction, may explain such a broad range of representation. Research now indicates that social life can also be mapped by these mechanisms: others' spatial locations, social memory, and even a two-dimensional social space framed by social power and affiliation. The systematic mapping of social life onto a relational social space facilitates adaptive social decision making, akin to social navigation. This emerging line of research has implications for cognitive mapping research, clinical disorders that feature hippocampal dysfunction, and the field of social cognitive neuroscience.

Human Vicarious Trial and Error Is Predictive of Spatial Navigation Performance

When learning new environments, rats often pause at decision points and look back and forth over their possible trajectories as if they were imagining the future outcome of their actions, a behavior termed “Vicarious trial and error” (VTE). As the animal learns the environmental configuration, rats change from deliberative to habitual behavior, and VTE tends to disappear, suggesting a functional relevance in the early stages of learning. Despite the extensive research on spatial navigation, learning and VTE in the rat model, fewer studies have focused on humans. Here, we tested whether head-scanning behaviors that humans typically exhibit during spatial navigation are as predictive of spatial learning as in the rat. Subjects performed a goal-oriented virtual navigation task in a symmetric environment. Spatial learning was assessed through the analysis of trajectories, timings, and head orientations, under habitual and deliberative spatial navigation conditions. As expected, we found that trajectory length and duration decreased with the trial number, implying that subjects learned the spatial configuration of the environment over trials. Interestingly, IdPhi (a standard metric of VTE) also decreased with the trial number, suggesting that humans benefit from the same head-orientation scanning behavior as rats at spatial decision-points. Moreover, IdPhi captured exclusively at the first decision-point of each trial, was correlated with trial trajectory duration and length. Our findings demonstrate that in VTE is a signature of the stage of spatial learning in humans, and can be used to predict performance in navigation tasks with high accuracy.

Decoding Cognitive Processes from Neural Ensembles

An intrinsic difficulty in studying cognitive processes is that they are unobservable states that exist in between observable responses to the sensory environment. Cognitive states must be inferred from indirect behavioral measures. Neuroscience potentially provides the tools necessary to measure cognitive processes directly, but it is challenged on two fronts. First, neuroscientific measures often lack the spatiotemporal resolution to identify the neural computations that underlie a cognitive process. Second, the activity of a single neuron, which is the fundamental building block of neural computation, is too noisy to provide accurate measurements of a cognitive process. In this paper, I examine recent developments in neurophysiological recording and analysis methods that provide a potential solution to these problems.

Sensitivity to "sunk costs" in mice, rats, and humans

Sunk costs are irrecoverable investments that should not influence decisions, because decisions should be made on the basis of expected future consequences. Both human and nonhuman animals can show sensitivity to sunk costs, but reports from across species are inconsistent. In a temporal context, a sensitivity to sunk costs arises when an individual resists ending an activity, even if it seems unproductive, because of the time already invested. In two parallel foraging tasks that we designed, we found that mice, rats, and humans show similar sensitivities to sunk costs in their decision-making. Unexpectedly, sensitivity to time invested accrued only after an initial decision had been made. These findings suggest that sensitivity to temporal sunk costs lies in a vulnerability distinct from deliberation processes and that this distinction is present across species.

The Hippocampus Promotes Effective Saccadic Information Gathering in Humans

It is well established that the hippocampus is critical for memory. Recent evidence suggests that one function of hippocampal memory processing is to optimize how people actively explore the world. Here we demonstrate that the link between the hippocampus and exploration extends even to the moment-to-moment use of eye movements during visuospatial memory encoding. In Experiment 1, we examined relationships between study-phase eye movements in healthy individuals and subsequent performance on a spatial reconstruction test. In addition to quantitative measures of viewing behaviors (e.g., how many fixations or saccades were deployed during study), we used the information-theoretic measure of entropy to assess the amount of randomness or disorganization in participants' scanning behaviors. We found that the use of scanpaths during study that were lower in entropy (e.g., more organized, less random) predicted more accurate spatial reconstruction both within and between participants. Scanpath entropy was a better predictor of reconstruction accuracy than were the quantitative measures of viewing. In Experiment 2, we found that individuals with hippocampal amnesia tended to engage in viewing patterns that were higher in entropy (less organized) relative to healthy comparisons. These findings reveal a critical role of the hippocampus in guiding eye movement exploration to optimize visuospatial relational memory.

Model-based spatial navigation in the hippocampus-ventral striatum circuit: A computational analysis

While the neurobiology of simple and habitual choices is relatively well known, our current understanding of goal-directed choices and planning in the brain is still limited. Theoretical work suggests that goal-directed computations can be productively associated to model-based (reinforcement learning) computations, yet a detailed mapping between computational processes and neuronal circuits remains to be fully established. Here we report a computational analysis that aligns Bayesian nonparametrics and model-based reinforcement learning (MB-RL) to the functioning of the hippocampus (HC) and the ventral striatum (vStr)-a neuronal circuit that increasingly recognized to be an appropriate model system to understand goal-directed (spatial) decisions and planning mechanisms in the brain. We test the MB-RL agent in a contextual conditioning task that depends on intact hippocampus and ventral striatal (shell) function and show that it solves the task while showing key behavioral and neuronal signatures of the HC-vStr circuit. Our simulations also explore the benefits of biological forms of look-ahead prediction (forward sweeps) during both learning and control. This article thus contributes to fill the gap between our current understanding of computational algorithms and biological realizations of (model-based) reinforcement learning.

Beyond simple tests of value: measuring addiction as a heterogeneous disease of computation-specific valuation processes

Addiction is considered to be a neurobiological disorder of learning and memory because addiction is capable of producing lasting changes in the brain. Recovering addicts chronically struggle with making poor decisions that ultimately lead to relapse, suggesting a view of addiction also as a neurobiological disorder of decision-making information processing. How the brain makes decisions depends on how decision-making processes access information stored as memories in the brain. Advancements in circuit-dissection tools and recent theories in neuroeconomics suggest that neurally dissociable valuation processes access distinct memories differently, and thus are uniquely susceptible as the brain changes during addiction. If addiction is to be considered a neurobiological disorder of memory, and thus decision-making, the heterogeneity with which information is both stored and processed must be taken into account in addiction studies. Addiction etiology can vary widely from person to person. We propose that addiction is not a single disease, nor simply a disorder of learning and memory, but rather a collection of symptoms of heterogeneous neurobiological diseases of distinct circuit-computation-specific decision-making processes.

Deliberation and Procedural Automation on a Two-Step Task for Rats

Current theories suggest that decision-making arises from multiple, competing action-selection systems. Rodent studies dissociate deliberation and procedural behavior, and find a transition from procedural to deliberative behavior with experience. However, it remains unknown how this transition from deliberative to procedural control evolves within single trials, or within blocks of repeated choices. We adapted for rats a two-step task which has been used to dissociate model-based from model-free decisions in humans. We found that a mixture of model-based and model-free algorithms was more likely to explain rat choice strategies on the task than either model-based or model-free algorithms alone. This task contained two choices per trial, which provides a more complex and non-discrete per-trial choice structure. This task structure enabled us to evaluate how deliberative and procedural behavior evolved within-trial and within blocks of repeated choice sequences. We found that vicarious trial and error (VTE), a behavioral correlate of deliberation in rodents, was correlated between the two choice points on a given lap. We also found that behavioral stereotypy, a correlate of procedural automation, increased with the number of repeated choices. While VTE at the first choice point decreased [corrected] with the number of repeated choices, VTE at the second choice point did not, and only increased after unexpected transitions within the task. This suggests that deliberation at the beginning of trials may correspond to changes in choice patterns, while mid-trial deliberation may correspond to an interruption of a procedural process.

Learning Naturalistic Temporal Structure in the Posterior Medial Network

The posterior medial network is at the apex of a temporal integration hierarchy in the brain, integrating information over many seconds of viewing intact, but not scrambled, movies. This has been interpreted as an effect of temporal structure. Such structure in movies depends on preexisting event schemas, but temporal structure can also arise de novo from learning. Here, we examined the relative role of schema-consistent temporal structure and arbitrary but consistent temporal structure on the human posterior medial network. We tested whether, with repeated viewing, the network becomes engaged by scrambled movies with temporal structure. Replicating prior studies, activity in posterior medial regions was immediately locked to stimulus structure upon exposure to intact, but not scrambled, movies. However, for temporally structured scrambled movies, functional coupling within the network increased across stimulus repetitions, rising to the level of intact movies. Thus, temporal structure is a key determinant of network dynamics and function in the posterior medial network.

Prolonged abstinence from cocaine or morphine disrupts separable valuations during decision conflict

Neuroeconomic theories propose changes in decision making drive relapse in recovering drug addicts, resulting in continued drug use despite stated wishes not to. Such conflict is thought to arise from multiple valuation systems dependent on separable neural components, yet many neurobiology of addiction studies employ only simple tests of value. Here, we tested in mice how prolonged abstinence from different drugs affects behavior in a neuroeconomic foraging task that reveals multiple tests of value. Abstinence from repeated cocaine and morphine disrupts separable decision-making processes. Cocaine alters deliberation-like behavior prior to choosing a preferred though economically unfavorable offer, while morphine disrupts re-evaluations after rapid initial decisions. These findings suggest that different drugs have long-lasting effects precipitating distinct decision-making vulnerabilities. Our approach can guide future refinement of decision-making behavioral paradigms and highlights how grossly similar behavioral maladaptations may mask multiple underlying, parallel, and dissociable processes that treatments for addiction could potentially target.

Neuroeconomic theories suggest that conflict during decision, such as exhibited by relapsing drug addicts who continue drug use despite stated wishes not to, might arise from separable processes in decision making. Here the authors test mice in a foraging task designed to separate these processes and find that mice show alterations in separable components of decision conflict following abstinence from cocaine versus morphine.

Mice learn to avoid regret

Regret can be defined as the subjective experience of recognizing that one has made a mistake and that a better alternative could have been selected. The experience of regret is thought to carry negative utility. This typically takes two distinct forms: augmenting immediate postregret valuations to make up for losses, and augmenting long-term changes in decision-making strategies to avoid future instances of regret altogether. While the short-term changes in valuation have been studied in human psychology, economics, neuroscience, and even recently in nonhuman-primate and rodent neurophysiology, the latter long-term process has received far less attention, with no reports of regret avoidance in nonhuman decision-making paradigms. We trained 31 mice in a novel variant of the Restaurant Row economic decision-making task, in which mice make decisions of whether to spend time from a limited budget to achieve food rewards of varying costs (delays). Importantly, we tested mice longitudinally for 70 consecutive days, during which the task provided their only source of food. Thus, decision strategies were interdependent across both trials and days. We separated principal commitment decisions from secondary reevaluation decisions across space and time and found evidence for regret-like behaviors following change-of-mind decisions that corrected prior economically disadvantageous choices. Immediately following change-of-mind events, subsequent decisions appeared to make up for lost effort by altering willingness to wait, decision speed, and pellet consumption speed, consistent with past reports of regret in rodents. As mice were exposed to an increasingly reward-scarce environment, we found they adapted and refined distinct economic decision-making strategies over the course of weeks to maximize reinforcement rate. However, we also found that even without changes in reinforcement rate, mice transitioned from an early strategy rooted in foraging to a strategy rooted in deliberation and planning that prevented future regret-inducing change-of-mind episodes from occurring. These data suggest that mice are learning to avoid future regret, independent of and separate from reinforcement rate maximization.

Regret describes a unique postdecision phenomenon in which losses are realized as a fault of one’s own actions. Regret is often hypothesized to have an inherent negative utility, and humans will often incur costs so as to avoid the risk of future regret. However, current models of nonhuman decision-making are based on reward maximization hypotheses. We recently found that rats express regret behaviorally and neurophysiologically on neuroeconomic foraging tasks; however, it remains unknown whether nonhuman animals will change strategies so as to avoid regret, even in the absence of changes in the achieved rate of reinforcement. Here, we provide the first evidence that mice change strategies to avoid future regret, independent of and separate from reinforcement rate maximization. Our data suggest mice accomplish this by shifting from a foraging decision-making strategy that produces change-of-mind decisions after investment mistakes to one rooted in deliberation that learns to plan ahead.

Temporal Dynamics of Hippocampal and Medial Prefrontal Cortex Interactions During the Delay Period of a Working Memory-Guided Foraging Task

Connections between the hippocampus (HC) and medial prefrontal cortex (mPFC) are critical for working memory; however, the precise contribution of this pathway is a matter of debate. One suggestion is that it may stabilize retrospective memories of recently encountered task-relevant information. Alternatively, it may be involved in encoding prospective memories, or the internal representation of future goals. To explore these possibilities, simultaneous extracellular recordings were made from mPFC and HC of rats performing the delayed spatial win-shift on a radial maze. Each trial consisted of a training-phase (when 4 randomly chosen arms were open) and test phase (all 8 arms were open but only previously blocked arms contained food) separated by a 60-s delay. Theta power was highest during the delay, and mPFC units were more likely to become entrained to hippocampal theta as the delay progressed. Training and test phase performance were accurately predicted by a linear classifier, and there was a transition in classification for training-phase to test-phase activity patterns throughout the delay on trials where the rats performed well. These data suggest that the HC and mPFC become more strongly synchronized as mPFC circuits preferentially shift from encoding retrospective to prospective information.

Altering gain of the infralimbic-to-accumbens shell circuit alters economically dissociable decision-making algorithms

The nucleus accumbens shell (NAcSh) is involved in reward valuation. Excitatory projections from infralimbic cortex (IL) to NAcSh undergo synaptic remodeling in rodent models of addiction and enable the extinction of disadvantageous behaviors. However, how the strength of synaptic transmission of the IL-NAcSh circuit affects decision-making information processing and reward valuation remains unknown, particularly because these processes can conflict within a given trial and particularly given recent data suggesting that decisions arise from separable information-processing algorithms. The approach of many neuromodulation studies is to disrupt information flow during on-going behaviors; however, this limits the interpretation of endogenous encoding of computational processes. Furthermore, many studies are limited by the use of simple behavioral tests of value which are unable to dissociate neurally distinct decision-making algorithms. We optogenetically altered the strength of synaptic transmission between glutamatergic IL-NAcSh projections in mice trained on a neuroeconomic task capable of separating multiple valuation processes. We found that induction of long-term depression in these synapses produced lasting changes in foraging processes without disrupting deliberative processes. Mice displayed inflated reevaluations to stay when deciding whether to abandon continued reward-seeking investments but displayed no changes during initial commitment decisions. We also developed an ensemble-level measure of circuit-specific plasticity that revealed individual differences in foraging valuation tendencies. Our results demonstrate that alterations in projection-specific synaptic strength between the IL and the NAcSh are capable of augmenting self-control economic valuations within a particular decision-making modality and suggest that the valuation mechanisms for these multiple decision-making modalities arise from different circuits.

Dorsolateral Striatum Engagement Interferes with Early Discrimination Learning

In current models, learning the relationship between environmental stimuli and the outcomes of actions involves both stimulus-driven and goal-directed systems, mediated in part by the DLS and DMS, respectively. However, though these models emphasize the importance of the DLS in governing actions after extensive experience has accumulated, there is growing evidence of DLS engagement from the onset of training. Here, we used in vivo photosilencing to reveal that DLS recruitment interferes with early touchscreen discrimination learning. We also show that the direct output pathway of the DLS is preferentially recruited and causally involved in early learning and find that silencing the normal contribution of the DLS produces plasticity-related alterations in a PL-DMS circuit. These data provide further evidence suggesting that the DLS is recruited in the construction of stimulus-elicited actions that ultimately automate behavior and liberate cognitive resources for other demands, but with a cost to performance at the outset of learning.

What does dopamine mean?

Dopamine is a critical modulator of both learning and motivation. This presents a problem: how can target cells know whether increased dopamine is a signal to learn or to move? It is often presumed that motivation involves slow ('tonic') dopamine changes, while fast ('phasic') dopamine fluctuations convey reward prediction errors for learning. Yet recent studies have shown that dopamine conveys motivational value and promotes movement even on subsecond timescales. Here I describe an alternative account of how dopamine regulates ongoing behavior. Dopamine release related to motivation is rapidly and locally sculpted by receptors on dopamine terminals, independently from dopamine cell firing. Target neurons abruptly switch between learning and performance modes, with striatal cholinergic interneurons providing one candidate switch mechanism. The behavioral impact of dopamine varies by subregion, but in each case dopamine provides a dynamic estimate of whether it is worth expending a limited internal resource, such as energy, attention, or time.

Adult neurogenesis affects motivation to obtain weak, but not strong, reward in operant tasks

Decreased motivation to seek rewards is a key feature of mood disorders that correlates with severity and treatment outcome. This anhedonia, or apathy, likely reflects impairment in reward circuitry, but the specific neuronal populations controlling motivation are unclear. Granule neurons generated in the adult hippocampus have been implicated in mood disorders, but are not generally considered as part of reward circuits. We investigated a possible role of these new neurons in motivation to work for food and sucrose rewards in operant conditioning tasks using GFAP-TK pharmacogenetic ablation of adult neurogenesis in both rats and mice. Rats and mice lacking adult neurogenesis showed normal lever press responding during fixed ratio training, reward devaluation, and Pavlovian Instrumental Transfer, suggesting no impairment in learning. However, on an exponentially progressive ratio schedule, or when regular chow was freely available in the testing chamber, TK rats and mice showed less effort to gain sucrose tablets. When working for balanced food tablets, which rats and mice of both genotypes strongly preferred over sucrose, the genotype effects on behavior were lost. This decrease in effort under conditions of low reward suggests that loss of adult neurogenesis decreases motivation to seek reward in a manner that may model behavioral apathy.

Answering hastily retards learning

Appropriate decisions involve at least two aspects: the speed of the decision and the correctness of the decision. Although a quick and correct decision is generally believed to work favorably, these two aspects may be interdependent in terms of overall task performance. In this study, we scrutinized learning behaviors in an operant task in which rats were required to poke their noses into either of two holes by referring to a light cue. All 22 rats reached the learning criterion, an 80% correct rate, within 4 days of testing, but they were diverse in the number of sessions spent to reach the learning criterion. Individual analyses revealed that the mean latency for responding was negatively correlated with the number of sessions until learning, suggesting that the rats that responded more rapidly to the cues learned the task more slowly. For individual trials, the mean latency for responding in correct trials (LC) was significantly longer than that in incorrect trials (LI), suggesting that, on average, long deliberation times led to correct answers in the trials. The success ratio before learning was not correlated with the learning speed. Thus, deliberative decision-making, rather than overall correctness, is critical for learning.

From Focused Thought to Reveries: A Memory System for a Conscious Robot

We introduce a memory model for robots that can account for many aspects of an inner world, ranging from object permanence, episodic memory, and planning to imagination and reveries. It is modeled after neurophysiological data and includes parts of the cerebral cortex together with models of arousal systems that are relevant for consciousness. The three central components are an identification network, a localization network, and a working memory network. Attention serves as the interface between the inner and the external world. It directs the flow of information from sensory organs to memory, as well as controlling top-down influences on perception. It also compares external sensations to internal top-down expectations. The model is tested in a number of computer simulations that illustrate how it can operate as a component in various cognitive tasks including perception, the A-not-B test, delayed matching to sample, episodic recall, and vicarious trial and error.