Interactions between deliberation and delay-discounting in rats

Deliberative Behaviors and Prefrontal-Hippocampal Coupling are Disrupted in a Rat Model of Fetal Alcohol Spectrum Disorders

Fetal alcohol spectrum disorders (FASDs) are characterized by a range of physical, cognitive, and behavioral impairments. Determining how temporally specific alcohol exposure (AE) affects neural circuits is crucial to understanding the FASD phenotype. Third trimester AE can be modeled in rats by administering alcohol during the first two postnatal weeks, which damages the medial prefrontal cortex (mPFC), thalamic nucleus reuniens, and hippocampus (HPC), structures whose functional interactions are required for working memory and executive function. Therefore, we hypothesized that AE during this period would impair working memory, disrupt choice behaviors, and alter mPFC-HPC oscillatory synchrony. To test this hypothesis, we recorded local field potentials from the mPFC and dorsal HPC as AE and sham intubated (SI) rats performed a spatial working memory task in adulthood and implemented algorithms to detect vicarious trial and errors (VTEs), behaviors associated with deliberative decision-making. We found that, compared to the SI group, the AE group performed fewer VTEs and demonstrated a disturbed relationship between VTEs and choice outcomes, while spatial working memory was unimpaired. This behavioral disruption was accompanied by alterations to mPFC and HPC oscillatory activity in the theta and beta bands, respectively, and a reduced prevalence of mPFC-HPC synchronous events. When trained on multiple behavioral variables, a machine learning algorithm could accurately predict whether rats were in the AE or SI group, thus characterizing a potential phenotype following third trimester AE. Together, these findings indicate that third trimester AE disrupts mPFC-HPC oscillatory interactions and choice behaviors.

Using synchronized brain rhythms to bias memory-guided decisions

Functional interactions between the prefrontal cortex and hippocampus, as revealed by strong oscillatory synchronization in the theta (6-11 Hz) frequency range, correlate with memory-guided decision-making. However, the degree to which this form of long-range synchronization influences memory-guided choice remains unclear. We developed a brain-machine interface that initiated task trials based on the magnitude of prefrontal-hippocampal theta synchronization, then measured choice outcomes. Trials initiated based on strong prefrontal-hippocampal theta synchrony were more likely to be correct compared to control trials on both working memory-dependent and -independent tasks. Prefrontal-thalamic neural interactions increased with prefrontal-hippocampal synchrony and optogenetic activation of the ventral midline thalamus primarily entrained prefrontal theta rhythms, but dynamically modulated synchrony. Together, our results show that prefrontal-hippocampal theta synchronization leads to a higher probability of a correct choice and strengthens prefrontal-thalamic dialogue. Our findings reveal new insights into the neural circuit dynamics underlying memory-guided choices and highlight a promising technique to potentiate cognitive processes or behavior via brain-machine interfacing.

Hippocampal-prefrontal communication subspaces align with behavioral and network patterns in a spatial memory task

Rhythmic network states have been theorized to facilitate communication between brain regions, but how these oscillations influence communication subspaces, i.e, the low-dimensional neural activity patterns that mediate inter-regional communication, and in turn how subspaces impact behavior remains unclear. Using a spatial memory task in rats, we simultaneously recorded ensembles from hippocampal CA1 and the prefrontal cortex (PFC) to address this question. We found that task behaviors best aligned with low-dimensional, shared subspaces between these regions, rather than local activity in either region. Critically, both network oscillations and speed modulated the structure and performance of this communication subspace. Contrary to expectations, theta coherence did not better predict CA1-PFC shared activity, while theta power played a more significant role. To understand the communication space, we visualized shared CA1-PFC communication geometry using manifold techniques and found ring-like structures. We hypothesize that these shared activity manifolds are utilized to mediate the task behavior. These findings suggest that memory-guided behaviors are driven by shared CA1-PFC interactions that are dynamically modulated by oscillatory states, offering a novel perspective on the interplay between rhythms and behaviorally relevant neural communication.

Flexible decision-making is related to strategy learning, vicarious trial and error, and medial prefrontal rhythms during spatial set-shifting

Flexible decision-making requires a balance between exploring features of an environment and exploiting prior knowledge. Behavioral flexibility is typically measured by how long it takes subjects to consistently make accurate choices after reward contingencies switch or task rules change. This measure, however, only allows for tracking flexibility across multiple trials, and does not assess the degree of flexibility. Plus, although increases in decision-making accuracy are strong indicators of learning, other decision-making behaviors have also been suggested as markers of flexibility, such as the on-the-fly decision reversals known as vicarious trial and error (VTE) or switches to a different, but incorrect, strategy. We sought to relate flexibility, learning, and neural activity by comparing choice history-derived evaluation of strategy use with changes in decision-making accuracy and VTE behavior while recording from the medial prefrontal cortex (mPFC) in rats. Using a set-shifting task that required rats to repeatedly switch between spatial decision-making strategies, we show that a previously developed strategy likelihood estimation procedure could identify putative learning points based on decision history. We confirm the efficacy of learning point estimation by showing increases in decision-making accuracy aligned to the learning point. Additionally, we show increases in the rate of VTE behavior surrounding identified learning points. By calculating changes in strategy likelihoods across trials, we tracked flexibility on a trial-by-trial basis and show that flexibility scores also increased around learning points. Further, we demonstrate that VTE behaviors could be separated into indecisive and deliberative subtypes depending on whether they occurred during periods of high or low flexibility and whether they led to correct or incorrect choice outcomes. Field potential recordings from the mPFC during decisions exhibited increased beta band activity on trials with VTE compared to non-VTE trials, as well as increased gamma during periods when learned strategies could be exploited compared to prelearning, exploratory periods. This study demonstrates that increased behavioral flexibility and VTE rates are often aligned to task learning. These relationships can break down, however, suggesting that VTE is not always an indicator of deliberative decision-making. Additionally, we further implicate the mPFC in decision-making and learning by showing increased beta-based activity on VTE trials and increased gamma after learning.

Memory consolidation affects the interplay of place and response navigation

Navigation through space is based on memory representations of landmarks ('place') or movement sequences ('response'). Over time, memory representations transform through consolidation. However, it is unclear how the transformation affects place and response navigation in humans. In the present study, healthy adults navigated to target locations in a virtual maze. The preference for using place and response strategies and the ability to recall place and response memories were tested after a delay of one hour (n = 31), one day (n = 30), or two weeks (n = 32). The different delays captured early-phase synaptic changes, changes after one night of sleep, and long-delay changes due to the reorganization of navigation networks. Our results show that the relative contributions of place and response navigation changed as a function of time. After a short delay of up to one day, participants preferentially used a place strategy and exhibited a high degree of visual landmark exploration. After a longer delay of two weeks, place strategy use decreased significantly. Participants now equally relied on place and response strategy use and increasingly repeated previously taken paths. Further analyses indicate that response strategy use predominantly occurred as a compensatory strategy in the absence of sufficient place memory. Over time, place memory faded before response memory. We suggest that the observed shift from place to response navigation is context-dependent since detailed landmark information, which strongly relied on hippocampal function, decayed faster than sequence information, which required less detail and depended on extra-hippocampal areas. We conclude that changes in place and response navigation likely reflect the reorganization of navigation networks during systems consolidation.

An Improved Dyna-Q Algorithm Inspired by the Forward Prediction Mechanism in the Rat Brain for Mobile Robot Path Planning

The traditional Model-Based Reinforcement Learning (MBRL) algorithm has high computational cost, poor convergence, and poor performance in robot spatial cognition and navigation tasks, and it cannot fully explain the ability of animals to quickly adapt to environmental changes and learn a variety of complex tasks. Studies have shown that vicarious trial and error (VTE) and the hippocampus forward prediction mechanism in rats and other mammals can be used as key components of action selection in MBRL to support "goal-oriented" behavior. Therefore, we propose an improved Dyna-Q algorithm inspired by the forward prediction mechanism of the hippocampus to solve the above problems and tackle the exploration-exploitation dilemma of Reinforcement Learning (RL). This algorithm alternately presents the potential path in the future for mobile robots and dynamically adjusts the sweep length according to the decision certainty, so as to determine action selection. We test the performance of the algorithm in a two-dimensional maze environment with static and dynamic obstacles, respectively. Compared with classic RL algorithms like State-Action-Reward-State-Action (SARSA) and Dyna-Q, the algorithm can speed up spatial cognition and improve the global search ability of path planning. In addition, our method reflects key features of how the brain organizes MBRL to effectively solve difficult tasks such as navigation, and it provides a new idea for spatial cognitive tasks from a biological perspective.

The medial prefrontal cortex during flexible decisions: Evidence for its role in distinct working memory processes

During decisions that involve working memory, task-related information must be encoded, maintained across delays, and retrieved. Few studies have attempted to causally disambiguate how different brain structures contribute to each of these components of working memory. In the present study, we used transient optogenetic disruptions of rat medial prefrontal cortex (mPFC) during a serial spatial reversal learning (SSRL) task to test its role in these specific working memory processes. By analyzing numerous performance metrics, we found: (1) mPFC disruption impaired performance during only the choice epoch of initial discrimination learning of the SSRL task; (2) mPFC disruption impaired performance in dissociable ways across all task epochs (delay, choice, return) during flexible decision-making; (3) mPFC disruption resulted in a reduction of the typical vicarious-trial-and-error rate modulation that was related to changes in task demands. Taken together, these findings suggest that the mPFC plays an outsized role in working memory retrieval, becomes involved in encoding and maintenance when recent memories conflict with task demands, and enables animals to flexibly utilize working memory to update behavior as environments change.

Flexible coding schemes in dorsomedial prefrontal cortex underlie decision-making during delay discounting

Determining how an agent decides between a small, immediate versus a larger, delayed reward has provided insight into the psychological and neural basis of decision-making. The tendency to excessively discount the value of delayed rewards is thought to reflect deficits in brain regions critical for impulse control such as the prefrontal cortex (PFC). This study tested the hypothesis that dorsomedial PFC (dmPFC) is critically involved in flexibly managing neural representations of strategies that limit impulsive choices. Optogenetic silencing of neurons in the rat dmPFC increased impulsive choices at an 8 sec, but not 4 sec, delay. Neural recordings from dmPFC ensembles revealed that, at the 8-sec delay, the encoding landscape transitions to reflect a deliberative-like process rather than the schema-like processes observed at the 4-sec delay. These findings show that changes in the encoding landscape reflect changes in task demands and that dmPFC is uniquely involved in decisions requiring deliberation.

Pausing and reorienting behaviors enhance the performance of a spatial working memory task

During spatial working memory tasks, animals need to retain information about a previous trial in order to successfully select their next trajectory. Specifically, the delayed non-match to position task requires rats to follow a cued sample trajectory, then select the opposite route after a delay period. When faced with this choice, rats will occasionally exhibit complex behaviors, such as pausing and sweeping their head back and forth. These behaviors, called vicarious trial and error (VTE), are thought to be a behavioral manifestation of deliberation. However, we identified similarly complex behaviors during sample-phase traversals, despite the fact that these laps do not require a decision. First, we identified that these behaviors occurred more often after incorrect trials than before them, indicating that rats are retaining information between trials. Next, we determined that these pause-and-reorient (PAR) behaviors increased the likelihood of the next choice being selected correctly, suggesting that these behaviors assist the rat in successful task performance. Finally, we identified similarities between PARs and choice-phase VTEs, suggesting that VTEs may not only be reflective of deliberation, but may also contribute to a strategy for successful performance of spatial working memory tasks.

Optogenetic disruption of the prelimbic cortex alters long-term decision strategy but not valuation on a spatial delay discounting task

Rats demonstrate a preference for smaller, immediate rewards over larger, delayed ones, a phenomenon known as delay-discounting (DD). Behavior arises from the interaction of multiple decision-making systems, and the medial prefrontal cortex (mPFC) has been identified as a central component in the mediation between these decision systems. To investigate the role of the prelimbic (PL) subregion of mPFC on decision strategy interaction, we compared two cohorts of rats (ChR2-opsin-expressing 'Active' and opsin-absent 'Control') on a spatial delay-discounting task while delivering in-vivo light stimulation into PL at the choice point of select trials. By analyzing the overall delay-adjustment along with deliberative and procedural behavioral strategy markers, our study revealed differences in the decision strategies used between the active and control animals despite both groups showing similar valuations. Control animals developed the expected shift from deliberative to procedural decision strategy on this task (indicated by reaching delay-stability, particularly during late-session laps); however, active-virus animals repeatedly over-adjusted around their preferred delay throughout the entire session, suggesting a significant deficit in procedural decision-making on this task. Active animals showed a significant decrease in proportion of vicarious trial and error events (VTE, a behavior correlated with deliberative processes) on delay adjustment laps relative to control animals. This points to a more nuanced role for VTE, not just in executing deliberation, but in shifting from deliberative to procedural processes. This opto-induced change in VTE was especially pronounced for late-session adjustment laps. We found no other session-by-session or lap-by-lap effects, leaving a particular role for PL in the long-term development of procedural strategies on this task.

Humans account for cognitive costs when finding shortcuts: An information-theoretic analysis of navigation

When faced with navigating back somewhere we have been before we might either retrace our steps or seek a shorter path. Both choices have costs. Here, we ask whether it is possible to characterize formally the choice of navigational plans as a bounded rational process that trades off the quality of the plan (e.g., its length) and the cognitive cost required to find and implement it. We analyze the navigation strategies of two groups of people that are firstly trained to follow a "default policy" taking a route in a virtual maze and then asked to navigate to various known goal destinations, either in the way they want ("Go To Goal") or by taking novel shortcuts ("Take Shortcut"). We address these wayfinding problems using InfoRL: an information-theoretic approach that formalizes the cognitive cost of devising a navigational plan, as the informational cost to deviate from a well-learned route (the "default policy"). In InfoRL, optimality refers to finding the best trade-off between route length and the amount of control information required to find it. We report five main findings. First, the navigational strategies automatically identified by InfoRL correspond closely to different routes (optimal or suboptimal) in the virtual reality map, which were annotated by hand in previous research. Second, people deliberate more in places where the value of investing cognitive resources (i.e., relevant goal information) is greater. Third, compared to the group of people who receive the "Go To Goal" instruction, those who receive the "Take Shortcut" instruction find shorter but less optimal solutions, reflecting the intrinsic difficulty of finding optimal shortcuts. Fourth, those who receive the "Go To Goal" instruction modulate flexibly their cognitive resources, depending on the benefits of finding the shortcut. Finally, we found a surprising amount of variability in the choice of navigational strategies and resource investment across participants. Taken together, these results illustrate the benefits of using InfoRL to address navigational planning problems from a bounded rational perspective.

Internally Triggered Experiences of Hedonic Valence in Nonhuman Animals: Cognitive and Welfare Considerations

Do any nonhuman animals have hedonically valenced experiences not directly caused by stimuli in their current environment? Do they, like us humans, experience anticipated or previously experienced pains and pleasures as respectively painful and pleasurable? We review evidence from comparative neuroscience about hippocampus-dependent simulation in relation to this question. Hippocampal sharp-wave ripples and theta oscillations have been found to instantiate previous and anticipated experiences. These hippocampal activations coordinate with neural reward and fear centers as well as sensory and cortical areas in ways that are associated with conscious episodic mental imagery in humans. Moreover, such hippocampal “re- and preplay” has been found to contribute to instrumental decision making, the learning of value representations, and the delay of rewards in rats. The functional and structural features of hippocampal simulation are highly conserved across mammals. This evidence makes it reasonable to assume that internally triggered experiences of hedonic valence (IHVs) are pervasive across (at least) all mammals. This conclusion has important welfare implications. Most prominently, IHVs act as a kind of “welfare multiplier” through which the welfare impacts of any given experience of pain or pleasure are increased through each future retrieval. However, IHVs also have practical implications for welfare assessment and cause prioritization.

The ventral midline thalamus coordinates prefrontal-hippocampal neural synchrony during vicarious trial and error

When faced with difficult choices, the possible outcomes are considered through a process known as deliberation. In rats, deliberation is thought to be reflected by pause-and-reorienting behaviors, better known as vicarious trial and errors (VTEs). While VTEs are thought to require medial prefrontal cortex (mPFC) and dorsal hippocampal (dHPC) interactions, no empirical evidence has yet demonstrated such a dual requirement. The nucleus reuniens (Re) of the ventral midline thalamus is anatomically connected with both the mPFC and dHPC, is required for HPC-dependent spatial memory tasks, and is critical for mPFC-dHPC neural synchronization. Currently, it is unclear if, or how, the Re is involved in deliberation. Therefore, by examining the role of the Re on VTE behaviors, we can better understand the anatomical and physiological mechanisms supporting deliberation. Here, we examined the impact of Re suppression on VTE behaviors and mPFC-dHPC theta synchrony during asymptotic performance of a HPC-dependent delayed alternation (DA) task. Pharmacological suppression of the Re increased VTE behaviors that occurred with repetitive choice errors. These errors were best characterized as perseverative behaviors, in which some rats repeatedly selected a goal arm that previously yielded no reward. We then examined the impact of Re suppression on mPFC-dHPC theta synchrony during VTEs. We found that during VTEs, Re inactivation was associated with a reduction in mPFC-dHPC theta coherence and mPFC-to-dHPC theta directionality. Our findings suggest that the Re contributes to deliberation by coordinating mPFC-dHPC neural interactions.

Computational validity: using computation to translate behaviours across species

We propose a new conceptual framework (computational validity) for translation across species and populations based on the computational similarity between the information processing underlying parallel tasks. Translating between species depends not on the superficial similarity of the tasks presented, but rather on the computational similarity of the strategies and mechanisms that underlie those behaviours. Computational validity goes beyond construct validity by directly addressing questions of information processing. Computational validity interacts with circuit validity as computation depends on circuits, but similar computations could be accomplished by different circuits. Because different individuals may use different computations to accomplish a given task, computational validity suggests that behaviour should be understood through the subject's point of view; thus, behaviour should be characterized on an individual level rather than a task level. Tasks can constrain the computational algorithms available to a subject and the observed subtleties of that behaviour can provide information about the computations used by each individual. Computational validity has especially high relevance for the study of psychiatric disorders, given the new views of psychiatry as identifying and mediating information processing dysfunctions that may show high inter-individual variability, as well as for animal models investigating aspects of human psychiatric disorders. This article is part of the theme issue 'Systems neuroscience through the lens of evolutionary theory'.

A Machine Learning Approach for Detecting Vicarious Trial and Error Behaviors

Vicarious trial and error behaviors (VTEs) indicate periods of indecision during decision-making, and have been proposed as a behavioral marker of deliberation. In order to understand the neural underpinnings of these putative bridges between behavior and neural dynamics, researchers need the ability to readily distinguish VTEs from non-VTEs. Here we utilize a small set of trajectory-based features and standard machine learning classifiers to identify VTEs from non-VTEs for rats performing a spatial delayed alternation task (SDA) on an elevated plus maze. We also show that previously reported features of the hippocampal field potential oscillation can be used in the same types of classifiers to separate VTEs from non-VTEs with above chance performance. However, we caution that the modest classifier success using hippocampal population dynamics does not identify many trials where VTEs occur, and show that combining oscillation-based features with trajectory-based features does not improve classifier performance compared to trajectory-based features alone. Overall, we propose a standard set of features useful for trajectory-based VTE classification in binary decision tasks, and support previous suggestions that VTEs are supported by a network including, but likely extending beyond, the hippocampus.

Disrupting the medial prefrontal cortex with designer receptors exclusively activated by designer drug alters hippocampal sharp-wave ripples and their associated cognitive processes

The hippocampus and medial prefrontal cortex (mPFC) interact during a myriad of cognitive processes including decision-making and long-term memory consolidation. Exactly how the mPFC and hippocampus interact during goal-directed decision-making remains to be fully elucidated. During periods of rest, bursts of high-frequency oscillations, termed sharp-wave ripple (SWR), appear in the local field potential. Impairing SWRs on the maze or during post-learning rest can interfere with memory-guided decision-making and memory consolidation. We hypothesize that the hippocampus and mPFC bidirectionally interact during SWRs to support memory consolidation and decision-making. Rats were trained on the neuroeconomic spatial decision-making task, Restaurant Row, to make serial stay-skip decisions where the amount of effort (delay to reward) varied upon entry to each restaurant. Hippocampal cells and SWRs were recorded in rats with the mPFC transduced with inhibitory DREADDs. We found that disrupting the mPFC impaired consolidating SWRs in the hippocampus. Hippocampal SWR rates depended on the internalized value of the reward (derived from individual flavor preferences), a parameter important in decision-making, and disrupting the mPFC changed this relationship. Additionally, we found a dissociation between SWRs that occurred while rats were on the maze dependent upon whether those SWRs occurred while the rat was anticipating food reward or during post-reward consumption.

Vicarious Trial-and-Error Is Enhanced During Deliberation in Human Virtual Navigation in a Translational Foraging Task

Foraging tasks provide valuable insights into decision-making as animals decide how to allocate limited resources (such as time). In rodents, vicarious trial-and-error (back and forth movements), or VTE, is an important behavioral measure of deliberation which is enhanced early in learning and when animals are presented with difficult decisions. Using new translational versions of a rodent foraging task (the "Movie Row" and "Candy Row"), humans navigated a virtual maze presented on standard computers to obtain rewards (either short videos or candy) offered after a variable delay. Decision latencies were longer when participants were presented with difficult offers, overrode their preferences, and when they accepted an offer after rejecting a previous offer. In these situations, humans showed VTE-like behavior, where they were more likely to pause and/or reorient one or more times before making a decision. Behavior on these tasks replicated previous results from the rodent foraging task ("Restaurant Row") and a human version lacking a navigation component ("Web-Surf") and revealed some species differences. Compared to survey measures of delay-discounting, willingness to wait for rewards in the foraging task was not related to willingness to wait for hypothetical rewards. And, smoking status (use of cigarettes or e-cigarettes) was associated with stronger discounting of hypothetical future rewards, but was not well-related to performance on the foraging tasks. In contrast, individuals with overweight or obese BMI (≥25) did not show stronger delay-discounting, but individuals with BMI ≥ 25, and especially females, showed reduced sensitivity to sunk-costs (where their decisions were less sensitive to irrecoverable investments of effort) and less deliberation when presented with difficult offers. These data indicate that VTE is a behavioral index of deliberation in humans, and further support the Movie and Candy Row as translational tools to study decision-making in humans with the potential to provide novel insights about decision-making that are relevant to public health.

Potential roles of the rodent medial prefrontal cortex in conflict resolution between multiple decision-making systems

Mammalian decision-making is mediated by the interaction of multiple, neurally and computationally separable decision systems. Having multiple systems requires a mechanism to manage conflict and converge onto the selection of singular actions. A long history of evidence has pointed to the prefrontal cortex as a central component in processing the interactions between distinct decision systems and resolving conflicts among them. In this chapter we review four theories of how that interaction might occur and identify how the medial prefrontal cortex in the rodent may be involved in each theory. We then present experimental predictions implied by the neurobiological data in the context of each theory as a starting point for future investigation of medial prefrontal cortex and decision-making.

Interindividual differences in memory system local field potential activity predict behavioral strategy on a dual-solution T-maze

Individuals can use diverse behavioral strategies to navigate their environment including hippocampal-dependent place strategies reliant upon cognitive maps and striatal-dependent response strategies reliant upon egocentric body turns. The existence of multiple memory systems appears to facilitate successful navigation across a wide range of environmental and physiological conditions. The mechanisms by which these systems interact to ultimately generate a unitary behavioral response, however, remain unclear. We trained 20 male, Sprague-Dawley rats on a dual-solution T-maze while simultaneously recording local field potentials that were targeted to the dorsolateral striatum and dorsal hippocampus. Eight rats spontaneously exhibited a place strategy while the remaining 12 rats exhibited a response strategy. Interindividual differences in behavioral strategy were associated with distinct patterns of LFP activity between the dorsolateral striatum and dorsal hippocampus. Specifically, striatal-hippocampal theta activity was in-phase in response rats and out-of-phase in place rats and response rats exhibited elevated striatal-hippocampal coherence across a wide range of frequency bands. These contrasting striatal-hippocampal activity regimes were (a) present during both maze-learning and a 30 min premaze habituation period and (b) could be used to train support vector machines to reliably predict behavioral strategy. Distinct patterns of neuronal activity across multiple memory systems, therefore, appear to bias behavioral strategy selection and thereby contribute to interindividual differences in behavior.

Spatial encoding in dorsomedial prefrontal cortex and hippocampus is related during deliberation

Deliberation is thought to involve the internal simulation of the outcomes of candidate actions, the valuation of those outcomes, and the selection of the actions with the highest expected value. While it is known that deliberation involves prefrontal cortical areas, specifically the dorsomedial prefrontal cortex (dmPFC), as well as the hippocampus (HPC) and other brain regions, how these areas process prospective information and select actions is not well understood. We recorded simultaneously from ensembles in dmPFC and CA1 of dorsal HPC in rats during performance of a spatial contingency switching task, and examined the relationships between spatial and reward encoding in these two areas during deliberation at the choice point. We found that CA1 and dmPFC represented either goal locations or the current position simultaneously, but that when goal locations were encoded, HPC and dmPFC did not always represent the same goal location. Ensemble activity in dmPFC predicted when HPC would represent goal locations, but on a broad timescale on the order of seconds. Also, reward encoding in dmPFC increased during hippocampal theta cycles where CA1 ensembles represented the goal location. These results suggest that dmPFC and HPC share prospective information during deliberation, that dmPFC may influence whether HPC represents prospective information, and that information recalled about goal locations by HPC may be integrated into dmPFC reward representations on fast timescales.

Stress drives deliberative tendencies by influencing vicarious trial and error in decision making

Previous studies have reported the effects of stress on decision making. However, the wide range of findings make it difficult to identify the fundamental effects of stress on decision making and, therefore, how stress affects decision making remains unknown. To investigate the influence of stress on decision making, we employed "vicarious trial and error" (VTE), which refers to a rat's behavior of orienting the head toward options at a decision point. VTE is thought to reflect mental simulation for possible options preceding a decision. We examined effects of acute restraint stress on VTE in a T-maze choice task. VTE depended on learning and past reward outcomes. Acute restraint stress before rats ran the T-maze choice task induced VTE, especially in trials with low demand of VTE, and increased the number of head orientations and time spent during each VTE. On the other hand, stress did not affect task performance (probability of advantageous choice) and patterns of behavioral choice (win-stay lose-shift, exploration-exploitation). In addition, stress activated serotonergic and noradrenergic neurons in the dorsal raphe nucleus and locus coeruleus, which are modulators of impulsivity and attentional control in decision making. These results suggest that stress in decision making drives the VTE process, which may lead to deep consideration, over-thinking, and indecisiveness.

Dorsomedial prefrontal cortex and hippocampus represent strategic context even while simultaneously changing representation throughout a task session

Dorsomedial prefrontal cortex (dmPFC) and hippocampus (HPC) are thought to play complementary roles in a spatial working memory and decision-making network, where spatial information from HPC informs representations in dmPFC, and contextual information from dmPFC biases how HPC recalls that information. We recorded simultaneously from neural ensembles in rodent dmPFC and HPC as rats performed a rule-switching task, and found that ensembles in dmPFC and HPC simultaneously encoded task contingencies and other time-varying information. While ensembles in HPC transitioned to represent new contingencies at the same time as rats updated their strategies to be consistent with the new contingency, dmPFC ensembles transitioned earlier. Neural representations of other time-varying information also changed faster in dmPFC than in HPC. Our results suggest that HPC and dmPFC represent contingencies while simultaneously representing other information which changes over time, and that this contextual information is integrated into hippocampal representations more slowly than in dmPFC.

Mouse Navigation Strategies for Odor Source Localization

Navigating an odor landscape is a critical behavior for the survival of many species, including mice. An ethologically relevant mouse behavior is locating food using information about odor concentration. To approximate this behavior, we used an open field odor-based spot-finding task indoors with little wind, examining navigation strategies as mice search for and approach an odor source. After mice were trained to navigate to odor sources paired with food reward, we observed behavioral changes consistent with localization 10-45 cm away from the source. These behaviors included orientation toward the source, decreased velocity, and increased exploration time. We also found that the amplitude of 'casting,' which we define as lateral back and forth movement of the nose, increased with proximity to the source. Based on these observations, we created a concentration-sensitive agent-based model to simulate mouse behavior. This model provided evidence for a binaral-sniffing strategy (inter-nostril comparison of concentration in a single sniff) and a serial-sniffing strategy (sampling concentration, moving in space, and then sampling again). Serial-sniffing may be accomplished at farther distances by moving the body and at closer distances by moving the head (casting). Together, these results elucidate components of behavioral strategies for odor-based navigation.

Hippocampal Theta-Gamma Coupling Reflects State-Dependent Information Processing in Decision Making

During decision making, hippocampal activity encodes information sometimes about present and sometimes about potential future plans. The mechanisms underlying this transition remain unknown. Building on the evidence that gamma oscillations at different frequencies (low gamma [LG], 30–55 Hz; high gamma [HG], 60–90 Hz; and epsilon, 100–140 Hz) reflect inputs from different circuits, we identified how changes in those frequencies reflect different information-processing states. Using a unique noradrenergic manipulation by clonidine, which shifted both neural representations and gamma states, we found that future representations depended on gamma components. These changes were identifiable on each cycle of theta as asymmetries in the theta cycle, which arose from changes within the ratio of LG and HG power and the underlying phases of those gamma rhythms within the theta cycle. These changes in asymmetry of the theta cycle reflected changes in representations of present and future on each theta cycle.

Disrupting the medial prefrontal cortex alters hippocampal sequences during deliberative decision making

Current theories of deliberative decision making suggest that deliberative decisions arise from imagined simulations that require interactions between the prefrontal cortex and hippocampus. In rodent navigation experiments, hippocampal theta sequences advance from the location of the rat ahead to the subsequent goal. To examine the role of the medial prefrontal cortex (mPFC) on the hippocampus, we disrupted the mPFC with DREADDs (designer receptors exclusively activated by designer drugs). Using the Restaurant Row foraging task, we found that mPFC disruption resulted in decreased vicarious trial and error behavior, reduced the number of theta sequences, and impaired theta sequences in hippocampus. mPFC disruption led to larger changes in the initiation of the hippocampal theta sequences that represent the current location of the rat rather than to the later portions that represent the future outcomes. These data suggest that the mPFC likely provides an important component to the initiation of deliberative sequences and provides support for an episodic-future thinking, working memory interpretation of deliberation. NEW & NOTEWORTHY The medial prefrontal cortex (mPFC) and hippocampus interact during deliberative decision making. Disruption of the mPFC impaired hippocampal processes, including the local and nonlocal representations of space along each theta cycle and the initiation of hippocampal theta sequences, while sparing place cell firing characteristics and phase precession. mPFC disruption reduced the deliberative behavioral process vicarious trial and error and improved economic behaviors on this task.

The perirhinal cortex supports spatial intertemporal choice stability

In order to optimize outcomes in the face of uncertainty, one must recall past experiences and extrapolate to the future by assigning values to different choice outcomes. This behavior requires an interplay between memory and reward valuation, necessitating communication across many brain regions. At the anatomical nexus of this interplay is the perirhinal cortex (PRC). The PRC is densely connected to the amygdala and orbital frontal cortex, regions that have been implicated in reward-based decision making, as well as the hippocampus. Thus, the PRC could serve as a hub for integrating memory, reward, and prediction. The PRC's role in value-based decision making, however, has not been empirically examined. Therefore, we tested the role of the PRC in a spatial delay discounting task, which allows rats to choose between a 1-s delay for a small food reward and a variable delay for a large food reward, with the delay to the large reward increasing after choice of each large reward and decreasing after each small reward. The rat can therefore adjust the delay by consecutively choosing the same reward or stabilize the delay by alternating between sides. The latter has been shown to occur once the 'temporal cost' of the large reward is established and is a decision-making process termed 'exploitation'. When the PRC was bilaterally inactivated with the GABA(A) agonist muscimol, rats spent fewer trials successfully exploiting to maintain a fixed delay compared to the vehicle control condition. Moreover, PRC inactivation resulted in an increased number of vicarious trial and error (VTE) events at the choice point, where rats had to decide between the two rewards. These behavioral patterns suggest that the PRC is critical for maintaining stability in linking a choice to a reward outcome in the face of a variable cost.

An improved within-session self-adjusting delay discounting procedure for the study of choice impulsivity in rats

RATIONALE

Delay-discounting procedures involving choice between small immediate rewards and large delayed rewards are used to study impulsivity in rodents. Improving existing procedures may provide new insights into the neurobiological mechanisms underlying decision-making processes.

OBJECTIVES

To develop a novel delay-discounting procedure that adjusts the delay value within individual sessions based on the rat's most recent choices.

METHODS

Compared to previously developed procedure, we required a more consistent demonstration of preference, five consecutive choices of the large or small reward, a criterion that is more likely to reflect deliberate choice by the animal, as opposed to two consecutive choices. In addition, delays were changed in steps of 5 s (rather than 1 s), because 5-s increments should be more easily discriminated and may produce a more distinct effect on choice. We characterized the procedure behaviorally by manipulating the duration of the session and the consecutive choice criterion, and we investigated the stability of the behavior upon interruption of training. We also characterized the procedure pharmacologically by investigating the effects of dopaminergic compounds.

RESULTS

Our procedures allowed obtaining two complementary measures of delay discounting: (1) the percentage of choices of the delay option and (2) the mean adjusting delay, an index of the delay that animals choose more frequently. We found that our procedure rapidly establishes a baseline of choice behavior that remains stable over time and is highly sensitive to manipulations of the dopaminergic system.

CONCLUSIONS

This procedure may provide a useful tool for investigating the neurobiology of inter-temporal choice and decision-making.

Mice learn to avoid regret

Regret can be defined as the subjective experience of recognizing that one has made a mistake and that a better alternative could have been selected. The experience of regret is thought to carry negative utility. This typically takes two distinct forms: augmenting immediate postregret valuations to make up for losses, and augmenting long-term changes in decision-making strategies to avoid future instances of regret altogether. While the short-term changes in valuation have been studied in human psychology, economics, neuroscience, and even recently in nonhuman-primate and rodent neurophysiology, the latter long-term process has received far less attention, with no reports of regret avoidance in nonhuman decision-making paradigms. We trained 31 mice in a novel variant of the Restaurant Row economic decision-making task, in which mice make decisions of whether to spend time from a limited budget to achieve food rewards of varying costs (delays). Importantly, we tested mice longitudinally for 70 consecutive days, during which the task provided their only source of food. Thus, decision strategies were interdependent across both trials and days. We separated principal commitment decisions from secondary reevaluation decisions across space and time and found evidence for regret-like behaviors following change-of-mind decisions that corrected prior economically disadvantageous choices. Immediately following change-of-mind events, subsequent decisions appeared to make up for lost effort by altering willingness to wait, decision speed, and pellet consumption speed, consistent with past reports of regret in rodents. As mice were exposed to an increasingly reward-scarce environment, we found they adapted and refined distinct economic decision-making strategies over the course of weeks to maximize reinforcement rate. However, we also found that even without changes in reinforcement rate, mice transitioned from an early strategy rooted in foraging to a strategy rooted in deliberation and planning that prevented future regret-inducing change-of-mind episodes from occurring. These data suggest that mice are learning to avoid future regret, independent of and separate from reinforcement rate maximization.

Regret describes a unique postdecision phenomenon in which losses are realized as a fault of one’s own actions. Regret is often hypothesized to have an inherent negative utility, and humans will often incur costs so as to avoid the risk of future regret. However, current models of nonhuman decision-making are based on reward maximization hypotheses. We recently found that rats express regret behaviorally and neurophysiologically on neuroeconomic foraging tasks; however, it remains unknown whether nonhuman animals will change strategies so as to avoid regret, even in the absence of changes in the achieved rate of reinforcement. Here, we provide the first evidence that mice change strategies to avoid future regret, independent of and separate from reinforcement rate maximization. Our data suggest mice accomplish this by shifting from a foraging decision-making strategy that produces change-of-mind decisions after investment mistakes to one rooted in deliberation that learns to plan ahead.

Altering gain of the infralimbic-to-accumbens shell circuit alters economically dissociable decision-making algorithms

The nucleus accumbens shell (NAcSh) is involved in reward valuation. Excitatory projections from infralimbic cortex (IL) to NAcSh undergo synaptic remodeling in rodent models of addiction and enable the extinction of disadvantageous behaviors. However, how the strength of synaptic transmission of the IL-NAcSh circuit affects decision-making information processing and reward valuation remains unknown, particularly because these processes can conflict within a given trial and particularly given recent data suggesting that decisions arise from separable information-processing algorithms. The approach of many neuromodulation studies is to disrupt information flow during on-going behaviors; however, this limits the interpretation of endogenous encoding of computational processes. Furthermore, many studies are limited by the use of simple behavioral tests of value which are unable to dissociate neurally distinct decision-making algorithms. We optogenetically altered the strength of synaptic transmission between glutamatergic IL-NAcSh projections in mice trained on a neuroeconomic task capable of separating multiple valuation processes. We found that induction of long-term depression in these synapses produced lasting changes in foraging processes without disrupting deliberative processes. Mice displayed inflated reevaluations to stay when deciding whether to abandon continued reward-seeking investments but displayed no changes during initial commitment decisions. We also developed an ensemble-level measure of circuit-specific plasticity that revealed individual differences in foraging valuation tendencies. Our results demonstrate that alterations in projection-specific synaptic strength between the IL and the NAcSh are capable of augmenting self-control economic valuations within a particular decision-making modality and suggest that the valuation mechanisms for these multiple decision-making modalities arise from different circuits.

The Web-Surf Task: A translational model of human decision-making

Animal models of decision-making are some of the most highly regarded psychological process models; however, there remains a disconnection between how these models are used for pre-clinical applications and the resulting treatment outcomes. This may be due to untested assumptions that different species recruit the same neural or psychological mechanisms. We propose a novel human foraging paradigm (Web-Surf Task) that we translated from a rat foraging paradigm (Restaurant Row) to evaluate cross-species decision-making similarities. We examined behavioral parallels in human and non-human animals using the respective tasks. We also compared two variants of the human task, one using videos and the other using photos as rewards, by correlating revealed and stated preferences. We demonstrate similarities in choice behaviors and decision reaction times in human and rat subjects. Findings also indicate that videos yielded more reliable and valid results. The joint use of the Web-Surf Task and Restaurant Row is therefore a promising approach for functional translational research, aiming to bridge pre-clinical and clinical lines of research using analogous tasks.

Interplay between Hippocampal Sharp-Wave-Ripple Events and Vicarious Trial and Error Behaviors in Decision Making

Current theories posit that memories encoded during experiences are subsequently consolidated into longer-term storage. Hippocampal sharp-wave-ripple (SWR) events have been linked to this consolidation process during sleep, but SWRs also occur during awake immobility, where their role remains unclear. We report that awake SWR rates at the reward site are inversely related to the prevalence of vicarious trial and error (VTE) behaviors, thought to be involved in deliberation processes. SWR rates were diminished immediately after VTE behaviors and an increase in the rate of SWR events at the reward site predicted a decrease in subsequent VTE behaviors at the choice point. Furthermore, SWR disruptions increased VTE behaviors. These results suggest an inverse relationship between SWRs and VTE behaviors and suggest that awake SWRs and associated planning and memory consolidation mechanisms are engaged specifically in the context of higher levels of behavioral certainty.

Representational changes of latent strategies in rat medial prefrontal cortex precede changes in behaviour

The ability to change behavioural strategies in the face of a changing world has been linked to the integrity of medial prefrontal cortex (mPFC) function in several species. While recording studies have found that mPFC representations reflect the strategy being used, lesion studies suggest that mPFC is necessary for changing strategy. Here we examine the relationship between representational changes in mPFC and behavioural strategy changes in the rat. We found that on tasks with a forced change in reward criterion, strategy-related representational transitions in mPFC occurred after animals learned that the reward contingency had changed, but before their behaviour changed. On tasks in which animals made their own strategic decisions, representational transitions in mPFC preceded changes in behaviour. These results suggest that mPFC does not merely reflect the action–selection policy of the animal, but rather that mPFC processes information related to a need for a change in strategy.

The medial prefrontal cortex (mPFC) is involved in changing behavioural strategies. Recording neural ensembles in rats, Powell and Redish find that the requirement for those changes is represented in mPFC before they manifest behaviourally, both in tasks that externally force a change and in tasks with self-determined change.

Rats value time differently on equivalent foraging and delay-discounting tasks

All organisms have to consider consequences that vary through time. Theories explaining how animals handle intertemporal choice include delay-discounting models, in which the value of future rewards is discounted by the delay until receipt, and foraging models, which predict that decision-makers maximize rate of reward. We measured the behavior of rats on a 2-option delay-discounting task and a stay/go foraging task that were equivalent for rate of reward and physical demand. Despite the highly shared features of the tasks, rats were willing to wait much longer on the foraging task than on the delay-discounting task. Moreover, choice performance by rats was less optimal in terms of total reward received on the foraging task compared to the delay-discounting task. We applied a suite of intertemporal choice models to the data but found that we needed a novel model incorporating interactions of decision-making systems to successfully explain behavior. Our findings (a) highlight the importance of factors that historically have been seen as irrelevant and (b) indicate the inadequacy of current general theories of intertemporal choice. (PsycINFO Database Record

Vicarious trial and error

When rats come to a decision point, they sometimes pause and look back and forth as if deliberating over the choice; at other times, they proceed as if they have already made their decision. In the 1930s, this pause-and-look behaviour was termed 'vicarious trial and error' (VTE), with the implication that the rat was 'thinking about the future'. The discovery in 2007 that the firing of hippocampal place cells gives rise to alternating representations of each of the potential path options in a serial manner during VTE suggested a possible neural mechanism that could underlie the representations of future outcomes. More-recent experiments examining VTE in rats suggest that there are direct parallels to human processes of deliberative decision making, working memory and mental time travel.

Active Inference, epistemic value, and vicarious trial and error

Balancing habitual and deliberate forms of choice entails a comparison of their respective merits—the former being faster but inflexible, and the latter slower but more versatile. Here, we show that arbitration between these two forms of control can be derived from first principles within an Active Inference scheme. We illustrate our arguments with simulations that reproduce rodent spatial decisions in T-mazes. In this context, deliberation has been associated with vicarious trial and error (VTE) behavior (i.e., the fact that rodents sometimes stop at decision points as if deliberating between choice alternatives), whose neurophysiological correlates are “forward sweeps” of hippocampal place cells in the arms of the maze under consideration. Crucially, forward sweeps arise early in learning and disappear shortly after, marking a transition from deliberative to habitual choice. Our simulations show that this transition emerges as the optimal solution to the trade-off between policies that maximize reward or extrinsic value (habitual policies) and those that also consider the epistemic value of exploratory behavior (deliberative or epistemic policies)—the latter requiring VTE and the retrieval of episodic information via forward sweeps. We thus offer a novel perspective on the optimality principles that engender forward sweeps and VTE, and on their role on deliberate choice.

Manipulating Decisiveness in Decision Making: Effects of Clonidine on Hippocampal Search Strategies

Decisiveness is the ability to commit to a decision quickly and efficiently; in contrast, indecision entails the repeated consideration of multiple alternative possibilities. In humans, the α2-adrenergic receptor agonist clonidine increases decisiveness in tasks that require planning through unknown neural mechanisms. In rats, indecision is manifested as reorienting behaviors at choice points (vicarious trial and error [VTE]), during which hippocampal representations alternate between prospective options. To determine whether the increase in decisiveness driven by clonidine also entails changes in hippocampal search processes, we compared the effect of clonidine on spatial representations in hippocampal neural ensembles as rats passed through a T-shaped decision point. Consistent with previous experiments, hippocampal representations reflected both chosen and unchosen paths during VTE events under saline control conditions. Also, consistent with previous experiments, hippocampal representations reflected the chosen path more than the unchosen path when the animal did not show VTE at the choice point. Injection of clonidine suppressed the spatial representation of the unchosen path at the choice point on VTE laps and hastened the differentiation of spatial representations of the chosen path from the unchosen path on non-VTE laps to appear before reaching the choice point. These results suggest that the decisiveness seen under clonidine is due to limited exploration of potential options in hippocampus, and suggest novel roles for noradrenaline as a modulator of the hippocampal search processes. Significance statement: Clonidine, an α2-adrenergic receptor agonist, which decreases the level of noradrenaline in vivo, has an interesting effect in humans and other animals: it makes them more decisive. However, the mechanisms by which clonidine makes them more decisive remain unknown. Researchers have speculated that clonidine limits the amount of mental search that subjects do when planning options. We test this hypothesis by measuring the mental search strategy in rats through hippocampal recordings. We find that clonidine limits the options searched by rats, suggesting that noradrenaline also plays a role in balancing exploration and exploitation in internally simulated behaviors, similar to its role in balancing exploration and exploitation in external behaviors.

Differential Arc expression in the hippocampus and striatum during the transition from attentive to automatic navigation on a plus maze

The strategies utilized to effectively perform a given task change with practice and experience. During a spatial navigation task, with relatively little training, performance is typically attentive enabling an individual to locate the position of a goal by relying on spatial landmarks. These (place) strategies require an intact hippocampus. With task repetition, performance becomes automatic; the same goal is reached using a fixed response or sequence of actions. These (response) strategies require an intact striatum. The current work aims to understand the activation patterns across these neural structures during this experience-dependent strategy transition. This was accomplished by region-specific measurement of activity-dependent immediate early gene expression among rats trained to different degrees on a dual-solution task (i.e., a task that can be solved using either place or response navigation). As expected, rats increased their reliance on response navigation with extended task experience. In addition, dorsal hippocampal expression of the immediate early gene Arc was considerably reduced in rats that used a response strategy late in training (as compared with hippocampal expression in rats that used a place strategy early in training). In line with these data, vicarious trial and error, a behavior linked to hippocampal function, also decreased with task repetition. Although Arc mRNA expression in dorsal medial or lateral striatum alone did not correlate with training stage, the ratio of expression in the medial striatum to that in the lateral striatum was relatively high among rats that used a place strategy early in training as compared with the ratio among over-trained response rats. Altogether, these results identify specific changes in the activation of dissociated neural systems that may underlie the experience-dependent emergence of response-based automatic navigation.

Multimodal Plasticity in Dorsal Striatum While Learning a Lateralized Navigation Task

Growing evidence supports a critical role for the dorsal striatum in cognitive as well as motor control. Both lesions and in vivo recordings demonstrate a transition in the engaged dorsal striatal subregion, from dorsomedial to dorsolateral, as skill performance shifts from an attentive phase to a more automatic or habitual phase. What are the neural mechanisms supporting the cognitive and behavioral transitions in skill learning? To pursue this question, we used T-maze training during which rats transition from early, attentive (dorsomedial) to late habitual (dorsolateral) performance. Following early or late training, we performed the first direct comparison of bidirectional synaptic plasticity in striatal brain slices, and the first evaluation of striatal synaptic plasticity by hemisphere relative to a learned turn. Consequently, we find that long-term potentiation and long-term depression are independently modulated with learning rather than reciprocally linked as previously suggested. Our results establish that modulation of evoked synaptic plasticity with learning depends on striatal subregion, training stage, and hemisphere relative to the learned turn direction. Exclusive to the contralateral hemisphere, intrinsic excitability is enhanced in dorsomedial relative to dorsolateral medium spiny neurons early in training and population responses are dampened late in training. Neuronal reconstructions indicate dendritic remodeling after training, which may represent a novel form of pruning. In conclusion, we describe region- and hemisphere-specific changes in striatal synaptic, intrinsic, and morphological plasticity which correspond to T-maze learning stages, and which may play a role in the cognitive transition between attentive and habitual strategies. Significance statement: We investigated neural plasticity in dorsal striatum from rats that were briefly or extensively trained on a directional T-maze task. Our results demonstrate that both the extent of training and the direction a rat learns to turn control the location and type of change in synaptic plasticity. In addition, brief training produces changes in neuron excitability only within one striatal subregion, whereas all training produces widespread changes in dendritic morphology. Our results suggest that activity in dorsomedial striatum strengthens the rewarded turn after brief training, whereas activity in dorsolateral striatum suppresses unrewarded turns after extensive training. This study illuminates how plasticity mediates learning using a task recognized for transitioning subjects from attentive to automatic performance.

Noradrenergic signaling in the medial prefrontal cortex and amygdala differentially regulates vicarious trial-and-error in a spatial decision-making task

In uncertain choice situations, we deliberately search and evaluate possible options before taking an action. Once we form a preference regarding the current situation, we take an action more automatically and with less deliberation. In rats, the deliberation process can be seen in vicarious trial-and-error behavior (VTE), which is a head-orienting behavior toward options at a choice point. Recent neurophysiological findings suggest that VTE reflects the rat's thinking about future options as deliberation, expectation, and planning when rats feel conflict. VTE occurs depending on the demand: an increase occurs during initial learning, and a decrease occurs with progression in learning. However, the brain circuit underlying the regulation of VTE has not been thoroughly examined. In situations in which VTE often appears, the medial prefrontal cortex (mPFC) and the amygdala (AMY) are crucial for learning and decision making. Our previous study reported that noradrenaline regulates VTE. Here, to investigate whether the mPFC and AMY are involved in regulation of VTE, we examined the effects of local injection of clonidine, an alpha2 adrenergic autoreceptor agonist, into either region in rats during VTE and choice behavior during a T-maze choice task. Injection of clonidine into either region impaired selection of the advantageous choice in the task. Furthermore, clonidine injection into the mPFC suppressed occurrence of VTE in the early phase of the task, whereas injection into the AMY inhibited the decrease in VTE in the later phase and thus maintained a high level of VTE throughout the task. These results suggest that the mPFC and AMY play a role in the increase and decrease in VTE, respectively, and that noradrenergic mechanisms mediate the dynamic regulation of VTE over experiences.

Hippocampus and subregions of the dorsal striatum respond differently to a behavioral strategy change on a spatial navigation task

Goal-directed and habit-based behaviors are driven by multiple but dissociable decision making systems involving several different brain areas, including the hippocampus and dorsal striatum. On repetitive tasks, behavior transitions from goal directed to habit based with experience. Hippocampus has been implicated in initial learning and dorsal striatum in automating behavior, but recent studies suggest that subregions within the dorsal striatum have distinct roles in mediating habit-based and goal-directed behavior. We compared neural activity in the CA1 region of hippocampus with anterior dorsolateral and posterior dorsomedial striatum in rats on a spatial choice task, in which subjects experienced reward delivery changes that forced them to adjust their behavioral strategy. Our results confirm the importance of the hippocampus in evaluating predictive steps during goal-directed behavior, while separate circuits in the basal ganglia integrated relevant information during automation of actions and recognized when new behaviors were needed to continue obtaining rewards.

A functional difference in information processing between orbitofrontal cortex and ventral striatum during decision-making behaviour

Both orbitofrontal cortex (OFC) and ventral striatum (vStr) have been identified as key structures that represent information about value in decision-making tasks. However, the dynamics of how this information is processed are not yet understood. We recorded ensembles of cells from OFC and vStr in rats engaged in the spatial adjusting delay-discounting task, a decision-making task that involves a trade-off between delay to and magnitude of reward. Ventral striatal neural activity signalled information about reward before the rat's decision, whereas such reward-related signals were absent in OFC until after the animal had committed to its decision. These data support models in which vStr is directly involved in action selection, but OFC processes decision-related information afterwards that can be used to compare the predicted and actual consequences of behaviour.

Cognitive effort: A neuroeconomic approach

Cognitive effort has been implicated in numerous theories regarding normal and aberrant behavior and the physiological response to engagement with demanding tasks. Yet, despite broad interest, no unifying, operational definition of cognitive effort itself has been proposed. Here, we argue that the most intuitive and epistemologically valuable treatment is in terms of effort-based decision-making, and advocate a neuroeconomics-focused research strategy. We first outline psychological and neuroscientific theories of cognitive effort. Then we describe the benefits of a neuroeconomic research strategy, highlighting how it affords greater inferential traction than do traditional markers of cognitive effort, including self-reports and physiologic markers of autonomic arousal. Finally, we sketch a future series of studies that can leverage the full potential of the neuroeconomic approach toward understanding the cognitive and neural mechanisms that give rise to phenomenal, subjective cognitive effort.

Choice impulsivity: Definitions, measurement issues, and clinical implications

Impulsivity critically relates to many psychiatric disorders. Given the multifaceted construct that impulsivity represents, defining core aspects of impulsivity is vital for the assessment and understanding of clinical conditions. Choice impulsivity (CI), involving the preferential selection of smaller sooner rewards over larger later rewards, represents one important type of impulsivity. The International Society for Research on Impulsivity (InSRI) convened to discuss the definition and assessment of CI and provide recommendations regarding measurement across species. Commonly used preclinical and clinical CI behavioral tasks are described, and considerations for each task are provided to guide CI task selection. Differences in assessment of CI (self-report, behavioral) and calculating CI indices (e.g., area-under-the-curve, indifference point, and steepness of discounting curve) are discussed along with properties of specific behavioral tasks used in preclinical and clinical settings. The InSRI group recommends inclusion of measures of CI in human studies examining impulsivity. Animal studies examining impulsivity should also include assessments of CI and these measures should be harmonized in accordance with human studies of the disorders being modeled in the preclinical investigations. The choice of specific CI measures to be included should be based on the goals of the study and existing preclinical and clinical literature using established CI measures.

Aging impairs deliberation and behavioral flexibility in inter-temporal choice

Inter-temporal choice depends on multiple, interacting systems, some of which may be compromised with age. Some of these systems may be responsible for ongoing trial-by-trial choice strategies. Some may represent the consequences of action. Some may be necessary for the coupling between anticipated consequences and strategies currently in use, flexibly guiding behavior. When faced with a difficult decision, rats will orient back and forth, a behavior termed "vicarious trial and error" (VTE). Recent experiments have linked the occurrence of VTE to hippocampal search processes and behavioral flexibility. We tested 5 month (n = 6), 9 month (n = 8) and over-27 month-old (n = 10) rats on a Spatial Adjusting Delay Discounting task to examine how aging impacted lap-by-lap strategies and VTE during inter-temporal choice. Rats chose between spatially separated food goals that provided a smaller-sooner or larger-later reward. On each lap, the delay to the larger-later reward was adjusted as a function of the rat's decisions, increasing by 1 s after delayed-side choices and decreasing by 1 s after non-delayed side choices. The strategies that aged rats used differed from those used in young and adult rats. Moreover, aged rats produced reliably more VTE behaviors, for protracted periods of time, uncoupled from behavioral flexibility.

Hippocampus, delay discounting, and vicarious trial-and-error

In decision-making, an immediate reward is usually preferred to a delayed reward, even if the latter is larger. We tested whether the hippocampus is necessary for this form of temporal discounting, and for vicarious trial-and-error at the decision point. Rats were trained on a recently developed, adjustable delay-discounting task (Papale et al. (2012) Cogn Affect Behav Neurosci 12:513-526), which featured a choice between a small, nearly immediate reward, and a larger, delayed reward. Rats then received either hippocampus or sham lesions. Animals with hippocampus lesions adjusted the delay for the larger reward to a level similar to that of sham-lesioned animals, suggesting a similar valuation capacity. However, the hippocampus lesion group spent significantly longer investigating the small and large rewards in the first part of the sessions, and were less sensitive to changes in the amount of reward in the large reward maze arm. Both sham- and hippocampus-lesioned rats showed a greater amount of vicarious trial-and-error on trials in which the delay was adjusted. In a nonadjusting version of the delay discounting task, animals with hippocampus lesions showed more variability in their preference for a larger reward that was delayed by 10 s compared with sham-lesioned animals. To verify the lesion behaviorally, rat were subsequently trained on a water maze task, and rats with hippocampus lesions were significantly impaired compared with sham-lesioned animals. The findings on the delay discounting tasks suggest that damage to the hippocampus may impair the detection of reward magnitude.

Noradrenergic modulation of vicarious trial-and-error behavior during a spatial decision-making task in rats

Deliberation between possible options before making a decision is crucial to responding with an optimal choice. However, the neural mechanisms regulating this deliberative decision-making process are still unclear. Recent studies have proposed that the locus coeruleus-noradrenaline (LC-NA) system plays a role in attention, behavioral flexibility, and exploration, which contribute to the search for an optimal choice under uncertain situations. In the present study, we examined whether the LC-NA system relates to the deliberative process in a T-maze spatial decision-making task in rats. To quantify deliberation in rats, we recorded vicarious trial-and-error behavior (VTE), which is considered to reflect a deliberative process exploring optimal choices. In experiment 1, we manipulated the difficulty of choice by varying the amount of reward pellets between the two maze arms (0 vs. 4, 1 vs. 3, 2 vs. 2). A difficulty-dependent increase in VTE was accompanied by a reduction of choice bias toward the high reward arm and an increase in time required to select one of the two arms in the more difficult manipulation. In addition, the increase of c-Fos-positive NA neurons in the LC depended on the task difficulty and the amount of c-Fos expression in LC-NA neurons positively correlated with the occurrence of VTE. In experiment 2, we inhibited LC-NA activity by injection of clonidine, an agonist of the alpha2 autoreceptor, during a decision-making task (1 vs. 3). The clonidine injection suppressed occurrence of VTE in the early phase of the task and subsequently impaired a valuable choice later in the task. These results suggest that the LC-NA system regulates the deliberative process during decision-making.