Triple dissociation of information processing in dorsal striatum, ventral striatum, and hippocampus on a learned spatial decision task

From ventral-medial to dorsal-lateral striatum: neural correlates of reward-guided decision-making

The striatum is critical for reward-guided and habitual behavior. Anatomical and interference studies suggest a functional heterogeneity within striatum. Medial regions, such as nucleus accumbens core and dorsal medial striatum play roles in goal-directed behavior, while dorsal lateral striatum is critical for control of habitual action. Subdivisions of striatum are topographically connected with different cortical and subcortical structures forming channels that carry information related to limbic, associative, and sensorimotor functions. Here, we describe data showing that as one progresses from ventral-medial to dorsal-lateral striatum, there is a shift from more prominent value encoding to activity more closely related to associative and motor aspects of decision-making. In addition, we will describe data suggesting that striatal circuits work in parallel to control behavior and that regions within striatum can compensate for each other when functions are disrupted.

Neurophysiology of Reward-Guided Behavior: Correlates Related to Predictions, Value, Motivation, Errors, Attention, and Action

Many brain areas are activated by the possibility and receipt of reward. Are all of these brain areas reporting the same information about reward? Or are these signals related to other functions that accompany reward-guided learning and decision-making? Through carefully controlled behavioral studies, it has been shown that reward-related activity can represent reward expectations related to future outcomes, errors in those expectations, motivation, and signals related to goal- and habit-driven behaviors. These dissociations have been accomplished by manipulating the predictability of positively and negatively valued events. Here, we review single neuron recordings in behaving animals that have addressed this issue. We describe data showing that several brain areas, including orbitofrontal cortex, anterior cingulate, and basolateral amygdala signal reward prediction. In addition, anterior cingulate, basolateral amygdala, and dopamine neurons also signal errors in reward prediction, but in different ways. For these areas, we will describe how unexpected manipulations of positive and negative value can dissociate signed from unsigned reward prediction errors. All of these signals feed into striatum to modify signals that motivate behavior in ventral striatum and guide responding via associative encoding in dorsolateral striatum.

How cognitive theory guides neuroscience

The field of cognitive science studies latent, unobservable cognitive processes that generate observable behaviors. Similarly, cognitive neuroscience attempts to link latent cognitive processes with the neural mechanisms that generate them. Although neural processes are partially observable (with imaging and electrophysiology), it would be a mistake to 'skip' the cognitive level and pursue a purely neuroscientific enterprise to studying behavior. In fact, virtually all of the major advances in understanding the neural basis of behavior over the last century have relied fundamentally on principles of cognition for guiding the appropriate measurements, manipulations, tasks, and interpretations. We provide several examples from the domains of episodic memory, working memory and cognitive control, and decision making in which cognitive theorizing and prior experimentation has been essential in guiding neuroscientific investigations and discoveries.

The nicotinic cholinergic system function in the human brain

Research on the nicotinic cholinergic system function in the brain was previously mainly derived from animal studies, yet, research in humans is growing. Up to date, findings allow significant advances on the understanding of nicotinic cholinergic effects on human cognition, emotion and behavior using a range of functional brain imaging approaches such as pharmacological functional magnetic resonance imaging or positron emission tomography. Studies provided insights across various mechanistic psychological domains using different tasks as well as at rest in both healthy individuals and patient populations, with so far partly mixed results reporting both enhancements and decrements of neural activity related to the nicotinic cholinergic system. Moreover, studies on the relation between brain structure and the nicotinic cholinergic system add important information in this context. The present review summarizes the current status of human brain imaging studies and presents the findings within a theoretical and clinical perspective as they may be useful not only for an advancement of the understanding of basic nicotinic cholinergic-related mechanisms, but also for the development and integration of psychological and pharmacological treatment approaches. Patterns of functional neuroanatomy and neural circuitry across various cognitive and emotional domains may be used as neuropsychological markers of mental disorders such as addiction, Alzheimer's disease, Parkinson disease or schizophrenia, where nicotinic cholinergic system changes are characteristic. This article is part of the Special Issue entitled 'The Nicotinic Acetylcholine Receptor: From Molecular Biology to Cognition'.

Oscillatory activity in the medial prefrontal cortex and nucleus accumbens correlates with impulsivity and reward outcome

Actions expressed prematurely without regard for their consequences are considered impulsive. Such behaviour is governed by a network of brain regions including the prefrontal cortex (PFC) and nucleus accumbens (NAcb) and is prevalent in disorders including attention deficit hyperactivity disorder (ADHD) and drug addiction. However, little is known of the relationship between neural activity in these regions and specific forms of impulsive behaviour. In the present study we investigated local field potential (LFP) oscillations in distinct sub-regions of the PFC and NAcb on a 5-choice serial reaction time task (5-CSRTT), which measures sustained, spatially-divided visual attention and action restraint. The main findings show that power in gamma frequency (50-60 Hz) LFP oscillations transiently increases in the PFC and NAcb during both the anticipation of a cue signalling the spatial location of a nose-poke response and again following correct responses. Gamma oscillations were coupled to low-frequency delta oscillations in both regions; this coupling strengthened specifically when an error response was made. Theta (7-9 Hz) LFP power in the PFC and NAcb increased during the waiting period and was also related to response outcome. Additionally, both gamma and theta power were significantly affected by upcoming premature responses as rats waited for the visual cue to respond. In a subgroup of rats showing persistently high levels of impulsivity we found that impulsivity was associated with increased error signals following a nose-poke response, as well as reduced signals of previous trial outcome during the waiting period. Collectively, these in-vivo neurophysiological findings further implicate the PFC and NAcb in anticipatory impulsive responses and provide evidence that abnormalities in the encoding of rewarding outcomes may underlie trait-like impulsive behaviour.

Perceiving active listening activates the reward system and improves the impression of relevant experiences

Although active listening is an influential behavior, which can affect the social responses of others, the neural correlates underlying its perception have remained unclear. Sensing active listening in social interactions is accompanied by an improvement in the recollected impressions of relevant experiences and is thought to arouse positive feelings. We therefore hypothesized that the recognition of active listening activates the reward system, and that the emotional appraisal of experiences that had been subject to active listening would be improved. To test these hypotheses, we conducted functional magnetic resonance imaging (fMRI) on participants viewing assessments of their own personal experiences made by evaluators with or without active listening attitude. Subjects rated evaluators who showed active listening more positively. Furthermore, they rated episodes more positively when they were evaluated by individuals showing active listening. Neural activation in the ventral striatum was enhanced by perceiving active listening, suggesting that this was processed as rewarding. It also activated the right anterior insula, representing positive emotional reappraisal processes. Furthermore, the mentalizing network was activated when participants were being evaluated, irrespective of active listening behavior. Therefore, perceiving active listening appeared to result in positive emotional appraisal and to invoke mental state attribution to the active listener.

Internally generated sequences in learning and executing goal-directed behavior

A network of brain structures including hippocampus (HC), prefrontal cortex, and striatum controls goal-directed behavior and decision making. However, the neural mechanisms underlying these functions are unknown. Here, we review the role of 'internally generated sequences': structured, multi-neuron firing patterns in the network that are not confined to signaling the current state or location of an agent, but are generated on the basis of internal brain dynamics. Neurophysiological studies suggest that such sequences fulfill functions in memory consolidation, augmentation of representations, internal simulation, and recombination of acquired information. Using computational modeling, we propose that internally generated sequences may be productively considered a component of goal-directed decision systems, implementing a sampling-based inference engine that optimizes goal acquisition at multiple timescales of on-line choice, action control, and learning.

Neurons in the ventral striatum exhibit cell-type-specific representations of outcome during learning

The ventromedial striatum (VMS) is a node in circuits underpinning both affect and reinforcement learning. The cellular bases of these functions and especially their potential linkages have been unclear. VMS cholinergic interneurons, however, have been singled out as being related both to affect and to reinforcement-based conditioning, raising the possibility that unique aspects of their signaling could account for these functions. Here we show that VMS tonically active neurons (TANs), including putative cholinergic interneurons, generate unique bidirectional outcome responses during reward-based learning, reporting both positive (reward) and negative (reward omission) outcomes when behavioral change is prompted by switches in reinforcement contingencies. VMS output neurons (SPNs), by contrast, are nearly insensitive to switches in reinforcement contingencies, gradually losing outcome signaling while maintaining responses at trial initiation and goal approach. Thus, TANs and SPNs in the VMS provide distinct signals optimized for different aspects of the learning process.

Vestibular pathways involved in cognition

Recent discoveries have emphasized the role of the vestibular system in cognitive processes such as memory, spatial navigation and bodily self-consciousness. A precise understanding of the vestibular pathways involved is essential to understand the consequences of vestibular diseases for cognition, as well as develop therapeutic strategies to facilitate recovery. The knowledge of the “vestibular cortical projection areas”, defined as the cortical areas activated by vestibular stimulation, has dramatically increased over the last several years from both anatomical and functional points of view. Four major pathways have been hypothesized to transmit vestibular information to the vestibular cortex: (1) the vestibulo-thalamo-cortical pathway, which probably transmits spatial information about the environment via the parietal, entorhinal and perirhinal cortices to the hippocampus and is associated with spatial representation and self-versus object motion distinctions; (2) the pathway from the dorsal tegmental nucleus via the lateral mammillary nucleus, the anterodorsal nucleus of the thalamus to the entorhinal cortex, which transmits information for estimations of head direction; (3) the pathway via the nucleus reticularis pontis oralis, the supramammillary nucleus and the medial septum to the hippocampus, which transmits information supporting hippocampal theta rhythm and memory; and (4) a possible pathway via the cerebellum, and the ventral lateral nucleus of the thalamus (perhaps to the parietal cortex), which transmits information for spatial learning. Finally a new pathway is hypothesized via the basal ganglia, potentially involved in spatial learning and spatial memory. From these pathways, progressively emerges the anatomical network of vestibular cognition.

A robotic approach to understanding the role and the mechanism of vicarious trial-and-error in a T-maze task

Vicarious trial-and-error (VTE) is a behavior observed in rat experiments that seems to suggest self-conflict. This behavior is seen mainly when the rats are uncertain about making a decision. The presence of VTE is regarded as an indicator of a deliberative decision-making process, that is, searching, predicting, and evaluating outcomes. This process is slower than automated decision-making processes, such as reflex or habituation, but it allows for flexible and ongoing control of behavior. In this study, we propose for the first time a robotic model of VTE to see if VTE can emerge just from a body-environment interaction and to show the underlying mechanism responsible for the observation of VTE and the advantages provided by it. We tried several robots with different parameters, and we have found that they showed three different types of VTE: high numbers of VTE at the beginning of learning, decreasing numbers afterward (similar VTE pattern to experiments with rats), low during the whole learning period, and high numbers all the time. Therefore, we were able to reproduce the phenomenon of VTE in a model robot using only a simple dynamical neural network with Hebbian learning, which suggests that VTE is an emergent property of a plastic and embodied neural network. From a comparison of the three types of VTE, we demonstrated that 1) VTE is associated with chaotic activity of neurons in our model and 2) VTE-showing robots were robust to environmental perturbations. We suggest that the instability of neuronal activity found in VTE allows ongoing learning to rebuild its strategy continuously, which creates robust behavior. Based on these results, we suggest that VTE is caused by a similar mechanism in biology and leads to robust decision making in an analogous way.

Caudate nucleus-dependent navigational strategies are associated with increased use of addictive drugs

This study aimed to investigate the relationship between navigational strategies and the use of abused substances in a sample of healthy young adults. Navigational strategies were assessed with the 4-on-8 virtual maze (4/8VM), a task previously shown to dissociate between hippocampal-dependent spatial navigational strategies and caudate nucleus-dependent stimulus-response navigational strategies. Spatial strategies involve learning the spatial relationships between the landmarks in an environment, while response learning strategies involve learning a rigid set of stimulus-response type associations, e.g., see the tree, turn left. We have shown that spatial learners have increased gray matter and fMRI activity in the hippocampus compared with response learners, while response learners have increased gray matter and fMRI activity in the caudate nucleus. We were interested in the prevalence of use of substances of abuse in spatial and response learners because of the evidence that people who score high on traits such as novelty seeking, sensation seeking, reward seeking, and impulsivity, are more cue-responsive and more likely to use substances of abuse. Since response learners show increased activity and gray matter in the caudate nucleus of the striatum, which is a brain area involved in addiction, we hypothesized that response learners would have a greater use of abused substances than spatial learners. Fifty-five young adults were tested on the 4/8VM and completed a time-line follow-back assessment of drug and alcohol use. We found that response learners had smoked a significantly greater number of cigarettes in their lifetime than spatial learners, were more likely to have used cannabis, and had double the lifetime alcohol consumption. We discuss the possible relationship between substance abuse and response strategies as well as the implications for the hippocampus, risks of neurological and psychiatric disorders, and healthy cognition. © 2013 The Authors. Hippocampus Published by Wiley Periodicals, Inc.

Experimental predictions drawn from a computational model of sign-trackers and goal-trackers

Gaining a better understanding of the biological mechanisms underlying the individual variation observed in response to rewards and reward cues could help to identify and treat individuals more prone to disorders of impulsive control, such as addiction. Variation in response to reward cues is captured in rats undergoing autoshaping experiments where the appearance of a lever precedes food delivery. Although no response is required for food to be delivered, some rats (goal-trackers) learn to approach and avidly engage the magazine until food delivery, whereas other rats (sign-trackers) come to approach and engage avidly the lever. The impulsive and often maladaptive characteristics of the latter response are reminiscent of addictive behaviour in humans. In a previous article, we developed a computational model accounting for a set of experimental data regarding sign-trackers and goal-trackers. Here we show new simulations of the model to draw experimental predictions that could help further validate or refute the model. In particular, we apply the model to new experimental protocols such as injecting flupentixol locally into the core of the nucleus accumbens rather than systemically, and lesioning of the core of the nucleus accumbens before or after conditioning. In addition, we discuss the possibility of removing the food magazine during the inter-trial interval. The predictions from this revised model will help us better understand the role of different brain regions in the behaviours expressed by sign-trackers and goal-trackers.

Multiple memory systems as substrates for multiple decision systems

It has recently become widely appreciated that value-based decision making is supported by multiple computational strategies. In particular, animal and human behavior in learning tasks appears to include habitual responses described by prominent model-free reinforcement learning (RL) theories, but also more deliberative or goal-directed actions that can be characterized by a different class of theories, model-based RL. The latter theories evaluate actions by using a representation of the contingencies of the task (as with a learned map of a spatial maze), called an "internal model." Given the evidence of behavioral and neural dissociations between these approaches, they are often characterized as dissociable learning systems, though they likely interact and share common mechanisms. In many respects, this division parallels a longstanding dissociation in cognitive neuroscience between multiple memory systems, describing, at the broadest level, separate systems for declarative and procedural learning. Procedural learning has notable parallels with model-free RL: both involve learning of habits and both are known to depend on parts of the striatum. Declarative memory, by contrast, supports memory for single events or episodes and depends on the hippocampus. The hippocampus is thought to support declarative memory by encoding temporal and spatial relations among stimuli and thus is often referred to as a relational memory system. Such relational encoding is likely to play an important role in learning an internal model, the representation that is central to model-based RL. Thus, insofar as the memory systems represent more general-purpose cognitive mechanisms that might subserve performance on many sorts of tasks including decision making, these parallels raise the question whether the multiple decision systems are served by multiple memory systems, such that one dissociation is grounded in the other. Here we investigated the relationship between model-based RL and relational memory by comparing individual differences across behavioral tasks designed to measure either capacity. Human subjects performed two tasks, a learning and generalization task (acquired equivalence) which involves relational encoding and depends on the hippocampus; and a sequential RL task that could be solved by either a model-based or model-free strategy. We assessed the correlation between subjects' use of flexible, relational memory, as measured by generalization in the acquired equivalence task, and their differential reliance on either RL strategy in the decision task. We observed a significant positive relationship between generalization and model-based, but not model-free, choice strategies. These results are consistent with the hypothesis that model-based RL, like acquired equivalence, relies on a more general-purpose relational memory system.

Complex neural codes in rat prelimbic cortex are stable across days on a spatial decision task

The rodent prelimbic cortex has been shown to play an important role in cognitive processing, and has been implicated in encoding many different parameters relevant to solving decision-making tasks. However, it is not known how the prelimbic cortex represents all these disparate variables, and if they are simultaneously represented when the task requires it. In order to investigate this question, we trained rats to run the Multiple-T Left Right Alternate (MT-LRA) task and recorded multi-unit ensembles from their prelimbic regions. Significant populations of cells in the prelimbic cortex represented the strategy controlling reward receipt on a given lap, whether the animal chose to go right or left on a given lap, and whether the animal made a correct decision or an error on a given lap. These populations overlapped in the cells recorded, with several cells demonstrating differential firing to all three variables. The spatial and strategic firing patterns of individual prelimbic cells were highly conserved across several days of running this task, indicating that each cell encoded the same information across days.

Impact of appetitive and aversive outcomes on brain responses: linking the animal and human literatures

Decision-making is motivated by the possibility of obtaining reward and/or avoiding punishment. Though many have investigated behavior associated with appetitive or aversive outcomes, few have examined behaviors that rely on both. Fewer still have addressed questions related to how anticipated appetitive and aversive outcomes interact to alter neural signals related to expected value, motivation, and salience. Here we review recent rodent, monkey, and human research that address these issues. Further development of this area will be fundamental to understanding the etiology behind human psychiatric diseases and cultivating more effective treatments.

Noradrenergic α2A-receptor stimulation in the ventral hippocampus reduces impulsive decision-making

RATIONALE

Guanfacine, an α2A-adrenergic receptor agonist, is currently in use for treatment of a variety of psychiatric disorders that are associated with impulsive decision-making (e.g., attention-deficit hyperactivity disorder; ADHD). In animals and humans, the behavioral effects of adrenergic agents are presumed to involve neuromodulation of the prefrontal cortex, consistent with the demonstrated actions of dopaminergic agents. However, recent experimental work has shown that the ventral hippocampus (vHC) contributes to decision-making and impulse control, raising the possibility that the hippocampus may be an important site of action for these drugs.

OBJECTIVE

The purpose of this study was to examine the effect of local vHC infusions of guanfacine and other neuropharmacological agents on behavioral decisions that involve a trade-off between reward size and delay.

METHODS

Different cohorts of rats were implanted with bilateral guide cannulae targeting the vHC. We examined the animals' behavior in a touchscreen version of a delay discounting task following intra-vHC infusions of: (a) guanfacine (α2A-adrenergic receptor agonist), (b) SCH 23390 (dopamine D1 receptor antagonist), and (c) muscimol/baclofen (GABAA/B agonists).

RESULTS

Guanfacine led to a dose-dependent reduction in impulsive decision-making, increasing the animals' tolerance for delay in exchange for a larger reward. By contrast, infusion of SCH 23390 had no behavioral effects. Consistent with previous lesion studies, reversible pharmacological inactivation with muscimol/baclofen increased impulsive decision-making.

CONCLUSIONS

These data provide the first evidence that guanfacine, a commonly used treatment for ADHD, may derive its clinical benefits through hippocampal stimulation, via α2A-adrenergic receptors.

Multiplexing signals in reinforcement learning with internal models and dopamine

A fundamental challenge for computational and cognitive neuroscience is to understand how reward-based learning and decision-making are made and how accrued knowledge and internal models of the environment are incorporated. Remarkable progress has been made in the field, guided by the midbrain dopamine reward prediction error hypothesis and the underlying reinforcement learning framework, which does not involve internal models ('model-free'). Recent studies, however, have begun not only to address more complex decision-making processes that are integrated with model-free decision-making, but also to include internal models about environmental reward structures and the minds of other agents, including model-based reinforcement learning and using generalized prediction errors. Even dopamine, a classic model-free signal, may work as multiplexed signals using model-based information and contribute to representational learning of reward structure.

Invigoration of reward seeking by cue and proximity encoding in the nucleus accumbens

A key function of the nucleus accumbens is to promote vigorous reward seeking, but the corresponding neural mechanism has not been identified despite many years of research. Here, we study cued flexible approach behavior, a form of reward seeking that strongly depends on the accumbens, and we describe a robust, single-cell neural correlate of behavioral vigor in the excitatory response of accumbens neurons to reward-predictive cues. Well before locomotion begins, this cue-evoked excitation predicts both the movement initiation latency and the speed of subsequent flexible approach responses, but not those of stereotyped, inflexible responses. Moreover, the excitation simultaneously signals the subject's proximity to the approach target, a signal that appears to mediate greater response vigor on trials that begin with the subject closer to the target. These results demonstrate a neural mechanism for response invigoration whereby accumbens neuronal encoding of reward availability and target proximity together drive the onset and speed of reward-seeking locomotion.

Separate populations of neurons in ventral striatum encode value and motivation

Neurons in the ventral striatum (VS) fire to cues that predict differently valued rewards. It is unclear whether this activity represents the value associated with the expected reward or the level of motivation induced by reward anticipation. To distinguish between the two, we trained rats on a task in which we varied value independently from motivation by manipulating the size of the reward expected on correct trials and the threat of punishment expected upon errors. We found that separate populations of neurons in VS encode expected value and motivation.

Vision, reanimated and reimagined

The publication in 1982 of David Marr's Vision has delivered a singular boost and a course correction to the science of vision. Thirty years later, cognitive science is being transformed by the new ways of thinking about what it is that the brain computes, how it does that, and, most importantly, why cognition requires these computations and not others. This ongoing process still owes much of its impetus and direction to the sound methodology, engaging style, and unique voice of Marr's Vision.

Hippocampal place-cell sequences depict future paths to remembered goals

Effective navigation requires planning extended routes to remembered goal locations. Hippocampal place cells have been proposed to have a role in navigational planning, but direct evidence has been lacking. Here we show that before goal-directed navigation in an open arena, the rat hippocampus generates brief sequences encoding spatial trajectories strongly biased to progress from the subject's current location to a known goal location. These sequences predict immediate future behaviour, even in cases in which the specific combination of start and goal locations is novel. These results indicate that hippocampal sequence events characterized previously in linearly constrained environments as 'replay' are also capable of supporting a goal-directed, trajectory-finding mechanism, which identifies important places and relevant behavioural paths, at specific times when memory retrieval is required, and in a manner that could be used to control subsequent navigational behaviour.

Conflict between place and response navigation strategies: effects on vicarious trial and error (VTE) behaviors

Navigation can be accomplished through multiple decision-making strategies, using different information-processing computations. A well-studied dichotomy in these decision-making strategies compares hippocampal-dependent "place" and dorsal-lateral striatal-dependent "response" strategies. A place strategy depends on the ability to flexibly respond to environmental cues, while a response strategy depends on the ability to quickly recognize and react to situations with well-learned action-outcome relationships. When rats reach decision points, they sometimes pause and orient toward the potential routes of travel, a process termed vicarious trial and error (VTE). VTE co-occurs with neurophysiological information processing, including sweeps of representation ahead of the animal in the hippocampus and transient representations of reward in the ventral striatum and orbitofrontal cortex. To examine the relationship between VTE and the place/response strategy dichotomy, we analyzed data in which rats were cued to switch between place and response strategies on a plus maze. The configuration of the maze allowed for place and response strategies to work competitively or cooperatively. Animals showed increased VTE on trials entailing competition between navigational systems, linking VTE with deliberative decision-making. Even in a well-learned task, VTE was preferentially exhibited when a spatial selection was required, further linking VTE behavior with decision-making associated with hippocampal processing.

Generalization of value in reinforcement learning by humans

Research in decision-making has focused on the role of dopamine and its striatal targets in guiding choices via learned stimulus-reward or stimulus-response associations, behavior that is well described by reinforcement learning theories. However, basic reinforcement learning is relatively limited in scope and does not explain how learning about stimulus regularities or relations may guide decision-making. A candidate mechanism for this type of learning comes from the domain of memory, which has highlighted a role for the hippocampus in learning of stimulus-stimulus relations, typically dissociated from the role of the striatum in stimulus-response learning. Here, we used functional magnetic resonance imaging and computational model-based analyses to examine the joint contributions of these mechanisms to reinforcement learning. Humans performed a reinforcement learning task with added relational structure, modeled after tasks used to isolate hippocampal contributions to memory. On each trial participants chose one of four options, but the reward probabilities for pairs of options were correlated across trials. This (uninstructed) relationship between pairs of options potentially enabled an observer to learn about option values based on experience with the other options and to generalize across them. We observed blood oxygen level-dependent (BOLD) activity related to learning in the striatum and also in the hippocampus. By comparing a basic reinforcement learning model to one augmented to allow feedback to generalize between correlated options, we tested whether choice behavior and BOLD activity were influenced by the opportunity to generalize across correlated options. Although such generalization goes beyond standard computational accounts of reinforcement learning and striatal BOLD, both choices and striatal BOLD activity were better explained by the augmented model. Consistent with the hypothesized role for the hippocampus in this generalization, functional connectivity between the ventral striatum and hippocampus was modulated, across participants, by the ability of the augmented model to capture participants' choice. Our results thus point toward an interactive model in which striatal reinforcement learning systems may employ relational representations typically associated with the hippocampus.

Homologous involvement of striatum and prefrontal cortex in rodent and human water maze learning

The multiple memory systems hypothesis posits that dorsal striatum and hippocampus are central nodes in independent memory systems, supporting response-based and place-based learning, respectively. Although our understanding of the function of hippocampus within this framework is relatively well established, the contribution of dorsal striatum is less clear. This in part seems to be due to the heterogeneous nature of dorsal striatum, which receives extensive topographically organized projections from higher cortical areas. Here we quantified neural activity in the intact brain while mice and humans acquired analogous versions of the Morris water maze. We found that dorsomedial striatum and medial prefrontal cortex support the initial acquisition of what is typically considered a hippocampus-dependent spatial learning task. We suggest that the circuit involving dorsomedial striatum and medial prefrontal cortex identified here plays a more task-independent role in early learning than currently thought. Furthermore, our results demonstrate that dorsomedial and dorsolateral striatum serve fundamentally different roles during place learning. The remarkably high degree of anatomical overlap in brain function between mouse and human observed in our study emphasizes the extent of convergence achievable with a well-matched multilevel approach.

A large-scale model of the functioning brain

A central challenge for cognitive and systems neuroscience is to relate the incredibly complex behavior of animals to the equally complex activity of their brains. Recently described, large-scale neural models have not bridged this gap between neural activity and biological function. In this work, we present a 2.5-million-neuron model of the brain (called "Spaun") that bridges this gap by exhibiting many different behaviors. The model is presented only with visual image sequences, and it draws all of its responses with a physically modeled arm. Although simplified, the model captures many aspects of neuroanatomy, neurophysiology, and psychological behavior, which we demonstrate via eight diverse tasks.

Integrating cortico-limbic-basal ganglia architectures for learning model-based and model-free navigation strategies

Behavior in spatial navigation is often organized into map-based (place-driven) vs. map-free (cue-driven) strategies; behavior in operant conditioning research is often organized into goal-directed vs. habitual strategies. Here we attempt to unify the two. We review one powerful theory for distinct forms of learning during instrumental conditioning, namely model-based (maintaining a representation of the world) and model-free (reacting to immediate stimuli) learning algorithms. We extend these lines of argument to propose an alternative taxonomy for spatial navigation, showing how various previously identified strategies can be distinguished as “model-based” or “model-free” depending on the usage of information and not on the type of information (e.g., cue vs. place). We argue that identifying “model-free” learning with dorsolateral striatum and “model-based” learning with dorsomedial striatum could reconcile numerous conflicting results in the spatial navigation literature. From this perspective, we further propose that the ventral striatum plays key roles in the model-building process. We propose that the core of the ventral striatum is positioned to learn the probability of action selection for every transition between states of the world. We further review suggestions that the ventral striatal core and shell are positioned to act as “critics” contributing to the computation of a reward prediction error for model-free and model-based systems, respectively.

The road not taken: neural correlates of decision making in orbitofrontal cortex

Empirical research links human orbitofrontal cortex (OFC) to the evaluation of outcomes during decision making and the representation of alternative (better) outcomes after failures. When faced with a difficult decision, rats sometimes pause and turn back-and-forth toward goals, until finally orienting toward the chosen direction. Neural representations of reward in rodent OFC increased immediately following each reorientation, implying a transient representation of the expected outcome following self-initiated decisions. Upon reaching reward locations and finding no reward (having made an error), OFC representations of reward decreased locally indicating a disappointment signal that then switched to represent the unrewarded, non-local, would-have-been rewarded site. These results illustrate that following a decision to act, neural ensembles in OFC represent reward, and upon the realization of an error, represent the reward that could have been.

Response inhibition signals and miscoding of direction in dorsomedial striatum

The ability to inhibit action is critical for everyday behavior and is affected by a variety of disorders. Behavioral control and response inhibition is thought to depend on a neural circuit that includes the dorsal striatum, yet the neural signals that lead to response inhibition and its failure are unclear. To address this issue, we recorded from neurons in rat dorsomedial striatum (mDS) in a novel task in which rats responded to a spatial cue that signaled that reward would be delivered either to the left or to the right. On 80% of trials rats were instructed to respond in the direction cued by the light (GO). On 20% of trials a second light illuminated instructing the rat to refrain from making the cued movement and move in the opposite direction (STOP). Many neurons in mDS encoded direction, firing more or less strongly for GO movements made ipsilateral or contralateral to the recording electrode. Neurons that fired more strongly for contralateral GO responses were more active when rats were faster, showed reduced activity on STOP trials, and miscoded direction on errors, suggesting that when these neurons were overly active, response inhibition failed. Neurons that decreased firing for contralateral movement were excited during trials in which the rat was required to stop the ipsilateral movement. For these neurons activity was reduced when errors were made and was negatively correlated with movement time suggesting that when these neurons were less active on STOP trials, response inhibition failed. Finally, the activity of a significant number of neurons represented a global inhibitory signal, firing more strongly during response inhibition regardless of response direction. Breakdown by cell type suggests that putative medium spiny neurons (MSNs) tended to fire more strongly under STOP trials, whereas putative interneurons exhibited both activity patterns.

The ubiquity of model-based reinforcement learning

The reward prediction error (RPE) theory of dopamine (DA) function has enjoyed great success in the neuroscience of learning and decision-making. This theory is derived from model-free reinforcement learning (RL), in which choices are made simply on the basis of previously realized rewards. Recently, attention has turned to correlates of more flexible, albeit computationally complex, model-based methods in the brain. These methods are distinguished from model-free learning by their evaluation of candidate actions using expected future outcomes according to a world model. Puzzlingly, signatures from these computations seem to be pervasive in the very same regions previously thought to support model-free learning. Here, we review recent behavioral and neural evidence about these two systems, in attempt to reconcile their enigmatic cohabitation in the brain.

Interactions between deliberation and delay-discounting in rats

When faced with decisions, rats sometimes pause and look back and forth between possible alternatives, a phenomenon termed vicarious trial and error (VTE). When it was first observed in the 1930s, VTE was theorized to be a mechanism for exploration. Later theories suggested that VTE aided the resolution of sensory or neuroeconomic conflict. In contrast, recent neurophysiological data suggest that VTE reflects a dynamic search and evaluation process. These theories make unique predictions about the timing of VTE on behavioral tasks. We tested these theories of VTE on a T-maze with return rails, where rats were given a choice between a smaller reward available after one delay or a larger reward available after an adjustable delay. Rats showed three clear phases of behavior on this task: investigation, characterized by discovery of task parameters; titration, characterized by iterative adjustment of the delay to a preferred interval; and exploitation, characterized by alternation to hold the delay at the preferred interval. We found that VTE events occurred during adjustment laps more often than during alternation laps. Results were incompatible with theories of VTE as an exploratory behavior, as reflecting sensory conflict, or as a simple neuroeconomic valuation process. Instead, our results were most consistent with VTE as reflecting a search process during deliberative decision making. This pattern of VTE that we observed is reminiscent of current navigational theories proposing a transition from a deliberative to a habitual decision-making mechanism.

Information processing in decision-making systems

Decisions result from an interaction between multiple functional systems acting in parallel to process information in very different ways, each with strengths and weaknesses. In this review, the authors address three action-selection components of decision-making: The Pavlovian system releases an action from a limited repertoire of potential actions, such as approaching learned stimuli. Like the Pavlovian system, the habit system is computationally fast but, unlike the Pavlovian system permits arbitrary stimulus-action pairings. These associations are a "forward'' mechanism; when a situation is recognized, the action is released. In contrast, the deliberative system is flexible but takes time to process. The deliberative system uses knowledge of the causal structure of the world to search into the future, planning actions to maximize expected rewards. Deliberation depends on the ability to imagine future possibilities, including novel situations, and it allows decisions to be taken without having previously experienced the options. Various anatomical structures have been identified that carry out the information processing of each of these systems: hippocampus constitutes a map of the world that can be used for searching/imagining the future; dorsal striatal neurons represent situation-action associations; and ventral striatum maintains value representations for all three systems. Each system presents vulnerabilities to pathologies that can manifest as psychiatric disorders. Understanding these systems and their relation to neuroanatomy opens up a deeper way to treat the structural problems underlying various disorders.

Reward-related learning via multiple memory systems

The application of a neuroeconomic approach to the study of reward-related processes has provided significant insights in our understanding of human learning and decision making. Much of this research has focused primarily on the contributions of the corticostriatal circuitry, involved in trial-and-error reward learning. As a result, less consideration has been allotted to the potential influence of different neural mechanisms such as the hippocampus or to more common ways in human society in which information is acquired and utilized to reach a decision, such as through explicit instruction rather than trial-and-error learning. This review examines the individual contributions of multiple learning and memory neural systems and their interactions during human decision making in both normal and neuropsychiatric populations. Specifically, the anatomical and functional connectivity across multiple memory systems are highlighted to suggest that probing the role of the hippocampus and its interactions with the corticostriatal circuitry via the application of model-based neuroeconomic approaches may provide novel insights into neuropsychiatric populations that suffer from damage to one of these structures and as a consequence have deficits in learning, memory, or decision making.

Negative symptoms and the failure to represent the expected reward value of actions: behavioral and computational modeling evidence

CONTEXT

Negative symptoms are a core feature of schizophrenia, but their pathogenesis remains unclear. Negative symptoms are defined by the absence of normal function. However, there must be a productive mechanism that leads to this absence.

OBJECTIVE

To test a reinforcement learning account suggesting that negative symptoms result from a failure in the representation of the expected value of rewards coupled with preserved loss-avoidance learning.

DESIGN

Participants performed a probabilistic reinforcement learning paradigm involving stimulus pairs in which choices resulted in reward or in loss avoidance. Following training, participants indicated their valuation of the stimuli in a transfer test phase. Computational modeling was used to distinguish between alternative accounts of the data.

SETTING

A tertiary care research outpatient clinic.

PATIENTS

In total, 47 clinically stable patients with a diagnosis of schizophrenia or schizoaffective disorder and 28 healthy volunteers participated in the study. Patients were divided into a high-negative symptom group and a low-negative symptom group.

MAIN OUTCOME MEASURES

The number of choices leading to reward or loss avoidance, as well as performance in the transfer test phase. Quantitative fits from 3 different models were examined.

RESULTS

Patients in the high-negative symptom group demonstrated impaired learning from rewards but intact loss-avoidance learning and failed to distinguish rewarding stimuli from loss-avoiding stimuli in the transfer test phase. Model fits revealed that patients in the high-negative symptom group were better characterized by an "actor-critic" model, learning stimulus-response associations, whereas control subjects and patients in the low-negative symptom group incorporated expected value of their actions ("Q learning") into the selection process.

CONCLUSIONS

Negative symptoms in schizophrenia are associated with a specific reinforcement learning abnormality: patients with high-negative symptoms do not represent the expected value of rewards when making decisions but learn to avoid punishments through the use of prediction errors. This computational framework offers the potential to understand negative symptoms at a mechanistic level.

Acquisition of "Start" and "Stop" response thresholds in peak-interval timing is differentially sensitive to protein synthesis inhibition in the dorsal and ventral striatum

Time-based decision-making in peak-interval timing procedures involves the setting of response thresholds for the initiation (“Start”) and termination (“Stop”) of a response sequence that is centered on a target duration. Using intracerebral infusions of the protein synthesis inhibitor anisomycin, we report that the acquisition of the “Start” response depends on normal functioning (including protein synthesis) in the dorsal striatum (DS), but not the ventral striatum (VS). Conversely, disruption of the VS, but not the DS, impairs the acquisition of the “Stop” response. We hypothesize that the dorsal and ventral regions of the striatum function as a competitive neural network that encodes the temporal boundaries marking the beginning and end of a timed response sequence.

Scarce means with alternative uses: robbins' definition of economics and its extension to the behavioral and neurobiological study of animal decision making

Almost 80 years ago, Lionel Robbins proposed a highly influential definition of the subject matter of economics: the allocation of scarce means that have alternative ends. Robbins confined his definition to human behavior, and he strove to separate economics from the natural sciences in general and from psychology in particular. Nonetheless, I extend his definition to the behavior of non-human animals, rooting my account in psychological processes and their neural underpinnings. Some historical developments are reviewed that render such a view more plausible today than would have been the case in Robbins’ time. To illustrate a neuroeconomic perspective on decision making in non-human animals, I discuss research on the rewarding effect of electrical brain stimulation. Central to this discussion is an empirically based, functional/computational model of how the subjective intensity of the electrical reward is computed and combined with subjective costs so as to determine the allocation of time to the pursuit of reward. Some successes achieved by applying the model are discussed, along with limitations, and evidence is presented regarding the roles played by several different neural populations in processes posited by the model. I present a rationale for marshaling convergent experimental methods to ground psychological and computational processes in the activity of identified neural populations, and I discuss the strengths, weaknesses, and complementarity of the individual approaches. I then sketch some recent developments that hold great promise for advancing our understanding of structure–function relationships in neuroscience in general and in the neuroeconomic study of decision making in particular.

Ventral striatum encodes past and predicted value independent of motor contingencies

The ventral striatum (VS) is thought to signal the predicted value of expected outcomes. However, it is still unclear whether VS can encode value independently from variables often yoked to value such as response direction and latency. Expectations of high value reward are often associated with a particular action and faster latencies. To address this issue we trained rats to perform a task in which the size of the predicted reward was signaled before the instrumental response was instructed. Instrumental directional cues were presented briefly at a variable onset to reduce accuracy and increase reaction time. Rats were more accurate and slower when a large versus small reward was at stake. We found that activity in VS was high during odors that predicted large reward even though reaction times were slower under these conditions. In addition to these effects, we found that activity before the reward predicting cue reflected past and predicted reward. These results demonstrate that VS can encode value independent of motor contingencies and that the role of VS in goal-directed behavior is not just to increase vigor of specific actions when more is at stake.

Striatum processes reward differently in adolescents versus adults

Adolescents often respond differently than adults to the same salient motivating contexts, such as peer interactions and pleasurable stimuli. Delineating the neural processing differences of adolescents is critical to understanding this phenomenon, as well as the bases of serious behavioral and psychiatric vulnerabilities, such as drug abuse, mood disorders, and schizophrenia. We believe that age-related changes in the ways salient stimuli are processed in key brain regions could underlie the unique predilections and vulnerabilities of adolescence. Because motivated behavior is the central issue, it is critical that age-related comparisons of brain activity be undertaken during motivational contexts. We compared single-unit activity and local field potentials in the nucleus accumbens (NAc) and dorsal striatum (DS) of adolescent and adult rats during a reward-motivated instrumental task. These regions are involved in motivated learning, reward processing, and action selection. We report adolescent neural processing differences in the DS, a region generally associated more with learning than reward processing in adults. Specifically, adolescents, but not adults, had a large proportion of neurons in the DS that activated in anticipation of reward. More similar response patterns were observed in NAc of the two age groups. DS single-unit activity differences were found despite similar local field potential oscillations. This study demonstrates that in adolescents, a region critically involved in learning and habit formation is highly responsive to reward. It thus suggests a mechanism for how rewards might shape adolescent behavior differently, and for their increased vulnerabilities to affective disorders.

Sequence learning and the role of the hippocampus in rodent navigation

The hippocampus has long been associated with navigation and spatial representations, but it has been difficult to link directly the neurophysiological correlates of hippocampal place cells with navigational planning and action. In recent years, large-scale population recordings of place cells have revealed that spatial sequences are stored and activated in ways that may support navigational strategies. Plasticity mechanisms allow the hippocampus to store learned sequences of locations that may allow predictions of future locations based on past experience. These sequences can also be activated during navigational behavior in ways that may allow the animal to learn trajectories toward goals. Task-dependent alterations in place cell firing patterns may reflect the operation of the hippocampus in associating locations with navigationally relevant decision variables.

Association between CHRNA5 genetic variation at rs16969968 and brain reactivity to smoking images in nicotine dependent women

BACKGROUND

Tobacco smoking is the leading preventable cause of death in the developed world. Identifying risk factors for smoking may lead to more effective treatments. Genome wide association studies revealed a relationship between development of nicotine dependence and a single-nucleotide polymorphism (SNP, rs16969968) of the nicotine acetylcholine receptor (nAChR) alpha-5 subunit gene (CHRNA5). The relationship between this SNP and other factors contributing to smoking behavior such as smoking cue reactivity is unclear.

METHODS

We assessed the role of rs16969968 on brain functional MRI (fMRI) reactivity to smoking cues by studying nicotine dependent women with the nicotine dependence 'risk' allele (A allele, N=14) and without the 'risk' allele (G/G smokers, N=10). Nicotine dependence severity, as assessed with the Fagerstrom test for nicotine dependence, smoking pack-years, and expired carbon monoxide levels, were equivalent in these groups.

RESULTS

We observed a group difference in fMRI reactivity; women without the A allele (G/G smokers) showed greater fMRI reactivity to smoking images in brain areas related to memory and habitual behavior such as the hippocampus and dorsal striatum.

CONCLUSIONS

Our finding suggests that nicotine-dependent smokers lacking the rs16969968 A allele are more likely to recall smoking-related memories and engage in habitual responding to smoking cues than A allele smokers. Although more studies are necessary to determine the mechanism underlying and significance of this cue reactivity difference, these data suggest that smokers may develop and remain nicotine dependent due to different factors including genetics and cue reactivity. This finding may have implications for personalizing smoking treatment.

Where is my reward and how do I get it? Interaction between the hippocampus and the basal ganglia during spatial learning

Spatial learning has been recognized over the years to be under the control of the hippocampus and related temporal lobe structures. Hippocampal damage often causes severe impairments in the ability to learn and remember a location in space defined by distal visual cues. Recent experimental evidence in rodents demonstrates, however, that other brain areas might also be involved in the acquisition of spatial information. Amongst these, the cortex--basal ganglia loop is known to be involved in reinforcement learning and has been identified as an important contributor to spatial learning. In particular, it has been shown that altered activity of the basal ganglia striatal complex can impair the ability to perform spatial learning tasks. Until recently, little was known about how the basal ganglia and the hippocampus interact and how their activities evolve during learning. The present review, focusing on rodent studies, provides a glimpse of the findings obtained over the past decade that support a dialog between these two structures during spatial learning. Based on these studies, we propose a new functional spatial decision network with three separate loops encompassing hippocampus and specific basal ganglia regions. Each of the three loops serves a different aspect of spatial decision making and all three are linked by their mutual connections and are under the control of the dopaminergic learning signal.

The hippocampus is functionally connected to the striatum and orbitofrontal cortex during context dependent decision making

Many of our everyday actions are only appropriate in certain situations and selecting the appropriate behavior requires that we use current context and previous experience to guide our decisions. The current study examined hippocampal functional connectivity with prefrontal and striatal regions during a task that required participants to make decisions based on the contextual retrieval of overlapping sequential representations. Participants learned four sequences comprised of six faces each. An overlapping condition was created by having two sequences with two identical faces as the middle images. A non-overlapping condition contained two sequences that did not share any faces between them. Hippocampal functional connectivity was assessed during the presentation period and at the critical choice, where participants had to make a contextually dependent decision. The left hippocampus showed significantly increased functional connectivity with dorsal and ventral striatum and anterior cingulate cortex during the presentation period of the overlapping compared to the non-overlapping condition after participants knew the sequences. At the critical choice point of the overlapping condition, the left hippocampus showed stronger functional connectivity with the orbitofrontal cortex. These functional connectivity results suggest that the hippocampus may play a role in decision making by predicting the possibilities of what might come next, allowing orbitofrontal and striatal regions to evaluate the expected choice options in order to make the correct action at the choice point.

Neural systems analysis of decision making during goal-directed navigation

The ability to make adaptive decisions during goal-directed navigation is a fundamental and highly evolved behavior that requires continual coordination of perceptions, learning and memory processes, and the planning of behaviors. Here, a neurobiological account for such coordination is provided by integrating current literatures on spatial context analysis and decision-making. This integration includes discussions of our current understanding of the role of the hippocampal system in experience-dependent navigation, how hippocampal information comes to impact midbrain and striatal decision making systems, and finally the role of the striatum in the implementation of behaviors based on recent decisions. These discussions extend across cellular to neural systems levels of analysis. Not only are key findings described, but also fundamental organizing principles within and across neural systems, as well as between neural systems functions and behavior, are emphasized. It is suggested that studying decision making during goal-directed navigation is a powerful model for studying interactive brain systems and their mediation of complex behaviors.

Multiplicity of control in the basal ganglia: computational roles of striatal subregions

The basal ganglia, in particular the striatum, are central to theories of behavioral control, and often identified as a seat of action selection. Reinforcement learning (RL) models--which have driven much recent experimental work on this region--cast striatum as a dynamic controller, integrating sensory and motivational information to construct efficient and enriching behavioral policies. Befitting this informationally central role, the BG sit at the nexus of multiple anatomical 'loops' of synaptic projections, connecting a wide range of cortical and subcortical structures. Numerous pioneering anatomical studies conducted over the past several decades have meticulously catalogued these loops, and labeled them according to the inferred functions of the connected regions. The specific cotermina of the projections are highly localized to several different subregions of the striatum, leading to the suggestion that these subregions perform complementary but distinct functions. However, until recently, the dominant computational framework outlined only a bipartite, dorsal/ventral, division of striatum. We review recent computational and experimental advances that argue for a more finely fractionated delineation. In particular, experimental data provide extensive insight into unique functions subserved by the dorsomedial striatum (DMS). These functions appear to correspond well with theories of a 'model-based' RL subunit, and may also shed light on the suborganization of ventral striatum. Finally, we discuss the limitations of these ideas and how they point the way toward future refinements of neurocomputational theories of striatal function, bringing them into contact with other areas of computational theory and other regions of the brain.

Changes in reward contingency modulate the trial-to-trial variability of hippocampal place cells

Pyramidal cells in the rodent hippocampus often exhibit clear spatial tuning. Theories of hippocampal function suggest that these "place cells" implement multiple, independent neural representations of position (maps), based on different reference frames or environmental features. Consistent with the "multiple maps" theory, previous studies have shown that manipulating spatial factors related to task performance modulates the within-session variability (overdispersion) of cells in the hippocampus. However, the influence of changes in reward contingency on overdispersion has not been examined. To test this, we first trained rats to collect food from three feeders positioned around a circular track (task(1)). When subjects were proficient, the reward contingency was altered such that every other feeder delivered food (task(2)). We recorded ensembles of hippocampal neurons as rats performed both tasks. Place cell overdispersion was high during task(1) but decreased significantly during task(2), and this increased reliability could not be accounted for by changes in running speed or familiarity with the task. Intuitively, decreased variability might be expected to improve neural representations of position. To test this, we used Bayesian decoding of hippocampal spike trains to estimate subjects' location. Neither the amount of probability decoded to subjects' position (local probability) nor the difference between estimated position and true location (decoding accuracy) differed between tasks. However, we found that hippocampal ensembles were significantly more self-consistent during task(2) performance. These results suggest that changes in task demands can affect the firing statistics of hippocampal neurons, leading to changes in the properties of decoded neural representations.

Attracting dynamics of frontal cortex ensembles during memory-guided decision-making

A common theoretical view is that attractor-like properties of neuronal dynamics underlie cognitive processing. However, although often proposed theoretically, direct experimental support for the convergence of neural activity to stable population patterns as a signature of attracting states has been sparse so far, especially in higher cortical areas. Combining state space reconstruction theorems and statistical learning techniques, we were able to resolve details of anterior cingulate cortex (ACC) multiple single-unit activity (MSUA) ensemble dynamics during a higher cognitive task which were not accessible previously. The approach worked by constructing high-dimensional state spaces from delays of the original single-unit firing rate variables and the interactions among them, which were then statistically analyzed using kernel methods. We observed cognitive-epoch-specific neural ensemble states in ACC which were stable across many trials (in the sense of being predictive) and depended on behavioral performance. More interestingly, attracting properties of these cognitively defined ensemble states became apparent in high-dimensional expansions of the MSUA spaces due to a proper unfolding of the neural activity flow, with properties common across different animals. These results therefore suggest that ACC networks may process different subcomponents of higher cognitive tasks by transiting among different attracting states.