Reinforcement learning: computing the temporal difference of values via distinct corticostriatal pathways

Enhancing reinforcement learning models by including direct and indirect pathways improves performance on striatal dependent tasks

A major advance in understanding learning behavior stems from experiments showing that reward learning requires dopamine inputs to striatal neurons and arises from synaptic plasticity of cortico-striatal synapses. Numerous reinforcement learning models mimic this dopamine-dependent synaptic plasticity by using the reward prediction error, which resembles dopamine neuron firing, to learn the best action in response to a set of cues. Though these models can explain many facets of behavior, reproducing some types of goal-directed behavior, such as renewal and reversal, require additional model components. Here we present a reinforcement learning model, TD2Q, which better corresponds to the basal ganglia with two Q matrices, one representing direct pathway neurons (G) and another representing indirect pathway neurons (N). Unlike previous two-Q architectures, a novel and critical aspect of TD2Q is to update the G and N matrices utilizing the temporal difference reward prediction error. A best action is selected for N and G using a softmax with a reward-dependent adaptive exploration parameter, and then differences are resolved using a second selection step applied to the two action probabilities. The model is tested on a range of multi-step tasks including extinction, renewal, discrimination; switching reward probability learning; and sequence learning. Simulations show that TD2Q produces behaviors similar to rodents in choice and sequence learning tasks, and that use of the temporal difference reward prediction error is required to learn multi-step tasks. Blocking the update rule on the N matrix blocks discrimination learning, as observed experimentally. Performance in the sequence learning task is dramatically improved with two matrices. These results suggest that including additional aspects of basal ganglia physiology can improve the performance of reinforcement learning models, better reproduce animal behaviors, and provide insight as to the role of direct- and indirect-pathway striatal neurons.

A computational model of learning flexible navigation in a maze by layout-conforming replay of place cells

Recent experimental observations have shown that the reactivation of hippocampal place cells (PC) during sleep or wakeful immobility depicts trajectories that can go around barriers and can flexibly adapt to a changing maze layout. However, existing computational models of replay fall short of generating such layout-conforming replay, restricting their usage to simple environments, like linear tracks or open fields. In this paper, we propose a computational model that generates layout-conforming replay and explains how such replay drives the learning of flexible navigation in a maze. First, we propose a Hebbian-like rule to learn the inter-PC synaptic strength during exploration. Then we use a continuous attractor network (CAN) with feedback inhibition to model the interaction among place cells and hippocampal interneurons. The activity bump of place cells drifts along paths in the maze, which models layout-conforming replay. During replay in sleep, the synaptic strengths from place cells to striatal medium spiny neurons (MSN) are learned by a novel dopamine-modulated three-factor rule to store place-reward associations. During goal-directed navigation, the CAN periodically generates replay trajectories from the animal's location for path planning, and the trajectory leading to a maximal MSN activity is followed by the animal. We have implemented our model into a high-fidelity virtual rat in the MuJoCo physics simulator. Extensive experiments have demonstrated that its superior flexibility during navigation in a maze is due to a continuous re-learning of inter-PC and PC-MSN synaptic strength.

Rigid reduced successor representation as a potential mechanism for addiction

Difficulty in cessation of drinking, smoking, or gambling has been widely recognized. Conventional theories proposed relative dominance of habitual over goal-directed control, but human studies have not convincingly supported them. Referring to the recently suggested "successor representation (SR)" of states that enables partially goal-directed control, we propose a dopamine-related mechanism that makes resistance to habitual reward-obtaining particularly difficult. We considered that long-standing behavior towards a certain reward without resisting temptation can (but not always) lead to a formation of rigid dimension-reduced SR based on the goal state, which cannot be updated. Then, in our model assuming such rigid reduced SR, whereas no reward prediction error (RPE) is generated at the goal while no resistance is made, a sustained large positive RPE is generated upon goal reaching once the person starts resisting temptation. Such sustained RPE is somewhat similar to the hypothesized sustained fictitious RPE caused by drug-induced dopamine. In contrast, if rigid reduced SR is not formed and states are represented individually as in simple reinforcement learning models, no sustained RPE is generated at the goal. Formation of rigid reduced SR also attenuates the resistance-dependent decrease in the value of the cue for behavior, makes subsequent introduction of punishment after the goal ineffective, and potentially enhances the propensity of nonresistance through the influence of RPEs via the spiral striatum-midbrain circuit. These results suggest that formation of rigid reduced SR makes cessation of habitual reward-obtaining particularly difficult and can thus be a mechanism for addiction, common to substance and nonsubstance reward.

Tonic dopamine, uncertainty and basal ganglia action selection

To make optimal decisions in uncertain circumstances flexible adaption of behaviour is required; exploring alternatives when the best choice is unknown, exploiting what is known when that is best. Using a computational model of the basal ganglia, we propose that switches between exploratory and exploitative decisions are mediated by the interaction between tonic dopamine and cortical input to the basal ganglia. We show that a biologically detailed action selection circuit model, endowed with dopamine dependant striatal plasticity, can optimally solve the explore-exploit problem, estimating the true underlying state of a noisy Gaussian diffusion process. Critical to the model's performance was a fluctuating level of tonic dopamine which increased under conditions of uncertainty. With an optimal range of tonic dopamine, explore-exploit decisions were mediated by the effects of tonic dopamine on the precision of the model action selection mechanism. Under conditions of uncertain reward pay-out, the model's reduced selectivity allowed disinhibition of multiple alternative actions to be explored at random. Conversely, when uncertainly about reward pay-out was low, enhanced selectivity of the action selection circuit facilitated exploitation of the high value choice. Model performance was at the level of a Kalman filter which provides an optimal solution for the task. These simulations support the idea that this subcortical neural circuit may have evolved to facilitate decision making in non-stationary reward environments. The model generates several experimental predictions with relevance to abnormal decision making in neuropsychiatric and neurological disease.

Modulation of Dopamine for Adaptive Learning: A Neurocomputational Model

There have been many proposals that learning rates in the brain are adaptive, in the sense that they increase or decrease depending on environmental conditions. The majority of these models are abstract and make no attempt to describe the neural circuitry that implements the proposed computations. This article describes a biologically detailed computational model that overcomes this shortcoming. Specifically, we propose a neural circuit that implements adaptive learning rates by modulating the gain on the dopamine response to reward prediction errors, and we model activity within this circuit at the level of spiking neurons. The model generates a dopamine signal that depends on the size of the tonically active dopamine neuron population and the phasic spike rate. The model was tested successfully against results from two single-neuron recording studies and a fast-scan cyclic voltammetry study. We conclude by discussing the general applicability of the model to dopamine mediated tasks that transcend the experimental phenomena it was initially designed to address.

Perseverative stereotypic behavior of Epac2 KO mice in a reward-based decision making task

Successfully navigating dynamic environments requires balancing the decision to stay at an optimal choice with that to switch to an alternative to acquire new knowledge. However, the genetic factors and cellular activity shaping this "stay or switch" action decision remains largely unidentified. Here we find that mice carrying a deletion of the exchange protein directly activated by cAMP 2 (Epac2) gene, a putative autism locus, exhibit perseverative "stay" behavior in a dynamic foraging task. Anatomical analysis found that the loss of Epac2 resulted in a significant decrease in the density of PV-expressing interneurons in the ventrolateral orbitofrontal cortex (OFC) and dorsal striatum (dSTR). Further, in vitro whole cell patch clamp recordings of PV⁺ GABAergic interneurons in the dSTR revealed altered neural activity in Epac2 KO mice in response to dopamine. Our findings highlight a potential role of Epac2 in structural changes and neural responses of PV-expressing GABAergic interneurons in the ventrolateral OFC and dSTR during value-based reinforcement learning and link Epac2 function to abnormal decision-making processes and perseverative behaviors seen in autism.

Towards Automaticity in Reinforcement Learning: A Model-Based Functional Magnetic Resonance Imaging Study

Introduction

Previous studies showed that over the course of learning many neurons in the medial prefrontal cortex adapt their firing rate towards the options with highest predicted value reward but it was showed that during later learning trials the brain switches to a more automatic processing mode governed by the basal ganglia. Based on this evidence, we hypothesized that during the early learning trials the predicted values of chosen options will be coded by a goal directed system in the medial frontal cortex but during the late trials the predicted values will be coded by the habitual learning system in the dorsal striatum.

Methods

In this study, using a 3 Tesla functional magnetic resonance imaging scanner (fMRI), blood oxygen level dependent signal (BOLD) data was collected whilst participants (N=12) performed a reinforcement learning task. The task consisted of instrumental conditioning trials wherein each trial a participant choose one of the two available options in order to win or avoid losing money. In addition to that, depending on the experimental condition, participants received either monetary reward (gain money), monetary penalty (lose money) or neural outcome.

Results

Using model-based analysis for functional magnetic resonance imaging (fMRI) event related designs; region of interest (ROI) analysis was performed to nucleus accumbens, medial frontal cortex, caudate nucleus, putamen and globus pallidus internal and external segments. In order to compare the difference in brain activity for early (goal directed) versus late learning (habitual, automatic) trials, separate ROI analyses were performed for each anatomical sub-region. For the reward condition, we found significant activity in the medial frontal cortex (p<0.05) only for early learning trials but activity is shifted to bilateral putamen (p<0.05) during later trials. However, for the loss condition no significant activity was found for early trials except globus pallidus internal segment showed a significant activity (p<0.05) for later trials.

Conclusion

We found that during reinforcement learning activation in the brain shifted from the medial frontal regions to dorsal regions of the striatum. These findings suggest that there are two separable (early goal directed and late habitual) learning systems in the brain.

Bipolar oscillations between positive and negative mood states in a computational model of Basal Ganglia

Bipolar disorder is characterized by mood swings-oscillations between manic and depressive states. The swings (oscillations) mark the length of an episode in a patient's mood cycle (period), and can vary from hours to years. The proposed modeling study uses decision making framework to investigate the role of basal ganglia network in generating bipolar oscillations. In this model, the basal ganglia system performs a two-arm bandit task in which one of the arms (action responses) leads to a positive outcome, while the other leads to a negative outcome. We explore the dynamics of key reward and risk related parameters in the system while the model agent receives various outcomes. Particularly, we study the system using a model that represents the fast dynamics of decision making, and a module to capture the slow dynamics that describe the variation of some meta-parameters of fast dynamics over long time scales. The model is cast at three levels of abstraction: (1) a two-dimensional dynamical system model, that is a simple two variable model capable of showing bistability for rewarding and punitive outcomes; (2) a phenomenological basal ganglia model, to extend the implications from the reduced model to a cortico-basal ganglia setup; (3) a detailed network model of basal ganglia, that incorporates detailed cellular level models for a more realistic understanding. In healthy conditions, the model chooses positive action and avoids negative one, whereas under bipolar conditions, the model exhibits slow oscillations in its choice of positive or negative outcomes, reminiscent of bipolar oscillations. Phase-plane analyses on the simple reduced dynamical system with two variables reveal the essential parameters that generate pathological 'bipolar-like' oscillations. Phenomenological and network models of the basal ganglia extend that logic, and interpret bipolar oscillations in terms of the activity of dopaminergic and serotonergic projections on the cortico-basal ganglia network dynamics. The network's dysfunction, specifically in terms of reward and risk sensitivity, is shown to be responsible for the pathological bipolar oscillations. The study proposes a computational model that explores the effects of impaired serotonergic neuromodulation on the dynamics of the cortico basal ganglia network, and relates this impairment to abstract mood states (manic and depressive episodes) and oscillations of bipolar disorder.

In Vivo Spiking Dynamics of Intra- and Extratelencephalic Projection Neurons in Rat Motor Cortex

In motor cortex, 2 types of deep layer pyramidal cells send their axons to other areas: intratelencephalic (IT)-type neurons specifically project bilaterally to the cerebral cortex and striatum, whereas neurons of the extratelencephalic (ET)-type, termed conventionally pyramidal tract-type, project ipsilaterally to the thalamus and other areas. Although they have totally different synaptic and membrane potential properties in vitro, little is known about the differences between them in ongoing spiking dynamics in vivo. We identified IT-type and ET-type neurons, as well as fast-spiking-type interneurons, using novel multineuronal analysis based on optogenetically evoked spike collision along their axons in behaving/resting rats expressing channelrhodopsin-2 (Multi-Linc method). We found "postspike suppression" (~100 ms) as a characteristic of ET-type neurons in spike auto-correlograms, and it remained constant independent of behavioral conditions in functionally different ET-type neurons. Postspike suppression followed even solitary spikes, and spike bursts significantly extended its duration. We also observed relatively strong spike synchrony in pairs containing IT-type neurons. Thus, spiking dynamics in IT-type and ET-type neurons may be optimized differently for precise and coordinated motor control.

Control of excitatory hierarchical circuits by parvalbumin-FS basket cells in layer 5 of the frontal cortex: insights for cortical oscillations

The cortex contains multiple neuron types with specific connectivity and functions. Recent progress has provided a better understanding of the interactions of these neuron types as well as their output organization particularly for the frontal cortex, with implications for the circuit mechanisms underlying cortical oscillations that have cognitive functions. Layer 5 pyramidal cells (PCs) in the frontal cortex comprise two major subtypes: crossed-corticostriatal (CCS) and corticopontine (CPn) cells. Functionally, CCS and CPn cells exhibit similar phase-dependent firing during gamma waves but participate in two distinct subnetworks that are linked unidirectionally from CCS to CPn cells. GABAergic parvalbumin-expressing fast-spiking (PV-FS) cells, necessary for gamma oscillation, innervate PCs, with stronger and global inhibition to somata and weaker and localized inhibitions to dendritic shafts/spines. While PV-FS cells form reciprocal connections with both CCS and CPn cells, the excitation from CPn to PV-FS cells exhibits short-term synaptic dynamics conducive for oscillation induction. The electrical coupling between PV-FS cells facilitates spike synchronization among PV-FS cells receiving common excitatory inputs from local PCs and inhibits other PV-FS cells via electrically communicated spike afterhyperpolarizations. These connectivity characteristics can promote synchronous firing in the local networks of CPn cells and firing of some CCS cells by anode-break excitation. Thus subsets of L5 CCS and CPn cells within different levels of connection hierarchy exhibit coordinated activity via their common connections with PV-FS cells, and the resulting PC output drives diverse neuronal targets in cortical layer 1 and the striatum with specific temporal precision, expanding the computational power of the cortical network.

Minimal Circuit Model of Reward Prediction Error Computations and Effects of Nicotinic Modulations

Dopamine (DA) neurons in the ventral tegmental area (VTA) are thought to encode reward prediction errors (RPE) by comparing actual and expected rewards. In recent years, much work has been done to identify how the brain uses and computes this signal. While several lines of evidence suggest the interplay of the DA and the inhibitory interneurons in the VTA implements the RPE computation, it still remains unclear how the DA neurons learn key quantities, for example the amplitude and the timing of primary rewards during conditioning tasks. Furthermore, endogenous acetylcholine and exogenous nicotine, also likely affect these computations by acting on both VTA DA and GABA (γ -aminobutyric acid) neurons via nicotinic-acetylcholine receptors (nAChRs). To explore the potential circuit-level mechanisms for RPE computations during classical-conditioning tasks, we developed a minimal computational model of the VTA circuitry. The model was designed to account for several reward-related properties of VTA afferents and recent findings on VTA GABA neuron dynamics during conditioning. With our minimal model, we showed that the RPE can be learned by a two-speed process computing reward timing and magnitude. By including models of nAChR-mediated currents in the VTA DA-GABA circuit, we showed that nicotine should reduce the acetylcholine action on the VTA GABA neurons by receptor desensitization and potentially boost DA responses to reward-related signals in a non-trivial manner. Together, our results delineate the mechanisms by which RPE are computed in the brain, and suggest a hypothesis on nicotine-mediated effects on reward-related perception and decision-making.

A Dual Role Hypothesis of the Cortico-Basal-Ganglia Pathways: Opponency and Temporal Difference Through Dopamine and Adenosine

The hypothesis that the basal-ganglia direct and indirect pathways represent goodness (or benefit) and badness (or cost) of options, respectively, explains a wide range of phenomena. However, this hypothesis, named the Opponent Actor Learning (OpAL), still has limitations. Structurally, the OpAL model does not incorporate differentiation of the two types of cortical inputs to the basal-ganglia pathways received from intratelencephalic (IT) and pyramidal-tract (PT) neurons. Functionally, the OpAL model does not describe the temporal-difference (TD)-type reward-prediction-error (RPE), nor explains how RPE is calculated in the circuitry connecting to the DA neurons. In fact, there is a different hypothesis on the basal-ganglia pathways and DA, named the Cortico-Striatal-Temporal-Difference (CS-TD) model. The CS-TD model differentiates the IT and PT inputs, describes the TD-type RPE, and explains how TD-RPE is calculated. However, a critical difficulty in this model lies in its assumption that DA induces the same direction of plasticity in both direct and indirect pathways, which apparently contradicts the experimentally observed opposite effects of DA on these pathways. Here, we propose a new hypothesis that integrates the OpAL and CS-TD models. Specifically, we propose that the IT-basal-ganglia pathways represent goodness/badness of current options while the PT-indirect pathway represents the overall value of the previously chosen option, and both of these have influence on the DA neurons, through the basal-ganglia output, so that a variant of TD-RPE is calculated. A key assumption is that opposite directions of plasticity are induced upon phasic activation of DA neurons in the IT-indirect pathway and PT-indirect pathway because of different profiles of IT and PT inputs. Specifically, at PT→indirect-pathway-medium-spiny-neuron (iMSN) synapses, sustained glutamatergic inputs generate rich adenosine, which allosterically prevents DA-D2 receptor signaling and instead favors adenosine-A2A receptor signaling. Then, phasic DA-induced phasic adenosine, which reflects TD-RPE, causes long-term synaptic potentiation. In contrast, at IT→iMSN synapses where adenosine is scarce, phasic DA causes long-term synaptic depression via D2 receptor signaling. This new Opponency and Temporal-Difference (OTD) model provides unique predictions, part of which is potentially in line with recently reported activity patterns of neurons in the globus pallidus externus on the indirect pathway.

Pyramidal Cell Subtypes and Their Synaptic Connections in Layer 5 of Rat Frontal Cortex

The frontal cortical areas make a coordinated response that generates appropriate behavior commands, using individual local circuits with corticostriatal and corticocortical connections in longer time scales than sensory areas. In secondary motor cortex (M2), situated between the prefrontal and primary motor areas, major subtypes of layer 5 corticostriatal cells are crossed-corticostriatal (CCS) cells innervating both sides of striatum, and corticopontine (CPn) cells projecting to the ipsilateral striatum and pontine nuclei. CCS cells innervate CPn cells unidirectionally: the former are therefore hierarchically higher than the latter among L5 corticostriatal cells. CCS cells project directly to both frontal and nonfrontal areas. On the other hand, CPn cells innervate the thalamus and layer 1a of frontal areas, where thalamic fibers relaying basal ganglia outputs are distributed. Thus, CCS cells can make activities of frontal areas in concert with those of nonfrontal area using corticocortical loops, whereas CPn cells are more involved in closed corticostriatal loops than CCS cells. Since reciprocal connections between CPn cells with facilitatory synapses may be related to persistent activity, CPn cells play a key role of longer time constant processes in corticostriatal as well as in corticocortical loops between the frontal areas.

Segregated Excitatory-Inhibitory Recurrent Subnetworks in Layer 5 of the Rat Frontal Cortex

A prominent feature of neocortical pyramidal cells (PCs) is their numerous projections to diverse brain areas. In layer 5 (L5) of the rat frontal cortex, there are 2 major subtypes of PCs that differ in their long-range axonal projections, corticopontine (CPn) cells and crossed corticostriatal (CCS) cells. The outputs of these L5 PCs can be regulated by feedback inhibition from neighboring cortical GABAergic cells. Two major subtypes of GABAergic cells are parvalbumin (PV)-positive and somatostatin (SOM)-positive cells. PV cells have a fast-spiking (FS) firing pattern, while SOM cells have a low threshold spike (LTS) and regular spiking. In this study, we found that the 2 PC subtypes in L5 selectively make recurrent connections with LTS cells. The connection patterns correlated with the morphological and physiological diversity of LTS cells. LTS cells with high input resistance (Ri) exhibited more compact dendrites and more rebound spikes than LTS cells with low Ri, which had vertically elongated dendrites. LTS subgroups differently inhibited the PC subtypes, although FS cells made nonselective connections with both projection subtypes. These results demonstrate a novel recurrent network of inhibitory and projection-specific excitatory neurons within the neocortex.

A Neural Circuit Mechanism for the Involvements of Dopamine in Effort-Related Choices: Decay of Learned Values, Secondary Effects of Depletion, and Calculation of Temporal Difference Error

Dopamine has been suggested to be crucially involved in effort-related choices. Key findings are that dopamine depletion (i) changed preference for a high-cost, large-reward option to a low-cost, small-reward option, (ii) but not when the large-reward option was also low-cost or the small-reward option gave no reward, (iii) while increasing the latency in all the cases but only transiently, and (iv) that antagonism of either dopamine D1 or D2 receptors also specifically impaired selection of the high-cost, large-reward option. The underlying neural circuit mechanisms remain unclear. Here we show that findings i–iii can be explained by the dopaminergic representation of temporal-difference reward-prediction error (TD-RPE), whose mechanisms have now become clarified, if (1) the synaptic strengths storing the values of actions mildly decay in time and (2) the obtained-reward-representing excitatory input to dopamine neurons increases after dopamine depletion. The former is potentially caused by background neural activity–induced weak synaptic plasticity, and the latter is assumed to occur through post-depletion increase of neural activity in the pedunculopontine nucleus, where neurons representing obtained reward exist and presumably send excitatory projections to dopamine neurons. We further show that finding iv, which is nontrivial given the suggested distinct functions of the D1 and D2 corticostriatal pathways, can also be explained if we additionally assume a proposed mechanism of TD-RPE calculation, in which the D1 and D2 pathways encode the values of actions with a temporal difference. These results suggest a possible circuit mechanism for the involvements of dopamine in effort-related choices and, simultaneously, provide implications for the mechanisms of TD-RPE calculation.

Consolidation of Goal-Directed Action Depends on MAPK/ERK Signaling in Rodent Prelimbic Cortex

The prelimbic prefrontal cortex (PL) has consistently been found to be necessary for the acquisition of goal-directed actions in rodents. Nevertheless, the specific cellular processes underlying this learning remain unknown. We assessed changes in learning-related expression of mitogen-activated protein kinase/extracellular signal-related kinase (MAPK/ERK1/2) phosphorylation (pERK) in layers 2-3 and 5-6 of the anterior and posterior PL and in the population of neurons projecting to posterior dorsomedial striatum (pDMS), also implicated in goal-directed learning. Rats were given either a single session of training to press a lever for a pellet reward or yoked reward deliveries without instrumental training and assessed 5 or 60 min after training for pERK expression. Relative to yoked training, instrumental training produced an increase in pERK expression in all regions of the PL both at 5 and 60 min, and this was accompanied by an increase in nuclear pERK expression in the posterior PL in rats given instrumental training. pDMS-projecting neurons showed a transient increase in pERK expression in posterior layer 5-6 projection neurons after 5 min, and a delayed increase in anterior layer 2-3 neurons after 60 min, suggesting that ERK expression in the PL is necessary for the consolidation of goal-directed learning. Consistent with this claim, we found that rats trained on two lever press actions for distinct outcomes and then infused with the MEK inhibitor PD98059 into the PL immediately after training failed to acquire specific action-outcome associations, suggesting that persistent pERK signaling in the PL is necessary for goal-directed learning.

SIGNIFICANCE STATEMENT

The prelimbic cortex is implicated in goal-directed learning in rodents; however, it is unclear whether it is involved in the consolidation of this learning, and what cellular processes are involved. We used pERK as a marker of activity-related synaptic plasticity to assess learning-induced changes in distinct layers and neuronal populations of the prelimbic prefrontal cortex (PL). Training produced long-lasting upregulation of pERK throughout the PL and specifically within neurons that project to the pDMS, another region critical for goal-directed learning. Next, we demonstrated that pERK signaling in the PL was necessary for the consolidation of goal-directed learning. Together, these results indicate that instrumental training induces ERK signaling in distinct layers and populations in the PL and this signaling underlies the consolidation of goal-directed learning.

Forgetting in Reinforcement Learning Links Sustained Dopamine Signals to Motivation

It has been suggested that dopamine (DA) represents reward-prediction-error (RPE) defined in reinforcement learning and therefore DA responds to unpredicted but not predicted reward. However, recent studies have found DA response sustained towards predictable reward in tasks involving self-paced behavior, and suggested that this response represents a motivational signal. We have previously shown that RPE can sustain if there is decay/forgetting of learned-values, which can be implemented as decay of synaptic strengths storing learned-values. This account, however, did not explain the suggested link between tonic/sustained DA and motivation. In the present work, we explored the motivational effects of the value-decay in self-paced approach behavior, modeled as a series of ‘Go’ or ‘No-Go’ selections towards a goal. Through simulations, we found that the value-decay can enhance motivation, specifically, facilitate fast goal-reaching, albeit counterintuitively. Mathematical analyses revealed that underlying potential mechanisms are twofold: (1) decay-induced sustained RPE creates a gradient of ‘Go’ values towards a goal, and (2) value-contrasts between ‘Go’ and ‘No-Go’ are generated because while chosen values are continually updated, unchosen values simply decay. Our model provides potential explanations for the key experimental findings that suggest DA's roles in motivation: (i) slowdown of behavior by post-training blockade of DA signaling, (ii) observations that DA blockade severely impairs effortful actions to obtain rewards while largely sparing seeking of easily obtainable rewards, and (iii) relationships between the reward amount, the level of motivation reflected in the speed of behavior, and the average level of DA. These results indicate that reinforcement learning with value-decay, or forgetting, provides a parsimonious mechanistic account for the DA's roles in value-learning and motivation. Our results also suggest that when biological systems for value-learning are active even though learning has apparently converged, the systems might be in a state of dynamic equilibrium, where learning and forgetting are balanced.

Dopamine (DA) has been suggested to have two reward-related roles: (1) representing reward-prediction-error (RPE), and (2) providing motivational drive. Role(1) is based on the physiological results that DA responds to unpredicted but not predicted reward, whereas role(2) is supported by the pharmacological results that blockade of DA signaling causes motivational impairments such as slowdown of self-paced behavior. So far, these two roles are considered to be played by two different temporal patterns of DA signals: role(1) by phasic signals and role(2) by tonic/sustained signals. However, recent studies have found sustained DA signals with features indicative of both roles (1) and (2), complicating this picture. Meanwhile, whereas synaptic/circuit mechanisms for role(1), i.e., how RPE is calculated in the upstream of DA neurons and how RPE-dependent update of learned-values occurs through DA-dependent synaptic plasticity, have now become clarified, mechanisms for role(2) remain unclear. In this work, we modeled self-paced behavior by a series of ‘Go’ or ‘No-Go’ selections in the framework of reinforcement-learning assuming DA's role(1), and demonstrated that incorporation of decay/forgetting of learned-values, which is presumably implemented as decay of synaptic strengths storing learned-values, provides a potential unified mechanistic account for the DA's two roles, together with its various temporal patterns.

Reinforcement learning with Marr

To many, the poster child for David Marr's famous three levels of scientific inquiry is reinforcement learning-a computational theory of reward optimization, which readily prescribes algorithmic solutions that evidence striking resemblance to signals found in the brain, suggesting a straightforward neural implementation. Here we review questions that remain open at each level of analysis, concluding that the path forward to their resolution calls for inspiration across levels, rather than a focus on mutual constraints.

Distributed and Mixed Information in Monosynaptic Inputs to Dopamine Neurons

Dopamine neurons encode the difference between actual and predicted reward, or reward prediction error (RPE). Although many models have been proposed to account for this computation, it has been difficult to test these models experimentally. Here we established an awake electrophysiological recording system, combined with rabies virus and optogenetic cell-type identification, to characterize the firing patterns of monosynaptic inputs to dopamine neurons while mice performed classical conditioning tasks. We found that each variable required to compute RPE, including actual and predicted reward, was distributed in input neurons in multiple brain areas. Further, many input neurons across brain areas signaled combinations of these variables. These results demonstrate that even simple arithmetic computations such as RPE are not localized in specific brain areas but, rather, distributed across multiple nodes in a brain-wide network. Our systematic method to examine both activity and connectivity revealed unexpected redundancy for a simple computation in the brain.

Functional Relevance of Different Basal Ganglia Pathways Investigated in a Spiking Model with Reward Dependent Plasticity

The brain enables animals to behaviorally adapt in order to survive in a complex and dynamic environment, but how reward-oriented behaviors are achieved and computed by its underlying neural circuitry is an open question. To address this concern, we have developed a spiking model of the basal ganglia (BG) that learns to dis-inhibit the action leading to a reward despite ongoing changes in the reward schedule. The architecture of the network features the two pathways commonly described in BG, the direct (denoted D1) and the indirect (denoted D2) pathway, as well as a loop involving striatum and the dopaminergic system. The activity of these dopaminergic neurons conveys the reward prediction error (RPE), which determines the magnitude of synaptic plasticity within the different pathways. All plastic connections implement a versatile four-factor learning rule derived from Bayesian inference that depends upon pre- and post-synaptic activity, receptor type, and dopamine level. Synaptic weight updates occur in the D1 or D2 pathways depending on the sign of the RPE, and an efference copy informs upstream nuclei about the action selected. We demonstrate successful performance of the system in a multiple-choice learning task with a transiently changing reward schedule. We simulate lesioning of the various pathways and show that a condition without the D2 pathway fares worse than one without D1. Additionally, we simulate the degeneration observed in Parkinson's disease (PD) by decreasing the number of dopaminergic neurons during learning. The results suggest that the D1 pathway impairment in PD might have been overlooked. Furthermore, an analysis of the alterations in the synaptic weights shows that using the absolute reward value instead of the RPE leads to a larger change in D1.

The functional logic of corticostriatal connections

Unidirectional connections from the cortex to the matrix of the corpus striatum initiate the cortico-basal ganglia (BG)-thalamocortical loop, thought to be important in momentary action selection and in longer-term fine tuning of behavioural repertoire; a discrete set of striatal compartments, striosomes, has the complementary role of registering or anticipating reward that shapes corticostriatal plasticity. Re-entrant signals traversing the cortico-BG loop impact predominantly frontal cortices, conveyed through topographically ordered output channels; by contrast, striatal input signals originate from a far broader span of cortex, and are far more divergent in their termination. The term 'disclosed loop' is introduced to describe this organisation: a closed circuit that is open to outside influence at the initial stage of cortical input. The closed circuit component of corticostriatal afferents is newly dubbed 'operative', as it is proposed to establish the bid for action selection on the part of an incipient cortical action plan; the broader set of converging corticostriatal afferents is described as contextual. A corollary of this proposal is that every unit of the striatal volume, including the long, C-shaped tail of the caudate nucleus, should receive a mandatory component of operative input, and hence include at least one area of BG-recipient cortex amongst the sources of its corticostriatal afferents. Individual operative afferents contact twin classes of GABAergic striatal projection neuron (SPN), distinguished by their neurochemical character, and onward circuitry. This is the basis of the classic direct and indirect pathway model of the cortico-BG loop. Each pathway utilises a serial chain of inhibition, with two such links, or three, providing positive and negative feedback, respectively. Operative co-activation of direct and indirect SPNs is, therefore, pictured to simultaneously promote action, and to restrain it. The balance of this rival activity is determined by the contextual inputs, which summarise the external and internal sensory environment, and the state of ongoing behavioural priorities. Notably, the distributed sources of contextual convergence upon a striatal locus mirror the transcortical network harnessed by the origin of the operative input to that locus, thereby capturing a similar set of contingencies relevant to determining action. The disclosed loop formulation of corticostriatal and subsequent BG loop circuitry, as advanced here, refines the operating rationale of the classic model and allows the integration of more recent anatomical and physiological data, some of which can appear at variance with the classic model. Equally, it provides a lucid functional context for continuing cellular studies of SPN biophysics and mechanisms of synaptic plasticity.

The Diversity of Cortical Inhibitory Synapses

The most typical and well known inhibitory action in the cortical microcircuit is a strong inhibition on the target neuron by axo-somatic synapses. However, it has become clear that synaptic inhibition in the cortex is much more diverse and complicated. Firstly, at least ten or more inhibitory non-pyramidal cell subtypes engage in diverse inhibitory functions to produce the elaborate activity characteristic of the different cortical states. Each distinct non-pyramidal cell subtype has its own independent inhibitory function. Secondly, the inhibitory synapses innervate different neuronal domains, such as axons, spines, dendrites and soma, and their inhibitory postsynaptic potential (IPSP) size is not uniform. Thus, cortical inhibition is highly complex, with a wide variety of anatomical and physiological modes. Moreover, the functional significance of the various inhibitory synapse innervation styles and their unique structural dynamic behaviors differ from those of excitatory synapses. In this review, we summarize our current understanding of the inhibitory mechanisms of the cortical microcircuit.

Neural circuitry involved in quitting after repeated failures: role of the cingulate and temporal parietal junction

The more times people fail the more likely they are to give up, however little is known about the neural mechanisms underlying this impact of repeated failure on decision making. Here we have used a visual shape discrimination task with computer-controlled feedback combined with functional magnetic resonance imaging (fMRI) to investigate the neural circuits involved. The behavioral task confirmed that the more times subjects experienced failure the more likely they were to give up, with three successive failures being the key threshold and the majority of subjects reaching the point where they decided to quit and try a new stimulus set after three or four failures. The fMRI analysis revealed activity changes in frontal, parietal, temporal, limbic and striatal regions, especially anterior cingulate cortex (ACC), posterior cingulate cortex (PCC) and temporal parietal junction (TPJ) associated with the number of previous failures experienced. Furthermore, their parameter estimates were predictive of subjects’ quitting rate. Thus, subjects reach the point where they decide to quit after three/four failures and this is associated with differential changes in brain regions involved in error monitoring and reward which regulate both failure detection and changes in decision-making strategy.

Dopamine Prediction Errors in Reward Learning and Addiction: From Theory to Neural Circuitry

Midbrain dopamine (DA) neurons are proposed to signal reward prediction error (RPE), a fundamental parameter in associative learning models. This RPE hypothesis provides a compelling theoretical framework for understanding DA function in reward learning and addiction. New studies support a causal role for DA-mediated RPE activity in promoting learning about natural reward; however, this question has not been explicitly tested in the context of drug addiction. In this review, we integrate theoretical models with experimental findings on the activity of DA systems, and on the causal role of specific neuronal projections and cell types, to provide a circuit-based framework for probing DA-RPE function in addiction. By examining error-encoding DA neurons in the neural network in which they are embedded, hypotheses regarding circuit-level adaptations that possibly contribute to pathological error signaling and addiction can be formulated and tested.

Temporal Structure of Neuronal Activity among Cortical Neuron Subtypes during Slow Oscillations in Anesthetized Rats

Slow-wave oscillations, the predominant brain rhythm during sleep, are composed of Up/Down cycles. Depolarizing Up-states involve activity in layer 5 (L5) of the neocortex, but it is unknown how diverse subtypes of neurons within L5 participate in generating and maintaining Up-states. Here we compare the in vivo firing patterns of corticopontine (CPn) pyramidal cells, crossed-corticostriatal (CCS) pyramidal cells, and fast-spiking (FS) GABAergic neurons in the rat frontal cortex, with those of thalamocortical neurons during Up/Down cycles in the anesthetized condition. During the transition from Down- to Up-states, increased activity in these neurons was highly temporally structured, with spiking occurring first in thalamocortical neurons, followed by cortical FS cells, CCS cells, and, finally, CPn cells. Activity in some FS, CCS, and CPn neurons occurred in phase with Up-nested gamma rhythms, with FS neurons showing phase delay relative to pyramidal neurons. These results suggest that thalamic and cortical pyramidal neurons are activated in a specific temporal sequence during Up/Down cycles, but cortical pyramidal cells are activated at a similar gamma phase. In addition to Up-state firing specificity, CCS and CPn cells exhibited differences in activity during cortical desynchronization, further indicating projection- and state-dependent information processing within L5.

SIGNIFICANCE STATEMENT

Patterned activity in neocortical electroencephalograms, including slow waves and gamma oscillations, is thought to reflect the organized activity of neocortical neurons that comprises many specialized neuron subtypes. We found that the timing of action potentials during slow waves in individual cortical neurons was correlated with their laminar positions and axonal targets. Within gamma cycles nested in the slow-wave depolarization, cortical pyramidal cells fired earlier than did interneurons. At the start of slow-wave depolarizations, activity in thalamic neurons receiving inhibition from the basal ganglia occurred earlier than activity in cortical neurons. Together, these findings reveal a temporally ordered pattern of output from diverse neuron subtypes in the frontal cortex and related thalamic nuclei during neocortical oscillations.

Computing reward-prediction error: an integrated account of cortical timing and basal-ganglia pathways for appetitive and aversive learning

There are two prevailing notions regarding the involvement of the corticobasal ganglia system in value‐based learning: (i) the direct and indirect pathways of the basal ganglia are crucial for appetitive and aversive learning, respectively, and (ii) the activity of midbrain dopamine neurons represents reward‐prediction error. Although (ii) constitutes a critical assumption of (i), it remains elusive how (ii) holds given (i), with the basal‐ganglia influence on the dopamine neurons. Here we present a computational neural‐circuit model that potentially resolves this issue. Based on the latest analyses of the heterogeneous corticostriatal neurons and connections, our model posits that the direct and indirect pathways, respectively, represent the values of upcoming and previous actions, and up‐regulate and down‐regulate the dopamine neurons via the basal‐ganglia output nuclei. This explains how the difference between the upcoming and previous values, which constitutes the core of reward‐prediction error, is calculated. Simultaneously, it predicts that blockade of the direct/indirect pathway causes a negative/positive shift of reward‐prediction error and thereby impairs learning from positive/negative error, i.e. appetitive/aversive learning. Through simulation of reward‐reversal learning and punishment‐avoidance learning, we show that our model could indeed account for the experimentally observed features that are suggested to support notion (i) and could also provide predictions on neural activity. We also present a behavioral prediction of our model, through simulation of inter‐temporal choice, on how the balance between the two pathways relates to the subject's time preference. These results indicate that our model, incorporating the heterogeneity of the cortical influence on the basal ganglia, is expected to provide a closed‐circuit mechanistic understanding of appetitive/aversive learning.

A network model of basal ganglia for understanding the roles of dopamine and serotonin in reward-punishment-risk based decision making

There is significant evidence that in addition to reward-punishment based decision making, the Basal Ganglia (BG) contributes to risk-based decision making (Balasubramani et al., 2014). Despite this evidence, little is known about the computational principles and neural correlates of risk computation in this subcortical system. We have previously proposed a reinforcement learning (RL)-based model of the BG that simulates the interactions between dopamine (DA) and serotonin (5HT) in a diverse set of experimental studies including reward, punishment and risk based decision making (Balasubramani et al., 2014). Starting with the classical idea that the activity of mesencephalic DA represents reward prediction error, the model posits that serotoninergic activity in the striatum controls risk-prediction error. Our prior model of the BG was an abstract model that did not incorporate anatomical and cellular-level data. In this work, we expand the earlier model into a detailed network model of the BG and demonstrate the joint contributions of DA-5HT in risk and reward-punishment sensitivity. At the core of the proposed network model is the following insight regarding cellular correlates of value and risk computation. Just as DA D1 receptor (D1R) expressing medium spiny neurons (MSNs) of the striatum were thought to be the neural substrates for value computation, we propose that DA D1R and D2R co-expressing MSNs are capable of computing risk. Though the existence of MSNs that co-express D1R and D2R are reported by various experimental studies, prior existing computational models did not include them. Ours is the first model that accounts for the computational possibilities of these co-expressing D1R-D2R MSNs, and describes how DA and 5HT mediate activity in these classes of neurons (D1R-, D2R-, D1R-D2R- MSNs). Starting from the assumption that 5HT modulates all MSNs, our study predicts significant modulatory effects of 5HT on D2R and co-expressing D1R-D2R MSNs which in turn explains the multifarious functions of 5HT in the BG. The experiments simulated in the present study relates 5HT to risk sensitivity and reward-punishment learning. Furthermore, our model is shown to capture reward-punishment and risk based decision making impairment in Parkinson's Disease (PD). The model predicts that optimizing 5HT levels along with DA medications might be essential for improving the patients' reward-punishment learning deficits.

Morphological elucidation of basal ganglia circuits contributing reward prediction

Electrophysiological studies in monkeys have shown that dopaminergic neurons respond to the reward prediction error. In addition, striatal neurons alter their responsiveness to cortical or thalamic inputs in response to the dopamine signal, via the mechanism of dopamine-regulated synaptic plasticity. These findings have led to the hypothesis that the striatum exhibits synaptic plasticity under the influence of the reward prediction error and conduct reinforcement learning throughout the basal ganglia circuits. The reinforcement learning model is useful; however, the mechanism by which such a process emerges in the basal ganglia needs to be anatomically explained. The actor–critic model has been previously proposed and extended by the existence of role sharing within the striatum, focusing on the striosome/matrix compartments. However, this hypothesis has been difficult to confirm morphologically, partly because of the complex structure of the striosome/matrix compartments. Here, we review recent morphological studies that elucidate the input/output organization of the striatal compartments.

Dorsal striatum is necessary for stimulus-value but not action-value learning in humans

Several lines of evidence implicate the striatum in learning from experience on the basis of positive and negative feedback. However, the necessity of the striatum for such learning has been difficult to demonstrate in humans, because brain damage is rarely restricted to this structure. Here we test a rare individual with widespread bilateral damage restricted to the dorsal striatum. His performance was impaired and not significantly different from chance on several classic learning tasks, consistent with current theories regarding the role of the striatum. However, he also exhibited remarkably intact performance on a different subset of learning paradigms. The tasks he could perform can all be solved by learning the value of actions, while those he could not perform can only be solved by learning the value of stimuli. Although dorsal striatum is often thought to play a specific role in action-value learning, we find surprisingly that dorsal striatum is necessary for stimulus-value but not action-value learning in humans.

Methamphetamine-induced neurotoxicity disrupts pharmacologically evoked dopamine transients in the dorsomedial and dorsolateral striatum

Phasic dopamine (DA) signaling, during which burst firing by DA neurons generates short-lived elevations in extracellular DA in terminal fields called DA transients, is implicated in reinforcement learning. Disrupted phasic DA signaling is proposed to link DA depletions and cognitive-behavioral impairment in methamphetamine (METH)-induced neurotoxicity. Here, we further investigated this disruption by assessing effects of METH pretreatment on DA transients elicited by a drug cocktail of raclopride, a D2 DA receptor antagonist, and nomifensine, an inhibitor of the dopamine transporter (DAT). One advantage of this approach is that pharmacological activation provides a large, high-quality data set of transients elicited by endogenous burst firing of DA neurons for analysis of regional differences and neurotoxicity. These pharmacologically evoked DA transients were measured in the dorsomedial (DM) and dorsolateral (DL) striatum of urethane-anesthetized rats by fast-scan cyclic voltammetry. Electrically evoked DA levels were also recorded to quantify DA release and uptake, and DAT binding was determined by means of autoradiography to index DA denervation. Pharmacologically evoked DA transients in intact animals exhibited a greater amplitude and frequency and shorter duration in the DM compared to the DL striatum, despite similar pre- and post-drug assessments of DA release and uptake in both sub-regions as determined from the electrically evoked DA signals. METH pretreatment reduced transient activity. The most prominent effect of METH pretreatment on transients across striatal sub-region was decreased amplitude, which mirrored decreased DAT binding and was accompanied by decreased DA release. Overall, these results identify marked intrastriatal differences in the activity of DA transients that appear independent of presynaptic mechanisms for DA release and uptake and further support disrupted phasic DA signaling mediated by decreased DA release in rats with METH-induced neurotoxicity.

Microstimulation of the human substantia nigra alters reinforcement learning

Animal studies have shown that substantia nigra (SN) dopaminergic (DA) neurons strengthen action-reward associations during reinforcement learning, but their role in human learning is not known. Here, we applied microstimulation in the SN of 11 patients undergoing deep brain stimulation surgery for the treatment of Parkinson's disease as they performed a two-alternative probability learning task in which rewards were contingent on stimuli, rather than actions. Subjects demonstrated decreased learning from reward trials that were accompanied by phasic SN microstimulation compared with reward trials without stimulation. Subjects who showed large decreases in learning also showed an increased bias toward repeating actions after stimulation trials; therefore, stimulation may have decreased learning by strengthening action-reward associations rather than stimulus-reward associations. Our findings build on previous studies implicating SN DA neurons in preferentially strengthening action-reward associations during reinforcement learning.

An extended reinforcement learning model of basal ganglia to understand the contributions of serotonin and dopamine in risk-based decision making, reward prediction, and punishment learning

Although empirical and neural studies show that serotonin (5HT) plays many functional roles in the brain, prior computational models mostly focus on its role in behavioral inhibition. In this study, we present a model of risk based decision making in a modified Reinforcement Learning (RL)-framework. The model depicts the roles of dopamine (DA) and serotonin (5HT) in Basal Ganglia (BG). In this model, the DA signal is represented by the temporal difference error (δ), while the 5HT signal is represented by a parameter (α) that controls risk prediction error. This formulation that accommodates both 5HT and DA reconciles some of the diverse roles of 5HT particularly in connection with the BG system. We apply the model to different experimental paradigms used to study the role of 5HT: (1) Risk-sensitive decision making, where 5HT controls risk assessment, (2) Temporal reward prediction, where 5HT controls time-scale of reward prediction, and (3) Reward/Punishment sensitivity, in which the punishment prediction error depends on 5HT levels. Thus the proposed integrated RL model reconciles several existing theories of 5HT and DA in the BG.

Striatal dopamine ramping may indicate flexible reinforcement learning with forgetting in the cortico-basal ganglia circuits

It has been suggested that the midbrain dopamine (DA) neurons, receiving inputs from the cortico-basal ganglia (CBG) circuits and the brainstem, compute reward prediction error (RPE), the difference between reward obtained or expected to be obtained and reward that had been expected to be obtained. These reward expectations are suggested to be stored in the CBG synapses and updated according to RPE through synaptic plasticity, which is induced by released DA. These together constitute the "DA=RPE" hypothesis, which describes the mutual interaction between DA and the CBG circuits and serves as the primary working hypothesis in studying reward learning and value-based decision-making. However, recent work has revealed a new type of DA signal that appears not to represent RPE. Specifically, it has been found in a reward-associated maze task that striatal DA concentration primarily shows a gradual increase toward the goal. We explored whether such ramping DA could be explained by extending the "DA=RPE" hypothesis by taking into account biological properties of the CBG circuits. In particular, we examined effects of possible time-dependent decay of DA-dependent plastic changes of synaptic strengths by incorporating decay of learned values into the RPE-based reinforcement learning model and simulating reward learning tasks. We then found that incorporation of such a decay dramatically changes the model's behavior, causing gradual ramping of RPE. Moreover, we further incorporated magnitude-dependence of the rate of decay, which could potentially be in accord with some past observations, and found that near-sigmoidal ramping of RPE, resembling the observed DA ramping, could then occur. Given that synaptic decay can be useful for flexibly reversing and updating the learned reward associations, especially in case the baseline DA is low and encoding of negative RPE by DA is limited, the observed DA ramping would be indicative of the operation of such flexible reward learning.

Differential cortical activation of the striatal direct and indirect pathway cells: reconciling the anatomical and optogenetic results by using a computational method

The corticostriatal system is considered to be crucially involved in learning and action selection. Anatomical studies have shown that two types of corticostriatal neurons, intratelencephalic (IT) and pyramidal tract (PT) cells, preferentially project to dopamine D1 or D2 receptor-expressing striatal projection neurons, respectively. In contrast, an optogenetic study has shown that stimulation of IT axons evokes comparable responses in D1 and D2 cells and that stimulation of PT axons evokes larger responses in D1 cells. Since the optogenetic study applied brief stimulation only, however, the overall impacts of repetitive inputs remain unclear. Moreover, the apparent contradiction between the anatomical and optogenetic results remains to be resolved. I addressed these issues by using a computational approach. Specifically, I constructed a model of striatal response to cortical inputs, with parameters regarding short-term synaptic plasticity and anatomical connection strength for each connection type. Under the constraint of the optogenetic results, I then explored the parameters that best explain the previously reported paired-pulse ratio of response in D1 and D2 cells to cortical and intrastriatal stimulations, which presumably recruit different compositions of IT and PT fibers. The results indicate that 1) IT→D1 and PT→D2 connections are anatomically stronger than IT→D2 and PT→D1 connections, respectively, consistent with the previous findings, and that 2) IT→D1 and PT→D2 synapses entail short-term facilitation, whereas IT→D2 and PT→D1 synapses would basically show depression, and thereby 3) repetitive IT or PT inputs have larger overall impacts on D1 or D2 cells, respectively, supporting a recently proposed hypothesis on the roles of corticostriatal circuits in reinforcement learning.

GENSAT BAC cre-recombinase driver lines to study the functional organization of cerebral cortical and basal ganglia circuits

Recent development of molecular genetic techniques are rapidly advancing understanding of the functional role of brain circuits in behavior. Critical to this approach is the ability to target specific neuron populations and circuits. The collection of over 250 BAC Cre-recombinase driver lines produced by the GENSAT project provides a resource for such studies. Here we provide characterization of GENSAT BAC-Cre driver lines with expression in specific neuroanatomical pathways within the cerebral cortex and basal ganglia.

Alterations of neocortical pyramidal neurons: turning points in the genesis of mental retardation

Pyramidal neurons (PNs) represent the majority of neocortical cells and their involvement in cognitive functions is decisive. Therefore, they are the most obvious target of developmental disorders characterized by mental retardation. Genetic and non-genetic forms of intellectual disability share a few basic pathogenetic signatures that result in the anomalous function of PNs. Here, we review the key mechanisms impairing these neurons and their participation in the cortical network, with special focus on experimental models of fetal exposure to alcohol. Due to the heterogeneity of PNs, some alterations affect selectively a given cell population, which may also differ depending on the considered pathology. These specific features open new possibilities for the interpretation of cognitive defects observed in mental retardation syndromes, as well as for novel therapeutic interventions.

The visual corticostriatal loop through the tail of the caudate: circuitry and function

Although high level visual cortex projects to a specific region of the striatum, the tail of the caudate, and participates in corticostriatal loops, the function of this visual corticostriatal system is not well understood. This article first reviews what is known about the anatomy of the visual corticostriatal loop across mammals, including rodents, cats, monkeys, and humans. Like other corticostriatal systems, the visual corticostriatal system includes both closed loop components (recurrent projections that return to the originating cortical location) and open loop components (projections that terminate in other neural regions). The article then reviews what previous empirical research has shown about the function of the tail of the caudate. The article finally addresses the possible functions of the closed and open loop connections of the visual loop in the context of theories and computational models of corticostriatal function.

Diversity of layer 5 projection neurons in the mouse motor cortex

In the primary motor cortex (M1), layer 5 projection neurons signal directly to distant motor structures to drive movement. Despite their pivotal position and acknowledged diversity these neurons are traditionally separated into broad commissural and corticofugal types, and until now no attempt has been made at resolving the basis for their diversity. We therefore probed the electrophysiological and morphological properties of retrogradely labeled M1 corticospinal (CSp), corticothalamic (CTh), and commissural projecting corticostriatal (CStr) and corticocortical (CC) neurons. An unsupervised cluster analysis established at least four phenotypes with additional differences between lumbar and cervical projecting CSp neurons. Distinguishing parameters included the action potential (AP) waveform, firing behavior, the hyperpolarisation-activated sag potential, sublayer position, and soma and dendrite size. CTh neurons differed from CSp neurons in showing spike frequency acceleration and a greater sag potential. CStr neurons had the lowest AP amplitude and maximum rise rate of all neurons. Temperature influenced spike train behavior in corticofugal neurons. At 26°C CTh neurons fired bursts of APs more often than CSp neurons, but at 36°C both groups fired regular APs. Our findings provide reliable phenotypic fingerprints to identify distinct M1 projection neuron classes as a tool to understand their unique contributions to motor function.

Direction- and distance-dependent interareal connectivity of pyramidal cell subpopulations in the rat frontal cortex

The frontal cortex plays an important role in the initiation and execution of movements via widespread projections to various cortical and subcortical areas. Layer 2/3 (L2/3) pyramidal cells in the frontal cortex send axons mainly to other ipsilateral/contralateral cortical areas. Subpopulations of layer 5 (L5) pyramidal cells that selectively project to the pontine nuclei or to the contralateral cortex [commissural (COM) cells] also target diverse and sometimes overlapping ipsilateral cortical areas. However, little is known about target area-dependent participation in ipsilateral corticocortical (iCC) connections by subclasses of L2/3 and L5 projection neurons. To better understand the functional hierarchy between cortical areas, we compared iCC connectivity between the secondary motor cortex (M2) and adjacent areas, such as the orbitofrontal and primary motor cortices, and distant non-frontal areas, such as the perirhinal and posterior parietal cortices. We particularly assessed the laminar distribution of iCC cells and fibers, and identified the subtypes of pyramidal cells participating in those projections. For connections between M2 and frontal areas, L2/3 and L5 cells in both areas contributed to reciprocal projections, which can be viewed as “bottom-up” or “top-down” on the basis of their differential targeting of cortical lamina. In connections between M2 and non-frontal areas, neurons participating in bottom-up and top-down projections were segregated into the different layers: bottom-up projections arose primarily from L2/3 cells, while top-down projections were dominated by L5 COM cells. These findings suggest that selective participation in iCC connections by pyramidal cell subtypes lead to directional connectivity between M2 and other cortical areas. Based on these findings, we propose a provisional unified framework of interareal hierarchy within the frontal cortex, and discuss the interaction of local circuits with long-range interareal connections.

MEG studies of motor cortex gamma oscillations: evidence for a gamma "fingerprint" in the brain?

The human motor cortex exhibits transient bursts of high frequency gamma oscillations in the 60–90 Hz range during movement. It has been proposed that gamma oscillations generally reflect local intracortical activity. However, movement-evoked gamma is observed simultaneously in both cortical and subcortical (basal ganglia) structures and thus appears to reflect long-range cortical-subcortical interactions. Recent evidence suggests that gamma oscillations do not simply reflect sensory reafference, but have a facilitative role in movement initiation. Here we summarize contributions of MEG to our understanding of movement-evoked gamma oscillations, including evidence that transient gamma bursts during the performance of specific movements constitutes a stereotyped spectral and temporal pattern within individuals—a gamma “fingerprint”—that is highly stable over time. Although their functional significance remains to be fully understood, movement-evoked gamma oscillations may represent frequency specific tuning within cortical-subcortical networks that can be monitored non-invasively using MEG during a variety of motor tasks, and may provide important information regarding cortical dynamics of ongoing motor control.

Bird brains and tool use: beyond instrumental conditioning

Few displays of complex cognition are as intriguing as nonhuman tool use. Long thought to be unique to humans, evidence for tool use and manufacture has now been gathered in chimpanzees, dolphins, and elephants. Outside of mammals, tool use is most common in birds, especially in corvids and parrots. The present paper reviews the evidence for avian tool use, both in the wild and in laboratory settings. It also places this behavioral evidence in the context of longstanding debates about the kinds of mental processes nonhumans can perform. Descartes argued that animals are unable to think because they are soulless machines, incapable of flexible behavior. Later, as human machines became more sophisticated and psychologists discovered classical and instrumental conditioning, skepticism about animal thinking decreased. However, behaviors that involve more than simple conditioning continued to elicit skepticism, especially among behaviorists. Nonetheless, as reviewed here, strong behavioral data now indicate that tool use in some birds cannot be explained as resulting entirely from instrumental conditioning. The neural substrates of tool use in birds remain unclear, but the available data point mainly to the caudolateral nidopallium, which shares both functional and structural features with the mammalian prefrontal cortex. As more data on the neural mechanisms of complex cognition in birds accrue, skepticism about those mental capacities should continue to wane.