A computational model of how cholinergic interneurons protect striatal-dependent learning

A mismatch between striatal cholinergic pauses and dopaminergic reward prediction errors

Striatal acetylcholine and dopamine critically regulate movement, motivation, and reward-related learning. Pauses in cholinergic interneuron (CIN) firing are thought to coincide with dopamine pulses encoding reward prediction errors (RPE) to jointly enable synaptic plasticity. Here, we examine the firing of identified CINs during reward-guided decision-making in freely moving rats and compare this firing to dopamine release. Relationships between CINs, dopamine, and behavior varied strongly by subregion. In the dorsal-lateral striatum, a Go! cue evoked burst-pause CIN spiking, followed by a brief dopamine pulse that was unrelated to RPE. In the dorsal-medial striatum, this cue evoked only a CIN pause, that was curtailed by a movement-selective rebound in firing. Finally, in the ventral striatum, a reward cue evoked RPE-coding increases in both dopamine and CIN firing, without a consistent pause. Our results demonstrate a spatial and temporal dissociation between CIN pauses and dopamine RPE signals and will inform future models of striatal information processing under both normal and pathological conditions.

Pathways for Naturalistic Looking Behavior in Primate II. Superior Colliculus Integrates Parallel Top-down and Bottom-up Inputs

Volitional signals for gaze control are provided by multiple parallel pathways converging on the midbrain superior colliculus (SC), whose deeper layers output to the brainstem gaze circuits. In the first of two papers (Takahashi and Veale, 2023), we described the properties of gaze behavior of several species under both laboratory and natural conditions, as well as the current understanding of the brainstem and spinal cord circuits implementing gaze control in primate. In this paper, we review the parallel pathways by which sensory and task information reaches SC and how these sensory and task signals interact within SC's multilayered structure. This includes both bottom-up (world statistics) signals mediated by sensory cortex, association cortex, and subcortical structures, as well as top-down (goal and task) influences which arrive via either direct excitatory pathways from cerebral cortex, or via indirect basal ganglia relays resulting in inhibition or dis-inhibition as appropriate for alternative behaviors. Models of attention such as saliency maps serve as convenient frameworks to organize our understanding of both the separate computations of each neural pathway, as well as the interaction between the multiple parallel pathways influencing gaze. While the spatial interactions between gaze's neural pathways are relatively well understood, the temporal interactions between and within pathways will be an important area of future study, requiring both improved technical methods for measurement and improvement of our understanding of how temporal dynamics results in the observed spatiotemporal allocation of gaze.

A spiking computational model for striatal cholinergic interneurons

Cholinergic interneurons in the striatum, also known as tonically active interneurons or TANs, are thought to have a strong effect on corticostriatal plasticity and on striatal activity and outputs, which in turn play a critical role in modulating downstream basal ganglia activity and movement. Striatal TANs can exhibit a variety of firing patterns and responses to synaptic inputs; furthermore, they have been found to display various surges and pauses in activity associated with sensory cues and reward delivery in learning as well as with motor tic production. To help explain the factors that contribute to TAN activity patterns and to provide a resource for future studies, we present a novel conductance-based computational model of a striatal TAN. We show that this model produces the various characteristic firing patterns observed in recordings of TANs. With a single baseline tuning associated with tonic firing, the model also captures a wide range of TAN behaviors found in previous experiments involving a variety of manipulations. In addition to demonstrating these results, we explain how various ionic currents in the model contribute to them. Finally, we use this model to explore the contributions of the acetylcholine released by TANs to the production of surges and pauses in TAN activity in response to strong excitatory inputs. These results provide predictions for future experimental testing that may help with efforts to advance our understanding of the role of TANs in reinforcement learning and in motor disorders such as Tourette's syndrome.

Modulation of Dopamine for Adaptive Learning: A Neurocomputational Model

There have been many proposals that learning rates in the brain are adaptive, in the sense that they increase or decrease depending on environmental conditions. The majority of these models are abstract and make no attempt to describe the neural circuitry that implements the proposed computations. This article describes a biologically detailed computational model that overcomes this shortcoming. Specifically, we propose a neural circuit that implements adaptive learning rates by modulating the gain on the dopamine response to reward prediction errors, and we model activity within this circuit at the level of spiking neurons. The model generates a dopamine signal that depends on the size of the tonically active dopamine neuron population and the phasic spike rate. The model was tested successfully against results from two single-neuron recording studies and a fast-scan cyclic voltammetry study. We conclude by discussing the general applicability of the model to dopamine mediated tasks that transcend the experimental phenomena it was initially designed to address.

An integrative model of Parkinson's disease treatment including levodopa pharmacokinetics, dopamine kinetics, basal ganglia neurotransmission and motor action throughout disease progression

Levodopa is considered the gold standard treatment of Parkinson's disease. Although very effective in alleviating symptoms at their onset, its chronic use with the progressive neuronal denervation in the basal ganglia leads to a decrease in levodopa's effect duration and to the appearance of motor complications. This evolution challenges the establishment of optimal regimens to manage the symptoms as the disease progresses. Based on up-to-date pathophysiological and pharmacological knowledge, we developed an integrative model for Parkinson's disease to evaluate motor function in response to levodopa treatment as the disease progresses. We combined a pharmacokinetic model of levodopa to a model of dopamine's kinetics and a neurocomputational model of basal ganglia. The parameter values were either measured directly or estimated from human and animal data. The concentrations and behaviors predicted by our model were compared to available information and data. Using this model, we were able to predict levodopa plasma concentration, its related dopamine concentration in the brain and the response performance of a motor task for different stages of disease.

Striatal Cholinergic Interneurons: How to Elucidate Their Function in Health and Disease

Striatal cholinergic interneurons (CINs) are the main source of acetylcholine in the striatum and are believed to play an important role in basal ganglia physiology and pathophysiology. The role of CINs in striatal function is known mostly from extracellular recordings of tonically active striatal neurons in monkeys, which are believed to correspond to CINs. Because these neurons transiently respond to motivationally cues with brief pauses, flanked by bursts of increased activity, they are classically viewed as key players in reward-related learning. However, CIN modulatory function within the striatal network has been mainly inferred from the action of acetylcholine agonists/antagonists or through CIN activation. These manipulations are far from recapitulating CIN activity in response to behaviorally-relevant stimuli. New technical tools such as optogenetics allow researchers to specifically manipulate this sparse neuronal population and to mimic their typical pause response. For example, it is now possible to investigate how short inhibition of CIN activity shapes striatal properties. Here, we review the most recent literature and show how these new techniques have brought considerable insights into the functional role of CINs in normal and pathological states, raising several interesting and novel questions. To continue moving forward, it is crucial to determine in detail CIN activity changes during behavior, particularly in rodents. We will also discuss how computational approaches combined with optogenetics will contribute to further our understanding of the CIN role in striatal circuits.

The Functional Role of Striatal Cholinergic Interneurons in Reinforcement Learning From Computational Perspective

In this study, we explore the functional role of striatal cholinergic interneurons, hereinafter referred to as tonically active neurons (TANs), via computational modeling; specifically, we investigate the mechanistic relationship between TAN activity and dopamine variations and how changes in this relationship affect reinforcement learning in the striatum. TANs pause their tonic firing activity after excitatory stimuli from thalamic and cortical neurons in response to a sensory event or reward information. During the pause striatal dopamine concentration excursions are observed. However, functional interactions between the TAN pause and striatal dopamine release are poorly understood. Here we propose a TAN activity-dopamine relationship model and demonstrate that the TAN pause is likely a time window to gate phasic dopamine release and dopamine variations reciprocally modulate the TAN pause duration. Furthermore, this model is integrated into our previously published model of reward-based motor adaptation to demonstrate how phasic dopamine release is gated by the TAN pause to deliver reward information for reinforcement learning in a timely manner. We also show how TAN-dopamine interactions are affected by striatal dopamine deficiency to produce poor performance of motor adaptation.

Increased cognitive load enables unlearning in procedural category learning

Interventions for drug abuse and other maladaptive habitual behaviors may yield temporary success but are often fragile and relapse is common. This implies that current interventions do not erase or substantially modify the representations that support the underlying addictive behavior-that is, they do not cause true unlearning. One example of an intervention that fails to induce true unlearning comes from Crossley, Ashby, and Maddox (2013, Journal of Experimental Psychology: General), who reported that a sudden shift to random feedback did not cause unlearning of category knowledge obtained through procedural systems, and they also reported results suggesting that this failure is because random feedback is noncontingent on behavior. These results imply the existence of a mechanism that (a) estimates feedback contingency and (b) protects procedural learning from modification when feedback contingency is low (i.e., during random feedback). This article reports the results of an experiment in which increasing cognitive load via an explicit dual task during the random feedback period facilitated unlearning. This result is consistent with the hypothesis that the mechanism that protects procedural learning when feedback contingency is low depends on executive function. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

Pauses in cholinergic interneuron firing exert an inhibitory control on striatal output in vivo

The cholinergic interneurons (CINs) of the striatum are crucial for normal motor and behavioral functions of the basal ganglia. Striatal CINs exhibit tonic firing punctuated by distinct pauses. Pauses occur in response to motivationally significant events, but their function is unknown. Here we investigated the effects of pauses in CIN firing on spiny projection neurons (SPNs) – the output neurons of the striatum – using in vivo whole cell and juxtacellular recordings in mice. We found that optogenetically-induced pauses in CIN firing inhibited subthreshold membrane potential activity and decreased firing of SPNs. During pauses, SPN membrane potential fluctuations became more hyperpolarized and UP state durations became shorter. In addition, short-term plasticity of corticostriatal inputs was decreased during pauses. Our results indicate that, in vivo, the net effect of the pause in CIN firing on SPNs activity is inhibition and provide a novel mechanism for cholinergic control of striatal output.

Nerve cells or neurons communicate with one another using electrical impulses and chemical messengers called neurotransmitters. Additional molecules known as neuromodulators regulate the communication process. In contrast to neurotransmitters, neuromodulators do not send messages directly from one neuron to the next. Instead they change the way that neurons respond to neurotransmitters.

For example, the neuromodulator acetylcholine is most abundant in a region called the striatum. Located deep within the brain, the striatum contributes to learning and memory, motivation, and movement. Studies in rodents show that neurons within the striatum called cholinergic interneurons are almost continuously active. Each time these cells fire, they release acetylcholine. But whenever an animal experiences something unusual or important, the interneurons temporarily stop firing. Zucca et al. wanted to know whether these pauses in firing also act as a signal within the striatum.

To find out, Zucca et al. inserted a light-sensitive ion channel into cholinergic interneurons in the mouse striatum. Activating the ion channels with a laser beam stopped the interneurons from firing. Zucca et al. showed that these pauses in firing reduced the activity of another group of neurons, the spiny projection neurons. These are the major output neurons of the striatum. They send messages from the striatum to other parts of the brain. The results thus suggest that cholinergic interneurons signal notable events by temporarily blocking output from the striatum.

Understanding how cholinergic interneurons work will help reveal how the striatum drives behavior. It may also lead to treatments for diseases caused by cholinergic system dysfunction. Many patients with Parkinson’s disease or schizophrenia take medicines to block the effects of acetylcholine. Understanding how acetylcholine affects the striatum may help clarify how these treatments work.

The impact of category structure and training methodology on learning and generalizing within-category representations

When interacting with categories, representations focused on within-category relationships are often learned, but the conditions promoting within-category representations and their generalizability are unclear. We report the results of three experiments investigating the impact of category structure and training methodology on the learning and generalization of within-category representations (i.e., correlational structure). Participants were trained on either rule-based or information-integration structures using classification (Is the stimulus a member of Category A or Category B?), concept (e.g., Is the stimulus a member of Category A, Yes or No?), or inference (infer the missing component of the stimulus from a given category) and then tested on either an inference task (Experiments 1 and 2) or a classification task (Experiment 3). For the information-integration structure, within-category representations were consistently learned, could be generalized to novel stimuli, and could be generalized to support inference at test. For the rule-based structure, extended inference training resulted in generalization to novel stimuli (Experiment 2) and inference training resulted in generalization to classification (Experiment 3). These data help to clarify the conditions under which within-category representations can be learned. Moreover, these results make an important contribution in highlighting the impact of category structure and training methodology on the generalization of categorical knowledge.

Quantitative modeling of category learning deficits in various patient populations

OBJECTIVE

To provide a select review of our applications of quantitative modeling to highlight the utility of such approaches to better understand the neuropsychological deficits associated with various neurologic and psychiatric diseases.

METHOD

We review our work examining category learning in various patient populations, including individuals with basal ganglia disorders (Huntington's Disease and Parkinson's disease), amnesia and Eating Disorders.

RESULTS

Our review suggests that the use of quantitative models has enabled a better understanding of the learning deficits often observed in these conditions and has allowed us to form novel hypotheses about the neurobiological bases of their deficits.

CONCLUSIONS

We feel that the use of neurobiologically inspired quantitative modeling holds great promise in neuropsychological assessment and that future clinical measures should incorporate the use of such models as part of their standard scoring. Appropriate studies need to be completed, however, to determine whether such modeling techniques adhere to the rigorous psychometric properties necessary for a valid and reliable application in a clinical setting. (PsycINFO Database Record

Functional neurochemical imaging of the human striatal cholinergic system during reversal learning

Animal studies have shown that acetylcholine (ACh) levels in the dorsal striatum play a role in reversal learning. However, this has not been studied in humans due to a lack of appropriate non-invasive techniques. Proton magnetic resonance spectroscopy (¹ H-MRS) can be used to measure metabolite levels in humans in vivo. Although it cannot be used to study ACh directly, ¹ H-MRS can be used to study choline, an ACh precursor, which is linked to activity-dependent ACh release. The aim of this study was to use functional-¹ H-MRS (fMRS) to measure changes in choline levels in the human dorsal striatum during performance of a probabilistic reversal learning task. We demonstrate a task-dependent decrease in choline, specifically during reversal, but not initial, learning. We interpret this to reflect a sustained increase in ACh levels, which is in line with findings from the animal literature. This task-dependent change was specific to choline and was not observed in control metabolites. These findings provide support for the use of fMRS in the in vivo study of the human cholinergic system.

Reward from bugs to bipeds: a comparative approach to understanding how reward circuits function

In a complex environment, animals learn from their responses to stimuli and events. Appropriate response to reward and punishment can promote survival, reproduction and increase evolutionary fitness. Interestingly, the neural processes underlying these responses are remarkably similar across phyla. In all species, dopamine is central to encoding reward and directing motivated behaviors, however, a comprehensive understanding of how circuits encode reward and direct motivated behaviors is still lacking. In part, this is a result of the sheer diversity of neurons, the heterogeneity of their responses and the complexity of neural circuits within which they are found. We argue that general features of reward circuitry are common across model organisms, and thus principles learned from invertebrate model organisms can inform research across species. In particular, we discuss circuit motifs that appear to be functionally equivalent from flies to primates. We argue that a comparative approach to studying and understanding reward circuit function provides a more comprehensive understanding of reward circuitry, and informs disorders that affect the brain’s reward circuitry.

The effect of training methodology on knowledge representation in categorization

Category representations can be broadly classified as containing within-category information or between-category information. Although such representational differences can have a profound impact on decision-making, relatively little is known about the factors contributing to the development and generalizability of different types of category representations. These issues are addressed by investigating the impact of training methodology and category structures using a traditional empirical approach as well as the novel adaptation of computational modeling techniques from the machine learning literature. Experiment 1 focused on rule-based (RB) category structures thought to promote between-category representations. Participants learned two sets of two categories during training and were subsequently tested on a novel categorization problem using the training categories. Classification training resulted in a bias toward between-category representations whereas concept training resulted in a bias toward within-category representations. Experiment 2 focused on information-integration (II) category structures thought to promote within-category representations. With II structures, there was a bias toward within-category representations regardless of training methodology. Furthermore, in both experiments, computational modeling suggests that only within-category representations could support generalization during the test phase. These data suggest that within-category representations may be dominant and more robust for supporting the reconfiguration of current knowledge to support generalization.

Cholinergic Interneurons Use Orbitofrontal Input to Track Beliefs about Current State

When conditions change, organisms need to learn about the changed conditions without interfering with what they already know. To do so, they can assign the new learning to a new "state" and the old learning to a previous state. This state assignment is fundamental to behavioral flexibility. Cholinergic interneurons (CINs) in the dorsomedial striatum (DMS) are necessary for associative information to be compartmentalized in this way, but the mechanism by which they do so is unknown. Here we addressed this question by recording putative CINs from the DMS in rats performing a task consisting of a series of trial blocks, or states, that required the recall and application of contradictory associative information. We found that individual CINs in the DMS represented the current state throughout each trial. These state correlates were not observed in dorsolateral striatal CINs recorded in the same rats. Notably, DMS CIN ensembles tracked rats' beliefs about the current state such that, when states were miscoded, rats tended to make suboptimal choices reflecting the miscoding. State information held by the DMS CINs also depended completely on the orbitofrontal cortex, an area that has been proposed to signal environmental states. These results suggest that CINs set the stage for recalling associative information relevant to the current environment by maintaining a real-time representation of the current state. Such a role has novel implications for understanding the neural basis of a variety of psychiatric diseases, such as addiction or anxiety disorders, in which patients generalize inappropriately (or fail to generalize) between different environments.

SIGNIFICANCE STATEMENT

Striatal cholinergic interneurons (CINs) are thought to be identical to tonically active neurons. These neurons have long been thought to have an important influence on striatal processing during reward-related learning. Recently, a more specific function for striatal CINs has been suggested, which is that they are necessary for striatal learning to be compartmentalized into different states as the state of the environment changes. Here we report that putative CINs appear to track rats' beliefs about which environmental state is current. We further show that this property of CINs depends on orbitofrontal cortex input and is correlated with choices made by rats. These findings could provide new insight into neuropsychiatric diseases that involve improper generalization between different contexts.

A neural interpretation of exemplar theory

Exemplar theory assumes that people categorize a novel object by comparing its similarity to the memory representations of all previous exemplars from each relevant category. Exemplar theory has been the most prominent cognitive theory of categorization for more than 30 years. Despite its considerable success in providing good quantitative fits to a wide variety of accuracy data, it has never had a detailed neurobiological interpretation. This article proposes a neural interpretation of exemplar theory in which category learning is mediated by synaptic plasticity at cortical-striatal synapses. In this model, categorization training does not create new memory representations, rather it alters connectivity between striatal neurons and neurons in sensory association cortex. The new model makes identical quantitative predictions as exemplar theory, yet it can account for many empirical phenomena that are either incompatible with or outside the scope of the cognitive version of exemplar theory. (PsycINFO Database Record

Perceptual category learning and visual processing: An exercise in computational cognitive neuroscience

The field of computational cognitive neuroscience (CCN) builds and tests neurobiologically detailed computational models that account for both behavioral and neuroscience data. This article leverages a key advantage of CCN-namely, that it should be possible to interface different CCN models in a plug-and-play fashion-to produce a new and biologically detailed model of perceptual category learning. The new model was created from two existing CCN models: the HMAX model of visual object processing and the COVIS model of category learning. Using bitmap images as inputs and by adjusting only a couple of learning-rate parameters, the new HMAX/COVIS model provides impressively good fits to human category-learning data from two qualitatively different experiments that used different types of category structures and different types of visual stimuli. Overall, the model provides a comprehensive neural and behavioral account of basal ganglia-mediated learning.

Hierarchical control of procedural and declarative category-learning systems

Substantial evidence suggests that human category learning is governed by the interaction of multiple qualitatively distinct neural systems. In this view, procedural memory is used to learn stimulus-response associations, and declarative memory is used to apply explicit rules and test hypotheses about category membership. However, much less is known about the interaction between these systems: how is control passed between systems as they interact to influence motor resources? Here, we used fMRI to elucidate the neural correlates of switching between procedural and declarative categorization systems. We identified a key region of the cerebellum (left Crus I) whose activity was bidirectionally modulated depending on switch direction. We also identified regions of the default mode network (DMN) that were selectively connected to left Crus I during switching. We propose that the cerebellum-in coordination with the DMN-serves a critical role in passing control between procedural and declarative memory systems.

Performance Pressure Enhances Speech Learning

Real-world speech learning often occurs in high pressure situations such as trying to communicate in a foreign country. However, the impact of pressure on speech learning success is largely unexplored. In this study, adult, native speakers of English learned non-native speech categories under pressure or no-pressure conditions. In the pressure conditions, participants were informed that they were paired with a (fictitious) partner, and that each had to independently exceed a performance criterion for both to receive a monetary bonus. They were then informed that their partner had exceeded the bonus and the fate of both bonuses depended upon the participant's performance. Our results demonstrate that pressure significantly enhanced speech learning success. In addition, neurobiologically-inspired computational modeling revealed that the performance advantage was due to faster and more frequent use of procedural learning strategies. These results integrate two well-studied research domains and suggest a facilitatory role of motivational factors in speech learning performance that may not be captured in traditional training paradigms.

Dopamine dependence in aggregate feedback learning: A computational cognitive neuroscience approach

Procedural learning of skills depends on dopamine-mediated striatal plasticity. Most prior work investigated single stimulus-response procedural learning followed by feedback. However, many skills include several actions that must be performed before feedback is available. A new procedural-learning task is developed in which three independent and successive unsupervised categorization responses receive aggregate feedback indicating either that all three responses were correct, or at least one response was incorrect. Experiment 1 showed superior learning of stimuli in position 3, and that learning in the first two positions was initially compromised, and then recovered. An extensive theoretical analysis that used parameter space partitioning found that a large class of procedural-learning models, which predict propagation of dopamine release from feedback to stimuli, and/or an eligibility trace, fail to fully account for these data. The analysis also suggested that any dopamine released to the second or third stimulus impaired categorization learning in the first and second positions. A second experiment tested and confirmed a novel prediction of this large class of procedural-learning models that if the to-be-learned actions are introduced one-by-one in succession then learning is much better if training begins with the first action (and works forwards) than if it begins with the last action (and works backwards).

Multiple stages of learning in perceptual categorization: evidence and neurocomputational theory

Virtually all current theories of category learning assume that humans learn new categories by gradually forming associations directly between stimuli and responses. In information-integration category-learning tasks, this purported process is thought to depend on procedural learning implemented via dopamine-dependent cortical-striatal synaptic plasticity. This article proposes a new, neurobiologically detailed model of procedural category learning that, unlike previous models, does not assume associations are made directly from stimulus to response. Rather, the traditional stimulus-response (S-R) models are replaced with a two-stage learning process. Multiple streams of evidence (behavioral, as well as anatomical and fMRI) are used as inspiration for the new model, which synthesizes evidence of multiple distinct cortical-striatal loops into a neurocomputational theory. An experiment is reported to test a priori predictions of the new model that: (1) recovery from a full reversal should be easier than learning new categories equated for difficulty, and (2) reversal learning in procedural tasks is mediated within the striatum via dopamine-dependent synaptic plasticity. The results confirm the predictions of the new two-stage model and are incompatible with existing S-R models.

A Mathematical Model of Levodopa Medication Effect on Basal Ganglia in Parkinson's Disease: An Application to the Alternate Finger Tapping Task

Malfunctions in the neural circuitry of the basal ganglia (BG), induced by alterations in the dopaminergic system, are responsible for an array of motor disorders and milder cognitive issues in Parkinson's disease (PD). Recently Baston and Ursino (2015a) presented a new neuroscience mathematical model aimed at exploring the role of basal ganglia in action selection. The model is biologically inspired and reproduces the main BG structures and pathways, modeling explicitly both the dopaminergic and the cholinergic system. The present work aims at interfacing this neurocomputational model with a compartmental model of levodopa, to propose a general model of medicated Parkinson's disease. Levodopa effect on the striatum was simulated with a two-compartment model of pharmacokinetics in plasma joined with a motor effect compartment. The latter is characterized by the levodopa removal rate and by a sigmoidal relationship (Hill law) between concentration and effect. The main parameters of this relationship are saturation, steepness, and the half-maximum concentration. The effect of levodopa is then summed to a term representing the endogenous dopamine effect, and is used as an external input for the neurocomputation model; this allows both the temporal aspects of medication and the individual patient characteristics to be simulated. The frequency of alternate tapping is then used as the outcome of the whole model, to simulate effective clinical scores. Pharmacokinetic-pharmacodynamic modeling was preliminary performed on data of six patients with Parkinson's disease (both "stable" and "wearing-off" responders) after levodopa standardized oral dosing over 4 h. Results show that the model is able to reproduce the temporal profiles of levodopa in plasma and the finger tapping frequency in all patients, discriminating between different patterns of levodopa motor response. The more influential parameters are the Hill coefficient, related with the slope of the effect sigmoidal relationship, the drug concentration at half-maximum effect, and the drug removal rate from the effect compartment. The model can be of value to gain a deeper understanding on the pharmacokinetics and pharmacodynamics of the medication, and on the way dopamine is exploited in the neural circuitry of the basal ganglia in patients at different stages of the disease progression.

A cholinergic feedback circuit to regulate striatal population uncertainty and optimize reinforcement learning

Convergent evidence suggests that the basal ganglia support reinforcement learning by adjusting action values according to reward prediction errors. However, adaptive behavior in stochastic environments requires the consideration of uncertainty to dynamically adjust the learning rate. We consider how cholinergic tonically active interneurons (TANs) may endow the striatum with such a mechanism in computational models spanning three Marr's levels of analysis. In the neural model, TANs modulate the excitability of spiny neurons, their population response to reinforcement, and hence the effective learning rate. Long TAN pauses facilitated robustness to spurious outcomes by increasing divergence in synaptic weights between neurons coding for alternative action values, whereas short TAN pauses facilitated stochastic behavior but increased responsiveness to change-points in outcome contingencies. A feedback control system allowed TAN pauses to be dynamically modulated by uncertainty across the spiny neuron population, allowing the system to self-tune and optimize performance across stochastic environments.

A Biologically Inspired Computational Model of Basal Ganglia in Action Selection

The basal ganglia (BG) are a subcortical structure implicated in action selection. The aim of this work is to present a new cognitive neuroscience model of the BG, which aspires to represent a parsimonious balance between simplicity and completeness. The model includes the 3 main pathways operating in the BG circuitry, that is, the direct (Go), indirect (NoGo), and hyperdirect pathways. The main original aspects, compared with previous models, are the use of a two-term Hebb rule to train synapses in the striatum, based exclusively on neuronal activity changes caused by dopamine peaks or dips, and the role of the cholinergic interneurons (affected by dopamine themselves) during learning. Some examples are displayed, concerning a few paradigmatic cases: action selection in basal conditions, action selection in the presence of a strong conflict (where the role of the hyperdirect pathway emerges), synapse changes induced by phasic dopamine, and learning new actions based on a previous history of rewards and punishments. Finally, some simulations show model working in conditions of altered dopamine levels, to illustrate pathological cases (dopamine depletion in parkinsonian subjects or dopamine hypermedication). Due to its parsimonious approach, the model may represent a straightforward tool to analyze BG functionality in behavioral experiments.

Expanding the role of striatal cholinergic interneurons and the midbrain dopamine system in appetitive instrumental conditioning

The basal ganglia are a collection of subcortical nuclei thought to underlie a wide variety of vertebrate behavior. Although a great deal is known about the functional and physiological properties of the basal ganglia, relatively few models have been formally developed that have been tested against both behavioral and physiological data. Our previous work (Ashby FG, Crossley MJ. J Cogn Neurosci 23: 1549-1566, 2011) showed that a model grounded in the neurobiology of the basal ganglia could account for basic single-neuron recording data, as well as behavioral phenomena such as fast reacquisition that constrain models of conditioning. In this article we show that this same model accounts for a variety of appetitive instrumental conditioning phenomena, including the partial reinforcement extinction (PRE) effect, rapid and slowed reacquisition following extinction, and renewal of previously extinguished instrumental responses by environmental context cues.

Criterion learning in rule-based categorization: simulation of neural mechanism and new data

In perceptual categorization, rule selection consists of selecting one or several stimulus-dimensions to be used to categorize the stimuli (e.g., categorize lines according to their length). Once a rule has been selected, criterion learning consists of defining how stimuli will be grouped using the selected dimension(s) (e.g., if the selected rule is line length, define 'long' and 'short'). Very little is known about the neuroscience of criterion learning, and most existing computational models do not provide a biological mechanism for this process. In this article, we introduce a new model of rule learning called Heterosynaptic Inhibitory Criterion Learning (HICL). HICL includes a biologically-based explanation of criterion learning, and we use new category-learning data to test key aspects of the model. In HICL, rule selective cells in prefrontal cortex modulate stimulus-response associations using pre-synaptic inhibition. Criterion learning is implemented by a new type of heterosynaptic error-driven Hebbian learning at inhibitory synapses that uses feedback to drive cell activation above/below thresholds representing ionic gating mechanisms. The model is used to account for new human categorization data from two experiments showing that: (1) changing rule criterion on a given dimension is easier if irrelevant dimensions are also changing (Experiment 1), and (2) showing that changing the relevant rule dimension and learning a new criterion is more difficult, but also facilitated by a change in the irrelevant dimension (Experiment 2). We conclude with a discussion of some of HICL's implications for future research on rule learning.

Context-dependent savings in procedural category learning

Environmental context can have a profound influence on the efficacy of intervention protocols designed to eliminate undesirable behaviors. This is clearly seen in drug rehabilitation clinics where patients often relapse soon after leaving the context of the treatment facility. A similar pattern is commonly observed in controlled laboratory studies of context-dependent savings in instrumental conditioning, where simply placing an animal back into the original conditioning chamber can renew an extinguished instrumental response. Surprisingly, context-dependent savings in human procedural learning has not been carefully examined in the laboratory. Here, we provide the first known empirical demonstration of context-dependent savings in a perceptual categorization task known to recruit procedural learning. We also present a computational account of these savings using a biologically detailed model in which a key role is played by cholinergic interneurons in the striatum.

Toward a dual-learning systems model of speech category learning

More than two decades of work in vision posits the existence of dual-learning systems of category learning. The reflective system uses working memory to develop and test rules for classifying in an explicit fashion, while the reflexive system operates by implicitly associating perception with actions that lead to reinforcement. Dual-learning systems models hypothesize that in learning natural categories, learners initially use the reflective system and, with practice, transfer control to the reflexive system. The role of reflective and reflexive systems in auditory category learning and more specifically in speech category learning has not been systematically examined. In this article, we describe a neurobiologically constrained dual-learning systems theoretical framework that is currently being developed in speech category learning and review recent applications of this framework. Using behavioral and computational modeling approaches, we provide evidence that speech category learning is predominantly mediated by the reflexive learning system. In one application, we explore the effects of normal aging on non-speech and speech category learning. Prominently, we find a large age-related deficit in speech learning. The computational modeling suggests that older adults are less likely to transition from simple, reflective, unidimensional rules to more complex, reflexive, multi-dimensional rules. In a second application, we summarize a recent study examining auditory category learning in individuals with elevated depressive symptoms. We find a deficit in reflective-optimal and an enhancement in reflexive-optimal auditory category learning. Interestingly, individuals with elevated depressive symptoms also show an advantage in learning speech categories. We end with a brief summary and description of a number of future directions.

A computational model of the temporal dynamics of plasticity in procedural learning: sensitivity to feedback timing

The evidence is now good that different memory systems mediate the learning of different types of category structures. In particular, declarative memory dominates rule-based (RB) category learning and procedural memory dominates information-integration (II) category learning. For example, several studies have reported that feedback timing is critical for II category learning, but not for RB category learning—results that have broad support within the memory systems literature. Specifically, II category learning has been shown to be best with feedback delays of 500 ms compared to delays of 0 and 1000 ms, and highly impaired with delays of 2.5 s or longer. In contrast, RB learning is unaffected by any feedback delay up to 10 s. We propose a neurobiologically detailed theory of procedural learning that is sensitive to different feedback delays. The theory assumes that procedural learning is mediated by plasticity at cortical-striatal synapses that are modified by dopamine-mediated reinforcement learning. The model captures the time-course of the biochemical events in the striatum that cause synaptic plasticity, and thereby accounts for the empirical effects of various feedback delays on II category learning.

Procedural-based category learning in patients with Parkinson's disease: impact of category number and category continuity

Previously we found that Parkinson's disease (PD) patients are impaired in procedural-based category learning when category membership is defined by a nonlinear relationship between stimulus dimensions, but these same patients are normal when the rule is defined by a linear relationship (Maddox and Filoteo, 2001; Filoteo et al., 2005a,b). We suggested that PD patients' impairment was due to a deficit in recruiting “striatal units” to represent complex nonlinear rules. In the present study, we further examined the nature of PD patients' procedural-based deficit in two experiments designed to examine the impact of (1) the number of categories, and (2) category discontinuity on learning. Results indicated that PD patients were impaired only under discontinuous category conditions but were normal when the number of categories was increased from two to four. The lack of impairment in the four-category condition suggests normal integrity of striatal medium spiny cells involved in procedural-based category learning. In contrast, and consistent with our previous observation of a nonlinear deficit, the finding that PD patients were impaired in the discontinuous condition suggests that these patients are impaired when they have to associate perceptually distinct exemplars with the same category. Theoretically, this deficit might be related to dysfunctional communication among medium spiny neurons within the striatum, particularly given that these are cholinergic neurons and a cholinergic deficiency could underlie some of PD patients' cognitive impairment.

Talker-specific learning in amnesia: Insight into mechanisms of adaptive speech perception

A hallmark of human speech perception is the ability to comprehend speech quickly and effortlessly despite enormous variability across talkers. However, current theories of speech perception do not make specific claims about the memory mechanisms involved in this process. To examine whether declarative memory is necessary for talker-specific learning, we tested the ability of amnesic patients with severe declarative memory deficits to learn and distinguish the accents of two unfamiliar talkers by monitoring their eye-gaze as they followed spoken instructions. Analyses of the time-course of eye fixations showed that amnesic patients rapidly learned to distinguish these accents and tailored perceptual processes to the voice of each talker. These results demonstrate that declarative memory is not necessary for this ability and points to the involvement of non-declarative memory mechanisms. These results are consistent with findings that other social and accommodative behaviors are preserved in amnesia and contribute to our understanding of the interactions of multiple memory systems in the use and understanding of spoken language.

Computational models of basal-ganglia pathway functions: focus on functional neuroanatomy

Over the past 15 years, computational models have had a considerable impact on basal-ganglia research. Most of these models implement multiple distinct basal-ganglia pathways and assume them to fulfill different functions. As there is now a multitude of different models, it has become complex to keep track of their various, sometimes just marginally different assumptions on pathway functions. Moreover, it has become a challenge to oversee to what extent individual assumptions are corroborated or challenged by empirical data. Focusing on computational, but also considering non-computational models, we review influential concepts of pathway functions and show to what extent they are compatible with or contradict each other. Moreover, we outline how empirical evidence favors or challenges specific model assumptions and propose experiments that allow testing assumptions against each other.

Exploring the cognitive and motor functions of the basal ganglia: an integrative review of computational cognitive neuroscience models

Many computational models of the basal ganglia (BG) have been proposed over the past twenty-five years. While computational neuroscience models have focused on closely matching the neurobiology of the BG, computational cognitive neuroscience (CCN) models have focused on how the BG can be used to implement cognitive and motor functions. This review article focuses on CCN models of the BG and how they use the neuroanatomy of the BG to account for cognitive and motor functions such as categorization, instrumental conditioning, probabilistic learning, working memory, sequence learning, automaticity, reaching, handwriting, and eye saccades. A total of 19 BG models accounting for one or more of these functions are reviewed and compared. The review concludes with a discussion of the limitations of existing CCN models of the BG and prescriptions for future modeling, including the need for computational models of the BG that can simultaneously account for cognitive and motor functions, and the need for a more complete specification of the role of the BG in behavioral functions.

A matched filter hypothesis for cognitive control

The prefrontal cortex exerts top-down influences on several aspects of higher-order cognition by functioning as a filtering mechanism that biases bottom-up sensory information toward a response that is optimal in context. However, research also indicates that not all aspects of complex cognition benefit from prefrontal regulation. Here we review and synthesize this research with an emphasis on the domains of learning and creative cognition, and outline how the appropriate level of cognitive control in a given situation can vary depending on the organism's goals and the characteristics of the given task. We offer a matched filter hypothesis for cognitive control, which proposes that the optimal level of cognitive control is task-dependent, with high levels of cognitive control best suited to tasks that are explicit, rule-based, verbal or abstract, and can be accomplished given the capacity limits of working memory and with low levels of cognitive control best suited to tasks that are implicit, reward-based, non-verbal or intuitive, and which can be accomplished irrespective of working memory limitations. Our approach promotes a view of cognitive control as a tool adapted to a subset of common challenges, rather than an all-purpose optimization system suited to every problem the organism might encounter.

Glycogen synthase kinase-3 reduces acetylcholine level in striatum via disturbing cellular distribution of choline acetyltransferase in cholinergic interneurons in rats

Cholinergic interneurons, which provide the main source of acetylcholine (ACh) in the striatum, control the striatal local circuits and deeply involve in the pathogenesis of neurodegenerative diseases. Glycogen synthase kinase-3 (GSK-3) is a crucial kinase with diverse fundamental functions and accepted that deregulation of GSK-3 activity also plays important roles in diverse neurodegenerative diseases. However, up to now, there is no direct proof indicating whether GSK-3 activation is responsible for cholinergic dysfunction. In the present study, with combined intracerebroventricular injection of Wortmannin and GF-109203X, we activated GSK-3 and demonstrated the increased phosphorylation level of microtubule-associated protein tau and neurofilaments (NFs) in the rat striatum. The activated GSK-3 consequently decreased ACh level in the striatum as a result of the reduction of choline acetyltransferase (ChAT) activity. The alteration of ChAT activity was due to impaired ChAT distribution rather than its expression. Furthermore, we proved that cellular ChAT distribution was dependent on low phosphorylation level of NFs. Nevertheless, the cholinergic dysfunction in the striatum failed to induce significant neuronal number reduction. In summary, our data demonstrates the link between GSK-3 activation and cholinergic dysfunction in the striatum and provided beneficial evidence for the pathogenesis study of relevant neurodegenerative diseases.

Erasing the engram: the unlearning of procedural skills

Huge amounts of money are spent every year on unlearning programs--in drug-treatment facilities, prisons, psychotherapy clinics, and schools. Yet almost all of these programs fail, since recidivism rates are high in each of these fields. Progress on this problem requires a better understanding of the mechanisms that make unlearning so difficult. Much cognitive neuroscience evidence suggests that an important component of these mechanisms also dictates success on categorization tasks that recruit procedural learning and depend on synaptic plasticity within the striatum. A biologically detailed computational model of this striatal-dependent learning is described (based on Ashby & Crossley, 2011). The model assumes that a key component of striatal-dependent learning is provided by interneurons in the striatum called the tonically active neurons (TANs), which act as a gate for the learning and expression of striatal-dependent behaviors. In their tonically active state, the TANs prevent the expression of any striatal-dependent behavior. However, they learn to pause in rewarding environments and thereby permit the learning and expression of striatal-dependent behaviors. The model predicts that when rewards are no longer contingent on behavior, the TANs cease to pause, which protects striatal learning from decay and prevents unlearning. In addition, the model predicts that when rewards are partially contingent on behavior, the TANs remain partially paused, leaving the striatum available for unlearning. The results from 3 human behavioral studies support the model predictions and suggest a novel unlearning protocol that shows promising initial signs of success.

The basal ganglia

Through its connections with widespread cortical areas and with dopaminergic midbrain areas, the basal ganglia are well situated to integrate patterns of cortical input with the dopaminergic reward signal originating in the midbrain. In this review, we consider the functions of the basal ganglia in relation to its gross and cellular anatomy, and discuss how these mechanisms subserve the thresholding and selection of motor and cognitive processes. We also discuss how the dopaminergic reward signal enables flexible task learning through modulation of striatal plasticity, and how reinforcement learning models have been used to account for various aspects of basal ganglia activity. Specifically, we will discuss the important role of the basal ganglia in instrumental learning, cognitive control, sequence learning, and categorization tasks. Finally, we will discuss the neurobiological and cognitive characteristics of Parkinson's disease, Huntington's disease and addiction to illustrate the relationship between the basal ganglia and cognitive function. WIREs Cogn Sci 2013, 4:135-148. doi: 10.1002/wcs.1217 For further resources related to this article, please visit the WIREs website.

A model of amygdala-hippocampal-prefrontal interaction in fear conditioning and extinction in animals

Empirical research has shown that the amygdala, hippocampus, and ventromedial prefrontal cortex (vmPFC) are involved in fear conditioning. However, the functional contribution of each brain area and the nature of their interactions are not clearly understood. Here, we extend existing neural network models of the functional roles of the hippocampus in classical conditioning to include interactions with the amygdala and prefrontal cortex. We apply the model to fear conditioning, in which animals learn physiological (e.g. heart rate) and behavioral (e.g. freezing) responses to stimuli that have been paired with a highly aversive event (e.g. electrical shock). The key feature of our model is that learning of these conditioned responses in the central nucleus of the amygdala is modulated by two separate processes, one from basolateral amygdala and signaling a positive prediction error, and one from the vmPFC, via the intercalated cells of the amygdala, and signaling a negative prediction error. In addition, we propose that hippocampal input to both vmPFC and basolateral amygdala is essential for contextual modulation of fear acquisition and extinction. The model is sufficient to account for a body of data from various animal fear conditioning paradigms, including acquisition, extinction, reacquisition, and context specificity effects. Consistent with studies on lesioned animals, our model shows that damage to the vmPFC impairs extinction, while damage to the hippocampus impairs extinction in a different context (e.g., a different conditioning chamber from that used in initial training in animal experiments). We also discuss model limitations and predictions, including the effects of number of training trials on fear conditioning.

A neurocomputational account of cognitive deficits in Parkinson's disease

Parkinson's disease (PD) is caused by the accelerated death of dopamine (DA) producing neurons. Numerous studies documenting cognitive deficits of PD patients have revealed impairments in a variety of tasks related to memory, learning, visuospatial skills, and attention. While there have been several studies documenting cognitive deficits of PD patients, very few computational models have been proposed. In this article, we use the COVIS model of category learning to simulate DA depletion and show that the model suffers from cognitive symptoms similar to those of human participants affected by PD. Specifically, DA depletion in COVIS produced deficits in rule-based categorization, non-linear information-integration categorization, probabilistic classification, rule maintenance, and rule switching. These were observed by simulating results from younger controls, older controls, PD patients, and severe PD patients in five well-known tasks. Differential performance among the different age groups and clinical populations was modeled simply by changing the amount of DA available in the model. This suggests that COVIS may not only be an adequate model of the simulated tasks and phenomena but also more generally of the role of DA in these tasks and phenomena.

Simulating the effects of dopamine imbalance on cognition: from positive affect to Parkinson's disease

Cools (2006) suggested that prefrontal dopamine levels are related to cognitive stability whereas striatal dopamine levels are related to cognitive plasticity. With such a wide ranging role, almost all cognitive activities should be affected by dopamine levels in the brain. Not surprisingly, factors influencing brain dopamine levels have been shown to improve/worsen performance in many behavioral experiments. On the one hand, Nadler, Rabi, and Minda (2010) showed that positive affect (which is thought to increase cortical dopamine levels) improves a type of categorization that depends on explicit reasoning (rule-based) but not another type that depends on procedural learning (information-integration). On the other hand, Parkinson's disease (which is known to decrease dopamine levels in both the striatum and cortex) produces proactive interference in the odd-man-out task (Flowers & Robertson, 1985) and renders subjects insensitive to negative feedback during reversal learning (Cools, Altamirano, & D'Esposito, 2006). This article uses the COVIS model of categorization to simulate the effects of different dopamine levels in categorization, reversal learning, and the odd-man-out task. The results show a good match between the simulated and human data, which suggests that the role of dopamine in COVIS can account for several cognitive enhancements and deficits related to dopamine levels in healthy and patient populations.

Acetylcholine-based entropy in response selection: a model of how striatal interneurons modulate exploration, exploitation, and response variability in decision-making

The basal ganglia play a fundamental role in decision-making. Their contribution is typically modeled within a reinforcement learning framework, with the basal ganglia learning to select the options associated with highest value and their dopamine inputs conveying performance feedback. This basic framework, however, does not account for the role of cholinergic interneurons in the striatum, and does not easily explain certain dynamic aspects of decision-making and skill acquisition like the generation of exploratory actions. This paper describes basal ganglia acetylcholine-based entropy (BABE), a model of the acetylcholine system in the striatum that provides a unified explanation for these phenomena. According to this model, cholinergic interneurons in the striatum control the level of variability in behavior by modulating the number of possible responses that are considered by the basal ganglia, as well as the level of competition between them. This mechanism provides a natural way to account for the role of basal ganglia in generating behavioral variability during the acquisition of certain cognitive skills, as well as for modulating exploration and exploitation in decision-making. Compared to a typical reinforcement learning model, BABE showed a greater modulation of response variability in the face of changes in the reward contingences, allowing for faster learning (and re-learning) of option values. Finally, the paper discusses the possible applications of the model to other domains.

The Neurodynamics of Cognition: A Tutorial on Computational Cognitive Neuroscience

Computational Cognitive Neuroscience (CCN) is a new field that lies at the intersection of computational neuroscience, machine learning, and neural network theory (i.e., connectionism). The ideal CCN model should not make any assumptions that are known to contradict the current neuroscience literature and at the same time provide good accounts of behavior and at least some neuroscience data (e.g., single-neuron activity, fMRI data). Furthermore, once set, the architecture of the CCN network and the models of each individual unit should remain fixed throughout all applications. Because of the greater weight they place on biological accuracy, CCN models differ substantially from traditional neural network models in how each individual unit is modeled, how learning is modeled, and how behavior is generated from the network. A variety of CCN solutions to these three problems are described. A real example of this approach is described, and some advantages and limitations of the CCN approach are discussed.

Shifting responsibly: the importance of striatal modularity to reinforcement learning in uncertain environments

We propose here that the modular organization of the striatum reflects a context-sensitive modular learning architecture in which clustered striosome-matrisome domains participate in modular reinforcement learning (RL). Based on anatomical and physiological evidence, it has been suggested that the modular organization of the striatum could represent a learning architecture. There is not, however, a coherent view of how such a learning architecture could relate to the organization of striatal outputs into the direct and indirect pathways of the basal ganglia, nor a clear formulation of how such a modular architecture relates to the RL functions attributed to the striatum. Here, we hypothesize that striosome-matrisome modules not only learn to bias behavior toward specific actions, as in standard RL, but also learn to assess their own relevance to the environmental context and modulate their own learning and activity on this basis. We further hypothesize that the contextual relevance or "responsibility" of modules is determined by errors in predictions of environmental features and that such responsibility is assigned by striosomes and conveyed to matrisomes via local circuit interneurons. To examine these hypotheses and to identify the general requirements for realizing this architecture in the nervous system, we developed a simple modular RL model. We then constructed a network model of basal ganglia circuitry that includes these modules and the direct and indirect pathways. Based on simple assumptions, this model suggests that while the direct pathway may promote actions based on striatal action values, the indirect pathway may act as a gating network that facilitates or suppresses behavioral modules on the basis of striatal responsibility signals. Our modeling functionally unites the modular compartmental organization of the striatum with the direct-indirect pathway divisions of the basal ganglia, a step that we suggest will have important clinical implications.