A Dual Role Hypothesis of the Cortico-Basal-Ganglia Pathways: Opponency and Temporal Difference Through Dopamine and Adenosine

The mechanistic effects of acupuncture in rodent neurodegenerative disease models: a literature review

Neurodegenerative diseases refer to a battery of medical conditions that affect the survival and function of neurons in the brain, which are mainly presented with progressive loss of cognitive and/or motor function. Acupuncture showed benign effects in improving neurological deficits, especially on movement and cognitive function impairment. Here, we reviewed the therapeutic mechanisms of acupuncture at the neural circuit level in movement and cognition disorders, summarizing the influence of acupuncture in the dopaminergic system, glutamatergic system, γ-amino butyric acid-ergic (GABAergic) system, serotonergic system, cholinergic system, and glial cells at the circuit and synaptic levels. These findings can provide targets for clinical treatment and perspectives for further studies.

Enhancing reinforcement learning models by including direct and indirect pathways improves performance on striatal dependent tasks

A major advance in understanding learning behavior stems from experiments showing that reward learning requires dopamine inputs to striatal neurons and arises from synaptic plasticity of cortico-striatal synapses. Numerous reinforcement learning models mimic this dopamine-dependent synaptic plasticity by using the reward prediction error, which resembles dopamine neuron firing, to learn the best action in response to a set of cues. Though these models can explain many facets of behavior, reproducing some types of goal-directed behavior, such as renewal and reversal, require additional model components. Here we present a reinforcement learning model, TD2Q, which better corresponds to the basal ganglia with two Q matrices, one representing direct pathway neurons (G) and another representing indirect pathway neurons (N). Unlike previous two-Q architectures, a novel and critical aspect of TD2Q is to update the G and N matrices utilizing the temporal difference reward prediction error. A best action is selected for N and G using a softmax with a reward-dependent adaptive exploration parameter, and then differences are resolved using a second selection step applied to the two action probabilities. The model is tested on a range of multi-step tasks including extinction, renewal, discrimination; switching reward probability learning; and sequence learning. Simulations show that TD2Q produces behaviors similar to rodents in choice and sequence learning tasks, and that use of the temporal difference reward prediction error is required to learn multi-step tasks. Blocking the update rule on the N matrix blocks discrimination learning, as observed experimentally. Performance in the sequence learning task is dramatically improved with two matrices. These results suggest that including additional aspects of basal ganglia physiology can improve the performance of reinforcement learning models, better reproduce animal behaviors, and provide insight as to the role of direct- and indirect-pathway striatal neurons.

Opponent learning with different representations in the cortico-basal ganglia pathways can develop obsession-compulsion cycle

Obsessive-compulsive disorder (OCD) has been suggested to be associated with impairment of model-based behavioral control. Meanwhile, recent work suggested shorter memory trace for negative than positive prediction errors (PEs) in OCD. We explored relations between these two suggestions through computational modeling. Based on the properties of cortico-basal ganglia pathways, we modeled human as an agent having a combination of successor representation (SR)-based system that enables model-based-like control and individual representation (IR)-based system that only hosts model-free control, with the two systems potentially learning from positive and negative PEs in different rates. We simulated the agent's behavior in the environmental model used in the recent work that describes potential development of obsession-compulsion cycle. We found that the dual-system agent could develop enhanced obsession-compulsion cycle, similarly to the agent having memory trace imbalance in the recent work, if the SR- and IR-based systems learned mainly from positive and negative PEs, respectively. We then simulated the behavior of such an opponent SR+IR agent in the two-stage decision task, in comparison with the agent having only SR-based control. Fitting of the agents' behavior by the model weighing model-based and model-free control developed in the original two-stage task study resulted in smaller weights of model-based control for the opponent SR+IR agent than for the SR-only agent. These results reconcile the previous suggestions about OCD, i.e., impaired model-based control and memory trace imbalance, raising a novel possibility that opponent learning in model(SR)-based and model-free controllers underlies obsession-compulsion. Our model cannot explain the behavior of OCD patients in punishment, rather than reward, contexts, but it could be resolved if opponent SR+IR learning operates also in the recently revealed non-canonical cortico-basal ganglia-dopamine circuit for threat/aversiveness, rather than reward, reinforcement learning, and the aversive SR + appetitive IR agent could actually develop obsession-compulsion if the environment is modeled differently.

Opponent Learning with Different Representations in the Cortico-Basal Ganglia Circuits

The direct and indirect pathways of the basal ganglia (BG) have been suggested to learn mainly from positive and negative feedbacks, respectively. Since these pathways unevenly receive inputs from different cortical neuron types and/or regions, they may preferentially use different state/action representations. We explored whether such a combined use of different representations, coupled with different learning rates from positive and negative reward prediction errors (RPEs), has computational benefits. We modeled animal as an agent equipped with two learning systems, each of which adopted individual representation (IR) or successor representation (SR) of states. With varying the combination of IR or SR and also the learning rates from positive and negative RPEs in each system, we examined how the agent performed in a dynamic reward navigation task. We found that combination of SR-based system learning mainly from positive RPEs and IR-based system learning mainly from negative RPEs could achieve a good performance in the task, as compared with other combinations. In such a combination of appetitive SR-based and aversive IR-based systems, both systems show activities of comparable magnitudes with opposite signs, consistent with the suggested profiles of the two BG pathways. Moreover, the architecture of such a combination provides a novel coherent explanation for the functional significance and underlying mechanism of diverse findings about the cortico-BG circuits. These results suggest that particularly combining different representations with appetitive and aversive learning could be an effective learning strategy in certain dynamic environments, and it might actually be implemented in the cortico-BG circuits.

Astrocyte-neuron interaction in the dorsal striatum-pallidal circuits and alcohol-seeking behaviors

In the striatum, two main types of GABAergic medium spiny neurons (MSNs), denoted striatonigral (or direct-pathway MSNs, dMSNs) and striatopallidal neurons (indirect-pathway MSNs, iMSNs), form circuits with distinct pallidal nuclei, which sends "GO" or "NO-GO" signals through the thalamus. These striatopallidal circuits evaluate and execute reward-seeking and taking behaviors. Especially, the dorsal striatum can be further divided into the dorsomedial striatum (DMS, equivalent to caudate in primates and humans) and dorsolateral striatum (DLS, equivalent to putamen), which orchestrates goal-directed and habitual reward-seeking and taking behaviors, respectively. Using optogenetics, chemogenetics and in vivo calcium imaging technologies combined with electrophysiology and digitalized behavior phenotyping, recent studies have revealed cell-, circuit- and context-specific functions of these microcircuits in addictive behaviors. Also, region-specific astrocytes regulate the homeostatic activities of the dMSNs and iMSNs as well as the downstream circuits, which determine the net balance of cortico-striato-pallidal activities to the thalamic neurons. This review will summarize the recent progress of striatopallidal circuits focusing on astrocyte-neuron interaction and, reward- and alcohol-seeking behaviors. Our review will also discuss the translational and clinical implications of these microcircuit studies. This article is part of the special Issue on "Neurocircuitry Modulating Drug and Alcohol Abuse".

Association of Brain Reward Response With Body Mass Index and Ventral Striatal-Hypothalamic Circuitry Among Young Women With Eating Disorders

IMPORTANCE

Eating disorders are severe psychiatric disorders; however, disease models that cross subtypes and integrate behavior and neurobiologic factors are lacking.

OBJECTIVE

To assess brain response during unexpected receipt or omission of a salient sweet stimulus across a large sample of individuals with eating disorders and healthy controls and test for evidence of whether this brain response is associated with the ventral striatal-hypothalamic circuitry, which has been associated with food intake control, and whether salient stimulus response and eating disorder related behaviors are associated.

DESIGN, SETTING, AND PARTICIPANTS

In this cross-sectional functional brain imaging study, young adults across the eating disorder spectrum were matched with healthy controls at a university brain imaging facility and eating disorder treatment program. During a sucrose taste classic conditioning paradigm, violations of learned associations between conditioned visual and unconditioned taste stimuli evoked the dopamine-related prediction error. Dynamic effective connectivity during expected sweet taste receipt was studied to investigate hierarchical brain activation between food intake relevant brain regions. The study was conducted from June 2014 to November 2019. Data were analyzed from December 2019 to February 2020.

MAIN OUTCOMES AND MEASURES

Prediction error brain reward response across insula and striatum; dynamic effective connectivity between hypothalamus and ventral striatum; and demographic and behavior variables and their correlations with prediction error brain response and connectivity edge coefficients.

RESULTS

Of 317 female participants (197 with eating disorders and 120 healthy controls), the mean (SD) age was 23.8 (5.6) years and mean (SD) body mass index was 20.8 (5.4). Prediction error response was elevated in participants with anorexia nervosa (Wilks λ, 0.843; P = .001) and in participants with eating disorders inversely correlated with body mass index (left nucleus accumbens: r = -0.291; 95% CI, -0.413 to -0.167; P < .001; right dorsal anterior insula: r = -0.228; 95% CI, -0.366 to -0.089; P = .001), eating disorder inventory-3 binge eating tendency (left nucleus accumbens: r = -0.207; 95% CI, -0.333 to -0.073; P = .004; right dorsal anterior insula: r = -0.220; 95% CI, -0.354 to -0.073; P = .002), and trait anxiety (left nucleus accumbens: r = -0.148; 95% CI, -0.288 to -0.003; P = .04; right dorsal anterior insula: r = -0.221; 95% CI, -0.357 to -0.076; P = .002). Ventral striatal to hypothalamus directed connectivity was positively correlated with ventral striatal prediction error in eating disorders (r = 0.189; 95% CI, 0.045-0.324; P = .01) and negatively correlated with feeling out of control after eating (right side: r = -0.328; 95% CI, -0.480 to -0.164; P < .001; left side: r = -0.297; 95% CI, -0.439 to -0.142; P = .001).

CONCLUSIONS AND RELEVANCE

The results of this cross-sectional imaging study support that body mass index modulates prediction error and food intake control circuitry in the brain. Once altered, this circuitry may reinforce eating disorder behaviors when paired with behavioral traits associated with overeating or undereating.

Rigid reduced successor representation as a potential mechanism for addiction

Difficulty in cessation of drinking, smoking, or gambling has been widely recognized. Conventional theories proposed relative dominance of habitual over goal-directed control, but human studies have not convincingly supported them. Referring to the recently suggested "successor representation (SR)" of states that enables partially goal-directed control, we propose a dopamine-related mechanism that makes resistance to habitual reward-obtaining particularly difficult. We considered that long-standing behavior towards a certain reward without resisting temptation can (but not always) lead to a formation of rigid dimension-reduced SR based on the goal state, which cannot be updated. Then, in our model assuming such rigid reduced SR, whereas no reward prediction error (RPE) is generated at the goal while no resistance is made, a sustained large positive RPE is generated upon goal reaching once the person starts resisting temptation. Such sustained RPE is somewhat similar to the hypothesized sustained fictitious RPE caused by drug-induced dopamine. In contrast, if rigid reduced SR is not formed and states are represented individually as in simple reinforcement learning models, no sustained RPE is generated at the goal. Formation of rigid reduced SR also attenuates the resistance-dependent decrease in the value of the cue for behavior, makes subsequent introduction of punishment after the goal ineffective, and potentially enhances the propensity of nonresistance through the influence of RPEs via the spiral striatum-midbrain circuit. These results suggest that formation of rigid reduced SR makes cessation of habitual reward-obtaining particularly difficult and can thus be a mechanism for addiction, common to substance and nonsubstance reward.

Neuropathology and pathogenesis of extrapyramidal movement disorders: a critical update-I. Hypokinetic-rigid movement disorders

Extrapyramidal movement disorders include hypokinetic rigid and hyperkinetic or mixed forms, most of them originating from dysfunction of the basal ganglia (BG) and their information circuits. The functional anatomy of the BG, the cortico-BG-thalamocortical, and BG-cerebellar circuit connections are briefly reviewed. Pathophysiologic classification of extrapyramidal movement disorder mechanisms distinguish (1) parkinsonian syndromes, (2) chorea and related syndromes, (3) dystonias, (4) myoclonic syndromes, (5) ballism, (6) tics, and (7) tremor syndromes. Recent genetic and molecular-biologic classifications distinguish (1) synucleinopathies (Parkinson's disease, dementia with Lewy bodies, Parkinson's disease-dementia, and multiple system atrophy); (2) tauopathies (progressive supranuclear palsy, corticobasal degeneration, FTLD-17; Guamian Parkinson-dementia; Pick's disease, and others); (3) polyglutamine disorders (Huntington's disease and related disorders); (4) pantothenate kinase-associated neurodegeneration; (5) Wilson's disease; and (6) other hereditary neurodegenerations without hitherto detected genetic or specific markers. The diversity of phenotypes is related to the deposition of pathologic proteins in distinct cell populations, causing neurodegeneration due to genetic and environmental factors, but there is frequent overlap between various disorders. Their etiopathogenesis is still poorly understood, but is suggested to result from an interaction between genetic and environmental factors. Multiple etiologies and noxious factors (protein mishandling, mitochondrial dysfunction, oxidative stress, excitotoxicity, energy failure, and chronic neuroinflammation) are more likely than a single factor. Current clinical consensus criteria have increased the diagnostic accuracy of most neurodegenerative movement disorders, but for their definite diagnosis, histopathological confirmation is required. We present a timely overview of the neuropathology and pathogenesis of the major extrapyramidal movement disorders in two parts, the first one dedicated to hypokinetic-rigid forms and the second to hyperkinetic disorders.

Control of excitatory hierarchical circuits by parvalbumin-FS basket cells in layer 5 of the frontal cortex: insights for cortical oscillations

The cortex contains multiple neuron types with specific connectivity and functions. Recent progress has provided a better understanding of the interactions of these neuron types as well as their output organization particularly for the frontal cortex, with implications for the circuit mechanisms underlying cortical oscillations that have cognitive functions. Layer 5 pyramidal cells (PCs) in the frontal cortex comprise two major subtypes: crossed-corticostriatal (CCS) and corticopontine (CPn) cells. Functionally, CCS and CPn cells exhibit similar phase-dependent firing during gamma waves but participate in two distinct subnetworks that are linked unidirectionally from CCS to CPn cells. GABAergic parvalbumin-expressing fast-spiking (PV-FS) cells, necessary for gamma oscillation, innervate PCs, with stronger and global inhibition to somata and weaker and localized inhibitions to dendritic shafts/spines. While PV-FS cells form reciprocal connections with both CCS and CPn cells, the excitation from CPn to PV-FS cells exhibits short-term synaptic dynamics conducive for oscillation induction. The electrical coupling between PV-FS cells facilitates spike synchronization among PV-FS cells receiving common excitatory inputs from local PCs and inhibits other PV-FS cells via electrically communicated spike afterhyperpolarizations. These connectivity characteristics can promote synchronous firing in the local networks of CPn cells and firing of some CCS cells by anode-break excitation. Thus subsets of L5 CCS and CPn cells within different levels of connection hierarchy exhibit coordinated activity via their common connections with PV-FS cells, and the resulting PC output drives diverse neuronal targets in cortical layer 1 and the striatum with specific temporal precision, expanding the computational power of the cortical network.