Multiple representations and algorithms for reinforcement learning in the cortico-basal ganglia circuit

Dopamine transients follow a striatal gradient of reward time horizons

Animals make predictions to guide their behavior and update those predictions through experience. Transient increases in dopamine (DA) are thought to be critical signals for updating predictions. However, it is unclear how this mechanism handles a wide range of behavioral timescales-from seconds or less (for example, if singing a song) to potentially hours or more (for example, if hunting for food). Here we report that DA transients in distinct rat striatal subregions convey prediction errors based on distinct time horizons. DA dynamics systematically accelerated from ventral to dorsomedial to dorsolateral striatum, in the tempo of spontaneous fluctuations, the temporal integration of prior rewards and the discounting of future rewards. This spectrum of timescales for evaluative computations can help achieve efficient learning and adaptive motivation for a broad range of behaviors.

Subthalamic nucleus deep brain stimulation alleviates oxidative stress via mitophagy in Parkinson's disease

Subthalamic nucleus deep brain stimulation (STN-DBS) has the potential to delay Parkinson's disease (PD) progression. Whether oxidative stress participates in the neuroprotective effects of DBS and related signaling pathways remains unknown. To address this, we applied STN-DBS to mice and monkey models of PD and collected brain tissue to evaluate mitophagy, oxidative stress, and related pathway. To confirm findings in animal experiments, a cohort of PD patients was recruited and oxidative stress was evaluated in cerebrospinal fluid. When PD mice received STN stimulation, the mTOR pathway was suppressed, accompanied by elevated LC3 II expression, increased mitophagosomes, and a decrease in p62 expression. The increase in mitophagy and balance of mitochondrial fission/fusion dynamics in the substantia nigra caused a marked enhancement of the antioxidant enzymes superoxide dismutase and glutathione levels. Subsequently, fewer mitochondrial apoptogenic factors were released to the cytoplasm, which resulted in a suppression of caspase activation and reservation of dopaminergic neurons. While interfaced with an mTOR activator, oxidative stress was no longer regulated by STN-DBS, with no neuroprotective effect. Similar results to those found in the rodent experiments were obtained in monkeys treated with chronic STN stimulation. Moreover, antioxidant enzymes in PD patients were increased after the operation, however, there was no relation between changes in antioxidant enzymes and motor impairment. Collectively, our study found that STN-DBS was able to increase mitophagy via an mTOR-dependent pathway, and oxidative stress was suppressed due to removal of damaged mitochondria, which was attributed to the dopaminergic neuroprotection of STN-DBS in PD.

Interaction between decision-making and motor learning when selecting reach targets in the presence of bias and noise

Motor errors can have both bias and noise components. Bias can be compensated for by adaptation and, in tasks in which the magnitude of noise varies across the environment, noise can be reduced by identifying and then acting in less noisy regions of the environment. Here we examine how these two processes interact when participants reach under a combination of an externally imposed visuomotor bias and noise. In a center-out reaching task, participants experienced noise (zero-mean random visuomotor rotations) that was target-direction dependent with a standard deviation that increased linearly from a least-noisy direction. They also experienced a constant bias, a visuomotor rotation that varied (across groups) from 0 to 40 degrees. Critically, on each trial, participants could select one of three targets to reach to, thereby allowing them to potentially select targets close to the least-noisy direction. The group who experienced no bias (0 degrees) quickly learned to select targets close to the least-noisy direction. However, groups who experienced a bias often failed to identify the least-noisy direction, even though they did partially adapt to the bias. When noise was introduced after participants experienced and adapted to a 40 degrees bias (without noise) in all directions, they exhibited an improved ability to find the least-noisy direction. We developed two models-one for reach adaptation and one for target selection-that could explain participants' adaptation and target-selection behavior. Our data and simulations indicate that there is a trade-off between adaptation and selection. Specifically, because bias learning is local, participants can improve performance, through adaptation, by always selecting targets that are closest to a chosen direction. However, this comes at the expense of improving performance, through selection, by reaching toward targets in different directions to find the least-noisy direction.

Selective encoding of reward predictions and prediction errors by globus pallidus subpopulations

Basal ganglia (BG) circuits help guide and invigorate actions using predictions of future rewards (values). Within the BG, the globus pallidus pars externa (GPe) may play an essential role in aggregating and distributing value information. We recorded from the GPe in unrestrained rats performing both Pavlovian and instrumental tasks to obtain rewards and distinguished neuronal subtypes by their firing properties across the wake/sleep cycle and optogenetic tagging. In both tasks, the parvalbumin-positive (PV+), faster-firing "prototypical" neurons showed strong, sustained modulation by value, unlike other subtypes, including the "arkypallidal" cells that project back to striatum. Furthermore, we discovered that a distinct minority (7%) of GP cells display slower, pacemaker-like firing and encode reward prediction errors (RPEs) almost identically to midbrain dopamine neurons. These cell-specific forms of GPe value representation help define the circuit mechanisms by which the BG contribute to motivation and reinforcement learning.

Contributions of the Basal Ganglia to Visual Perceptual Decisions

The basal ganglia (BG) make up a prominent nexus between visual and motor-related brain regions. In contrast to the BG's well-established roles in movement control and value-based decision making, their contributions to the transformation of visual input into an action remain unclear, especially in the context of perceptual decisions based on uncertain visual evidence. This article reviews recent progress in our understanding of the BG's contributions to the formation, evaluation, and adjustment of such decisions. From theoretical and experimental perspectives, the review focuses on four key stations in the BG network, namely, the striatum, pallidum, subthalamic nucleus, and midbrain dopamine neurons, which can have different roles and together support the decision process.

Virtual reality to improve low-back pain and pelvic pain during pregnancy: a pilot RCT for a multicenter randomized controlled trial

A significant proportion of women experience low back and pelvic pain during and after pregnancy, which can negatively impact their daily lives. Various factors are attributed to these complaints, and many affected women do not receive adequate healthcare. However, there is evidence to support the use of different physiotherapeutic interventions to alleviate these conditions. Virtual reality is a promising complementary treatment to physiotherapy, particularly in improving pain perception and avoidance. The primary objective of this study is to evaluate the efficacy of a four-week program combining VR and physiotherapy compared to standard physiotherapy in pregnant women with low back and pelvic pain, in terms of improving pain avoidance, intensity, disability, and functional level. The study also aims to investigate patient satisfaction with the VR intervention. This research will be conducted through a multi-center randomized controlled clinical trial involving pregnant patients residing in the provinces of Seville and Malaga with a diagnosis of low back and pelvic pain during pregnancy. The alternative hypothesis is that the implementation of a Virtual Reality program in combination with standard physiotherapy will result in better clinical outcomes compared to the current standard intervention, which could lead to the development of new policies and interventions for these pathologies and their consequences. Clinical trial registration: clinicaltrials.gov, identifier NCT05571358.

Reinforcement learning establishes a minimal metacognitive process to monitor and control motor learning performance

Humans and animals develop learning-to-learn strategies throughout their lives to accelerate learning. One theory suggests that this is achieved by a metacognitive process of controlling and monitoring learning. Although such learning-to-learn is also observed in motor learning, the metacognitive aspect of learning regulation has not been considered in classical theories of motor learning. Here, we formulated a minimal mechanism of this process as reinforcement learning of motor learning properties, which regulates a policy for memory update in response to sensory prediction error while monitoring its performance. This theory was confirmed in human motor learning experiments, in which the subjective sense of learning-outcome association determined the direction of up- and down-regulation of both learning speed and memory retention. Thus, it provides a simple, unifying account for variations in learning speeds, where the reinforcement learning mechanism monitors and controls the motor learning process.

The Situated Assessment Method (SAM2): Establishing individual differences in habitual behavior

From the perspectives of grounded, situated, and embodied cognition, we have developed a new approach for assessing individual differences. Because this approach is grounded in two dimensions of situatedness-situational experience and the Situated Action Cycle-we refer to it as the Situated Assessment Method (SAM2). Rather than abstracting over situations during assessment of a construct (as in traditional assessment instruments), SAM2 assesses a construct in situations where it occurs, simultaneously measuring factors from the Situated Action Cycle known to influence it. To demonstrate this framework, we developed the SAM2 Habitual Behavior Instrument (SAM2 HBI). Across three studies with a total of 442 participants, the SAM2 HBI produced a robust and replicable pattern of results at both the group and individual levels. Trait-level measures of habitual behavior exhibited large reliable individual differences in the regularity of performing positive versus negative habits. Situational assessments established large effects of situations and large situation by individual interactions. Several sources of evidence demonstrated construct and content validity for SAM2 measures of habitual behavior. At both the group and individual levels, these measures were associated with factors from the Situated Action Cycle known to influence habitual behavior in the literature (consistency, automaticity, immediate reward, long-term reward). Regressions explained approximately 65% of the variance at the group level and a median of approximately 75% at the individual level. SAM2 measures further exhibited well-established interactions with personality measures for self-control and neuroticism. Cognitive-affective processes from the Situated Action Cycle explained nearly all the variance in these interactions. Finally, a composite measure of habitualness established habitual behaviors at both the group and individual levels. Additionally, a composite measure of reward was positively related to the composite measure of habitualness, increasing with self-control and decreasing with neuroticism.

Somatosensory Information in Skilled Motor Performance: A Narrative Review

Historically, research aimed at improving motor performance has largely focused on the neural processes involved in motor execution due to their role in muscle activation. However, accompanying somatosensory and proprioceptive sensory information is also vitally involved in performing motor skills. Here we review research from interdisciplinary fields to provide a description for how somatosensation informs the successful performance of motor skills as well as emphasize the need for careful selection of study methods to isolate the neural processes involved in somatosensory perception. We also discuss upcoming strategies of intervention that have been used to improve performance via somatosensory targets. We believe that a greater appreciation for somatosensation's role in motor learning and control will enable researchers and practitioners to develop and apply methods for the enhancement of human performance that will benefit clinical, healthy, and elite populations alike.

Hierarchical categorization learning is associated with representational changes in the dorsal striatum and posterior frontal and parietal cortex

Learning and recognition can be improved by sorting novel items into categories and subcategories. Such hierarchical categorization is easy when it can be performed according to learned rules (e.g., "if car, then automatic or stick shift" or "if boat, then motor or sail"). Here, we present results showing that human participants acquire categorization rules for new visual hierarchies rapidly, and that, as they do, corresponding hierarchical representations of the categorized stimuli emerge in patterns of neural activation in the dorsal striatum and in posterior frontal and parietal cortex. Participants learned to categorize novel visual objects into a hierarchy with superordinate and subordinate levels based on the objects' shape features, without having been told the categorization rules for doing so. On each trial, participants were asked to report the category and subcategory of the object, after which they received feedback about the correctness of their categorization responses. Participants trained over the course of a one-hour-long session while their brain activation was measured using functional magnetic resonance imaging. Over the course of training, significant hierarchy learning took place as participants discovered the nested categorization rules, as evidenced by the occurrence of a learning trial, after which performance suddenly increased. This learning was associated with increased representational strength of the newly acquired hierarchical rules in a corticostriatal network including the posterior frontal and parietal cortex and the dorsal striatum. We also found evidence suggesting that reinforcement learning in the dorsal striatum contributed to hierarchical rule learning.

Virtual Reality-Augmented Physiotherapy for Chronic Pain in Youth: Protocol for a Randomized Controlled Trial Enhanced With a Single-Case Experimental Design

BACKGROUND

Chronic musculoskeletal (MSK) pain is a prominent health concern, resulting in pain-related disability, loss of functioning, and high health care costs. Physiotherapy rehabilitation is a gold-standard treatment for improving functioning in youth with chronic MSK pain. However, increasing physical activity can feel unattainable for many adolescents because of pain-related fear and movement avoidance. Virtual reality (VR) offers an immersive experience that can interrupt the fear-avoidance cycle and improve engagement in physiotherapy. Despite promising initial findings, data are limited and often lack the rigor required to establish VR as an evidence-based treatment for MSK pain.

OBJECTIVE

This trial evaluates physiorehabilitation with VR in adolescents with MSK pain. This protocol outlines the rationale, design, and implementation of a randomized controlled trial enhanced with a single-case experimental design.

METHODS

This study is a 2-group randomized controlled trial assessing the use of physiorehabilitation with VR in adolescents with MSK pain. The authors will collaborate with physical therapists to integrate VR into their standard clinical care. For participants enrolled in standard physiotherapy, there will be no VR integrated into their physical therapy program. Primary outcomes include physical function and engagement in VR. Secondary outcomes include pain-related fear and treatment adherence. Moreover, we will obtain clinician perspectives regarding the feasibility of integrating the intervention into the flow of clinical practice.

RESULTS

The pilot study implementing physiorehabilitation with VR demonstrated that high engagement and use of physiorehabilitation with VR were associated with improvements in pain, fear, avoidance, and function. Coupled with qualitative feedback from patients, families, and clinicians, the pilot study results provide support for this trial to evaluate physiorehabilitation with VR for youth with chronic MSK pain. Analysis of results from the main clinical trial will begin as recruitment progresses, and results are expected in early 2024.

CONCLUSIONS

Significant breakthroughs for treating MSK pain require mechanistically informed innovative approaches. Physiorehabilitation with VR provides exposure to progressive challenges, real-time feedback, and reinforcement for movement and can include activities that are difficult to achieve in the real world. It has the added benefit of sustaining patient motivation and adherence while enabling clinicians to use objective benchmarks to influence progression. These findings will inform the decision of whether to proceed with a hybrid effectiveness-dissemination trial of physiorehabilitation with VR, serving as the basis for potential large-scale implementation of physiorehabilitation with VR.

TRIAL REGISTRATION

ClinicalTrials.gov NCT04636177; https://clinicaltrials.gov/ct2/show/NCT04636177.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/40705.

Cerebellar and basal ganglia motor network predicts trait depression and hyperactivity

In the human brain, the cerebellum (CB) and basal ganglia (BG) are implicated in cognition-, emotion-, and motor-related cortical processes and are highly interconnected, both to cortical regions via separate, trans-thalamic pathways and to each other via subcortical disynaptic pathways. We previously demonstrated a distinction between cognitive and motor CB-BG networks (CCBN, MCBN, respectively) as it relates to cortical network integration in healthy young adults, suggesting the subcortical networks separately support cortical networks. The CB and BG are also implicated in the pathophysiology of schizophrenia, Parkinson's, and compulsive behavior; thus, integration within subcortical CB-BG networks may be related to transdiagnostic symptomology. Here, we asked whether CCBN or MCBN integration predicted Achenbach Self-Report scores for anxiety, depression, intrusive thoughts, hyperactivity and inactivity, and cognitive performance in a community sample of young adults. We computed global efficiency for each CB-BG network and 7 canonical resting-state networks for all right-handed participants in the Human Connectome Project 1200 release with a complete set of preprocessed resting-state functional MRI data (N = 783). We used multivariate regression to control for substance abuse and age, and permutation testing with exchangeability blocks to control for family relationships. MCBN integration negatively predicted depression and hyperactivity, and positively predicted cortical network integration. CCBN integration predicted cortical network integration (except for the emotional network) and marginally predicted a positive relationship with hyperactivity, indicating a potential dichotomy between cognitive and motor CB-BG networks and hyperactivity. These results highlight the importance of CB-BG interactions as they relate to motivation and symptoms of depression.

Ramping activity in the striatum

Control of the timing of behavior is thought to require the basal ganglia (BG) and BG pathologies impair performance in timing tasks. Temporal interval discrimination depends on the ramping activity of medium spiny neurons (MSN) in the main BG input structure, the striatum, but the underlying mechanisms driving this activity are unclear. Here, we combine an MSN dynamical network model with an action selection system applied to an interval discrimination task. We find that when network parameters are appropriate for the striatum so that slowly fluctuating marginally stable dynamics are intrinsically generated, up and down ramping populations naturally emerge which enable significantly above chance task performance. We show that emergent population activity is in very good agreement with empirical studies and discuss how MSN network dysfunction in disease may alter temporal perception.

Representation learning in the artificial and biological neural networks underlying sensorimotor integration

The integration of deep learning and theories of reinforcement learning (RL) is a promising avenue to explore novel hypotheses on reward-based learning and decision-making in humans and other animals. Here, we trained deep RL agents and mice in the same sensorimotor task with high-dimensional state and action space and studied representation learning in their respective neural networks. Evaluation of thousands of neural network models with extensive hyperparameter search revealed that learning-dependent enrichment of state-value and policy representations of the task-performance-optimized deep RL agent closely resembled neural activity of the posterior parietal cortex (PPC). These representations were critical for the task performance in both systems. PPC neurons also exhibited representations of the internally defined subgoal, a feature of deep RL algorithms postulated to improve sample efficiency. Such striking resemblance between the artificial and biological networks and their functional convergence in sensorimotor integration offers new opportunities to better understand respective intelligent systems.

An insula hierarchical network architecture for active interoceptive inference

In the brain, the insular cortex receives a vast amount of interoceptive information, ascending through deep brain structures, from multiple visceral organs. The unique hierarchical and modular architecture of the insula suggests specialization for processing interoceptive afferents. Yet, the biological significance of the insula's neuroanatomical architecture, in relation to deep brain structures, remains obscure. In this opinion piece, we propose the Insula Hierarchical Modular Adaptive Interoception Control (IMAC) model to suggest that insula modules (granular, dysgranular and agranular), forming parallel networks with the prefrontal cortex and striatum, are specialized to form higher order interoceptive representations. These interoceptive representations are recruited in a context-dependent manner to support habitual, model-based and exploratory control of visceral organs and physiological processes. We discuss how insula interoceptive representations may give rise to conscious feelings that best explain lower order deep brain interoceptive representations, and how the insula may serve to defend the body and mind against pathological depression.

The neuroecology of the water-to-land transition and the evolution of the vertebrate brain

The water-to-land transition in vertebrate evolution offers an unusual opportunity to consider computational affordances of a new ecology for the brain. All sensory modalities are changed, particularly a greatly enlarged visual sensorium owing to air versus water as a medium, and expanded by mobile eyes and neck. The multiplication of limbs, as evolved to exploit aspects of life on land, is a comparable computational challenge. As the total mass of living organisms on land is a hundredfold larger than the mass underwater, computational improvements promise great rewards. In water, the midbrain tectum coordinates approach/avoid decisions, contextualized by water flow and by the animal's body state and learning. On land, the relative motions of sensory surfaces and effectors must be resolved, adding on computational architectures from the dorsal pallium, such as the parietal cortex. For the large-brained and long-living denizens of land, making the right decision when the wrong one means death may be the basis of planning, which allows animals to learn from hypothetical experience before enactment. Integration of value-weighted, memorized panoramas in basal ganglia/frontal cortex circuitry, with allocentric cognitive maps of the hippocampus and its associated cortices becomes a cognitive habit-to-plan transition as substantial as the change in ecology. This article is part of the theme issue 'Systems neuroscience through the lens of evolutionary theory'.

Human Variation in Error-Based and Reinforcement Motor Learning Is Associated With Entorhinal Volume

Error-based and reward-based processes are critical for motor learning and are thought to be mediated via distinct neural pathways. However, recent behavioral work in humans suggests that both learning processes can be bolstered by the use of cognitive strategies, which may mediate individual differences in motor learning ability. It has been speculated that medial temporal lobe regions, which have been shown to support motor sequence learning, also support the use of cognitive strategies in error-based and reinforcement motor learning. However, direct evidence in support of this idea remains sparse. Here we first show that better overall learning during error-based visuomotor adaptation is associated with better overall learning during the reward-based shaping of reaching movements. Given the cognitive contribution to learning in both of these tasks, these results support the notion that strategic processes, associated with better performance, drive intersubject variation in both error-based and reinforcement motor learning. Furthermore, we show that entorhinal cortex volume is larger in better learning individuals-characterized across both motor learning tasks-compared with their poorer learning counterparts. These results suggest that individual differences in learning performance during error and reinforcement learning are related to neuroanatomical differences in entorhinal cortex.

Brain signals of a Surprise-Actor-Critic model: Evidence for multiple learning modules in human decision making

Learning how to reach a reward over long series of actions is a remarkable capability of humans, and potentially guided by multiple parallel learning modules. Current brain imaging of learning modules is limited by (i) simple experimental paradigms, (ii) entanglement of brain signals of different learning modules, and (iii) a limited number of computational models considered as candidates for explaining behavior. Here, we address these three limitations and (i) introduce a complex sequential decision making task with surprising events that allows us to (ii) dissociate correlates of reward prediction errors from those of surprise in functional magnetic resonance imaging (fMRI); and (iii) we test behavior against a large repertoire of model-free, model-based, and hybrid reinforcement learning algorithms, including a novel surprise-modulated actor-critic algorithm. Surprise, derived from an approximate Bayesian approach for learning the world-model, is extracted in our algorithm from a state prediction error. Surprise is then used to modulate the learning rate of a model-free actor, which itself learns via the reward prediction error from model-free value estimation by the critic. We find that action choices are well explained by pure model-free policy gradient, but reaction times and neural data are not. We identify signatures of both model-free and surprise-based learning signals in blood oxygen level dependent (BOLD) responses, supporting the existence of multiple parallel learning modules in the brain. Our results extend previous fMRI findings to a multi-step setting and emphasize the role of policy gradient and surprise signalling in human learning.

Reinforcement learning and its connections with neuroscience and psychology

Reinforcement learning methods have recently been very successful at performing complex sequential tasks like playing Atari games, Go and Poker. These algorithms have outperformed humans in several tasks by learning from scratch, using only scalar rewards obtained through interaction with their environment. While there certainly has been considerable independent innovation to produce such results, many core ideas in reinforcement learning are inspired by phenomena in animal learning, psychology and neuroscience. In this paper, we comprehensively review a large number of findings in both neuroscience and psychology that evidence reinforcement learning as a promising candidate for modeling learning and decision making in the brain. In doing so, we construct a mapping between various classes of modern RL algorithms and specific findings in both neurophysiological and behavioral literature. We then discuss the implications of this observed relationship between RL, neuroscience and psychology and its role in advancing research in both AI and brain science.

Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making

Classic reinforcement learning (RL) theories cannot explain human behavior in the absence of external reward or when the environment changes. Here, we employ a deep sequential decision-making paradigm with sparse reward and abrupt environmental changes. To explain the behavior of human participants in these environments, we show that RL theories need to include surprise and novelty, each with a distinct role. While novelty drives exploration before the first encounter of a reward, surprise increases the rate of learning of a world-model as well as of model-free action-values. Even though the world-model is available for model-based RL, we find that human decisions are dominated by model-free action choices. The world-model is only marginally used for planning, but it is important to detect surprising events. Our theory predicts human action choices with high probability and allows us to dissociate surprise, novelty, and reward in EEG signals.

More dynamical and more symbiotic: Cortico-striatal models of resolve, suppression, and routine habit

I extend Ainslie's core claims with three cortico-striatal models that respectively subserve the key constructs of resolve, suppression, and routine habit. I show that these models suggest a more dynamical and symbiotic relation among the constructs: there are more ways they interact to reinforce willpower, and the temporal dimension of the interactions can often determine the effectiveness of the reinforcement.

Robust and distributed neural representation of action values

Studies in rats, monkeys, and humans have found action-value signals in multiple regions of the brain. These findings suggest that action-value signals encoded in these brain structures bias choices toward higher expected rewards. However, previous estimates of action-value signals might have been inflated by serial correlations in neural activity and also by activity related to other decision variables. Here, we applied several statistical tests based on permutation and surrogate data to analyze neural activity recorded from the striatum, frontal cortex, and hippocampus. The results show that previously identified action-value signals in these brain areas cannot be entirely accounted for by concurrent serial correlations in neural activity and action value. We also found that neural activity related to action value is intermixed with signals related to other decision variables. Our findings provide strong evidence for broadly distributed neural signals related to action value throughout the brain.

Dynamic Modulation of a Learned Motor Skill for Its Recruitment

Humans learn motor skills (MSs) through practice and experience and may then retain them for recruitment, which is effective as a rapid response for novel contexts. For an MS to be recruited for novel contexts, its recruitment range must be extended. In addressing this issue, we hypothesized that an MS is dynamically modulated according to the feedback context to expand its recruitment range into novel contexts, which do not involve the learning of an MS. The following two sub-issues are considered. We previously demonstrated that the learned MS could be recruited in novel contexts through its modulation, which is driven by dynamically regulating the synergistic redundancy between muscles according to the feedback context. However, this modulation is trained in the dynamics under the MS learning context. Learning an MS in a specific condition naturally causes movement deviation from the desired state when the MS is executed in a novel context. We hypothesized that this deviation can be reduced with the additional modulation of an MS, which tunes the MS-produced muscle activities by using the feedback gain signals driven by the deviation from the desired state. Based on this hypothesis, we propose a feedback gain signal-driven tuning model of a learned MS for its robust recruitment. This model is based on the neurophysiological architecture in the cortico-basal ganglia circuit, in which an MS is plausibly retained as it was learned and is then recruited by tuning its muscle control signals according to the feedback context. In this study, through computational simulation, we show that the proposed model may be used to neurophysiologically describe the recruitment of a learned MS in novel contexts.

Actor-critic reinforcement learning in the songbird

It feels rewarding to ace your opponent on match point. Here, we propose common mechanisms underlie reward and performance learning. First, when a singing bird unexpectedly hits the right note, its dopamine (DA) neurons are activated as when a thirsty monkey receives an unexpected juice reward. Second, these DA signals reinforce vocal variations much as they reinforce stimulus-response associations. Third, limbic inputs to DA neurons signal the predicted quality of song syllables much like they signal the predicted reward value of a place or a stimulus during foraging. Finally, songbirds may solve difficult problems in reinforcement learning - such as credit assignment and catastrophic forgetting - with node perturbation and consolidation of reinforced vocal patterns in motor cortical circuits. Consolidation occurs downstream of a canonical 'actor-critic' circuit motif that learns to maximize performance quality in essentially the same way it learns to maximize reward: by computing and learning from prediction errors.

Deep brain stimulation of the anterior nuclei of the thalamus relieves basal ganglia dysfunction in monkeys with temporal lobe epilepsy

Aims

Deep brain stimulation of the anterior nuclei of the thalamus (ANT‐DBS) is effective in temporal lobe epilepsy (TLE). Previous studies have shown that the basal ganglia are involved in seizure propagation in TLE, but the effects of ANT‐DBS on the basal ganglia have not been clarified.

Methods

ANT‐DBS was applied to monkeys with kainic acid–induced TLE using a robot‐assisted system. Behavior was monitored continuously. Immunofluorescence analysis and Western blotting were used to estimate protein expression levels in the basal ganglia and the effects of ANT stimulation.

Results

The seizure frequency decreased after ANT‐DBS. D1 and D2 receptor levels in the putamen and caudate were significantly higher in the ANT‐DBS group than in the epilepsy (EP) model. Neuronal loss and apoptosis were less severe in the ANT‐DBS group. Glutamate receptor 1 (GluR1) in the nucleus accumbens (NAc) shell and globus pallidus internus (GPi) increased in the EP group but decreased after ANT‐DBS. γ‐Aminobutyric acid receptor A (GABA_A‐R) decreased and glutamate decarboxylase 67 (GAD67) increased in the GPi of the EP group, whereas the reverse tendencies were observed after ANT‐DBS.

Conclusion

ANT‐DBS exerts neuroprotective effects on the caudate and putamen, enhances D1 and D2 receptor expression, and downregulates GPi overactivation, which enhanced the antiepileptic function of the basal ganglia.

Spatial organization of functional clusters representing reward and movement information in the striatal direct and indirect pathways

To obtain insights into striatal neural processes underlying reward-based learning and movement control, we examined spatial organizations of striatal neurons related to movement and reward-based learning. For this, we recorded the activity of direct- and indirect-pathway neurons (D1 and A2a receptor-expressing neurons, respectively) in mice engaged in probabilistic classical conditioning and open-field free exploration. We found broadly organized functional clusters of striatal neurons in the direct as well as indirect pathways for both movement- and reward-related variables. Functional clusters for different variables were partially overlapping in both pathways, but the overlap between outcome- and value-related functional clusters was greater in the indirect than direct pathway. Also, value-related spatial clusters were progressively refined during classical conditioning. Our study shows the broad and learning-dependent spatial organization of functional clusters of dorsal striatal neurons in the direct and indirect pathways. These findings further argue against the classic model of the basal ganglia and support the importance of spatiotemporal patterns of striatal neuronal ensemble activity in the control of behavior.

Parvalbumin+ and Npas1+ Pallidal Neurons Have Distinct Circuit Topology and Function

The external globus pallidus (GPe) is a critical node within the basal ganglia circuit. Phasic changes in the activity of GPe neurons during movement and their alterations in Parkinson's disease (PD) argue that the GPe is important in motor control. Parvalbumin-positive (PV+) neurons and Npas1+ neurons are the two principal neuron classes in the GPe. The distinct electrophysiological properties and axonal projection patterns argue that these two neuron classes serve different roles in regulating motor output. However, the causal relationship between GPe neuron classes and movement remains to be established. Here, by using optogenetic approaches in mice (both males and females), we showed that PV+ neurons and Npas1+ neurons promoted and suppressed locomotion, respectively. Moreover, PV+ neurons and Npas1+ neurons are under different synaptic influences from the subthalamic nucleus (STN). Additionally, we found a selective weakening of STN inputs to PV+ neurons in the chronic 6-hydroxydopamine lesion model of PD. This finding reinforces the idea that the reciprocally connected GPe-STN network plays a key role in disease symptomatology and thus provides the basis for future circuit-based therapies.SIGNIFICANCE STATEMENT The external pallidum is a key, yet an understudied component of the basal ganglia. Neural activity in the pallidum goes awry in neurologic diseases, such as Parkinson's disease. While this strongly argues that the pallidum plays a critical role in motor control, it has been difficult to establish the causal relationship between pallidal activity and motor function/dysfunction. This was in part because of the cellular complexity of the pallidum. Here, we showed that the two principal neuron types in the pallidum have opposing roles in motor control. In addition, we described the differences in their synaptic influence. Importantly, our research provides new insights into the cellular and circuit mechanisms that explain the hypokinetic features of Parkinson's disease.

Challenges and Opportunities for Grounding Cognition

According to the grounded perspective, cognition emerges from the interaction of classic cognitive processes with the modalities, the body, and the environment. Rather than being an autonomous impenetrable module, cognition incorporates these other domains intrinsically into its operation. The Situated Action Cycle offers one way of understanding how the modalities, the body, and the environment become integrated to ground cognition. Seven challenges and opportunities are raised for this perspective: (1) How does cognition emerge from the Situated Action Cycle and in turn support it? (2) How can we move beyond simply equating embodiment with action, additionally establishing how embodiment arises in the autonomic, neuroendocrine, immune, cardiovascular, respiratory, digestive, and integumentary systems? (3) How can we better understand the mechanisms underlying multimodal simulation, its functions across the Situated Action Cycle, and its integration with other representational systems? (4) How can we develop and assess theoretical accounts of symbolic processing from the grounded perspective (perhaps using the construct of simulators)? (5) How can we move beyond the simplistic distinction between concrete and abstract concepts, instead addressing how concepts about the external and internal worlds pattern to support the Situated Action Cycle? (6) How do individual differences emerge from different populations of situational memories as the Situated Action Cycle manifests itself differently across individuals? (7) How can constructs from grounded cognition provide insight into the replication and generalization crises, perhaps from a quantum perspective on mechanisms (as exemplified by simulators).

A Genetically Defined Compartmentalized Striatal Direct Pathway for Negative Reinforcement

The striosome compartment within the dorsal striatum has been implicated in reinforcement learning and regulation of motivation, but how striosomal neurons contribute to these functions remains elusive. Here, we show that a genetically identified striosomal population, which expresses the Teashirt family zinc finger 1 (Tshz1) and belongs to the direct pathway, drives negative reinforcement and is essential for aversive learning in mice. Contrasting a "conventional" striosomal direct pathway, the Tshz1 neurons cause aversion, movement suppression, and negative reinforcement once activated, and they receive a distinct set of synaptic inputs. These neurons are predominantly excited by punishment rather than reward and represent the anticipation of punishment or the motivation for avoidance. Furthermore, inhibiting these neurons impairs punishment-based learning without affecting reward learning or movement. These results establish a major role of striosomal neurons in behaviors reinforced by punishment and moreover uncover functions of the direct pathway unaccounted for in classic models.

Topographic distinction in long-term value signals between presumed dopamine neurons and presumed striatal projection neurons in behaving monkeys

Nigrostriatal dopamine (DA) projections are anatomically organized along the dorsolateral-ventromedial axis, conveying long-term value signals to the striatum for shaping actions toward multiple future rewards. The present study examines whether the topographic organization of long-term value signals are observed upon activity of presumed DA neurons and presumed striatal projection neurons (phasically active neurons, PANs), as predicted based on anatomical literature. Our results indicate that DA neurons in the dorsolateral midbrain encode long-term value signals on a short timescale, while ventromedial midbrain DA neurons encode such signals on a relatively longer timescale. Activity of the PANs in the dorsal striatum is more heterogeneous for encoding long-term values, although significant differences in long-term value signals were observed between the caudate nucleus and putamen. These findings suggest that topographic DA signals for long-term values are not simply transferred to striatal neurons, possibly due to the contribution of other projections to the striatum.

Habit learning in hierarchical cortex-basal ganglia loops

How do the multiple cortico-basal ganglia-thalamo-cortical loops interact? Are they parallel and fully independent or controlled by an arbitrator, or are they hierarchically organized? We introduce here a set of four key concepts, integrated and evaluated by means of a neuro-computational model, that bring together current ideas regarding cortex-basal ganglia interactions in the context of habit learning. According to key concept 1, each loop learns to select an intermediate objective at a different abstraction level, moving from goals in the ventral striatum to motor in the putamen. Key concept 2 proposes that the cortex integrates the basal ganglia selection with environmental information regarding the achieved objective. Key concept 3 claims shortcuts between loops, and key concept 4 predicts that loops compute their own prediction error signal for learning. Computational benefits of the key concepts are demonstrated. Contrasting with former concepts of habit learning, the loops collaborate to select goal-directed actions while training slower shortcuts develops habitual responses.

Anterior thalamic nucleus stimulation protects hippocampal neurons by activating autophagy in epileptic monkeys

Deep brain stimulation of the anterior nucleus of the thalamus (ANT-DBS) is effective in treating temporal lobe epilepsy (TLE) and protects hippocampal neurons. Autophagy plays an essential role in epileptogenesis; however, the underlying effect of autophagy on ANT-DBS-mediated neuroprotection remains unclear. A monkey model of epilepsy was established by injecting kainic acid into the hippocampus and amygdala using a robot-assisted system. ANT-DBS was delivered in the chronic stage of the epileptic model and continued for 8 weeks. We found that ANT-DBS reduced the frequency of seizures and exerted neuroprotective effects via activating autophagy in hippocampal neurons. ANT-DBS increased light chain 3 (LC3) II level and co-localization of LC3 and lysosomal-associated membrane protein-1, accompanied by decreased expression of the autophagy substrate ubiquitin-binding protein p62, suggesting increased autophagosome formation. Most importantly, brain-derived neurotrophic factor (BDNF) –tropomyosin-related kinase type B (TrkB) pathway were involved in the regulation of autophagy. Both protein levels were reduced by ANT-DBS, and there was less phosphorylation of downstream regulators, extracellular signal-regulated kinase and Akt, followed by inactivation of mammalian target of rapamycin complex 1. Taken together, chronic ANT-DBS exerts neuroprotective effects on hippocampal neurons through inducing autophagy via suppressing the BDNF–TrkB pathway in a TLE monkey model.

Npas1+-Nkx2.1+ Neurons Are an Integral Part of the Cortico-pallido-cortical Loop

Within the basal ganglia circuit, the external globus pallidus (GPe) is critically involved in motor control. Aside from Foxp2+ neurons and ChAT+ neurons that have been established as unique neuron types, there is little consensus on the classification of GPe neurons. Properties of the remaining neuron types are poorly defined. In this study, we leverage new mouse lines, viral tools, and molecular markers to better define GPe neuron subtypes. We found that Sox6 represents a novel, defining marker for GPe neuron subtypes. Lhx6+ neurons that lack the expression of Sox6 were devoid of both parvalbumin and Npas1. This result confirms previous assertions of the existence of a unique Lhx6+ population. Neurons that arise from the Dbx1+ lineage were similarly abundant in the GPe and displayed a heterogeneous makeup. Importantly, tracing experiments revealed that Npas1+-Nkx2.1+ neurons represent the principal noncholinergic, cortically-projecting neurons. In other words, they form the pallido-cortical arm of the cortico-pallido-cortical loop. Our data further show that pyramidal-tract neurons in the cortex collateralized within the GPe, forming a closed-loop system between the two brain structures. Overall, our findings reconcile some of the discrepancies that arose from differences in techniques or the reliance on preexisting tools. Although spatial distribution and electrophysiological properties of GPe neurons reaffirm the diversification of GPe subtypes, statistical analyses strongly support the notion that these neuron subtypes can be categorized under the two principal neuron classes: PV+ neurons and Npas1+ neurons.SIGNIFICANCE STATEMENT The poor understanding of the neuronal composition in the external globus pallidus (GPe) undermines our ability to interrogate its precise behavioral and disease involvements. In this study, 12 different genetic crosses were used, hundreds of neurons were electrophysiologically characterized, and >100,000 neurons were histologically- and/or anatomically-profiled. Our current study further establishes the segregation of GPe neuron classes and illustrates the complexity of GPe neurons in adult mice. Our results support the idea that Npas1+-Nkx2.1+ neurons are a distinct GPe neuron subclass. By providing a detailed analysis of the organization of the cortico-pallidal-cortical projection, our findings establish the cellular and circuit substrates that can be important for motor function and dysfunction.

Virtual reality and chronic low back pain

AIM

Chronic low back pain (CLBP) is a highly prevalent and significant cause of disability which is often resistant to pharmacological management. Virtual reality (VR) is an emerging technology with the potential to influence CLBP, and has been suggested as an alternative to opioids for pain management. VR is a goalfocused, computer-simulated reality allowing modification of the user's experience of their perceived world.

MATERIALS/METHODS

A narrative review of peer-reviewed literature using a systematic search strategy, and sole reviewer for data extraction.

CONCLUSIONS

VR has demonstrated effectiveness in reducing acute, experimental and chronic pain. This review describes the theoretical basis of the therapeutic effects of VR on CLBP via three distinct mechanisms: distraction, neuromodulation and graded exposure therapy. Furthermore, clinical application will be considered, including discussion of ethical issues associated with the technology.Implications for rehabilitationVirtual reality (VR) is suggested as an alternative for opioids in the management of acute and chronic pain.The therapeutic mechanisms of VR in chronic low back pain (CLBP) are equivocal but include distraction, neuromodulation of body perception and graded exposure therapy.VR may show greater efficacy in patients with CLBP with associated kinesiophobia.VR may show greater effect with increased immersion.

Learning the structure of the world: The adaptive nature of state-space and action representations in multi-stage decision-making

State-space and action representations form the building blocks of decision-making processes in the brain; states map external cues to the current situation of the agent whereas actions provide the set of motor commands from which the agent can choose to achieve specific goals. Although these factors differ across environments, it is currently unknown whether or how accurately state and action representations are acquired by the agent because previous experiments have typically provided this information a priori through instruction or pre-training. Here we studied how state and action representations adapt to reflect the structure of the world when such a priori knowledge is not available. We used a sequential decision-making task in rats in which they were required to pass through multiple states before reaching the goal, and for which the number of states and how they map onto external cues were unknown a priori. We found that, early in training, animals selected actions as if the task was not sequential and outcomes were the immediate consequence of the most proximal action. During the course of training, however, rats recovered the true structure of the environment and made decisions based on the expanded state-space, reflecting the multiple stages of the task. Similarly, we found that the set of actions expanded with training, although the emergence of new action sequences was sensitive to the experimental parameters and specifics of the training procedure. We conclude that the profile of choices shows a gradual shift from simple representations to more complex structures compatible with the structure of the world.

Distinct roles of striatal direct and indirect pathways in value-based decision making

The striatum is critically involved in value-based decision making. However, it is unclear how striatal direct and indirect pathways work together to make optimal choices in a dynamic and uncertain environment. Here, we examined the effects of selectively inactivating D1 receptor (D1R)- or D2 receptor (D2R)-expressing dorsal striatal neurons (corresponding to direct- and indirect-pathway neurons, respectively) on mouse choice behavior in a reversal task with progressively increasing reversal frequency and a dynamic two-armed bandit task. Inactivation of either D1R- or D2R-expressing striatal neurons impaired performance in both tasks, but the pattern of altered choice behavior differed between the two animal groups. A reinforcement learning model-based analysis indicated that inactivation of D1R- and D2R-expressing striatal neurons selectively impairs value-dependent action selection and value learning, respectively. Our results suggest differential contributions of striatal direct and indirect pathways to two distinct steps in value-based decision making.

Reward associations do not explain transitive inference performance in monkeys

Most accounts of behavior in nonhuman animals assume that they make choices to maximize expected reward value. However, model-free reinforcement learning based on reward associations cannot account for choice behavior in transitive inference paradigms. We manipulated the amount of reward associated with each item of an ordered list, so that maximizing expected reward value was always in conflict with decision rules based on the implicit list order. Under such a schedule, model-free reinforcement algorithms cannot achieve high levels of accuracy, even after extensive training. Monkeys nevertheless learned to make correct rule-based choices. These results show that monkeys' performance in transitive inference paradigms is not driven solely by expected reward and that appropriate inferences are made despite discordant reward incentives. We show that their choices can be explained by an abstract, model-based representation of list order, and we provide a method for inferring the contents of such representations from observed data.

An Analysis of Decision under Risk in Rats

In 1979, Daniel Kahneman and Amos Tversky published a ground-breaking paper titled "Prospect Theory: An Analysis of Decision under Risk," which presented a behavioral economic theory that accounted for the ways in which humans deviate from economists' normative workhorse model, Expected Utility Theory [1, 2]. For example, people exhibit probability distortion (they overweight low probabilities), loss aversion (losses loom larger than gains), and reference dependence (outcomes are evaluated as gains or losses relative to an internal reference point). We found that rats exhibited many of these same biases, using a task in which rats chose between guaranteed and probabilistic rewards. However, prospect theory assumes stable preferences in the absence of learning, an assumption at odds with alternative frameworks such as animal learning theory and reinforcement learning [3-7]. Rats also exhibited trial history effects, consistent with ongoing learning. A reinforcement learning model in which state-action values were updated by the subjective value of outcomes according to prospect theory reproduced rats' nonlinear utility and probability weighting functions and also captured trial-by-trial learning dynamics.

Efficacy of virtual reality to reduce chronic low back pain: Proof-of-concept of a non-pharmacological approach on pain, quality of life, neuropsychological and functional outcome

Objectives

Chronic pain, such as low-back pain, can be a highly disabling condition degrading people’s quality of life (QoL). Not every patient responds to pharmacological therapies, thus alternative treatments have to be developed. The chronicity of pain can lead to a somatic dysperception, meaning a mismatch between patients’ own body perception and its actual physical state. Since clinical evaluation of pain relies on patients’ subjective reports, a body image disruption can be associated with an incorrect pain rating inducing incorrect treatment and a possible risk of drug abuse. Our aim was to reduce chronic low-back pain through a multimodal neurorehabilitative strategy using innovative technologies to help patients regain a correct body image.

Methods

Twenty patients with chronic low-back pain were included. Before and after treatment, patients underwent: a neurological exam; a neuro-psychological evaluation testing cognitive functions (memory, attention, executive functions) and personality traits, QoL and mood; pain ratings; sensorimotor functional abilities’ testing. Patients underwent a 6 week-neurorehabilitative treatment (total 12 sessions) using virtual reality (VRRS system, Khymeia, Italy). Treatment consisted on teaching patients to execute correct movements with the painful body parts to regain a correct body image, based on the augmented multisensory feedback (auditory, visual) provided by the VRRS.

Results

Our data showed significant reductions in all pain rating scale scores (p<0.05); significant improvements of QoL in the domains of physical functioning, physical role functioning, bodily pain, vitality, and social role functioning; improvements in cognitive functions (p<0.05); improvements in functional scales (p<0.05) and mood (p = 0.04).

Conclusion

This non-pharmacological approach was able to act on the multi-dimensional aspects of pain and improved patients’ QoL, pain intensity, mood and patient’s functional abilities.

Role of the hippocampal CA1 region in incremental value learning

It is generally believed that the hippocampus plays a crucial role in declarative memory—remembering facts and events—but not in gradual stimulus-response association or incremental value learning. Based on the finding that CA1 conveys strong value signals during dynamic foraging, we investigated the possibility that the hippocampus contributes to incremental value learning. Specifically, we examined effects of inactivating different subregions of the dorsal hippocampus on behavioral performance of mice performing a dynamic foraging task in a modified T-maze. A reinforcement learning model-based analysis indicated that inactivation of CA1, but not dentate gyrus, CA3, or CA2, impaired trial-by-trial updating of chosen value without affecting value-dependent action selection. As a result, it took longer for CA1-inactivated mice to bias their choices toward the higher-reward-probability target after changes in reward probability. Our results indicate, contrary to the traditional view, that the hippocampus, especially CA1, might contribute to incremental value learning under certain circumstances.

Muscle Synergy-Driven Robust Motion Control

Humans are able to robustly maintain desired motion and posture under dynamically changing circumstances, including novel conditions. To accomplish this, the brain needs to optimize the synergistic control between muscles against external dynamic factors. However, previous related studies have usually simplified the control of multiple muscles using two opposing muscles, which are minimum actuators to simulate linear feedback control. As a result, they have been unable to analyze how muscle synergy contributes to motion control robustness in a biological system. To address this issue, we considered a new muscle synergy concept used to optimize the synergy between muscle units against external dynamic conditions, including novel conditions. We propose that two main muscle control policies synergistically control muscle units to maintain the desired motion against external dynamic conditions. Our assumption is based on biological evidence regarding the control of multiple muscles via the corticospinal tract. One of the policies is the group control policy (GCP), which is used to control muscle group units classified based on functional similarities in joint control. This policy is used to effectively resist external dynamic circumstances, such as disturbances. The individual control policy (ICP) assists the GCP in precisely controlling motion by controlling individual muscle units. To validate this hypothesis, we simulated the reinforcement of the synergistic actions of the two control policies during the reinforcement learning of feedback motion control. Using this learning paradigm, the two control policies were synergistically combined to result in robust feedback control under novel transient and sustained disturbances that did not involve learning. Further, by comparing our data to experimental data generated by human subjects under the same conditions as those of the simulation, we showed that the proposed synergy concept may be used to analyze muscle synergy-driven motion control robustness in humans.

Differential coding of reward and movement information in the dorsomedial striatal direct and indirect pathways

The direct and indirect pathways of the basal ganglia have long been thought to mediate behavioral promotion and inhibition, respectively. However, this classic dichotomous model has been recently challenged. To better understand neural processes underlying reward-based learning and movement control, we recorded from direct (dSPNs) and indirect (iSPNs) pathway spiny projection neurons in the dorsomedial striatum of D1-Cre and D2-Cre mice performing a probabilistic Pavlovian conditioning task. dSPNs tend to increase activity while iSPNs decrease activity as a function of reward value, suggesting the striatum represents value in the relative activity levels of dSPNs versus iSPNs. Lick offset-related activity increase is largely dSPN selective, suggesting dSPN involvement in suppressing ongoing licking behavior. Rapid responses to negative outcome and previous reward-related responses are more frequent among iSPNs than dSPNs, suggesting stronger contributions of iSPNs to outcome-dependent behavioral adjustment. These findings provide new insights into striatal neural circuit operations.

Classically the direct and indirect pathways of basal ganglia are understood to have opposing roles in movement and reward learning, but recent work suggests a more complicated view. Here the authors further study indirect and direct pathway neurons, in the context of a probabilistic reward task.

Different involvement of medial prefrontal cortex and dorso-lateral striatum in automatic and controlled processing of a future conditioned stimulus

Recent studies support the idea that stimulus processing in latent inhibition can vary during the course of preexposure. Controlled attentional mechanisms are said to be important in the early stages of preexposure, while in later stages animals adopt automatic processing of the stimulus to be used for conditioning. Given this distinction, it is possible that both types of processing are governed by different neural systems, affecting differentially the retrieval of information about the stimulus. In the present study we tested if a lesion to the dorso-lateral striatum or to the medial prefrontal cortex has a selective effect on exposure to the future conditioned stimulus (CS). With this aim, animals received different amounts of exposure to the future CS. The results showed that a lesion to the medial prefrontal cortex enhanced latent inhibition in animals receiving limited preexposure to the CS, but had no effect in animals receiving extended preexposure to the CS. The lesion of the dorso-lateral striatum produced a decrease in latent inhibition, but only in animals with an extended exposure to the future conditioned stimulus. These results suggest that the dorsal striatum and medial prefrontal cortex play essential roles in controlled and automatic processes. Automatic attentional processes appear to be impaired by a lesion to the dorso-lateral striatum and facilitated by a lesion to the prefrontal cortex.

Illuminating Neural Circuits: From Molecules to MRI

Neurological disease drives symptoms through pathological changes to circuit functions. Therefore, understanding circuit mechanisms that drive behavioral dysfunction is of critical importance for quantitative diagnosis and systematic treatment of neurological disease. Here, we describe key technologies that enable measurement and manipulation of neural activity and neural circuits. Applying these approaches led to the discovery of circuit mechanisms underlying pathological motor behavior, arousal regulation, and protein accumulation. Finally, we discuss how optogenetic functional magnetic resonance imaging reveals global scale circuit mechanisms, and how circuit manipulations could lead to new treatments of neurological diseases.

Adaptive coordination of working-memory and reinforcement learning in non-human primates performing a trial-and-error problem solving task

Accumulating evidence suggest that human behavior in trial-and-error learning tasks based on decisions between discrete actions may involve a combination of reinforcement learning (RL) and working-memory (WM). While the understanding of brain activity at stake in this type of tasks often involve the comparison with non-human primate neurophysiological results, it is not clear whether monkeys use similar combined RL and WM processes to solve these tasks. Here we analyzed the behavior of five monkeys with computational models combining RL and WM. Our model-based analysis approach enables to not only fit trial-by-trial choices but also transient slowdowns in reaction times, indicative of WM use. We found that the behavior of the five monkeys was better explained in terms of a combination of RL and WM despite inter-individual differences. The same coordination dynamics we used in a previous study in humans best explained the behavior of some monkeys while the behavior of others showed the opposite pattern, revealing a possible different dynamics of WM process. We further analyzed different variants of the tested models to open a discussion on how the long pretraining in these tasks may have favored particular coordination dynamics between RL and WM. This points towards either inter-species differences or protocol differences which could be further tested in humans.

The role of the basal ganglia in the control of seizure

Epilepsy is a network disorder and each type of seizure involves distinct cortical and subcortical network, differently implicated in the control and propagation of the ictal activity. The role of the basal ganglia has been revealed in several cases of focal and generalized seizures. Here, we review the data that show the implication of the basal ganglia in absence, temporal lobe, and neocortical seizures in animal models (rodent, cat, and non-human primate) and in human. Based on these results and the advancement of deep brain stimulation for Parkinson's disease, basal ganglia neuromodulation has been tested with some success that can be equally seen as promising or disappointing. The effect of deep brain stimulation can be considered promising with a 76% in seizure reduction in temporal lobe epilepsy patients, but also disappointing, since only few patients have become seizure free and the antiepileptic effects have been highly variable among patients. This variability could probably be explained by the heterogeneity among the patients included in these clinical studies. To illustrate the importance of specific network identification, electrophysiological activity of the putamen and caudate nucleus has been recorded during penicillin-induced pre-frontal and motor seizures in one monkey. While an increase of the firing rate was found in putamen and caudate nucleus during pre-frontal seizures, only the activity of the putamen cells was increased during motor seizures. These preliminary results demonstrate the implication of the basal ganglia in two types of neocortical seizures and the necessity of studying the network to identify the important nodes implicated in the propagation and control of each type of seizure.

Computational Properties of the Hippocampus Increase the Efficiency of Goal-Directed Foraging through Hierarchical Reinforcement Learning

The mammalian brain is thought to use a version of Model-based Reinforcement Learning (MBRL) to guide “goal-directed” behavior, wherein animals consider goals and make plans to acquire desired outcomes. However, conventional MBRL algorithms do not fully explain animals' ability to rapidly adapt to environmental changes, or learn multiple complex tasks. They also require extensive computation, suggesting that goal-directed behavior is cognitively expensive. We propose here that key features of processing in the hippocampus support a flexible MBRL mechanism for spatial navigation that is computationally efficient and can adapt quickly to change. We investigate this idea by implementing a computational MBRL framework that incorporates features inspired by computational properties of the hippocampus: a hierarchical representation of space, “forward sweeps” through future spatial trajectories, and context-driven remapping of place cells. We find that a hierarchical abstraction of space greatly reduces the computational load (mental effort) required for adaptation to changing environmental conditions, and allows efficient scaling to large problems. It also allows abstract knowledge gained at high levels to guide adaptation to new obstacles. Moreover, a context-driven remapping mechanism allows learning and memory of multiple tasks. Simulating dorsal or ventral hippocampal lesions in our computational framework qualitatively reproduces behavioral deficits observed in rodents with analogous lesions. The framework may thus embody key features of how the brain organizes model-based RL to efficiently solve navigation and other difficult tasks.

Human motor adaptation in whole body motion

The main role of the sensorimotor system of an organism is to increase the survival of the species. Therefore, to understand the adaptation and optimality mechanisms of motor control, it is necessary to study the sensorimotor system in terms of ecological fitness. We designed an experimental paradigm that exposed sensorimotor system to risk of injury. We studied human subjects performing uncon- strained squat-to-stand movements that were systematically subjected to non-trivial perturbation. We found that subjects adapted by actively compensating the perturbations, converging to movements that were different from their normal unperturbed squat-to-stand movements. Furthermore, the adapted movements had clear intrinsic inter-subject differences which could be explained by different adapta- tion strategies employed by the subjects. These results suggest that classical optimality measures of physical energy and task satisfaction should be seen as part of a hierarchical organization of optimality with safety being at the highest level. Therefore, in addition to physical energy and task fulfillment, the risk of injury and other possible costs such as neural computational overhead have to be considered when analyzing human movement.