Model-based action planning involves cortico-cerebellar and basal ganglia networks

The role of training variability for model-based and model-free learning of an arbitrary visuomotor mapping

A fundamental feature of the human brain is its capacity to learn novel motor skills. This capacity requires the formation of vastly different visuomotor mappings. Using a grid navigation task, we investigated whether training variability would enhance the flexible use of a visuomotor mapping (key-to-direction rule), leading to better generalization performance. Experiments 1 and 2 show that participants trained to move between multiple start-target pairs exhibited greater generalization to both distal and proximal targets compared to participants trained to move between a single pair. This finding suggests that limited variability can impair decisions even in simple tasks without planning. In addition, during the training phase, participants exposed to higher variability were more inclined to choose options that, counterintuitively, moved the cursor away from the target while minimizing its actual distance under the constrained mapping, suggesting a greater engagement in model-based computations. In Experiments 3 and 4, we showed that the limited generalization performance in participants trained with a single pair can be enhanced by a short period of variability introduced early in learning or by incorporating stochasticity into the visuomotor mapping. Our computational modeling analyses revealed that a hybrid model between model-free and model-based computations with different mixing weights for the training and generalization phases, best described participants' data. Importantly, the differences in the model-based weights between our experimental groups, paralleled the behavioral findings during training and generalization. Taken together, our results suggest that training variability enables the flexible use of the visuomotor mapping, potentially by preventing the consolidation of habits due to the continuous demand to change responses.

Secondary cerebro-cerebellar and intra-cerebellar dysfunction in cerebellar mutism syndrome

BACKGROUND

Cerebellar mutism syndrome (CMS) is characterized by deficits of speech, movement, and affect that can occur following tumor removal from the posterior fossa. The role of cerebro-cerebellar tract injuries in the etiology of CMS remains unclear, with recent studies suggesting that cerebro-cerebellar dysfunction may be related to chronic, rather than transient, symptomatology.

METHODS

We measured functional connectivity between the cerebellar cortex and functional nodes throughout the brain using fMRI acquired after tumor removal but prior to adjuvant therapy in a cohort of 70 patients diagnosed with medulloblastoma. Surgical lesions were mapped to the infratentorial anatomy, and connectivity with cerebral cortex was tested for statistical dependence on extent of cerebellar outflow pathway injury.

RESULTS

CMS diagnosis was associated with an increase in connectivity between the right cerebellar and left cerebral hemisphere, maximally between cerebellum and ventromedial prefrontal cortex (VM-PFC). Connectivity dependence on cerebellar outflow was significant for some speech nodes but not for VM-PFC, suggesting altered input to the cerebellum. Connectivity between posterior regions of cerebellar cortex and ipsilateral dentate nuclei was abnormal in CMS participants, maximally within the right cerebellar hemisphere.

CONCLUSIONS

The functional abnormalities we identified are notably upstream of where causal surgical injury is thought to occur, indicating a secondary phenomenon. The VM-PFC is involved in several functions that may be relevant to the symptomatology of CMS, including emotional control and motor learning. We hypothesize that these abnormalities may reflect maladaptive learning within the cerebellum consequent to disordered motor and limbic function by the periaqueductal gray and other critical midbrain targets.

Synergizing habits and goals with variational Bayes

Behaving efficiently and flexibly is crucial for biological and artificial embodied agents. Behavior is generally classified into two types: habitual (fast but inflexible), and goal-directed (flexible but slow). While these two types of behaviors are typically considered to be managed by two distinct systems in the brain, recent studies have revealed a more sophisticated interplay between them. We introduce a theoretical framework using variational Bayesian theory, incorporating a Bayesian intention variable. Habitual behavior depends on the prior distribution of intention, computed from sensory context without goal-specification. In contrast, goal-directed behavior relies on the goal-conditioned posterior distribution of intention, inferred through variational free energy minimization. Assuming that an agent behaves using a synergized intention, our simulations in vision-based sensorimotor tasks explain the key properties of their interaction as observed in experiments. Our work suggests a fresh perspective on the neural mechanisms of habits and goals, shedding light on future research in decision making.

Doubling down on dual systems: A cerebellum-amygdala route towards action- and outcome-based social and affective behavior

The amygdala and cerebellum are both evolutionary preserved brain structures containing cortical as well as subcortical properties. For decades, the amygdala has been considered the fear-center of the brain, but recent advances have shown that the amygdala acts as a critical hub between cortical and subcortical systems and shapes social and affective behaviors beyond fear. Likewise, the cerebellum is a dedicated control unit that fine-tunes motor behavior to fit contextual requirements. There is however increasing evidence that the cerebellum strongly influences subcortical as well as cortical processes beyond the motor domain. These insights broadened the view on the cerebellum's functions to also include social and affective behavior. Here we explore how the amygdala and cerebellum might interact in shaping social and affective behaviors based on their roles in threat reactivity and reinforcement learning. A novel mechanistic neural framework of cerebellum-amygdala interactions will be presented which provides testable hypotheses for future social and affective neuroscientific research in humans.

An Integrated Neurorobotics Model of the Cerebellar-Basal Ganglia Circuitry

This work presents a neurorobotics model of the brain that integrates the cerebellum and the basal ganglia regions to coordinate movements in a humanoid robot. This cerebellar-basal ganglia circuitry is well known for its relevance to the motor control used by most mammals. Other computational models have been designed for similar applications in the robotics field. However, most of them completely ignore the interplay between neurons from the basal ganglia and cerebellum. Recently, neuroscientists indicated that neurons from both regions communicate not only at the level of the cerebral cortex but also at the subcortical level. In this work, we built an integrated neurorobotics model to assess the capacity of the network to predict and adjust the motion of the hands of a robot in real time. Our model was capable of performing different movements in a humanoid robot by respecting the sensorimotor loop of the robot and the biophysical features of the neuronal circuitry. The experiments were executed in simulation and the real world. We believe that our proposed neurorobotics model can be an important tool for new studies on the brain and a reference toward new robot motor controllers.

Enhancing reinforcement learning models by including direct and indirect pathways improves performance on striatal dependent tasks

A major advance in understanding learning behavior stems from experiments showing that reward learning requires dopamine inputs to striatal neurons and arises from synaptic plasticity of cortico-striatal synapses. Numerous reinforcement learning models mimic this dopamine-dependent synaptic plasticity by using the reward prediction error, which resembles dopamine neuron firing, to learn the best action in response to a set of cues. Though these models can explain many facets of behavior, reproducing some types of goal-directed behavior, such as renewal and reversal, require additional model components. Here we present a reinforcement learning model, TD2Q, which better corresponds to the basal ganglia with two Q matrices, one representing direct pathway neurons (G) and another representing indirect pathway neurons (N). Unlike previous two-Q architectures, a novel and critical aspect of TD2Q is to update the G and N matrices utilizing the temporal difference reward prediction error. A best action is selected for N and G using a softmax with a reward-dependent adaptive exploration parameter, and then differences are resolved using a second selection step applied to the two action probabilities. The model is tested on a range of multi-step tasks including extinction, renewal, discrimination; switching reward probability learning; and sequence learning. Simulations show that TD2Q produces behaviors similar to rodents in choice and sequence learning tasks, and that use of the temporal difference reward prediction error is required to learn multi-step tasks. Blocking the update rule on the N matrix blocks discrimination learning, as observed experimentally. Performance in the sequence learning task is dramatically improved with two matrices. These results suggest that including additional aspects of basal ganglia physiology can improve the performance of reinforcement learning models, better reproduce animal behaviors, and provide insight as to the role of direct- and indirect-pathway striatal neurons.

Decision heuristics in contexts integrating action selection and execution

Heuristics can inform human decision making in complex environments through a reduction of computational requirements (accuracy-resource trade-off) and a robustness to overparameterisation (less-is-more). However, tasks capturing the efficiency of heuristics typically ignore action proficiency in determining rewards. The requisite movement parameterisation in sensorimotor control questions whether heuristics preserve efficiency when actions are nontrivial. We developed a novel action selection-execution task requiring joint optimisation of action selection and spatio-temporal skillful execution. State-appropriate choices could be determined by a simple spatial heuristic, or by more complex planning. Computational models of action selection parsimoniously distinguished human participants who adopted the heuristic from those using a more complex planning strategy. Broader comparative analyses then revealed that participants using the heuristic showed combined decisional (selection) and skill (execution) advantages, consistent with a less-is-more framework. In addition, the skill advantage of the heuristic group was predominantly in the core spatial features that also shaped their decision policy, evidence that the dimensions of information guiding action selection might be yoked to salient features in skill learning.

Hippocampal and motor regions contribute to memory benefits after enacted encoding: cross-sectional and longitudinal evidence

The neurobiological underpinnings of action-related episodic memory and how enactment contributes to efficient memory encoding are not well understood. We examine whether individual differences in level (n = 338) and 5-year change (n = 248) in the ability to benefit from motor involvement during memory encoding are related to gray matter (GM) volume, white matter (WM) integrity, and dopamine-regulating genes in a population-based cohort (age range = 25-80 years). A latent profile analysis identified 2 groups with similar performance on verbal encoding but with marked differences in the ability to benefit from motor involvement during memory encoding. Impaired ability to benefit from enactment was paired with smaller HC, parahippocampal, and putamen volume along with lower WM microstructure in the fornix. Individuals with reduced ability to benefit from encoding enactment over 5 years were characterized by reduced HC and motor cortex GM volume along with reduced WM microstructure in several WM tracts. Moreover, the proportion of catechol-O-methyltransferase-Val-carriers differed significantly between classes identified from the latent-profile analysis. These results provide converging evidence that individuals with low or declining ability to benefit from motor involvement during memory encoding are characterized by low and reduced GM volume in regions critical for memory and motor functions along with altered WM microstructure.

Structural atrophy and functional dysconnectivity patterns in the cerebellum relate to cerebral networks in svMCI

Subcortical vascular mild cognitive impairment (svMCI) is associated with structural and functional changes in the cerebral cortex affecting major brain networks. While recent studies have shown that the intrinsic cerebral connectivity networks can be mapped onto the cerebellum, and the cortex and cerebellum are interconnected via the cortico-basal ganglia-cerebellar circuit, structural and functional disruptions in cerebellum in svMCI are rarely studied. In this study, we conducted voxel-based morphometry analysis to investigate gray matter atrophy pattern across cerebellar regions in 40 svMCI patients, and explored alterations in functional connectivity between the basal ganglia and cerebellum. The results showed that the amount of cerebellar atrophy within the default mode, salience, and frontoparietal networks correlated with their counterpart in the cerebral cortex. Moreover, key regions of the cerebellum, including the lobule VI, VIIb, VIII, and Crus I, which are reported to have a role in cognitive function, showed both anatomical atrophy and decreased functional connectivity with the striatum. These atrophy and connectivity patterns in the cerebellum also correlated with memory performances. These findings demonstrate that there are coupled changes in cerebral and cerebellar circuits, reflecting that degeneration patterns in svMCI are not limited to the cerebral cortex but similarly extend to the cerebellum as well, and suggest the cortico-basal ganglia-cerebellar circuit may play an important role in the pathology of svMCI.

Ventrolateral Prefrontal Cortex Contributes to Human Motor Learning

This study assesses the involvement in human motor learning, of the ventrolateral prefrontal cortex (BA 9/46v), a somatic region in the middle frontal gyrus. The potential involvement of this cortical area in motor learning is suggested by studies in nonhuman primates which have found anatomic connections between this area and sensorimotor regions in frontal and parietal cortex, and also with basal ganglia output zones. It is likewise suggested by electrophysiological studies which have shown that activity in this region is implicated in somatic sensory memory and is also influenced by reward. We directly tested the hypothesis that area 9/46v is involved in reinforcement-based motor learning in humans. Participants performed reaching movements to a hidden target and received positive feedback when successful. Before the learning task, we applied continuous theta burst stimulation (cTBS) to disrupt activity in 9/46v in the left or right hemisphere. A control group received sham cTBS. The data showed that cTBS to left 9/46v almost entirely eliminated motor learning, whereas learning was not different from sham stimulation when cTBS was applied to the same zone in the right hemisphere. Additional analyses showed that the basic reward-history-dependent pattern of movements was preserved but more variable following left hemisphere stimulation, which suggests an overall deficit in somatic memory for target location or target directed movement rather than reward processing per se. The results indicate that area 9/46v is part of the human motor learning circuit.

The effect of self-generated versus externally generated actions on timing, duration, and amplitude of blood oxygen level dependent response for visual feedback processing

It has been widely assumed that internal forward models use efference copies to create predictions about the sensory consequences of our own actions. While these predictions have frequently been associated with a reduced blood oxygen level dependent (BOLD) response in sensory cortices, the timing and duration of the hemodynamic response for the processing of video feedback of self‐generated (active) versus externally generated (passive) movements is poorly understood. In the present study, we tested the hypothesis that predictive mechanisms for self‐generated actions lead to early and shorter neural processing compared with externally generated movements. We investigated active and passive movements using a custom‐made fMRI‐compatible movement device. Visual video feedback of the active and passive movements was presented in real time or with variable delays. Participants had to judge whether the feedback was delayed. Timing and duration of BOLD impulse response was calculated using a first (temporal derivative [TD]) and second‐order (dispersion derivative [DD]) Taylor approximation. Our reanalysis confirmed our previous finding of reduced BOLD response for active compared to passive movements. Moreover, we found positive effects of the TD and DD in the supplementary motor area, cerebellum, visual cortices, and subcortical structures, indicating earlier and shorter hemodynamic responses for active compared to passive movements. Furthermore, earlier activation in the putamen for active compared to passive conditions was associated with reduced delay detection performance. These findings indicate that efference copy‐based predictive mechanisms enable earlier processing of action feedback, which might have reduced the ability to detect short delays between action and feedback.

Atypical cerebellar functional connectivity at 9 months of age predicts delayed socio-communicative profiles in infants at high and low risk for autism

BACKGROUND

While the cerebellum is traditionally known for its role in sensorimotor control, emerging research shows that particular subregions, such as right Crus I (RCrusI), support language and social processing. Indeed, cerebellar atypicalities are commonly reported in autism spectrum disorder (ASD), a neurodevelopmental disorder characterized by socio-communicative impairments. However, the cerebellum's contribution to early socio-communicative development remains virtually unknown.

METHODS

Here, we characterized functional connectivity within cerebro-cerebellar networks implicated in language/social functions in 9-month-old infants who exhibit distinct 3-year socio-communicative developmental profiles. We employed a data-driven clustering approach to stratify our sample of infants at high (n = 82) and low (n = 37) familial risk for ASD into three cohorts-Delayed, Late-Blooming, and Typical-who showed unique socio-communicative trajectories. We then compared the cohorts on indices of language and social development. Seed-based functional connectivity analyses with RCrusI were conducted on infants with fMRI data (n = 66). Cohorts were compared on connectivity estimates from a-priori regions, selected on the basis of reported coactivation with RCrusI during language/social tasks.

RESULTS

The three trajectory-based cohorts broadly differed in social communication development, as evidenced by robust differences on numerous indices of language and social skills. Importantly, at 9 months, the cohorts showed striking differences in cerebro-cerebellar circuits implicated in language/social functions. For all regions examined, the Delayed cohort exhibited significantly weaker RCrusI connectivity compared to both the Late-Blooming and Typical cohorts, with no significant differences between the latter cohorts.

CONCLUSIONS

We show that hypoconnectivity within distinct cerebro-cerebellar networks in infancy predicts altered socio-communicative development before delays overtly manifest, which may be relevant for early detection and intervention. As the cerebellum is implicated in prediction, our findings point to probabilistic learning as a potential intermediary mechanism that may be disrupted in infancy, cascading into alterations in social communication.

An fMRI meta-analysis of the role of the striatum in everyday-life vs laboratory-developed habits

The dorsolateral striatum plays a critical role in the acquisition and expression of stimulus-response habits that are learned in experimental laboratories. Here, we use meta-analytic procedures to contrast the neural circuits activated by laboratory-acquired habits with those activated by stimulus-response behaviours acquired in everyday-life. We confirmed that newly learned habits rely more on the anterior putamen with activation extending into caudate and nucleus accumbens. Motor and associative components of everyday-life habits were identified. We found that motor-dominant stimulus-response associations developed outside the laboratory primarily engaged posterior dorsal putamen, supplementary motor area (SMA) and cerebellum. Importantly, associative components were also represented in the posterior putamen. Thus, common neural representations for both naturalistic and laboratory-based habits were found in the left posterior and right anterior putamen. These findings suggest a partial common striatal substrate for habitual actions that are performed predominantly by stimulus-response associations represented in the posterior striatum. The overlapping neural substrates for laboratory and everyday-life habits supports the use of both methods for the analysis of habitual behaviour.

Reward prediction errors, not sensory prediction errors, play a major role in model selection in human reinforcement learning

Model-based reinforcement learning enables an agent to learn in variable environments and tasks by optimizing its actions based on the predicted states and outcomes. This mechanism has also been considered in the brain. However, exactly how the brain selects an appropriate model for confronting environments has remained unclear. Here, we investigated the model selection algorithm in the human brain during a reinforcement learning task. One primary theory of model selection in the brain is based on sensory prediction errors. Here, we compared this theory with an alternative possibility of internal model selection with reward prediction errors. To compare these two theories, we devised a switching experiment from a first-order Markov decision process to a second-order Markov decision process that provides either reward- or sensory prediction error regarding environmental change. We tested two representative computational models driven by different prediction errors. One is the sensory prediction-error-driven Bayesian algorithm, which has been discussed as a representative internal model selection algorithm in the animal reinforcement learning task. The other is the reward-prediction-error-driven policy gradient algorithm. We compared the simulation results of these two computational models with human reinforcement learning behaviors. The model fitting result supports that the policy gradient algorithm is preferable to the Bayesian algorithm. This suggests that the human brain employs the reward prediction error to select an appropriate internal model in the reinforcement learning task.

An insula hierarchical network architecture for active interoceptive inference

In the brain, the insular cortex receives a vast amount of interoceptive information, ascending through deep brain structures, from multiple visceral organs. The unique hierarchical and modular architecture of the insula suggests specialization for processing interoceptive afferents. Yet, the biological significance of the insula's neuroanatomical architecture, in relation to deep brain structures, remains obscure. In this opinion piece, we propose the Insula Hierarchical Modular Adaptive Interoception Control (IMAC) model to suggest that insula modules (granular, dysgranular and agranular), forming parallel networks with the prefrontal cortex and striatum, are specialized to form higher order interoceptive representations. These interoceptive representations are recruited in a context-dependent manner to support habitual, model-based and exploratory control of visceral organs and physiological processes. We discuss how insula interoceptive representations may give rise to conscious feelings that best explain lower order deep brain interoceptive representations, and how the insula may serve to defend the body and mind against pathological depression.

Linking Pain and Motor Control: Conceptualization of Movement Deficits in Patients With Painful Conditions

When people experience or expect pain, they move differently. Pain-altered movement strategies, collectively described here as pain-related movement dysfunction (PRMD), may persist well after pain resolves and, ultimately, may result in altered kinematics and kinetics, future reinjury, and disability. Although PRMD may manifest as abnormal movements that are often evident in clinical assessment, the underlying mechanisms are complex, engaging sensory-perceptual, cognitive, psychological, and motor processes. Motor control theories provide a conceptual framework to determine, assess, and target processes that contribute to normal and abnormal movement and thus are important for physical therapy and rehabilitation practice. Contemporary understanding of motor control has evolved from reflex-based understanding to a more complex task-dependent interaction between cognitive and motor systems, each with distinct neuroanatomic substrates. Though experts have recognized the importance of motor control in the management of painful conditions, there is no comprehensive framework that explicates the processes engaged in the control of goal-directed actions, particularly in the presence of pain. This Perspective outlines sensory-perceptual, cognitive, psychological, and motor processes in the contemporary model of motor control, describing the neural substrates underlying each process and highlighting how pain and anticipation of pain influence motor control processes and consequently contribute to PRMD. Finally, potential lines of future inquiry-grounded in the contemporary model of motor control-are outlined to advance understanding and improve the assessment and treatment of PRMD.

IMPACT

This Perspective proposes that approaching PRMD from a contemporary motor control perspective will uncover key mechanisms, identify treatment targets, inform assessments, and innovate treatments across sensory-perceptual, cognitive, and motor domains, all of which have the potential to improve movement and functional outcomes in patients with painful conditions.

A whole brain probabilistic generative model: Toward realizing cognitive architectures for developmental robots

Building a human-like integrative artificial cognitive system, that is, an artificial general intelligence (AGI), is the holy grail of the artificial intelligence (AI) field. Furthermore, a computational model that enables an artificial system to achieve cognitive development will be an excellent reference for brain and cognitive science. This paper describes an approach to develop a cognitive architecture by integrating elemental cognitive modules to enable the training of the modules as a whole. This approach is based on two ideas: (1) brain-inspired AI, learning human brain architecture to build human-level intelligence, and (2) a probabilistic generative model (PGM)-based cognitive architecture to develop a cognitive system for developmental robots by integrating PGMs. The proposed development framework is called a whole brain PGM (WB-PGM), which differs fundamentally from existing cognitive architectures in that it can learn continuously through a system based on sensory-motor information. In this paper, we describe the rationale for WB-PGM, the current status of PGM-based elemental cognitive modules, their relationship with the human brain, the approach to the integration of the cognitive modules, and future challenges. Our findings can serve as a reference for brain studies. As PGMs describe explicit informational relationships between variables, WB-PGM provides interpretable guidance from computational sciences to brain science. By providing such information, researchers in neuroscience can provide feedback to researchers in AI and robotics on what the current models lack with reference to the brain. Further, it can facilitate collaboration among researchers in neuro-cognitive sciences as well as AI and robotics.

The basal ganglia and the cerebellum in human emotion

The basal ganglia (BG) and the cerebellum historically have been relegated to a functional role in producing or modulating motor output. Recent research, however, has emphasized the importance of these subcortical structures in multiple functional domains, including affective processes such as emotion recognition, subjective feeling elicitation and reward valuation. The pathways through the thalamus that connect the BG and cerebellum directly to each other and with extensive regions of the cortex provide a structural basis for their combined influence on limbic function. By regulating cortical oscillations to guide learning and strengthening rewarded behaviors or thought patterns to achieve a desired goal state, these regions can shape the way an individual processes emotional stimuli. This review will discuss the basic structure and function of the BG and cerebellum and propose an updated view of their functional role in human affective processing.

Stress, memory, and implications for major depression

The stress response comprises a phylogenetically conserved set of cognitive, physiological, and behavioral responses that evolved as a survival strategy. In this context, the memory of stressful events would be adaptive as it could avoid re-exposure to an adverse event, otherwise the event would be facilitated in positively stressful or non-distressful conditions. However, the interaction between stress and memory comprises complex responses, some of them which are not yet completely understood, and which depend on several factors such as the memory system that is recruited, the nature and duration of the stressful event, as well as the timing in which this interaction takes place. In this narrative review, we briefly discuss the mechanisms of the stress response, the main memory systems, and its neural correlates. Then, we show how stress, through the action of its biochemical mediators, influences memory systems and mnemonic processes. Finally, we make use of major depressive disorder to explore the possible implications of non-adaptive interactions between stress and memory to psychiatric disorders, as well as possible roles for memory studies in the field of psychiatry.

Cognitive and Motor Learning in Internally-Guided Motor Skills

Several canonical experimental paradigms (e.g., serial reaction time task, discrete sequence production task, m × n task) have been proposed to study the typical behavioral phenomenon and the nature of learning in sequential keypress tasks. A characteristic feature of most paradigms is that they are representative of externally-specified sequencing—motor tasks where the environment or task paradigm extrinsically provides the sequence of stimuli, i.e., the responses are stimulus-driven. Previous studies utilizing such canonical paradigms have largely overlooked the learning behaviors in a more realistic class of motor tasks that involve internally-guided sequencing—where the sequence of motor actions is self-generated or internally-specified. In this work, we use the grid-navigation task as an instance of internally-guided sequencing to investigate the nature of learning in such paradigms. The participants performed Grid-Sailing Task (GST), which required navigating (by executing sequential keypresses) a 5 × 5 grid from start to goal (SG) position while using a particular key-mapping (KM) among the three cursor-movement directions and the three keyboard buttons. The participants performed two behavioral experiments—Single-SG and Mixed-SG condition. The Single-SG condition required performing GST on a single SG position repeatedly, whereas the Mixed-SG condition involved performing GST using the same KM on two novel SG positions presented in a random, inter-mixed manner. In the Single-SG condition, we show that motor learning contributes to the sequence-specific learning in GST with the repeated execution of the same trajectories. In the Mixed-SG condition, since the participants utilize the previously learned KM, we anticipate a transfer of learning from the Single-SG condition. The acquisition and transfer of a KM-specific internal model facilitates efficient trajectory planning on novel SG conditions. The acquisition of such a KM-specific internal model amounts to trajectory-independent cognitive learning in GST. We show that cognitive learning contributes to the learning in GST by showing transfer-related performance improvements in the Mixed-SG condition. In sum, we show the role of cognitive and motor learning processes in internally-guided sequencing and further make a case for using GST-like grid-navigation paradigms in investigating internally guided skill learning.

Enhanced activations in syntax-related regions for multilinguals while acquiring a new language

The neuroscientific foundation of multilingualism, a unique cognitive capacity, necessitates further elucidation. We conducted an fMRI experiment to evaluate the acquisition of syntactic features in a new language (Kazakh) for multilinguals and bilinguals. Results showed that the multilinguals who were more proficient in their second/third languages needed fewer task trials to acquire Kazakh phonology. Regarding group differences, the reduction in response times during the initial exposure to Kazakh were significantly larger for the multilinguals than the bilinguals. For the multilinguals, activations in the bilateral frontal/temporal regions were maintained at a higher level than the initial level during subsequent new grammar conditions. For the bilinguals, activations in the basal ganglia/thalamus and cerebellum decreased to the initial level each time. Direct group comparisons showed significantly enhanced activations for the multilinguals in the left ventral inferior frontal gyrus. These results indicate that both syntax-related and domain-general brain networks were more enhanced for the multilinguals. We also unexpectedly observed significant activations in the visual areas for the multilinguals, implying the use of visual representation even when listening to speech sounds alone. Because the multilinguals were able to successfully utilize acquired knowledge in an accumulated manner, the results support the cumulative-enhancement model of language acquisition.

Motor Chunking in Internally Guided Sequencing

Motor skill learning involves the acquisition of sequential motor movements with practice. Studies have shown that we learn to execute these sequences efficiently by chaining several elementary actions in sub-sequences called motor chunks. Several experimental paradigms, such as serial reaction task, discrete sequence production, and m × n task, have investigated motor chunking in externally specified sequencing where the environment or task paradigm provides the sequence of stimuli, i.e., the responses are stimulus driven. In this study, we examine motor chunking in a class of more realistic motor tasks that involve internally guided sequencing where the sequence of motor actions is self-generated or internally specified. We employ a grid-navigation task as an exemplar of internally guided sequencing to investigate practice-driven performance improvements due to motor chunking. The participants performed the grid-sailing task (GST) (Fermin et al., 2010), which required navigating (by executing sequential keypresses) a 10 × 10 grid from start to goal position while using a particular type of key mapping between the three cursor movement directions and the three keyboard buttons. We provide empirical evidence for motor chunking in grid-navigation tasks by showing the emergence of subject-specific, unique temporal patterns in response times. Our findings show spontaneous chunking without pre-specified or externally guided structures while replicating the earlier results with a less constrained, internally guided sequencing paradigm.

Cortical activity during sensorial tactile stimulation in healthy adults through Vojta therapy. A randomized pilot controlled trial

Background

Brain’s is stimulated by Vojta Therapy through selected body areas activating stored innate motor programs which are exported as coordinate movement and muscle contractions to trunk and limbs. The aim of this pilot study is to know the responses at cortical level to a specific tactile input, assessed by electroencephalography (EEG), compared to a sham stimulation, in healthy subjects.

Methods

A randomized-controlled trial was conducted. Participants were randomly distributed into two groups: a non-specific tactile input-group (non-STI-group) (n = 20) and a Vojta specific tactile input-group (V-STI-group) (n = 20). The non-STI-group was stimulated in a non specific area (quadriceps distal area) and V-STI-group was stimulated in a specific area (intercostal space, at the mammillary line between the 7th and 8th ribs) according to the Vojta therapy. Recording was performed with EEG for 10 min considering a first minute of rest, 8 min during the stimulus and 1 min after the stimulus. EEG activity was recorded from 32 positions with active Ag/AgCl scalp electrodes following the 10–20 system. The continuous EEG signal was split into consecutive segments of one minute.

Results

The V-STI-group showed statistically significant differences in the theta, low alpha and high alpha bands, bilaterally in the supplementary motor (SMA) and premotor (PMA) areas (BA6 and BA8), superior parietal cortex (BA5, BA7) and the posterior cingulate cortex (BA23, BA31). For the V-STI-group, all frequency bands presented an initial bilateral activation of the superior and medial SMA (BA6) during the first minute. This activation was maintained until the fourth minute. During the fourth minute, the activation decreased in the three frequency bands. From the fifth minute, the activation in the superior and medial SMA rose again in the three frequency bands

Conclusions

Our findings highlight that the specific stimulation area at intercostal space, on the mammillary line between 7 and 8th ribs according to Vojta therapy differentially increased bilateral activation in SMA (BA6) and Pre-SMA (BA8), BA5, BA7, BA23 and BA31 in the theta, low and high alpha bands in healthy subjects. These results could indicate the activation of innate locomotor circuits during stimulation of the pectoral area according to the Vojta therapy.

Trial registration Retrospectively registered. This randomized controlled trial has been registered at ClinicalTrials.gov Identifier: NCT04317950 (March 23, 2020).

Beyond dichotomies in reinforcement learning

Reinforcement learning (RL) is a framework of particular importance to psychology, neuroscience and machine learning. Interactions between these fields, as promoted through the common hub of RL, has facilitated paradigm shifts that relate multiple levels of analysis in a singular framework (for example, relating dopamine function to a computationally defined RL signal). Recently, more sophisticated RL algorithms have been proposed to better account for human learning, and in particular its oft-documented reliance on two separable systems: a model-based (MB) system and a model-free (MF) system. However, along with many benefits, this dichotomous lens can distort questions, and may contribute to an unnecessarily narrow perspective on learning and decision-making. Here, we outline some of the consequences that come from overconfidently mapping algorithms, such as MB versus MF RL, with putative cognitive processes. We argue that the field is well positioned to move beyond simplistic dichotomies, and we propose a means of refocusing research questions towards the rich and complex components that comprise learning and decision-making.

Goal-oriented and habitual decisions: Neural signatures of model-based and model-free learning

Human decision-making is mainly driven by two fundamental learning processes: a slow, deliberative, goal-directed model-based process that maps out the potential outcomes of all options and a rapid habitual model-free process that enables reflexive repetition of previously successful choices. Although many model-informed neuroimaging studies have examined the neural correlates of model-based and model-free learning, the concordant activity among these two processes remains unclear. We used quantitative meta-analyses of functional magnetic resonance imaging experiments to identify the concordant activity pertaining to model-based and model-free learning over a range of reward-related paradigms. We found that: 1) both processes yielded concordant ventral striatum activity, 2) model-based learning activated the medial prefrontal cortex and orbital frontal cortex, and 3) model-free learning specifically activated the left globus pallidus and right caudate head. Our findings suggest that model-free and model-based decision making engage overlapping yet distinct neural regions. These stereotaxic maps improve our understanding of how deliberative goal-directed and reflexive habitual learning are implemented in the brain.

Generalizing to generalize: Humans flexibly switch between compositional and conjunctive structures during reinforcement learning

Humans routinely face novel environments in which they have to generalize in order to act adaptively. However, doing so involves the non-trivial challenge of deciding which aspects of a task domain to generalize. While it is sometimes appropriate to simply re-use a learned behavior, often adaptive generalization entails recombining distinct components of knowledge acquired across multiple contexts. Theoretical work has suggested a computational trade-off in which it can be more or less useful to learn and generalize aspects of task structure jointly or compositionally, depending on previous task statistics, but it is unknown whether humans modulate their generalization strategy accordingly. Here we develop a series of navigation tasks that separately manipulate the statistics of goal values ("what to do") and state transitions ("how to do it") across contexts and assess whether human subjects generalize these task components separately or conjunctively. We find that human generalization is sensitive to the statistics of the previously experienced task domain, favoring compositional or conjunctive generalization when the task statistics are indicative of such structures, and a mixture of the two when they are more ambiguous. These results support a normative "meta-generalization" account and suggests that people not only generalize previous task components but also generalize the statistical structure most likely to support generalization.

The basal ganglia and the cerebellum: nodes in an integrated network

The basal ganglia and the cerebellum are considered to be distinct subcortical systems that perform unique functional operations. The outputs of the basal ganglia and the cerebellum influence many of the same cortical areas but do so by projecting to distinct thalamic nuclei. As a consequence, the two subcortical systems were thought to be independent and to communicate only at the level of the cerebral cortex. Here, we review recent data showing that the basal ganglia and the cerebellum are interconnected at the subcortical level. The subthalamic nucleus in the basal ganglia is the source of a dense disynaptic projection to the cerebellar cortex. Similarly, the dentate nucleus in the cerebellum is the source of a dense disynaptic projection to the striatum. These observations lead to a new functional perspective that the basal ganglia, the cerebellum and the cerebral cortex form an integrated network. This network is topographically organized so that the motor, cognitive and affective territories of each node in the network are interconnected. This perspective explains how synaptic modifications or abnormal activity at one node can have network-wide effects. A future challenge is to define how the unique learning mechanisms at each network node interact to improve performance.

Corticocortical Systems Underlying High-Order Motor Control

Cortical networks are characterized by the origin, destination, and reciprocity of their connections, as well as by the diameter, conduction velocity, and synaptic efficacy of their axons. The network formed by parietal and frontal areas lies at the core of cognitive-motor control because the outflow of parietofrontal signaling is conveyed to the subcortical centers and spinal cord through different parallel pathways, whose orchestration determines, not only when and how movements will be generated, but also the nature of forthcoming actions. Despite intensive studies over the last 50 years, the role of corticocortical connections in motor control and the principles whereby selected cortical networks are recruited by different task demands remain elusive. Furthermore, the synaptic integration of different cortical signals, their modulation by transthalamic loops, and the effects of conduction delays remain challenging questions that must be tackled to understand the dynamical aspects of parietofrontal operations. In this article, we evaluate results from nonhuman primate and selected rodent experiments to offer a viewpoint on how corticocortical systems contribute to learning and producing skilled actions. Addressing this subject is not only of scientific interest but also essential for interpreting the devastating consequences for motor control of lesions at different nodes of this integrated circuit. In humans, the study of corticocortical motor networks is currently based on MRI-related methods, such as resting-state connectivity and diffusion tract-tracing, which both need to be contrasted with histological studies in nonhuman primates.

Compositional clustering in task structure learning

Humans are remarkably adept at generalizing knowledge between experiences in a way that can be difficult for computers. Often, this entails generalizing constituent pieces of experiences that do not fully overlap, but nonetheless share useful similarities with, previously acquired knowledge. However, it is often unclear how knowledge gained in one context should generalize to another. Previous computational models and data suggest that rather than learning about each individual context, humans build latent abstract structures and learn to link these structures to arbitrary contexts, facilitating generalization. In these models, task structures that are more popular across contexts are more likely to be revisited in new contexts. However, these models can only re-use policies as a whole and are unable to transfer knowledge about the transition structure of the environment even if only the goal has changed (or vice-versa). This contrasts with ecological settings, where some aspects of task structure, such as the transition function, will be shared between context separately from other aspects, such as the reward function. Here, we develop a novel non-parametric Bayesian agent that forms independent latent clusters for transition and reward functions, affording separable transfer of their constituent parts across contexts. We show that the relative performance of this agent compared to an agent that jointly clusters reward and transition functions depends environmental task statistics: the mutual information between transition and reward functions and the stochasticity of the observations. We formalize our analysis through an information theoretic account of the priors, and propose a meta learning agent that dynamically arbitrates between strategies across task domains to optimize a statistical tradeoff.

Mapping the human brain during a specific Vojta's tactile input: the ipsilateral putamen's role

A century of research in human brain parcellation has demonstrated that different brain areas are associated with functional tasks. New neuroscientist perspectives to achieve the parcellation of the human brain have been developed to know the brain areas activation and its relationship with different stimuli. This descriptive study aimed to compare brain regions activation by specific tactile input (STI) stimuli according to the Vojta protocol (STI-group) to a non-STI stimulation (non-STI-group). An exploratory functional magnetic resonance imaging (fMRI) study was performed. The 2 groups of participants were passively stimulated by an expert physical therapist using the same paradigm structure, although differing in the place of stimulation. The stimulation was presented to participants using a block design in all cases. A sample of 16 healthy participants, 5 men and 11 women, with mean age 31.31 ± 8.13 years was recruited. Indeed, 12 participants were allocated in the STI-group and 4 participants in the non-STI-group. fMRI was used to map the human brain in vivo while these tactile stimuli were being applied. Data were analyzed using a general linear model in SPM12 implemented in MATLAB. Differences between groups showed a greater activation in the right cortical areas (temporal and frontal lobes), subcortical regions (thalamus, brainstem, and basal nuclei), and in the cerebellum (anterior lobe). STI-group had specific difference brain activation areas, such as the ipsilateral putamen. Future studies should study clinical implications in neurorehabilitation patients.

A hippocampo-cerebellar centred network for the learning and execution of sequence-based navigation

How do we translate self-motion into goal-directed actions? Here we investigate the cognitive architecture underlying self-motion processing during exploration and goal-directed behaviour. The task, performed in an environment with limited and ambiguous external landmarks, constrained mice to use self-motion based information for sequence-based navigation. The post-behavioural analysis combined brain network characterization based on c-Fos imaging and graph theory analysis as well as computational modelling of the learning process. The study revealed a widespread network centred around the cerebral cortex and basal ganglia during the exploration phase, while a network dominated by hippocampal and cerebellar activity appeared to sustain sequence-based navigation. The learning process could be modelled by an algorithm combining memory of past actions and model-free reinforcement learning, which parameters pointed toward a central role of hippocampal and cerebellar structures for learning to translate self-motion into a sequence of goal-directed actions.

Effect of Error Augmentation on Brain Activation and Motor Learning of a Complex Locomotor Task

Up to date, the functional gains obtained after robot-aided gait rehabilitation training are limited. Error augmenting strategies have a great potential to enhance motor learning of simple motor tasks. However, little is known about the effect of these error modulating strategies on complex tasks, such as relearning to walk after a neurologic accident. Additionally, neuroimaging evaluation of brain regions involved in learning processes could provide valuable information on behavioral outcomes. We investigated the effect of robotic training strategies that augment errors—error amplification and random force disturbance—and training without perturbations on brain activation and motor learning of a complex locomotor task. Thirty-four healthy subjects performed the experiment with a robotic stepper (MARCOS) in a 1.5 T MR scanner. The task consisted in tracking a Lissajous figure presented on a display by coordinating the legs in a gait-like movement pattern. Behavioral results showed that training without perturbations enhanced motor learning in initially less skilled subjects, while error amplification benefited better-skilled subjects. Training with error amplification, however, hampered transfer of learning. Randomly disturbing forces induced learning and promoted transfer in all subjects, probably because the unexpected forces increased subjects' attention. Functional MRI revealed main effects of training strategy and skill level during training. A main effect of training strategy was seen in brain regions typically associated with motor control and learning, such as, the basal ganglia, cerebellum, intraparietal sulcus, and angular gyrus. Especially, random disturbance and no perturbation lead to stronger brain activation in similar brain regions than error amplification. Skill-level related effects were observed in the IPS, in parts of the superior parietal lobe (SPL), i.e., precuneus, and temporal cortex. These neuroimaging findings indicate that gait-like motor learning depends on interplay between subcortical, cerebellar, and fronto-parietal brain regions. An interesting observation was the low activation observed in the brain's reward system after training with error amplification compared to training without perturbations. Our results suggest that to enhance learning of a locomotor task, errors should be augmented based on subjects' skill level. The impacts of these strategies on motor learning, brain activation, and motivation in neurological patients need further investigation.

The Neural Basis of Aversive Pavlovian Guidance during Planning

Important real-world decisions are often arduous as they frequently involve sequences of choices, with initial selections affecting future options. Evaluating every possible combination of choices is computationally intractable, particularly for longer multistep decisions. Therefore, humans frequently use heuristics to reduce the complexity of decisions. We recently used a goal-directed planning task to demonstrate the profound behavioral influence and ubiquity of one such shortcut, namely aversive pruning, a reflexive Pavlovian process that involves neglecting parts of the decision space residing beyond salient negative outcomes. However, how the brain implements this important decision heuristic and what underlies individual differences have hitherto remained unanswered. Therefore, we administered an adapted version of the same planning task to healthy male and female volunteers undergoing functional magnetic resonance imaging (fMRI) to determine the neural basis of aversive pruning. Through both computational and standard categorical fMRI analyses, we show that when planning was influenced by aversive pruning, the subgenual cingulate cortex was robustly recruited. This neural signature was distinct from those associated with general planning and valuation, two fundamental cognitive components elicited by our task but which are complementary to aversive pruning. Furthermore, we found that individual variation in levels of aversive pruning was associated with the responses of insula and dorsolateral prefrontal cortices to the receipt of large monetary losses, and also with subclinical levels of anxiety. In summary, our data reveal the neural signatures of an important reflexive Pavlovian process that shapes goal-directed evaluations and thereby determines the outcome of high-level sequential cognitive processes.SIGNIFICANCE STATEMENT Multistep decisions are complex because initial choices constrain future options. Evaluating every path for long decision sequences is often impractical; thus, cognitive shortcuts are often essential. One pervasive and powerful heuristic is aversive pruning, in which potential decision-making avenues are curtailed at immediate negative outcomes. We used neuroimaging to examine how humans implement such pruning. We found it to be associated with activity in the subgenual cingulate cortex, with neural signatures that were distinguishable from those covarying with planning and valuation. Individual variations in aversive pruning levels related to subclinical anxiety levels and insular cortex activation. These findings reveal the neural mechanisms by which basic negative Pavlovian influences guide decision-making during planning, with implications for disrupted decision-making in psychiatric disorders.

Predictive representations can link model-based reinforcement learning to model-free mechanisms

Humans and animals are capable of evaluating actions by considering their long-run future rewards through a process described using model-based reinforcement learning (RL) algorithms. The mechanisms by which neural circuits perform the computations prescribed by model-based RL remain largely unknown; however, multiple lines of evidence suggest that neural circuits supporting model-based behavior are structurally homologous to and overlapping with those thought to carry out model-free temporal difference (TD) learning. Here, we lay out a family of approaches by which model-based computation may be built upon a core of TD learning. The foundation of this framework is the successor representation, a predictive state representation that, when combined with TD learning of value predictions, can produce a subset of the behaviors associated with model-based learning, while requiring less decision-time computation than dynamic programming. Using simulations, we delineate the precise behavioral capabilities enabled by evaluating actions using this approach, and compare them to those demonstrated by biological organisms. We then introduce two new algorithms that build upon the successor representation while progressively mitigating its limitations. Because this framework can account for the full range of observed putatively model-based behaviors while still utilizing a core TD framework, we suggest that it represents a neurally plausible family of mechanisms for model-based evaluation.