Reward Expectation Modulates Local Field Potentials, Spiking Activity and Spike-Field Coherence in the Primary Motor Cortex

Dorsomedial frontal cortex damage impairs error-based, but not reinforcement-based motor learning in humans

We adapt our movements to new and changing environments through multiple processes. Sensory error-based learning counteracts environmental perturbations that affect the sensory consequences of movements. Sensory errors also cause the upregulation of reflexes and muscle co-contraction. Reinforcement-based learning enhances the selection of movements that produce rewarding outcomes. Although some findings have identified dissociable neural substrates of sensory error- and reinforcement-based learning, correlative methods have implicated dorsomedial frontal cortex in both. Here, we tested the causal contributions of dorsomedial frontal to adaptive motor control, studying people with chronic damage to this region. Seven human participants with focal brain lesions affecting the dorsomedial frontal and 20 controls performed a battery of arm movement tasks. Three experiments tested: (i) the upregulation of visuomotor reflexes and muscle co-contraction in response to unpredictable mechanical perturbations, (ii) sensory error-based learning in which participants learned to compensate predictively for mechanical force-field perturbations, and (iii) reinforcement-based motor learning based on binary feedback in the absence of sensory error feedback. Participants with dorsomedial frontal damage were impaired in the early stages of force field adaptation, but performed similarly to controls in all other measures. These results provide evidence for a specific and selective causal role for the dorsomedial frontal in sensory error-based learning.

Electrophysiological population dynamics reveal context dependencies during decision making in human frontal cortex

Evidence from monkeys and humans suggests that the orbitofrontal cortex (OFC) encodes the subjective value of options under consideration during choice. Data from non-human primates suggests that these value signals are context-dependent, representing subjective value in a way influenced by the decision makers' recent experience. Using electrodes distributed throughout cortical and subcortical structures, human epilepsy patients performed an auction task where they repeatedly reported the subjective values they placed on snack food items. High-gamma activity in many cortical and subcortical sites including the OFC positively correlated with subjective value. Other OFC sites showed signals contextually modulated by the subjective value of previously offered goods-a context dependency predicted by theory but not previously observed in humans. These results suggest that value and value-context signals are simultaneously present but separately represented in human frontal cortical activity.

Neural representation and modulation of volitional motivation in response to escalating efforts

Task-dependent volitional control of the selected neural activity in the cortex is critical to neuroprosthetic learning to achieve reliable and robust control of the external device. The volitional control of neural activity is driven by a motivational factor (volitional motivation), which directly reinforces the target neurons via real-time biofeedback. However, in the absence of motor behaviour, how do we evaluate volitional motivation? Here, we defined the criterion (ΔF/F) of the calcium fluorescence signal in a volitionally controlled neural task, then escalated the efforts by progressively increasing the number of reaching the criterion or holding time after reaching the criterion. We devised calcium-based progressive threshold-crossing events (termed 'Calcium PTE') and calcium-based progressive threshold-crossing holding-time (termed 'Calcium PTH') for quantitative assessment of volitional motivation in response to progressively escalating efforts. Furthermore, we used this novel neural representation of volitional motivation to explore the neural circuit and neuromodulator bases for volitional motivation. As with behavioural motivation, chemogenetic activation and pharmacological blockade of the striatopallidal pathway decreased and increased, respectively, the breakpoints of the 'Calcium PTE' and 'Calcium PTH' in response to escalating efforts. Furthermore, volitional and behavioural motivation shared similar dopamine dynamics in the nucleus accumbens in response to trial-by-trial escalating efforts. In general, the development of a neural representation of volitional motivation may open a new avenue for smooth and effective control of brain-machine interface tasks. KEY POINTS: Volitional motivation is quantitatively evaluated by M1 neural activity in response to progressively escalating volitional efforts. The striatopallidal pathway and adenosine A_2A receptor modulate volitional motivation in response to escalating efforts. Dopamine dynamics encode prediction signal for reward in response to repeated escalating efforts during motor and volitional conditioning. Mice learn to modulate neural activity to compensate for repeated escalating efforts in volitional control.

Alterations in a cross-hemispheric circuit associates with novelty discrimination deficits in mouse models of neurodegeneration

A major pathological hallmark of neurodegenerative diseases, including Alzheimer's, is a significant reduction in the white matter connecting the two cerebral hemispheres, as well as in the correlated activity between anatomically corresponding bilateral brain areas. However, the underlying circuit mechanisms and the cognitive relevance of cross-hemispheric (CH) communication remain poorly understood. Here, we show that novelty discrimination behavior activates CH neurons and enhances homotopic synchronized neural oscillations in the visual cortex. CH neurons provide excitatory drive required for synchronous neural oscillations between hemispheres, and unilateral inhibition of the CH circuit is sufficient to impair synchronous oscillations and novelty discrimination behavior. In the 5XFAD and Tau P301S mouse models, CH communication is altered, and novelty discrimination is impaired. These data reveal a hitherto uncharacterized CH circuit in the visual cortex, establishing a causal link between this circuit and novelty discrimination behavior and highlighting its impairment in mouse models of neurodegeneration.

Neural Decoders Using Reinforcement Learning in Brain Machine Interfaces: A Technical Review

Creating flexible and robust brain machine interfaces (BMIs) is currently a popular topic of research that has been explored for decades in medicine, engineering, commercial, and machine-learning communities. In particular, the use of techniques using reinforcement learning (RL) has demonstrated impressive results but is under-represented in the BMI community. To shine more light on this promising relationship, this article aims to provide an exhaustive review of RL's applications to BMIs. Our primary focus in this review is to provide a technical summary of various algorithms used in RL-based BMIs to decode neural intention, without emphasizing preprocessing techniques on the neural signals and reward modeling for RL. We first organize the literature based on the type of RL methods used for neural decoding, and then each algorithm's learning strategy is explained along with its application in BMIs. A comparative analysis highlighting the similarities and uniqueness among neural decoders is provided. Finally, we end this review with a discussion about the current stage of RLBMIs including their limitations and promising directions for future research.

Improved Grip Force Prediction Using a Loss Function that Penalizes Reward Related Neural Information

Neural activity in the sensorimotor cortices has been previously shown to correlate with kinematics, kinetics, and non-sensorimotor variables, such as reward. In this work, we compare the grip force offline Brain Machine Interface (BMI) prediction performance, of a simple artificial neural network (ANN), under two loss functions: the standard mean squared error (MSE) and a modified reward penalized mean squared error (RP_MSE), which penalizes for correlation between reward and grip force. Our results show that the ANN performs significantly better under the RP_MSE loss function in three brain regions: dorsal premotor cortex (PMd), primary motor cortex (M1) and the primary somatosensory cortex (S1) by approximately 6%.

Reward-dependent Graded Suppression of Sensorimotor Beta-band Local Field Potentials During an Arm Reaching Task in NHP

A better understanding of reward signaling in the sensorimotor cortices can aid in developing Reinforcement Learning-based Brain-Computer Interfaces (RLBCI) for restoration of movement functions with fewer implants. Brain-computer interfaces (BCIs) using local field potentials (LFPs) have recently achieved performance comparable to spike-BCIs [1]. With superior stability over time, LFPs may be the preferred signal for BCIs. We show that sensorimotor LFPs can provide reward level information (R1 - R3) like spikes[2]. We used a cued reward-level reaching task in which reward information was temporally dissociated from movement information. This allowed the study of reward- and movement-related modulations in LFPs. We recorded simultaneously from contralateral primary -somatosensory (S1), -motor (M1), and the dorsal premotor (PMd) cortices in a female Macaca Mulatta. We found that all three cortices' average beta band (14-30 Hz) amplitude showed robust modulation with reward levels during the cue presentation period. Such modulation was consistently observed after controlling for cue color, differences in behavioral variables, and electromyogram (EMG) activity. Statistical amplitude analysis showed that reward level could be extracted from the simple LFP feature of beta band amplitude, even before a reaching target appeared, and no specific reach plan could be developed. Clinical Relevance - The availability of reward-related signals in the sensorimotor cortical (S1, M1,and PMd) LFPs' prior to movement planning opens new avenues to build RLBCIs with fewer implants recording fewer sites among different cortices Reward and motivational representations derived from LFPs compared to spikes allow the development of long-term clinical applications given LFP's stability and ease of recording over long periods.

A multisynaptic pathway from the ventral midbrain toward spinal motoneurons in monkeys

Abstract

Motivation boosts motor performance. Activity of the ventral midbrain (VM), consisting of the ventral tegmental area (VTA), the substantia nigra pars compacta (SNc) and the retrorubral field (RRF), plays an important role in processing motivation. However, little is known about the neural substrate bridging the VM and the spinal motor output. We hypothesized that the VM might exert a modulatory influence over the descending motor pathways. By retrograde transneuronal labelling with rabies virus, we demonstrated the existence of multisynaptic projections from the VM to the cervical enlargement in monkeys. The distribution pattern of spinal projection neurons in the VM exhibited a caudorostral gradient, in that the RRF and the caudal part of the SNc contained more retrogradely labelled neurons than the VTA and the rostral part of the SNc. Electrical stimulation of the VM induced muscle responses in the contralateral forelimb with a delay of a few milliseconds following the responses of the ipsilateral primary motor cortex (M1). The magnitude and number of evoked muscle responses were associated with the stimulus intensity and number of pulses. The muscle responses were diminished during M1 inactivation. Thus, the present study has identified a multisynaptic VM–spinal pathway that is mediated, at least in part, by the M1 and might play a pivotal role in modulatory control of the spinal motor output.

Key points

Motivation to obtain reward is thought to boost motor performance, and activity in the ventral midbrain is important to the motivational process.

Little is known about a neural substrate bridging the ventral midbrain and the spinal motor output.

Retrograde trans‐synaptic experiments revealed that the ventral midbrain projects multisynaptically to the spinal cord in macaque monkeys.

Ventral midbrain activation by electrical stimulation generated cortical activity in the motor cortex and forelimb muscle activity.

A multisynaptic ventral midbrain–spinal pathway most probably plays a pivotal role in modulatory control of the spinal motor output.

Normalization by valence and motivational intensity in the sensorimotor cortices (PMd, M1, and S1)

Our brain's ability to represent vast amounts of information, such as continuous ranges of reward spanning orders of magnitude, with limited dynamic range neurons, may be possible due to normalization. Recently our group and others have shown that the sensorimotor cortices are sensitive to reward value. Here we ask if psychological affect causes normalization of the sensorimotor cortices by modulating valence and motivational intensity. We had two non-human primates (NHP) subjects (one male bonnet macaque and one female rhesus macaque) make visually cued grip-force movements while simultaneously cueing the level of possible reward if successful, or timeout punishment, if unsuccessful. We recorded simultaneously from 96 electrodes in each the following: caudal somatosensory, rostral motor, and dorsal premotor cortices (cS1, rM1, PMd). We utilized several normalization models for valence and motivational intensity in all three regions. We found three types of divisive normalized relationships between neural activity and the representation of valence and motivation, linear, sigmodal, and hyperbolic. The hyperbolic relationships resemble receptive fields in psychological affect space, where a unit is susceptible to a small range of the valence/motivational space. We found that these cortical regions have both strong valence and motivational intensity representations.

Mirror neurons are modulated by grip force and reward expectation in the sensorimotor cortices (S1, M1, PMd, PMv)

Mirror Neurons (MNs) respond similarly when primates make or observe grasping movements. Recent work indicates that reward expectation influences rostral M1 (rM1) during manual, observational, and Brain Machine Interface (BMI) reaching movements. Previous work showed MNs are modulated by subjective value. Here we expand on the above work utilizing two non-human primates (NHPs), one male Macaca Radiata (NHP S) and one female Macaca Mulatta (NHP P), that were trained to perform a cued reward level isometric grip-force task, where the NHPs had to apply visually cued grip-force to move and transport a virtual object. We found a population of (S1 area 1–2, rM1, PMd, PMv) units that significantly represented grip-force during manual and observational trials. We found the neural representation of visually cued force was similar during observational trials and manual trials for the same units; however, the representation was weaker during observational trials. Comparing changes in neural time lags between manual and observational tasks indicated that a subpopulation fit the standard MN definition of observational neural activity lagging the visual information. Neural activity in (S1 areas 1–2, rM1, PMd, PMv) significantly represented force and reward expectation. In summary, we present results indicating that sensorimotor cortices have MNs for visually cued force and value.

Noise-Correlation Is Modulated by Reward Expectation in the Primary Motor Cortex Bilaterally During Manual and Observational Tasks in Primates

Reward modulation is represented in the motor cortex (M1) and could be used to implement more accurate decoding models to improve brain-computer interfaces (BCIs; Zhao et al., 2018). Analyzing trial-to-trial noise-correlations between neural units in the presence of rewarding (R) and non-rewarding (NR) stimuli adds to our understanding of cortical network dynamics. We utilized Pearson's correlation coefficient to measure shared variability between simultaneously recorded units (32-112) and found significantly higher noise-correlation and positive correlation between the populations' signal- and noise-correlation during NR trials as compared to R trials. This pattern is evident in data from two non-human primates (NHPs) during single-target center out reaching tasks, both manual and action observation versions. We conducted a mean matched noise-correlation analysis to decouple known interactions between event-triggered firing rate changes and neural correlations. Isolated reward discriminatory units demonstrated stronger correlational changes than units unresponsive to reward firing rate modulation, however, the qualitative response was similar, indicating correlational changes within the network as a whole can serve as another information channel to be exploited by BCIs that track the underlying cortical state, such as reward expectation, or attentional modulation. Reward expectation and attention in return can be utilized with reinforcement learning (RL) towards autonomous BCI updating.

Task Learning Over Multi-Day Recording via Internally Rewarded Reinforcement Learning Based Brain Machine Interfaces

Autonomous brain machine interfaces (BMIs) aim to enable paralyzed people to self-evaluate their movement intention to control external devices. Previous reinforcement learning (RL)-based decoders interpret the mapping between neural activity and movements using the external reward for well-trained subjects, and have not investigated the task learning procedure. The brain has developed a learning mechanism to identify the correct actions that lead to rewards in the new task. This internal guidance can be utilized to replace the external reference to advance BMIs as an autonomous system. In this study, we propose to build an internally rewarded reinforcement learning-based BMI framework using the multi-site recording to demonstrate the autonomous learning ability of the BMI decoder on the new task. We test the model on the neural data collected over multiple days while the rats were learning a new lever discrimination task. The primary motor cortex (M1) and medial prefrontal cortex (mPFC) spikes are interpreted by the proposed RL framework into the discrete lever press actions. The neural activity of the mPFC post the action duration is interpreted as the internal reward information, where a support vector machine is implemented to classify the reward vs. non-reward trials with a high accuracy of 87.5% across subjects. This internal reward is used to replace the external water reward to update the decoder, which is able to adapt to the nonstationary neural activity during subject learning. The multi-cortical recording allows us to take in more cortical recordings as input and uses internal critics to guide the decoder learning. Comparing with the classic decoder using M1 activity as the only input and external guidance, the proposed system with multi-cortical recordings shows a better decoding accuracy. More importantly, our internally rewarded decoder demonstrates the autonomous learning ability on the new task as the decoder successfully addresses the time-variant neural patterns while subjects are learning, and works asymptotically as the subjects' behavioral learning progresses. It reveals the potential of endowing BMIs with autonomous task learning ability in the RL framework.

Reward uncertainty asymmetrically affects information transmission within the monkey fronto-parietal network

A central hypothesis in research on executive function is that controlled information processing is costly and is allocated according to the behavioral benefits it brings. However, while computational theories predict that the benefits of new information depend on prior uncertainty, the cellular effects of uncertainty on the executive network are incompletely understood. Using simultaneous recordings in monkeys, we describe several mechanisms by which the fronto-parietal network reacts to uncertainty. We show that the variance of expected rewards, independently of the value of the rewards, was encoded in single neuron and population spiking activity and local field potential (LFP) oscillations, and, importantly, asymmetrically affected fronto-parietal information transmission (measured through the coherence between spikes and LFPs). Higher uncertainty selectively enhanced information transmission from the parietal to the frontal lobe and suppressed it in the opposite direction, consistent with Bayesian principles that prioritize sensory information according to a decision maker’s prior uncertainty.

Bahareh Taghizadeh and Nicholas Foley et al. show that individual neuronal responses, population spiking activity, and local field potential oscillations encode the variance of expected rewards independent of their value. They also demonstrate that reward uncertainty asymmetrically affects neuronal transmission within the monkey fronto-parietal network.

Basal ganglia role in learning rewarded actions and executing previously learned choices: Healthy and diseased states

The basal ganglia (BG) is a collection of nuclei located deep beneath the cerebral cortex that is involved in learning and selection of rewarded actions. Here, we analyzed BG mechanisms that enable these functions. We implemented a rate model of a BG-thalamo-cortical loop and simulated its performance in a standard action selection task. We have shown that potentiation of corticostriatal synapses enables learning of a rewarded option. However, these synapses became redundant later as direct connections between prefrontal and premotor cortices (PFC-PMC) were potentiated by Hebbian learning. After we switched the reward to the previously unrewarded option (reversal), the BG was again responsible for switching to the new option. Due to the potentiated direct cortical connections, the system was biased to the previously rewarded choice, and establishing the new choice required a greater number of trials. Guided by physiological research, we then modified our model to reproduce pathological states of mild Parkinson's and Huntington's diseases. We found that in the Parkinsonian state PMC activity levels become extremely variable, which is caused by oscillations arising in the BG-thalamo-cortical loop. The model reproduced severe impairment of learning and predicted that this is caused by these oscillations as well as a reduced reward prediction signal. In the Huntington state, the potentiation of the PFC-PMC connections produced better learning, but altered BG output disrupted expression of the rewarded choices. This resulted in random switching between rewarded and unrewarded choices resembling an exploratory phase that never ended. Along with other computational studies, our results further reconcile the apparent contradiction between the critical involvement of the BG in execution of previously learned actions and yet no impairment of these actions after BG output is ablated by lesions or deep brain stimulation. We predict that the cortico-BG-thalamo-cortical loop conforms to previously learned choice in healthy conditions, but impedes those choices in disease states.