1
|
Pulferer HS, Kostoglou K, Müller-Putz GR. Improving non-invasive trajectory decoding via neural correlates of continuous erroneous feedback processing. J Neural Eng 2024; 21:056010. [PMID: 39231465 DOI: 10.1088/1741-2552/ad7762] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/17/2024] [Accepted: 09/04/2024] [Indexed: 09/06/2024]
Abstract
Objective. Over the last decades, error-related potentials (ErrPs) have repeatedly proven especially useful as corrective mechanisms in invasive and non-invasive brain-computer interfaces (BCIs). However, research in this context exclusively investigated the distinction of discrete events intocorrectorerroneousto the present day. Due to this predominant formulation as a binary classification problem, classical ErrP-based BCIs fail to monitor tasks demanding quantitative information on error severity rather than mere qualitative decisions on error occurrence. As a result, fine-tuned and natural feedback control based on continuously perceived deviations from an intended target remains beyond the capabilities of previously used BCI setups.Approach.To address this issue for future BCI designs, we investigated the feasibility of regressing rather than classifying error-related activity non-invasively from the brain.Main results.Using pre-recorded data from ten able-bodied participants in three sessions each and a multi-output convolutional neural network, we demonstrated the above-chance regression of ongoing target-feedback discrepancies from brain signals in a pseudo-online fashion. In a second step, we used this inferred information about the target deviation to correct the initially displayed feedback accordingly, reporting significant improvements in correlations between corrected feedback and target trajectories across feedback conditions.Significance.Our results indicate that continuous information on target-feedback discrepancies can be successfully regressed from cortical activity, paving the way to increasingly naturalistic, fine-tuned correction mechanisms for future BCI applications.
Collapse
Affiliation(s)
- Hannah S Pulferer
- Institute of Neural Engineering, TU Graz, Stremayrgasse 16/4, Graz, 8010 Styria, Austria
| | - Kyriaki Kostoglou
- Institute of Neural Engineering, TU Graz, Stremayrgasse 16/4, Graz, 8010 Styria, Austria
| | - Gernot R Müller-Putz
- Institute of Neural Engineering, TU Graz, Stremayrgasse 16/4, Graz, 8010 Styria, Austria
- BioTechMed-Graz, Graz, Styria, Austria
| |
Collapse
|
2
|
Boucher PO, Wang T, Carceroni L, Kane G, Shenoy KV, Chandrasekaran C. Initial conditions combine with sensory evidence to induce decision-related dynamics in premotor cortex. Nat Commun 2023; 14:6510. [PMID: 37845221 PMCID: PMC10579235 DOI: 10.1038/s41467-023-41752-2] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2022] [Accepted: 09/18/2023] [Indexed: 10/18/2023] Open
Abstract
We used a dynamical systems perspective to understand decision-related neural activity, a fundamentally unresolved problem. This perspective posits that time-varying neural activity is described by a state equation with an initial condition and evolves in time by combining at each time step, recurrent activity and inputs. We hypothesized various dynamical mechanisms of decisions, simulated them in models to derive predictions, and evaluated these predictions by examining firing rates of neurons in the dorsal premotor cortex (PMd) of monkeys performing a perceptual decision-making task. Prestimulus neural activity (i.e., the initial condition) predicted poststimulus neural trajectories, covaried with RT and the outcome of the previous trial, but not with choice. Poststimulus dynamics depended on both the sensory evidence and initial condition, with easier stimuli and fast initial conditions leading to the fastest choice-related dynamics. Together, these results suggest that initial conditions combine with sensory evidence to induce decision-related dynamics in PMd.
Collapse
Affiliation(s)
- Pierre O Boucher
- Department of Biomedical Engineering, Boston University, Boston, 02115, MA, USA
| | - Tian Wang
- Department of Biomedical Engineering, Boston University, Boston, 02115, MA, USA
| | - Laura Carceroni
- Undergraduate Program in Neuroscience, Boston University, Boston, 02115, MA, USA
| | - Gary Kane
- Department of Psychological and Brain Sciences, Boston University, Boston, 02115, MA, USA
| | - Krishna V Shenoy
- Department of Electrical Engineering, Stanford University, Stanford, 94305, CA, USA
- Department of Neurobiology, Stanford University, Stanford, 94305, CA, USA
- Howard Hughes Medical Institute, HHMI, Chevy Chase, 20815-6789, MD, USA
- Department of Bioengineering, Stanford University, Stanford, 94305, CA, USA
- Stanford Neurosciences Institute, Stanford University, Stanford, 94305, CA, USA
- Bio-X Program, Stanford University, Stanford, 94305, CA, USA
- Department of Neurosurgery, Stanford University, Stanford, 94305, CA, USA
| | - Chandramouli Chandrasekaran
- Department of Biomedical Engineering, Boston University, Boston, 02115, MA, USA.
- Department of Psychological and Brain Sciences, Boston University, Boston, 02115, MA, USA.
- Center for Systems Neuroscience, Boston University, Boston, 02115, MA, USA.
- Department of Anatomy & Neurobiology, Boston University, Boston, 02118, MA, USA.
| |
Collapse
|
3
|
Ye J, Collinger JL, Wehbe L, Gaunt R. Neural Data Transformer 2: Multi-context Pretraining for Neural Spiking Activity. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.09.18.558113. [PMID: 37781630 PMCID: PMC10541112 DOI: 10.1101/2023.09.18.558113] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/03/2023]
Abstract
The neural population spiking activity recorded by intracortical brain-computer interfaces (iBCIs) contain rich structure. Current models of such spiking activity are largely prepared for individual experimental contexts, restricting data volume to that collectable within a single session and limiting the effectiveness of deep neural networks (DNNs). The purported challenge in aggregating neural spiking data is the pervasiveness of context-dependent shifts in the neural data distributions. However, large scale unsupervised pretraining by nature spans heterogeneous data, and has proven to be a fundamental recipe for successful representation learning across deep learning. We thus develop Neural Data Transformer 2 (NDT2), a spatiotemporal Transformer for neural spiking activity, and demonstrate that pretraining can leverage motor BCI datasets that span sessions, subjects, and experimental tasks. NDT2 enables rapid adaptation to novel contexts in downstream decoding tasks and opens the path to deployment of pretrained DNNs for iBCI control. Code: https://github.com/joel99/context_general_bci.
Collapse
Affiliation(s)
- Joel Ye
- Rehab Neural Engineering Labs, University of Pittsburgh
- Neuroscience Institute, Carnegie Mellon University
- Center for the Neural Basis of Cognition, Pittsburgh
| | - Jennifer L. Collinger
- Rehab Neural Engineering Labs, University of Pittsburgh
- Center for the Neural Basis of Cognition, Pittsburgh
- Department of Physical Medicine and Rehabilitation, University of Pittsburgh
- Department of Bioengineering, University of Pittsburgh
- Department of Biomedical Engineering, Carnegie Mellon University
| | - Leila Wehbe
- Neuroscience Institute, Carnegie Mellon University
- Center for the Neural Basis of Cognition, Pittsburgh
- Machine Learning Department, Carnegie Mellon University
| | - Robert Gaunt
- Rehab Neural Engineering Labs, University of Pittsburgh
- Center for the Neural Basis of Cognition, Pittsburgh
- Department of Physical Medicine and Rehabilitation, University of Pittsburgh
- Department of Bioengineering, University of Pittsburgh
- Department of Biomedical Engineering, Carnegie Mellon University
| |
Collapse
|
4
|
Wallace DM, Benyamini M, Nason-Tomaszewski SR, Costello JT, Cubillos LH, Mender MJ, Temmar H, Willsey MS, Patil PG, Chestek CA, Zacksenhouse M. Error detection and correction in intracortical brain-machine interfaces controlling two finger groups. J Neural Eng 2023; 20:046037. [PMID: 37567222 PMCID: PMC10594236 DOI: 10.1088/1741-2552/acef95] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/23/2023] [Revised: 08/01/2023] [Accepted: 08/11/2023] [Indexed: 08/13/2023]
Abstract
Objective.While brain-machine interfaces (BMIs) are promising technologies that could provide direct pathways for controlling the external world and thus regaining motor capabilities, their effectiveness is hampered by decoding errors. Previous research has demonstrated the detection and correction of BMI outcome errors, which occur at the end of trials. Here we focus on continuous detection and correction of BMI execution errors, which occur during real-time movements.Approach.Two adult male rhesus macaques were implanted with Utah arrays in the motor cortex. The monkeys performed single or two-finger group BMI tasks where a Kalman filter decoded binned spiking-band power into intended finger kinematics. Neural activity was analyzed to determine how it depends not only on the kinematics of the fingers, but also on the distance of each finger-group to its target. We developed a method to detect erroneous movements, i.e. consistent movements away from the target, from the same neural activity used by the Kalman filter. Detected errors were corrected by a simple stopping strategy, and the effect on performance was evaluated.Mainresults.First we show that including distance to target explains significantly more variance of the recorded neural activity. Then, for the first time, we demonstrate that neural activity in motor cortex can be used to detect execution errors during BMI controlled movements. Keeping false positive rate below5%, it was possible to achieve mean true positive rate of28.1%online. Despite requiring 200 ms to detect and react to suspected errors, we were able to achieve a significant improvement in task performance via reduced orbiting time of one finger group.Significance.Neural activity recorded in motor cortex for BMI control can be used to detect and correct BMI errors and thus to improve performance. Further improvements may be obtained by enhancing classification and correction strategies.
Collapse
Affiliation(s)
- Dylan M Wallace
- Department of Robotics, University of Michigan, Ann Arbor, MI, United States of America
| | - Miri Benyamini
- BCI for Rehabilitation Lab., Faculty of Mechanical Engineering, Technion - Israel Institute of Technology, Haifa, Israel
| | - Samuel R Nason-Tomaszewski
- Cortical Neural Prosthetics Lab., Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI, United States of America
| | - Joseph T Costello
- Department of Electrical and Computer Engineering, University of Michigan, Ann Arbor, MI, United States of America
| | - Luis H Cubillos
- Department of Robotics, University of Michigan, Ann Arbor, MI, United States of America
| | - Matthew J Mender
- Cortical Neural Prosthetics Lab., Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI, United States of America
| | - Hisham Temmar
- Cortical Neural Prosthetics Lab., Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI, United States of America
| | - Matthew S Willsey
- Cortical Neural Prosthetics Lab., Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI, United States of America
- Department of Neurosurgery, University of Michigan, Ann Arbor, MI, United States of America
| | - Parag G Patil
- Cortical Neural Prosthetics Lab., Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI, United States of America
- Department of Neurosurgery, University of Michigan, Ann Arbor, MI, United States of America
| | - Cynthia A Chestek
- Department of Robotics, University of Michigan, Ann Arbor, MI, United States of America
- Cortical Neural Prosthetics Lab., Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI, United States of America
- Department of Electrical and Computer Engineering, University of Michigan, Ann Arbor, MI, United States of America
| | - Miriam Zacksenhouse
- BCI for Rehabilitation Lab., Faculty of Mechanical Engineering, Technion - Israel Institute of Technology, Haifa, Israel
| |
Collapse
|
5
|
Hadjidimitrakis K, De Vitis M, Ghodrati M, Filippini M, Fattori P. Anterior-posterior gradient in the integrated processing of forelimb movement direction and distance in macaque parietal cortex. Cell Rep 2022; 41:111608. [DOI: 10.1016/j.celrep.2022.111608] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/29/2021] [Revised: 07/16/2022] [Accepted: 10/14/2022] [Indexed: 11/09/2022] Open
|
6
|
Li Y, Qi Y, Wang Y, Wang Y, Xu K, Pan G. Robust neural decoding by kernel regression with Siamese representation learning. J Neural Eng 2021; 18. [PMID: 34663771 DOI: 10.1088/1741-2552/ac2c4e] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/09/2021] [Accepted: 10/01/2021] [Indexed: 11/12/2022]
Abstract
Objective. Brain-machine interfaces (BMIs) provide a direct pathway between the brain and external devices such as computer cursors and prosthetics, which have great potential in motor function restoration. One critical limitation of current BMI systems is the unstable performance, partly due to the variability of neural signals. Studies showed that neural activities exhibit trial-to-trial variability, and the preferred direction of neurons frequently changes under different conditions. Therefore, a fixed decoding function does not work well.Approach. To deal with the problems, we propose a novel kernel regression framework. The nonparametric kernel regression is used to fit diverse decoding functions by finding similar neural patterns to handle neural variations caused by varying tuning functions. Further, the representations of raw neural signals are learned by Siamese networks and constrained by kinematic parameters, which can alleviate neural variations caused by intrinsic noises and task-irrelevant information. The representations are jointly learned with the kernel regression framework in an end-to-end manner so that neural variations can be tackled effectively.Main results. Experiments on two datasets demonstrate that our approach outperforms most existing methods and significantly improves the robustness in challenging situations such as limited samples and missing channels.Significance. The proposed approach demonstrates robust performance with different conditions and provides a new and inspiring perspective toward robust BMI control.
Collapse
Affiliation(s)
- Yangang Li
- Qiushi Academy for Advanced Studies, Zhejiang University, Hangzhou, People's Republic of China.,College of Computer Science and Technology, Zhejiang University, Hangzhou, People's Republic of China
| | - Yu Qi
- College of Computer Science and Technology, Zhejiang University, Hangzhou, People's Republic of China.,MOE Frontier Science Center for Brain Science and Brain-machine Integration, Zhejiang University, Hangzhou, People's Republic of China
| | - Yiwen Wang
- Department of Chemical and Biological Engineering, Hong Kong University of Science and Technology, Hong Kong, People's Republic of China.,Department of Electronic and Computer Engineering, Hong Kong University of Science and Technology, Hong Kong, People's Republic of China
| | - Yueming Wang
- Qiushi Academy for Advanced Studies, Zhejiang University, Hangzhou, People's Republic of China
| | - Kedi Xu
- Qiushi Academy for Advanced Studies, Zhejiang University, Hangzhou, People's Republic of China.,Key Laboratory for Biomedical Engineering of Ministry of Education, Zhejiang University, Hangzhou, People's Republic of China.,Zhejiang Provincial Key Laboratory of Cardio-Cerebral Vascular Detection Technology and Medicinal Effectiveness Appraisal, Zhejiang University, Hangzhou, People's Republic of China
| | - Gang Pan
- College of Computer Science and Technology, Zhejiang University, Hangzhou, People's Republic of China
| |
Collapse
|
7
|
Norman SL, Maresca D, Christopoulos VN, Griggs WS, Demene C, Tanter M, Shapiro MG, Andersen RA. Single-trial decoding of movement intentions using functional ultrasound neuroimaging. Neuron 2021; 109:1554-1566.e4. [PMID: 33756104 DOI: 10.1016/j.neuron.2021.03.003] [Citation(s) in RCA: 28] [Impact Index Per Article: 9.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/02/2020] [Revised: 12/29/2020] [Accepted: 03/01/2021] [Indexed: 12/18/2022]
Abstract
New technologies are key to understanding the dynamic activity of neural circuits and systems in the brain. Here, we show that a minimally invasive approach based on ultrasound can be used to detect the neural correlates of movement planning, including directions and effectors. While non-human primates (NHPs) performed memory-guided movements, we used functional ultrasound (fUS) neuroimaging to record changes in cerebral blood volume with 100 μm resolution. We recorded from outside the dura above the posterior parietal cortex, a brain area important for spatial perception, multisensory integration, and movement planning. We then used fUS signals from the delay period before movement to decode the animals' intended direction and effector. Single-trial decoding is a prerequisite to brain-machine interfaces, a key application that could benefit from this technology. These results are a critical step in the development of neuro-recording and brain interface tools that are less invasive, high resolution, and scalable.
Collapse
Affiliation(s)
- Sumner L Norman
- Biology & Biological Engineering, California Institute of Technology, Pasadena, CA 91125, USA
| | - David Maresca
- Chemistry & Chemical Engineering, California Institute of Technology, Pasadena, CA 91125, USA
| | - Vassilios N Christopoulos
- Biology & Biological Engineering, California Institute of Technology, Pasadena, CA 91125, USA; T&C Chen Brain-Machine Interface Center, California Institute of Technology, Pasadena, CA 91125, USA
| | - Whitney S Griggs
- Biology & Biological Engineering, California Institute of Technology, Pasadena, CA 91125, USA
| | - Charlie Demene
- Physics for Medicine Paris, INSERM, CNRS, ESPCI Paris, PSL Research University, 75012 Paris, France; INSERM Technology Research Accelerator in Biomedical Ultrasound, Paris, France
| | - Mickael Tanter
- Physics for Medicine Paris, INSERM, CNRS, ESPCI Paris, PSL Research University, 75012 Paris, France; INSERM Technology Research Accelerator in Biomedical Ultrasound, Paris, France
| | - Mikhail G Shapiro
- Chemistry & Chemical Engineering, California Institute of Technology, Pasadena, CA 91125, USA.
| | - Richard A Andersen
- Biology & Biological Engineering, California Institute of Technology, Pasadena, CA 91125, USA; T&C Chen Brain-Machine Interface Center, California Institute of Technology, Pasadena, CA 91125, USA.
| |
Collapse
|
8
|
Shen X, Zhang X, Huang Y, Chen S, Wang Y. Task Learning Over Multi-Day Recording via Internally Rewarded Reinforcement Learning Based Brain Machine Interfaces. IEEE Trans Neural Syst Rehabil Eng 2020; 28:3089-3099. [PMID: 33232240 DOI: 10.1109/tnsre.2020.3039970] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
Autonomous brain machine interfaces (BMIs) aim to enable paralyzed people to self-evaluate their movement intention to control external devices. Previous reinforcement learning (RL)-based decoders interpret the mapping between neural activity and movements using the external reward for well-trained subjects, and have not investigated the task learning procedure. The brain has developed a learning mechanism to identify the correct actions that lead to rewards in the new task. This internal guidance can be utilized to replace the external reference to advance BMIs as an autonomous system. In this study, we propose to build an internally rewarded reinforcement learning-based BMI framework using the multi-site recording to demonstrate the autonomous learning ability of the BMI decoder on the new task. We test the model on the neural data collected over multiple days while the rats were learning a new lever discrimination task. The primary motor cortex (M1) and medial prefrontal cortex (mPFC) spikes are interpreted by the proposed RL framework into the discrete lever press actions. The neural activity of the mPFC post the action duration is interpreted as the internal reward information, where a support vector machine is implemented to classify the reward vs. non-reward trials with a high accuracy of 87.5% across subjects. This internal reward is used to replace the external water reward to update the decoder, which is able to adapt to the nonstationary neural activity during subject learning. The multi-cortical recording allows us to take in more cortical recordings as input and uses internal critics to guide the decoder learning. Comparing with the classic decoder using M1 activity as the only input and external guidance, the proposed system with multi-cortical recordings shows a better decoding accuracy. More importantly, our internally rewarded decoder demonstrates the autonomous learning ability on the new task as the decoder successfully addresses the time-variant neural patterns while subjects are learning, and works asymptotically as the subjects' behavioral learning progresses. It reveals the potential of endowing BMIs with autonomous task learning ability in the RL framework.
Collapse
|
9
|
Mousavi M, Krol LR, de Sa VR. Hybrid brain-computer interface with motor imagery and error-related brain activity. J Neural Eng 2020; 17:056041. [PMID: 32726757 DOI: 10.1088/1741-2552/abaa9d] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
Abstract
OBJECTIVE Brain-computer interface (BCI) systems read and interpret brain activity directly from the brain. They can provide a means of communication or locomotion for patients suffering from neurodegenerative diseases or stroke. However, non-stationarity of brain activity limits the reliable transfer of the algorithms that were trained during a calibration session to real-time BCI control. One source of non-stationarity is the user's brain response to the BCI output (feedback), for instance, whether the BCI feedback is perceived as an error by the user or not. By taking such sources of non-stationarity into account, the reliability of the BCI can be improved. APPROACH In this work, we demonstrate a real-time implementation of a hybrid motor imagery BCI combining the information from the motor imagery signal and the error-related brain activity simultaneously so as to gain benefit from both sources. MAIN RESULTS We show significantly improved performance in real-time BCI control across 12 participants, compared to a conventional motor imagery BCI. The significant improvement is in terms of classification accuracy, target hit rate, subjective perception of control and information-transfer rate. Moreover, our offline analyses of the recorded EEG data show that the error-related brain activity provides a more reliable source of information than the motor imagery signal. SIGNIFICANCE This work shows, for the first time, that the error-related brain activity classifier compared to the motor imagery classifier is more consistent when trained on calibration data and tested during online control. This likely explains why the proposed hybrid BCI allows for a more reliable means of communication or rehabilitation for patients in need.
Collapse
Affiliation(s)
- Mahta Mousavi
- Department of Cognitive Science, UC San Diego, La Jolla, CA, United States of America
| | | | | |
Collapse
|
10
|
Even-Chen N, Muratore DG, Stavisky SD, Hochberg LR, Henderson JM, Murmann B, Shenoy KV. Power-saving design opportunities for wireless intracortical brain-computer interfaces. Nat Biomed Eng 2020; 4:984-996. [PMID: 32747834 PMCID: PMC8286886 DOI: 10.1038/s41551-020-0595-9] [Citation(s) in RCA: 32] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/02/2018] [Accepted: 06/30/2020] [Indexed: 12/17/2022]
Abstract
The efficacy of wireless intracortical brain-computer interfaces (iBCIs) is limited in part by the number of recording channels, which is constrained by the power budget of the implantable system. Designing wireless iBCIs that provide the high-quality recordings of today's wired neural interfaces may lead to inadvertent over-design at the expense of power consumption and scalability. Here, we report analyses of neural signals collected from experimental iBCI measurements in rhesus macaques and from a clinical-trial participant with implanted 96-channel Utah multielectrode arrays to understand the trade-offs between signal quality and decoder performance. Moreover, we propose an efficient hardware design for clinically viable iBCIs, and suggest that the circuit design parameters of current recording iBCIs can be relaxed considerably without loss of performance. The proposed design may allow for an order-of-magnitude power savings and lead to clinically viable iBCIs with a higher channel count.
Collapse
Affiliation(s)
- Nir Even-Chen
- Department of Electrical Engineering, Stanford University, Stanford, CA, USA.
| | - Dante G Muratore
- Department of Electrical Engineering, Stanford University, Stanford, CA, USA
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA
| | - Sergey D Stavisky
- Department of Electrical Engineering, Stanford University, Stanford, CA, USA
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA
- Department of Neurosurgery, Stanford University, Stanford, CA, USA
| | - Leigh R Hochberg
- Department of Veterans Affairs Medical Center, Center for Neurorestoration and Neurotechnology, Providence, RI, USA
- School of Engineering and Carney Institute for Brain Science, Brown University, Providence, RI, USA
- Center for Neurotechnology and Neurorecovery, Department of Neurology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
| | - Jaimie M Henderson
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA
- Department of Neurosurgery, Stanford University, Stanford, CA, USA
| | - Boris Murmann
- Department of Electrical Engineering, Stanford University, Stanford, CA, USA
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA
| | - Krishna V Shenoy
- Department of Electrical Engineering, Stanford University, Stanford, CA, USA
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA
- Department of Bioengineering, Stanford University, Stanford, CA, USA
- Department of Neurobiology, Stanford University, Stanford, CA, USA
- Howard Hughes Medical Institute at Stanford University, Stanford, CA, USA
- The Bio-X Institute, Stanford University, Stanford, CA, USA
| |
Collapse
|
11
|
|
12
|
Cell-Type-Specific Outcome Representation in the Primary Motor Cortex. Neuron 2020; 107:954-971.e9. [PMID: 32589878 DOI: 10.1016/j.neuron.2020.06.006] [Citation(s) in RCA: 34] [Impact Index Per Article: 8.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/12/2019] [Revised: 04/14/2020] [Accepted: 06/02/2020] [Indexed: 12/18/2022]
Abstract
Adaptive movements are critical for animal survival. To guide future actions, the brain monitors various outcomes, including achievement of movement and appetitive goals. The nature of these outcome signals and their neuronal and network realization in the motor cortex (M1), which directs skilled movements, is largely unknown. Using a dexterity task, calcium imaging, optogenetic perturbations, and behavioral manipulations, we studied outcome signals in the murine forelimb M1. We found two populations of layer 2-3 neurons, termed success- and failure-related neurons, that develop with training, and report end results of trials. In these neurons, prolonged responses were recorded after success or failure trials independent of reward and kinematics. In addition, the initial state of layer 5 pyramidal tract neurons contained a memory trace of the previous trial's outcome. Intertrial cortical activity was needed to learn new task requirements. These M1 layer-specific performance outcome signals may support reinforcement motor learning of skilled behavior.
Collapse
|
13
|
Vyas S, O'Shea DJ, Ryu SI, Shenoy KV. Causal Role of Motor Preparation during Error-Driven Learning. Neuron 2020; 106:329-339.e4. [PMID: 32053768 PMCID: PMC7185427 DOI: 10.1016/j.neuron.2020.01.019] [Citation(s) in RCA: 25] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/23/2019] [Revised: 11/12/2019] [Accepted: 01/16/2020] [Indexed: 11/28/2022]
Abstract
Current theories suggest that an error-driven learning process updates trial-by-trial to facilitate motor adaptation. How this process interacts with motor cortical preparatory activity-which current models suggest plays a critical role in movement initiation-remains unknown. Here, we evaluated the role of motor preparation during visuomotor adaptation. We found that preparation time was inversely correlated to variance of errors on current trials and mean error on subsequent trials. We also found causal evidence that intracortical microstimulation during motor preparation was sufficient to disrupt learning. Surprisingly, stimulation did not affect current trials, but instead disrupted the update computation of a learning process, thereby affecting subsequent trials. This is consistent with a Bayesian estimation framework where the motor system reduces its learning rate by virtue of lowering error sensitivity when faced with uncertainty. This interaction between motor preparation and the error-driven learning system may facilitate new probes into mechanisms underlying trial-by-trial adaptation.
Collapse
Affiliation(s)
- Saurabh Vyas
- Department of Bioengineering, Stanford University, Stanford, CA 94305, USA.
| | - Daniel J O'Shea
- Department of Electrical Engineering, Stanford University, Stanford, CA 94305, USA; Neurosciences Graduate Program, Stanford University, Stanford, CA 94305, USA
| | - Stephen I Ryu
- Department of Electrical Engineering, Stanford University, Stanford, CA 94305, USA; Palo Alto Medical Foundation, Palo Alto, CA 94301, USA
| | - Krishna V Shenoy
- Department of Bioengineering, Stanford University, Stanford, CA 94305, USA; Department of Electrical Engineering, Stanford University, Stanford, CA 94305, USA; Department of Neurobiology, Stanford University, Stanford, CA 94305, USA; Bio-X Program, Stanford University, Stanford, CA 94305, USA; Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA 94305, USA; Howard Hughes Medical Institute, Stanford University, Stanford, CA 94305, USA
| |
Collapse
|
14
|
Even-Chen N, Sheffer B, Vyas S, Ryu SI, Shenoy KV. Structure and variability of delay activity in premotor cortex. PLoS Comput Biol 2019; 15:e1006808. [PMID: 30794541 PMCID: PMC6402694 DOI: 10.1371/journal.pcbi.1006808] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2018] [Revised: 03/06/2019] [Accepted: 01/21/2019] [Indexed: 11/18/2022] Open
Abstract
Voluntary movements are widely considered to be planned before they are executed. Recent studies have hypothesized that neural activity in motor cortex during preparation acts as an ‘initial condition’ which seeds the proceeding neural dynamics. Here, we studied these initial conditions in detail by investigating 1) the organization of neural states for different reaches and 2) the variance of these neural states from trial to trial. We examined population-level responses in macaque premotor cortex (PMd) during the preparatory stage of an instructed-delay center-out reaching task with dense target configurations. We found that after target onset the neural activity on single trials converges to neural states that have a clear low-dimensional structure which is organized by both the reach endpoint and maximum speed of the following reach. Further, we found that variability of the neural states during preparation resembles the spatial variability of reaches made in the absence of visual feedback: there is less variability in direction than distance in neural state space. We also used offline decoding to understand the implications of this neural population structure for brain-machine interfaces (BMIs). We found that decoding of angle between reaches is dependent on reach distance, while decoding of arc-length is independent. Thus, it might be more appropriate to quantify decoding performance for discrete BMIs by using arc-length between reach end-points rather than the angle between them. Lastly, we show that in contrast to the common notion that direction can better be decoded than distance, their decoding capabilities are comparable. These results provide new insights into the dynamical neural processes that underline motor control and can inform the design of BMIs. Early studies of premotor cortex explored how individual neurons directly encode aspects of an upcoming movement during preparation. Recent developments have proposed that the dynamics of populations of neurons underlie motor control, and that neural activity during preparation serves to set up these dynamics. While the dynamics of motor control have been studied extensively, several aspects of preparatory activity remain unresolved. Here, we ask how the patterns of neural activity during preparation for different reaches are related to one another. We found that the neural activity during preparation for reaches to different targets has a clear ‘structure’. Additionally, we found that the activity on a given trial was predictive of the initial trajectory of the reach. Lastly, we assessed the implications of our findings for predicting upcoming movements from neural activity, as in brain-machine interfaces.
Collapse
Affiliation(s)
- Nir Even-Chen
- Department of Electrical Engineering, Stanford University, Stanford, CA, United States of America
- * E-mail:
| | - Blue Sheffer
- Department of Computer Science, Stanford University, Stanford, CA, United States of America
| | - Saurabh Vyas
- Department of Bioengineering, Stanford University, Stanford, CA, United States of America
| | - Stephen I. Ryu
- Department of Neurosurgery, Palo Alto Medical Foundation, Palo Alto, CA, United States of America
| | - Krishna V. Shenoy
- Department of Electrical Engineering, Stanford University, Stanford, CA, United States of America
- Department of Bioengineering, Stanford University, Stanford, CA, United States of America
- Department of Neurobiology, Stanford University, Stanford, CA, United States of America
- The Bio-X Program, Stanford University, Stanford, CA, United States of America
- The Stanford Neurosciences Institute, Stanford University, Stanford, CA, United States of America
- Howard Hughes Medical Institute, Stanford University, Stanford, CA, United States of America
| |
Collapse
|
15
|
Pandarinath C, Ames KC, Russo AA, Farshchian A, Miller LE, Dyer EL, Kao JC. Latent Factors and Dynamics in Motor Cortex and Their Application to Brain-Machine Interfaces. J Neurosci 2018; 38:9390-9401. [PMID: 30381431 PMCID: PMC6209846 DOI: 10.1523/jneurosci.1669-18.2018] [Citation(s) in RCA: 56] [Impact Index Per Article: 9.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/06/2018] [Revised: 09/24/2018] [Accepted: 09/25/2018] [Indexed: 01/07/2023] Open
Abstract
In the 1960s, Evarts first recorded the activity of single neurons in motor cortex of behaving monkeys (Evarts, 1968). In the 50 years since, great effort has been devoted to understanding how single neuron activity relates to movement. Yet these single neurons exist within a vast network, the nature of which has been largely inaccessible. With advances in recording technologies, algorithms, and computational power, the ability to study these networks is increasing exponentially. Recent experimental results suggest that the dynamical properties of these networks are critical to movement planning and execution. Here we discuss this dynamical systems perspective and how it is reshaping our understanding of the motor cortices. Following an overview of key studies in motor cortex, we discuss techniques to uncover the "latent factors" underlying observed neural population activity. Finally, we discuss efforts to use these factors to improve the performance of brain-machine interfaces, promising to make these findings broadly relevant to neuroengineering as well as systems neuroscience.
Collapse
Affiliation(s)
- Chethan Pandarinath
- Wallace H. Coulter Department of Biomedical Engineering, Emory University and Georgia Institute of Technology, Atlanta, Georgia 30322,
- Department of Neurosurgery, Emory University, Atlanta, Georgia 30322
| | - K Cora Ames
- Department of Neuroscience
- Center for Theoretical Neuroscience
- Grossman Center for the Statistics of Mind
- Zuckerman Institute, Columbia University, New York, New York 10027
| | - Abigail A Russo
- Department of Neuroscience
- Grossman Center for the Statistics of Mind
- Zuckerman Institute, Columbia University, New York, New York 10027
| | - Ali Farshchian
- Department of Physiology, Northwestern University, Chicago, Illinois 60611
| | - Lee E Miller
- Department of Physiology, Northwestern University, Chicago, Illinois 60611
| | - Eva L Dyer
- Wallace H. Coulter Department of Biomedical Engineering, Emory University and Georgia Institute of Technology, Atlanta, Georgia 30322
- Department of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, Georgia 30332
| | - Jonathan C Kao
- Department of Electrical and Computer Engineering, and
- Neurosciences Program, University of California, Los Angeles, California 90095
| |
Collapse
|
16
|
Even-Chen N, Stavisky SD, Pandarinath C, Nuyujukian P, Blabe CH, Hochberg LR, Henderson JM, Shenoy KV. Feasibility of Automatic Error Detect-and-Undo System in Human Intracortical Brain-Computer Interfaces. IEEE Trans Biomed Eng 2017; 65:1771-1784. [PMID: 29989931 DOI: 10.1109/tbme.2017.2776204] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
Abstract
OBJECTIVE Brain-computer interfaces (BCIs) aim to help people with impaired movement ability by directly translating their movement intentions into command signals for assistive technologies. Despite large performance improvements over the last two decades, BCI systems still make errors that need to be corrected manually by the user. This decreases system performance and is also frustrating for the user. The deleterious effects of errors could be mitigated if the system automatically detected when the user perceives that an error was made and automatically intervened with a corrective action; thus, sparing users from having to make the correction themselves. Our previous preclinical work with monkeys demonstrated that task-outcome correlates exist in motor cortical spiking activity and can be utilized to improve BCI performance. Here, we asked if these signals also exist in the human hand area of motor cortex, and whether they can be decoded with high accuracy. METHODS We analyzed posthoc the intracortical neural activity of two BrainGate2 clinical trial participants who were neurally controlling a computer cursor to perform a grid target selection task and a keyboard-typing task. RESULTS Our key findings are that: 1) there exists a putative outcome error signal reflected in both the action potentials and local field potentials of the human hand area of motor cortex, and 2) target selection outcomes can be classified with high accuracy (70-85%) of errors successfully detected with minimal (0-3%) misclassifications of success trials, based on neural activity alone. SIGNIFICANCE These offline results suggest that it will be possible to improve the performance of clinical intracortical BCIs by incorporating a real-time error detect-and-undo system alongside the decoding of movement intention.
Collapse
|