26
|
Wilson GH, Stavisky SD, Willett FR, Avansino DT, Kelemen JN, Hochberg LR, Henderson JM, Druckmann S, Shenoy KV. Decoding spoken English from intracortical electrode arrays in dorsal precentral gyrus. J Neural Eng 2020; 17:066007. [PMID: 33236720 PMCID: PMC8293867 DOI: 10.1088/1741-2552/abbfef] [Citation(s) in RCA: 24] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/08/2023]
Abstract
OBJECTIVE To evaluate the potential of intracortical electrode array signals for brain-computer interfaces (BCIs) to restore lost speech, we measured the performance of decoders trained to discriminate a comprehensive basis set of 39 English phonemes and to synthesize speech sounds via a neural pattern matching method. We decoded neural correlates of spoken-out-loud words in the 'hand knob' area of precentral gyrus, a step toward the eventual goal of decoding attempted speech from ventral speech areas in patients who are unable to speak. APPROACH Neural and audio data were recorded while two BrainGate2 pilot clinical trial participants, each with two chronically-implanted 96-electrode arrays, spoke 420 different words that broadly sampled English phonemes. Phoneme onsets were identified from audio recordings, and their identities were then classified from neural features consisting of each electrode's binned action potential counts or high-frequency local field potential power. Speech synthesis was performed using the 'Brain-to-Speech' pattern matching method. We also examined two potential confounds specific to decoding overt speech: acoustic contamination of neural signals and systematic differences in labeling different phonemes' onset times. MAIN RESULTS A linear decoder achieved up to 29.3% classification accuracy (chance = 6%) across 39 phonemes, while an RNN classifier achieved 33.9% accuracy. Parameter sweeps indicated that performance did not saturate when adding more electrodes or more training data, and that accuracy improved when utilizing time-varying structure in the data. Microphonic contamination and phoneme onset differences modestly increased decoding accuracy, but could be mitigated by acoustic artifact subtraction and using a neural speech onset marker, respectively. Speech synthesis achieved r = 0.523 correlation between true and reconstructed audio. SIGNIFICANCE The ability to decode speech using intracortical electrode array signals from a nontraditional speech area suggests that placing electrode arrays in ventral speech areas is a promising direction for speech BCIs.
Collapse
|
27
|
Even-Chen N, Muratore DG, Stavisky SD, Hochberg LR, Henderson JM, Murmann B, Shenoy KV. Power-saving design opportunities for wireless intracortical brain-computer interfaces. Nat Biomed Eng 2020; 4:984-996. [PMID: 32747834 PMCID: PMC8286886 DOI: 10.1038/s41551-020-0595-9] [Citation(s) in RCA: 32] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/02/2018] [Accepted: 06/30/2020] [Indexed: 12/17/2022]
Abstract
The efficacy of wireless intracortical brain-computer interfaces (iBCIs) is limited in part by the number of recording channels, which is constrained by the power budget of the implantable system. Designing wireless iBCIs that provide the high-quality recordings of today's wired neural interfaces may lead to inadvertent over-design at the expense of power consumption and scalability. Here, we report analyses of neural signals collected from experimental iBCI measurements in rhesus macaques and from a clinical-trial participant with implanted 96-channel Utah multielectrode arrays to understand the trade-offs between signal quality and decoder performance. Moreover, we propose an efficient hardware design for clinically viable iBCIs, and suggest that the circuit design parameters of current recording iBCIs can be relaxed considerably without loss of performance. The proposed design may allow for an order-of-magnitude power savings and lead to clinically viable iBCIs with a higher channel count.
Collapse
|
28
|
Jiang X, Saggar H, Ryu SI, Shenoy KV, Kao JC. Structure in Neural Activity during Observed and Executed Movements Is Shared at the Neural Population Level, Not in Single Neurons. Cell Rep 2020; 32:108148. [PMID: 32905762 DOI: 10.1016/j.celrep.2020.108148] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/23/2022] Open
|
29
|
Jiang X, Saggar H, Ryu SI, Shenoy KV, Kao JC. Structure in Neural Activity during Observed and Executed Movements Is Shared at the Neural Population Level, Not in Single Neurons. Cell Rep 2020; 32:108006. [DOI: 10.1016/j.celrep.2020.108006] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2020] [Revised: 05/24/2020] [Accepted: 07/16/2020] [Indexed: 12/30/2022] Open
|
30
|
Nason SR, Vaskov AK, Willsey MS, Welle EJ, An H, Vu PP, Bullard AJ, Nu CS, Kao JC, Shenoy KV, Jang T, Kim HS, Blaauw D, Patil PG, Chestek CA. A low-power band of neuronal spiking activity dominated by local single units improves the performance of brain-machine interfaces. Nat Biomed Eng 2020; 4:973-983. [PMID: 32719512 DOI: 10.1038/s41551-020-0591-0] [Citation(s) in RCA: 52] [Impact Index Per Article: 13.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2018] [Accepted: 06/24/2020] [Indexed: 12/18/2022]
Abstract
The large power requirement of current brain-machine interfaces is a major hindrance to their clinical translation. In basic behavioural tasks, the downsampled magnitude of the 300-1,000 Hz band of spiking activity can predict movement similarly to the threshold crossing rate (TCR) at 30 kilo-samples per second. However, the relationship between such a spiking-band power (SBP) and neural activity remains unclear, as does the capability of using the SBP to decode complicated behaviour. By using simulations of recordings of neural activity, here we show that the SBP is dominated by local single-unit spikes with spatial specificity comparable to or better than that of the TCR, and that the SBP correlates better with the firing rates of lower signal-to-noise-ratio units than the TCR. With non-human primates, in an online task involving the one-dimensional decoding of the movement of finger groups and in an offline two-dimensional cursor-control task, the SBP performed equally well or better than the TCR. The SBP may enhance the decoding performance of neural interfaces while enabling substantial cuts in power consumption.
Collapse
|
31
|
Vyas S, O'Shea DJ, Ryu SI, Shenoy KV. Causal Role of Motor Preparation during Error-Driven Learning. Neuron 2020; 106:329-339.e4. [PMID: 32053768 PMCID: PMC7185427 DOI: 10.1016/j.neuron.2020.01.019] [Citation(s) in RCA: 25] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/23/2019] [Revised: 11/12/2019] [Accepted: 01/16/2020] [Indexed: 11/28/2022]
Abstract
Current theories suggest that an error-driven learning process updates trial-by-trial to facilitate motor adaptation. How this process interacts with motor cortical preparatory activity-which current models suggest plays a critical role in movement initiation-remains unknown. Here, we evaluated the role of motor preparation during visuomotor adaptation. We found that preparation time was inversely correlated to variance of errors on current trials and mean error on subsequent trials. We also found causal evidence that intracortical microstimulation during motor preparation was sufficient to disrupt learning. Surprisingly, stimulation did not affect current trials, but instead disrupted the update computation of a learning process, thereby affecting subsequent trials. This is consistent with a Bayesian estimation framework where the motor system reduces its learning rate by virtue of lowering error sensitivity when faced with uncertainty. This interaction between motor preparation and the error-driven learning system may facilitate new probes into mechanisms underlying trial-by-trial adaptation.
Collapse
|
32
|
Stavisky SD, Willett FR, Avansino DT, Hochberg LR, Shenoy KV, Henderson JM. Speech-related dorsal motor cortex activity does not interfere with iBCI cursor control. J Neural Eng 2020; 17:016049. [PMID: 32023225 PMCID: PMC8288044 DOI: 10.1088/1741-2552/ab5b72] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/25/2022]
Abstract
OBJECTIVE Speech-related neural modulation was recently reported in 'arm/hand' area of human dorsal motor cortex that is used as a signal source for intracortical brain-computer interfaces (iBCIs). This raises the concern that speech-related modulation might deleteriously affect the decoding of arm movement intentions, for instance by affecting velocity command outputs. This study sought to clarify whether or not speaking would interfere with ongoing iBCI use. APPROACH A participant in the BrainGate2 iBCI clinical trial used an iBCI to control a computer cursor; spoke short words in a stand-alone speech task; and spoke short words during ongoing iBCI use. We examined neural activity in all three behaviors and compared iBCI performance with and without concurrent speech. MAIN RESULTS Dorsal motor cortex firing rates modulated strongly during stand-alone speech, but this activity was largely attenuated when speaking occurred during iBCI cursor control using attempted arm movements. 'Decoder-potent' projections of the attenuated speech-related neural activity were small, explaining why cursor task performance was similar between iBCI use with and without concurrent speaking. SIGNIFICANCE These findings indicate that speaking does not directly interfere with iBCIs that decode attempted arm movements. This suggests that patients who are able to speak will be able to use motor cortical-driven computer interfaces or prostheses without needing to forgo speaking while using these devices.
Collapse
|
33
|
Stavisky SD, Willett FR, Wilson GH, Murphy BA, Rezaii P, Avansino DT, Memberg WD, Miller JP, Kirsch RF, Hochberg LR, Ajiboye AB, Druckmann S, Shenoy KV, Henderson JM. Neural ensemble dynamics in dorsal motor cortex during speech in people with paralysis. eLife 2019; 8:e46015. [PMID: 31820736 PMCID: PMC6954053 DOI: 10.7554/elife.46015] [Citation(s) in RCA: 43] [Impact Index Per Article: 8.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/14/2019] [Accepted: 11/14/2019] [Indexed: 01/20/2023] Open
Abstract
Speaking is a sensorimotor behavior whose neural basis is difficult to study with single neuron resolution due to the scarcity of human intracortical measurements. We used electrode arrays to record from the motor cortex 'hand knob' in two people with tetraplegia, an area not previously implicated in speech. Neurons modulated during speaking and during non-speaking movements of the tongue, lips, and jaw. This challenges whether the conventional model of a 'motor homunculus' division by major body regions extends to the single-neuron scale. Spoken words and syllables could be decoded from single trials, demonstrating the potential of intracortical recordings for brain-computer interfaces to restore speech. Two neural population dynamics features previously reported for arm movements were also present during speaking: a component that was mostly invariant across initiating different words, followed by rotatory dynamics during speaking. This suggests that common neural dynamical motifs may underlie movement of arm and speech articulators.
Collapse
|
34
|
Williams AH, Poole B, Maheswaranathan N, Dhawale AK, Fisher T, Wilson CD, Brann DH, Trautmann EM, Ryu S, Shusterman R, Rinberg D, Ölveczky BP, Shenoy KV, Ganguli S. Discovering Precise Temporal Patterns in Large-Scale Neural Recordings through Robust and Interpretable Time Warping. Neuron 2019; 105:246-259.e8. [PMID: 31786013 DOI: 10.1016/j.neuron.2019.10.020] [Citation(s) in RCA: 34] [Impact Index Per Article: 6.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/16/2019] [Revised: 09/17/2019] [Accepted: 10/10/2019] [Indexed: 12/22/2022]
Abstract
Though the temporal precision of neural computation has been studied intensively, a data-driven determination of this precision remains a fundamental challenge. Reproducible spike patterns may be obscured on single trials by uncontrolled temporal variability in behavior and cognition and may not be time locked to measurable signatures in behavior or local field potentials (LFP). To overcome these challenges, we describe a general-purpose time warping framework that reveals precise spike-time patterns in an unsupervised manner, even when these patterns are decoupled from behavior or are temporally stretched across single trials. We demonstrate this method across diverse systems: cued reaching in nonhuman primates, motor sequence production in rats, and olfaction in mice. This approach flexibly uncovers diverse dynamical firing patterns, including pulsatile responses to behavioral events, LFP-aligned oscillatory spiking, and even unanticipated patterns, such as 7 Hz oscillations in rat motor cortex that are not time locked to measured behaviors or LFP.
Collapse
|
35
|
Brandman DM, Hosman T, Saab J, Burkhart MC, Shanahan BE, Ciancibello JG, Sarma AA, Milstein DJ, Vargas-Irwin CE, Franco B, Kelemen J, Blabe C, Murphy BA, Young DR, Willett FR, Pandarinath C, Stavisky SD, Kirsch RF, Walter BL, Bolu Ajiboye A, Cash SS, Eskandar EN, Miller JP, Sweet JA, Shenoy KV, Henderson JM, Jarosiewicz B, Harrison MT, Simeral JD, Hochberg LR. Rapid calibration of an intracortical brain-computer interface for people with tetraplegia. J Neural Eng 2019; 15:026007. [PMID: 29363625 DOI: 10.1088/1741-2552/aa9ee7] [Citation(s) in RCA: 63] [Impact Index Per Article: 12.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/15/2022]
Abstract
OBJECTIVE Brain-computer interfaces (BCIs) can enable individuals with tetraplegia to communicate and control external devices. Though much progress has been made in improving the speed and robustness of neural control provided by intracortical BCIs, little research has been devoted to minimizing the amount of time spent on decoder calibration. APPROACH We investigated the amount of time users needed to calibrate decoders and achieve performance saturation using two markedly different decoding algorithms: the steady-state Kalman filter, and a novel technique using Gaussian process regression (GP-DKF). MAIN RESULTS Three people with tetraplegia gained rapid closed-loop neural cursor control and peak, plateaued decoder performance within 3 min of initializing calibration. We also show that a BCI-naïve user (T5) was able to rapidly attain closed-loop neural cursor control with the GP-DKF using self-selected movement imagery on his first-ever day of closed-loop BCI use, acquiring a target 37 s after initiating calibration. SIGNIFICANCE These results demonstrate the potential for an intracortical BCI to be used immediately after deployment by people with paralysis, without the need for user learning or extensive system calibration.
Collapse
|
36
|
Stavisky SD, Rezaii P, Willett FR, Hochberg LR, Shenoy KV, Henderson JM. Decoding Speech from Intracortical Multielectrode Arrays in Dorsal "Arm/Hand Areas" of Human Motor Cortex. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2019; 2018:93-97. [PMID: 30440349 DOI: 10.1109/embc.2018.8512199] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
Neural prostheses are being developed to restore speech to people with neurological injury or disease. A key design consideration is where and how to access neural correlates of intended speech. Most prior work has examined cortical field potentials at a coarse resolution using electroencephalography (EEG) or medium resolution using electrocorticography (ECoG). The few studies of speech with single-neuron resolution recorded from ventral areas known to be part of the speech network. Here, we recorded from two 96- electrode arrays chronically implanted into the 'hand knob' area of motor cortex while a person with tetraplegia spoke. Despite being located in an area previously demonstrated to modulate during attempted arm movements, many electrodes' neuronal firing rates responded to speech production. In offline analyses, we could classify which of 9 phonemes (plus silence) was spoken with 81% single-trial accuracy using a combination of spike rate and local field potential (LFP) power. This suggests that high-fidelity speech prostheses may be possible using large-scale intracortical recordings in motor cortical areas involved in controlling speech articulators.
Collapse
|
37
|
Ames KC, Ryu SI, Shenoy KV. Simultaneous motor preparation and execution in a last-moment reach correction task. Nat Commun 2019; 10:2718. [PMID: 31221968 PMCID: PMC6586876 DOI: 10.1038/s41467-019-10772-2] [Citation(s) in RCA: 27] [Impact Index Per Article: 5.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/04/2017] [Accepted: 05/31/2019] [Indexed: 11/28/2022] Open
Abstract
Motor preparation typically precedes movement and is thought to determine properties of upcoming movements. However, preparation has mostly been studied in point-to-point delayed reaching tasks. Here, we ask whether preparation is engaged during mid-reach modifications. Monkeys reach to targets that occasionally jump locations prior to movement onset, requiring a mid-reach correction. In motor cortex and dorsal premotor cortex, we find that the neural activity that signals when to reach predicts monkeys’ jump responses on a trial-by-trial basis. We further identify neural patterns that signal where to reach, either during motor preparation or during motor execution. After a target jump, neural activity responds in both preparatory and movement-related dimensions, even though error in preparatory dimensions can be small at that time. This suggests that the same preparatory process used in delayed reaching is also involved in reach correction. Furthermore, it indicates that motor preparation and execution can be performed simultaneously. Motor preparation processes guide movement. Here, by recording neural activity in monkeys reaching toward targets that can change location, the authors provide evidence that changing a prepared movement midway through completion reengages motor preparation.
Collapse
|
38
|
Willett FR, Young DR, Murphy BA, Memberg WD, Blabe CH, Pandarinath C, Stavisky SD, Rezaii P, Saab J, Walter BL, Sweet JA, Miller JP, Henderson JM, Shenoy KV, Simeral JD, Jarosiewicz B, Hochberg LR, Kirsch RF, Bolu Ajiboye A. Principled BCI Decoder Design and Parameter Selection Using a Feedback Control Model. Sci Rep 2019; 9:8881. [PMID: 31222030 PMCID: PMC6586941 DOI: 10.1038/s41598-019-44166-7] [Citation(s) in RCA: 18] [Impact Index Per Article: 3.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/19/2018] [Accepted: 03/04/2019] [Indexed: 02/01/2023] Open
Abstract
Decoders optimized offline to reconstruct intended movements from neural recordings sometimes fail to achieve optimal performance online when they are used in closed-loop as part of an intracortical brain-computer interface (iBCI). This is because typical decoder calibration routines do not model the emergent interactions between the decoder, the user, and the task parameters (e.g. target size). Here, we investigated the feasibility of simulating online performance to better guide decoder parameter selection and design. Three participants in the BrainGate2 pilot clinical trial controlled a computer cursor using a linear velocity decoder under different gain (speed scaling) and temporal smoothing parameters and acquired targets with different radii and distances. We show that a user-specific iBCI feedback control model can predict how performance changes under these different decoder and task parameters in held-out data. We also used the model to optimize a nonlinear speed scaling function for the decoder. When used online with two participants, it increased the dynamic range of decoded speeds and decreased the time taken to acquire targets (compared to an optimized standard decoder). These results suggest that it is feasible to simulate iBCI performance accurately enough to be useful for quantitative decoder optimization and design.
Collapse
|
39
|
Trautmann EM, Stavisky SD, Lahiri S, Ames KC, Kaufman MT, O'Shea DJ, Vyas S, Sun X, Ryu SI, Ganguli S, Shenoy KV. Accurate Estimation of Neural Population Dynamics without Spike Sorting. Neuron 2019; 103:292-308.e4. [PMID: 31171448 DOI: 10.1016/j.neuron.2019.05.003] [Citation(s) in RCA: 121] [Impact Index Per Article: 24.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/22/2017] [Revised: 02/06/2019] [Accepted: 04/30/2019] [Indexed: 11/25/2022]
Abstract
A central goal of systems neuroscience is to relate an organism's neural activity to behavior. Neural population analyses often reduce the data dimensionality to focus on relevant activity patterns. A major hurdle to data analysis is spike sorting, and this problem is growing as the number of recorded neurons increases. Here, we investigate whether spike sorting is necessary to estimate neural population dynamics. The theory of random projections suggests that we can accurately estimate the geometry of low-dimensional manifolds from a small number of linear projections of the data. We recorded data using Neuropixels probes in motor cortex of nonhuman primates and reanalyzed data from three previous studies and found that neural dynamics and scientific conclusions are quite similar using multiunit threshold crossings rather than sorted neurons. This finding unlocks existing data for new analyses and informs the design and use of new electrode arrays for laboratory and clinical use.
Collapse
|
40
|
Wang M, Montanède C, Chandrasekaran C, Peixoto D, Shenoy KV, Kalaska JF. Macaque dorsal premotor cortex exhibits decision-related activity only when specific stimulus-response associations are known. Nat Commun 2019; 10:1793. [PMID: 30996222 PMCID: PMC6470163 DOI: 10.1038/s41467-019-09460-y] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2018] [Accepted: 03/12/2019] [Indexed: 01/16/2023] Open
Abstract
How deliberation on sensory cues and action selection interact in decision-related brain areas is still not well understood. Here, monkeys reached to one of two targets, whose colors alternated randomly between trials, by discriminating the dominant color of a checkerboard cue composed of different numbers of squares of the two target colors in different trials. In a Targets First task the colored targets appeared first, followed by the checkerboard; in a Checkerboard First task, this order was reversed. After both cues appeared in both tasks, responses of dorsal premotor cortex (PMd) units covaried with action choices, strength of evidence for action choices, and RTs- hallmarks of decision-related activity. However, very few units were modulated by checkerboard color composition or the color of the chosen target, even during the checkerboard deliberation epoch of the Checkerboard First task. These findings implicate PMd in the action-selection but not the perceptual components of the decision-making process in these tasks.
Collapse
|
41
|
Even-Chen N, Sheffer B, Vyas S, Ryu SI, Shenoy KV. Structure and variability of delay activity in premotor cortex. PLoS Comput Biol 2019; 15:e1006808. [PMID: 30794541 PMCID: PMC6402694 DOI: 10.1371/journal.pcbi.1006808] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2018] [Revised: 03/06/2019] [Accepted: 01/21/2019] [Indexed: 11/18/2022] Open
Abstract
Voluntary movements are widely considered to be planned before they are executed. Recent studies have hypothesized that neural activity in motor cortex during preparation acts as an ‘initial condition’ which seeds the proceeding neural dynamics. Here, we studied these initial conditions in detail by investigating 1) the organization of neural states for different reaches and 2) the variance of these neural states from trial to trial. We examined population-level responses in macaque premotor cortex (PMd) during the preparatory stage of an instructed-delay center-out reaching task with dense target configurations. We found that after target onset the neural activity on single trials converges to neural states that have a clear low-dimensional structure which is organized by both the reach endpoint and maximum speed of the following reach. Further, we found that variability of the neural states during preparation resembles the spatial variability of reaches made in the absence of visual feedback: there is less variability in direction than distance in neural state space. We also used offline decoding to understand the implications of this neural population structure for brain-machine interfaces (BMIs). We found that decoding of angle between reaches is dependent on reach distance, while decoding of arc-length is independent. Thus, it might be more appropriate to quantify decoding performance for discrete BMIs by using arc-length between reach end-points rather than the angle between them. Lastly, we show that in contrast to the common notion that direction can better be decoded than distance, their decoding capabilities are comparable. These results provide new insights into the dynamical neural processes that underline motor control and can inform the design of BMIs. Early studies of premotor cortex explored how individual neurons directly encode aspects of an upcoming movement during preparation. Recent developments have proposed that the dynamics of populations of neurons underlie motor control, and that neural activity during preparation serves to set up these dynamics. While the dynamics of motor control have been studied extensively, several aspects of preparatory activity remain unresolved. Here, we ask how the patterns of neural activity during preparation for different reaches are related to one another. We found that the neural activity during preparation for reaches to different targets has a clear ‘structure’. Additionally, we found that the activity on a given trial was predictive of the initial trajectory of the reach. Lastly, we assessed the implications of our findings for predicting upcoming movements from neural activity, as in brain-machine interfaces.
Collapse
|
42
|
Milekovic T, Bacher D, Sarma AA, Simeral JD, Saab J, Pandarinath C, Yvert B, Sorice BL, Blabe C, Oakley EM, Tringale KR, Eskandar E, Cash SS, Shenoy KV, Henderson JM, Hochberg LR, Donoghue JP. Volitional control of single-electrode high gamma local field potentials by people with paralysis. J Neurophysiol 2019; 121:1428-1450. [PMID: 30785814 DOI: 10.1152/jn.00131.2018] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/13/2022] Open
Abstract
Intracortical brain-computer interfaces (BCIs) can enable individuals to control effectors, such as a computer cursor, by directly decoding the user's movement intentions from action potentials and local field potentials (LFPs) recorded within the motor cortex. However, the accuracy and complexity of effector control achieved with such "biomimetic" BCIs will depend on the degree to which the intended movements used to elicit control modulate the neural activity. In particular, channels that do not record distinguishable action potentials and only record LFP modulations may be of limited use for BCI control. In contrast, a biofeedback approach may surpass these limitations by letting the participants generate new control signals and learn strategies that improve the volitional control of signals used for effector control. Here, we show that, by using a biofeedback paradigm, three individuals with tetraplegia achieved volitional control of gamma LFPs (40-400 Hz) recorded by a single microelectrode implanted in the precentral gyrus. Control was improved over a pair of consecutive sessions up to 3 days apart. In all but one session, the channel used to achieve control lacked distinguishable action potentials. Our results indicate that biofeedback LFP-based BCIs may potentially contribute to the neural modulation necessary to obtain reliable and useful control of effectors. NEW & NOTEWORTHY Our study demonstrates that people with tetraplegia can volitionally control individual high-gamma local-field potential (LFP) channels recorded from the motor cortex, and that this control can be improved using biofeedback. Motor cortical LFP signals are thought to be both informative and stable intracortical signals and, thus, of importance for future brain-computer interfaces.
Collapse
|
43
|
Young D, Willett F, Memberg WD, Murphy B, Rezaii P, Walter B, Sweet J, Miller J, Shenoy KV, Hochberg LR, Kirsch RF, Ajiboye AB. Closed-loop cortical control of virtual reach and posture using Cartesian and joint velocity commands. J Neural Eng 2018; 16:026011. [PMID: 30523839 DOI: 10.1088/1741-2552/aaf606] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/09/2023]
Abstract
OBJECTIVE Brain-computer interfaces (BCIs) are a promising technology for the restoration of function to people with paralysis, especially for controlling coordinated reaching. Typical BCI studies decode Cartesian endpoint velocities as commands, but human arm movements might be better controlled in a joint-based coordinate frame, which may match underlying movement encoding in the motor cortex. A better understanding of BCI controlled reaching by people with paralysis may lead to performance improvements in brain-controlled assistive devices. APPROACH Two intracortical BCI participants in the BrainGate2 pilot clinical trial performed a visual 3D endpoint virtual reality reaching task using two decoders: Cartesian and joint velocity. Task performance metrics (i.e. success rate and path efficiency) and single feature and population tuning were compared across the two decoder conditions. The participants also demonstrated the first BCI control of a fourth dimension of reaching, the arm's swivel angle, in a 4D posture matching task. MAIN RESULTS Both users achieved significantly higher success rates using Cartesian velocity control, and joint controlled trajectories were more variable and significantly more curved. Neural tuning analyses showed that most single feature activity was best described by a Cartesian kinematic encoding model, and population analyses revealed only slight differences in aggregate activity between the decoder conditions. Simulations of a BCI user reproduced trajectory features seen during closed-loop joint control when assuming only Cartesian-tuned features passed through a joint decoder. With minimal training, both participants controlled the virtual arm's swivel angle to complete a 4D posture matching task, and achieved significantly higher success using a Cartesian + swivel velocity decoder compared to a joint velocity decoder. SIGNIFICANCE These results suggest that Cartesian velocity command interfaces may provide better BCI control of arm movements than other kinematic variables, even in 4D posture tasks with swivel angle targets.
Collapse
|
44
|
Nuyujukian P, Albites Sanabria J, Saab J, Pandarinath C, Jarosiewicz B, Blabe CH, Franco B, Mernoff ST, Eskandar EN, Simeral JD, Hochberg LR, Shenoy KV, Henderson JM. Cortical control of a tablet computer by people with paralysis. PLoS One 2018; 13:e0204566. [PMID: 30462658 PMCID: PMC6248919 DOI: 10.1371/journal.pone.0204566] [Citation(s) in RCA: 64] [Impact Index Per Article: 10.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/10/2017] [Accepted: 09/11/2018] [Indexed: 12/16/2022] Open
Abstract
General-purpose computers have become ubiquitous and important for everyday life, but they are difficult for people with paralysis to use. Specialized software and personalized input devices can improve access, but often provide only limited functionality. In this study, three research participants with tetraplegia who had multielectrode arrays implanted in motor cortex as part of the BrainGate2 clinical trial used an intracortical brain-computer interface (iBCI) to control an unmodified commercial tablet computer. Neural activity was decoded in real time as a point-and-click wireless Bluetooth mouse, allowing participants to use common and recreational applications (web browsing, email, chatting, playing music on a piano application, sending text messages, etc.). Two of the participants also used the iBCI to "chat" with each other in real time. This study demonstrates, for the first time, high-performance iBCI control of an unmodified, commercially available, general-purpose mobile computing device by people with tetraplegia.
Collapse
|
45
|
Jiang X, Ryu SI, Shenoy KV, Kao JC. Single Neuron Firing Rate Statistics in Motor Cortex During Execution and Observation of Movement. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2018; 2018:981-986. [PMID: 30440555 DOI: 10.1109/embc.2018.8512445] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
Mirror neurons, which fire during both the execution and observation of movement, are believed to play an important role in motor processing and learning. However, much work still remains to understand the similarities and differences in how these neurons compute in the motor cortex during movement execution and observation. Here, we performed experiments where a monkey both executes and observes a center-out-and-back task within the same experimental session. By recording from putatively the same neural population, we were able to analyze and compare single neuron statistics between movement execution and observation. We found that a majority of neurons in the primary motor cortex (M1) and dorsal premotor cortex (PMd) have statistically different firing rate statistics between movement execution and observation. As a result of this difference, we then wondered if neurons during movement observation exhibited a similar characteristic to those during movement execution: changing of preferred directions as a function of movement speed. Interestingly, we found that while observed movement speed is encoded in the neural population, it only alters a small proportion of the neuron's firing rate statistics. These results suggest that neural populations in Ml and PMd process information related to movement differently between execution and observation.
Collapse
|
46
|
Stavisky SD, Kao JC, Nuyujukian P, Pandarinath C, Blabe C, Ryu SI, Hochberg LR, Henderson JM, Shenoy KV. Brain-machine interface cursor position only weakly affects monkey and human motor cortical activity in the absence of arm movements. Sci Rep 2018; 8:16357. [PMID: 30397281 PMCID: PMC6218537 DOI: 10.1038/s41598-018-34711-1] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2018] [Accepted: 10/24/2018] [Indexed: 12/26/2022] Open
Abstract
Brain-machine interfaces (BMIs) that decode movement intentions should ignore neural modulation sources distinct from the intended command. However, neurophysiology and control theory suggest that motor cortex reflects the motor effector's position, which could be a nuisance variable. We investigated motor cortical correlates of BMI cursor position with or without concurrent arm movement. We show in two monkeys that subtracting away estimated neural correlates of position improves online BMI performance only if the animals were allowed to move their arm. To understand why, we compared the neural variance attributable to cursor position when the same task was performed using arm reaching, versus arms-restrained BMI use. Firing rates correlated with both BMI cursor and hand positions, but hand positional effects were greater. To examine whether BMI position influences decoding in people with paralysis, we analyzed data from two intracortical BMI clinical trial participants and performed an online decoder comparison in one participant. We found only small motor cortical correlates, which did not affect performance. These results suggest that arm movement and proprioception are the major contributors to position-related motor cortical correlates. Cursor position visual feedback is therefore unlikely to affect the performance of BMI-driven prosthetic systems being developed for people with paralysis.
Collapse
|
47
|
Pandarinath C, O'Shea DJ, Collins J, Jozefowicz R, Stavisky SD, Kao JC, Trautmann EM, Kaufman MT, Ryu SI, Hochberg LR, Henderson JM, Shenoy KV, Abbott LF, Sussillo D. Inferring single-trial neural population dynamics using sequential auto-encoders. Nat Methods 2018; 15:805-815. [PMID: 30224673 PMCID: PMC6380887 DOI: 10.1038/s41592-018-0109-9] [Citation(s) in RCA: 242] [Impact Index Per Article: 40.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/28/2018] [Accepted: 06/28/2018] [Indexed: 01/28/2023]
Abstract
Neuroscience is experiencing a revolution in which simultaneous recording
of many thousands of neurons is revealing population dynamics that are not
apparent from single-neuron responses. This structure is typically extracted
from trial-averaged data, but deeper understanding requires studying
single-trial phenomena, which is challenging due to incomplete sampling of the
neural population, trial-to-trial variability, and fluctuations in action
potential timing. We introduce Latent Factor Analysis via Dynamical Systems
(LFADS), a deep learning method to infer latent dynamics from single-trial
neural spiking data. LFADS uses a nonlinear dynamical system to infer the
dynamics underlying observed spiking activity and to extract
‘de-noised’ single-trial firing rates. When applied to a variety
of monkey and human motor cortical datasets, LFADS predicts observed behavioral
variables with unprecedented accuracy, extracts precise estimates of neural
dynamics on single trials, infers perturbations to those dynamics that correlate
with behavioral choices, and combines data from non-overlapping recording
sessions spanning months to improve inference of underlying dynamics.
Collapse
|
48
|
Willett FR, Murphy BA, Young DR, Memberg WD, Blabe CH, Pandarinath C, Franco B, Saab J, Walter BL, Sweet JA, Miller JP, Henderson JM, Shenoy KV, Simeral JD, Jarosiewicz B, Hochberg LR, Kirsch RF, Ajiboye AB. A Comparison of Intention Estimation Methods for Decoder Calibration in Intracortical Brain-Computer Interfaces. IEEE Trans Biomed Eng 2018; 65:2066-2078. [PMID: 29989927 PMCID: PMC6043406 DOI: 10.1109/tbme.2017.2783358] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/27/2023]
Abstract
OBJECTIVE Recent reports indicate that making better assumptions about the user's intended movement can improve the accuracy of decoder calibration for intracortical brain-computer interfaces. Several methods now exist for estimating user intent, including an optimal feedback control model, a piecewise-linear feedback control model, ReFIT, and other heuristics. Which of these methods yields the best decoding performance? METHODS Using data from the BrainGate2 pilot clinical trial, we measured how a steady-state velocity Kalman filter decoder was affected by the choice of intention estimation method. We examined three separate components of the Kalman filter: dimensionality reduction, temporal smoothing, and output gain (speed scaling). RESULTS The decoder's dimensionality reduction properties were largely unaffected by the intention estimation method. Decoded velocity vectors differed by <5% in terms of angular error and speed vs. target distance curves across methods. In contrast, the smoothing and gain properties of the decoder were greatly affected (> 50% difference in average values). Since the optimal gain and smoothing properties are task-specific (e.g. lower gains are better for smaller targets but worse for larger targets), no one method was better for all tasks. CONCLUSION Our results show that, when gain and smoothing differences are accounted for, current intention estimation methods yield nearly equivalent decoders and that simple models of user intent, such as a position error vector (target position minus cursor position), perform comparably to more elaborate models. Our results also highlight that simple differences in gain and smoothing properties have a large effect on online performance and can confound decoder comparisons.
Collapse
|
49
|
Williams AH, Kim TH, Wang F, Vyas S, Ryu SI, Shenoy KV, Schnitzer M, Kolda TG, Ganguli S. Unsupervised Discovery of Demixed, Low-Dimensional Neural Dynamics across Multiple Timescales through Tensor Component Analysis. Neuron 2018; 98:1099-1115.e8. [PMID: 29887338 DOI: 10.1016/j.neuron.2018.05.015] [Citation(s) in RCA: 112] [Impact Index Per Article: 18.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/13/2017] [Revised: 03/18/2018] [Accepted: 05/08/2018] [Indexed: 01/19/2023]
Abstract
Perceptions, thoughts, and actions unfold over millisecond timescales, while learned behaviors can require many days to mature. While recent experimental advances enable large-scale and long-term neural recordings with high temporal fidelity, it remains a formidable challenge to extract unbiased and interpretable descriptions of how rapid single-trial circuit dynamics change slowly over many trials to mediate learning. We demonstrate a simple tensor component analysis (TCA) can meet this challenge by extracting three interconnected, low-dimensional descriptions of neural data: neuron factors, reflecting cell assemblies; temporal factors, reflecting rapid circuit dynamics mediating perceptions, thoughts, and actions within each trial; and trial factors, describing both long-term learning and trial-to-trial changes in cognitive state. We demonstrate the broad applicability of TCA by revealing insights into diverse datasets derived from artificial neural networks, large-scale calcium imaging of rodent prefrontal cortex during maze navigation, and multielectrode recordings of macaque motor cortex during brain machine interface learning.
Collapse
|
50
|
O'Shea DJ, Kalanithi P, Ferenczi EA, Hsueh B, Chandrasekaran C, Goo W, Diester I, Ramakrishnan C, Kaufman MT, Ryu SI, Yeom KW, Deisseroth K, Shenoy KV. Development of an optogenetic toolkit for neural circuit dissection in squirrel monkeys. Sci Rep 2018; 8:6775. [PMID: 29712920 PMCID: PMC5928036 DOI: 10.1038/s41598-018-24362-7] [Citation(s) in RCA: 21] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/24/2017] [Accepted: 04/03/2018] [Indexed: 01/01/2023] Open
Abstract
Optogenetic tools have opened a rich experimental landscape for understanding neural function and disease. Here, we present the first validation of eight optogenetic constructs driven by recombinant adeno-associated virus (AAV) vectors and a WGA-Cre based dual injection strategy for projection targeting in a widely-used New World primate model, the common squirrel monkey Saimiri sciureus. We observed opsin expression around the local injection site and in axonal projections to downstream regions, as well as transduction to thalamic neurons, resembling expression patterns observed in macaques. Optical stimulation drove strong, reliable excitatory responses in local neural populations for two depolarizing opsins in anesthetized monkeys. Finally, we observed continued, healthy opsin expression for at least one year. These data suggest that optogenetic tools can be readily applied in squirrel monkeys, an important first step in enabling precise, targeted manipulation of neural circuits in these highly trainable, cognitively sophisticated animals. In conjunction with similar approaches in macaques and marmosets, optogenetic manipulation of neural circuits in squirrel monkeys will provide functional, comparative insights into neural circuits which subserve dextrous motor control as well as other adaptive behaviors across the primate lineage. Additionally, development of these tools in squirrel monkeys, a well-established model system for several human neurological diseases, can aid in identifying novel treatment strategies.
Collapse
|