A bilingual speech neuroprosthesis driven by cortical articulatory representations shared between languages

Advancements in decoding speech from brain activity have focused on decoding a single language. Hence, the extent to which bilingual speech production relies on unique or shared cortical activity across languages has remained unclear. Here, we leveraged electrocorticography, along with deep-learning and statistical natural-language models of English and Spanish, to record and decode activity from speech-motor cortex of a Spanish-English bilingual with vocal-tract and limb paralysis into sentences in either language. This was achieved without requiring the participant to manually specify the target language. Decoding models relied on shared vocal-tract articulatory representations across languages, which allowed us to build a syllable classifier that generalized across a shared set of English and Spanish syllables. Transfer learning expedited training of the bilingual decoder by enabling neural data recorded in one language to improve decoding in the other language. Overall, our findings suggest shared cortical articulatory representations that persist after paralysis and enable the decoding of multiple languages without the need to train separate language-specific decoders.

A high-performance neuroprosthesis for speech decoding and avatar control

Speech neuroprostheses have the potential to restore communication to people living with paralysis, but naturalistic speed and expressivity are elusive1. Here we use high-density surface recordings of the speech cortex in a clinical-trial participant with severe limb and vocal paralysis to achieve high-performance real-time decoding across three complementary speech-related output modalities: text, speech audio and facial-avatar animation. We trained and evaluated deep-learning models using neural data collected as the participant attempted to silently speak sentences. For text, we demonstrate accurate and rapid large-vocabulary decoding with a median rate of 78 words per minute and median word error rate of 25%. For speech audio, we demonstrate intelligible and rapid speech synthesis and personalization to the participant's pre-injury voice. For facial-avatar animation, we demonstrate the control of virtual orofacial movements for speech and non-speech communicative gestures. The decoders reached high performance with less than two weeks of training. Our findings introduce a multimodal speech-neuroprosthetic approach that has substantial promise to restore full, embodied communication to people living with severe paralysis.

Generalizable spelling using a speech neuroprosthesis in an individual with severe limb and vocal paralysis

Neuroprostheses have the potential to restore communication to people who cannot speak or type due to paralysis. However, it is unclear if silent attempts to speak can be used to control a communication neuroprosthesis. Here, we translated direct cortical signals in a clinical-trial participant (ClinicalTrials.gov; NCT03698149) with severe limb and vocal-tract paralysis into single letters to spell out full sentences in real time. We used deep-learning and language-modeling techniques to decode letter sequences as the participant attempted to silently spell using code words that represented the 26 English letters (e.g. "alpha" for "a"). We leveraged broad electrode coverage beyond speech-motor cortex to include supplemental control signals from hand cortex and complementary information from low- and high-frequency signal components to improve decoding accuracy. We decoded sentences using words from a 1,152-word vocabulary at a median character error rate of 6.13% and speed of 29.4 characters per minute. In offline simulations, we showed that our approach generalized to large vocabularies containing over 9,000 words (median character error rate of 8.23%). These results illustrate the clinical viability of a silently controlled speech neuroprosthesis to generate sentences from a large vocabulary through a spelling-based approach, complementing previous demonstrations of direct full-word decoding.

High-density single-unit human cortical recordings using the Neuropixels probe

The action potential is a fundamental unit of neural computation. Even though significant advances have been made in recording large numbers of individual neurons in animal models, translation of these methodologies to humans has been limited because of clinical constraints and electrode reliability. Here, we present a reliable method for intraoperative recording of dozens of neurons in humans using the Neuropixels probe, yielding up to ∼100 simultaneously recorded single units. Most single units were active within 1 min of reaching target depth. The motion of the electrode array had a strong inverse correlation with yield, identifying a major challenge and opportunity to further increase the probe utility. Cell pairs active close in time were spatially closer in most recordings, demonstrating the power to resolve complex cortical dynamics. Altogether, this approach provides access to population single-unit activity across the depth of human neocortex at scales previously only accessible in animal models.

Columnar Localization and Laminar Origin of Cortical Surface Electrical Potentials

Electrocorticography (ECoG) methodologically bridges basic neuroscience and understanding of human brains in health and disease. However, the localization of ECoG signals across the surface of the brain and the spatial distribution of their generating neuronal sources are poorly understood. To address this gap, we recorded from rat auditory cortex using customized μECoG, and simulated cortical surface electrical potentials with a full-scale, biophysically detailed cortical column model. Experimentally, μECoG-derived auditory representations were tonotopically organized and signals were anisotropically localized to less than or equal to ±200 μm, that is, a single cortical column. Biophysical simulations reproduce experimental findings and indicate that neurons in cortical layers V and VI contribute ∼85% of evoked high-gamma signal recorded at the surface. Cell number and synchrony were the primary biophysical properties determining laminar contributions to evoked μECoG signals, whereas distance was only a minimal factor. Thus, evoked μECoG signals primarily originate from neurons in the infragranular layers of a single cortical column.SIGNIFICANCE STATEMENT ECoG methodologically bridges basic neuroscience and understanding of human brains in health and disease. However, the localization of ECoG signals across the surface of the brain and the spatial distribution of their generating neuronal sources are poorly understood. We investigated the localization and origins of sensory-evoked ECoG responses. We experimentally found that ECoG responses were anisotropically localized to a cortical column. Biophysically detailed simulations revealed that neurons in layers V and VI were the primary sources of evoked ECoG responses. These results indicate that evoked ECoG high-gamma responses are primarily generated by the population spike rate of pyramidal neurons in layers V and VI of single cortical columns and highlight the possibility of understanding how microscopic sources produce mesoscale signals.

Neuroprosthesis for Decoding Speech in a Paralyzed Person with Anarthria

BACKGROUND

Technology to restore the ability to communicate in paralyzed persons who cannot speak has the potential to improve autonomy and quality of life. An approach that decodes words and sentences directly from the cerebral cortical activity of such patients may represent an advancement over existing methods for assisted communication.

METHODS

We implanted a subdural, high-density, multielectrode array over the area of the sensorimotor cortex that controls speech in a person with anarthria (the loss of the ability to articulate speech) and spastic quadriparesis caused by a brain-stem stroke. Over the course of 48 sessions, we recorded 22 hours of cortical activity while the participant attempted to say individual words from a vocabulary set of 50 words. We used deep-learning algorithms to create computational models for the detection and classification of words from patterns in the recorded cortical activity. We applied these computational models, as well as a natural-language model that yielded next-word probabilities given the preceding words in a sequence, to decode full sentences as the participant attempted to say them.

RESULTS

We decoded sentences from the participant's cortical activity in real time at a median rate of 15.2 words per minute, with a median word error rate of 25.6%. In post hoc analyses, we detected 98% of the attempts by the participant to produce individual words, and we classified words with 47.1% accuracy using cortical signals that were stable throughout the 81-week study period.

CONCLUSIONS

In a person with anarthria and spastic quadriparesis caused by a brain-stem stroke, words and sentences were decoded directly from cortical activity during attempted speech with the use of deep-learning models and a natural-language model. (Funded by Facebook and others; ClinicalTrials.gov number, NCT03698149.).

Improved inference in coupling, encoding, and decoding models and its consequence for neuroscientific interpretation

BACKGROUND

A central goal of systems neuroscience is to understand the relationships amongst constituent units in neural populations, and their modulation by external factors, using high-dimensional and stochastic neural recordings. Parametric statistical models (e.g., coupling, encoding, and decoding models), play an instrumental role in accomplishing this goal. However, extracting conclusions from a parametric model requires that it is fit using an inference algorithm capable of selecting the correct parameters and properly estimating their values. Traditional approaches to parameter inference have been shown to suffer from failures in both selection and estimation. The recent development of algorithms that ameliorate these deficiencies raises the question of whether past work relying on such inference procedures have produced inaccurate systems neuroscience models, thereby impairing their interpretation.

NEW METHOD

We used algorithms based on Union of Intersections, a statistical inference framework based on stability principles, capable of improved selection and estimation.

COMPARISON

We fit functional coupling, encoding, and decoding models across a battery of neural datasets using both UoI and baseline inference procedures (e.g., ℓ₁-penalized GLMs), and compared the structure of their fitted parameters.

RESULTS

Across recording modality, brain region, and task, we found that UoI inferred models with increased sparsity, improved stability, and qualitatively different parameter distributions, while maintaining predictive performance. We obtained highly sparse functional coupling networks with substantially different community structure, more parsimonious encoding models, and decoding models that relied on fewer single-units.

CONCLUSIONS

Together, these results demonstrate that improved parameter inference, achieved via UoI, reshapes interpretation in diverse neuroscience contexts.

Laminar origin of evoked ECoG high-gamma activity

High-gamma (Hγ) activity from electrocorticography (ECoG) is a common-used signal for understanding the human brain, but its interpretation is impeded by a lack of spatial localization. To address this, we developed a novel recording approach to simultaneously record μECoG cortical surface electrical potentials (CSEPs) and laminar multiunit activity (MUA). We demonstrate that stimulus evoked CSEPs carry a multi-modal frequency response, peaking in the Hγ range. Laminar MUA responses exhibited similar tuning to CSEP Hγ directly over the intracortical recording site, suggesting a functional relationship. We fit CSEP Hγ to the simultaneously-recorded laminar MUA using a state-of-the-art sparse multi-linear regression model to identify laminar contributions to cortical surface Hγ. Our results indicate that CSEP Hγ recorded by ECoG reflects spiking activity from neurons in layer 3. These results provide initial insight into localizing the sources of CSEPs, which will guide clinical and BMI device decisions.

A Thirty-Year Review of Oral Appliances Used to Manage Microstomia, 1972 to 2002

Patients who have sustained a facial burn often contend with cosmetic and functional challenges, including microstomia. Many devices have been created by a variety of professionals to provide oral stretching. These devices can be classified as intraoral or extraoral and deliver a stretch either horizontally, vertically, or circumorally. The purpose of this article is to present a comprehensive review of oral-stretching devices.