Stable Decoding from a Speech BCI Enables Control for an Individual with ALS without Recalibration for 3 Months

Neurosurgical and BCI approaches to visual rehabilitation in occipital lobe tumor patients

This study investigates the effects of occipital lobe tumors on visual processing and the role of brain-computer interface (BCI) technologies in post-surgical visual rehabilitation. Through a combination of pre-surgical functional magnetic resonance imaging (fMRI) and Diffusion Tensor Imaging (DTI), intra-operative direct cortical stimulation (DCS) and Electrocorticography (ECoG), and post-surgical BCI interventions, we provide insight into the complex dynamics between occipital lobe tumors and visual function. Our results highlight a discrepancy between clinical assessments of visual field damage and the patient's reported visual experiences, suggesting a residual functional capacity within the damaged occipital regions. Additionally, the absence of expected visual phenomena during surgery and the promising outcomes from BCI-driven rehabilitation underscore the complexity of visual processing and the potential of technology-enhanced rehabilitation strategies. This work emphasizes the need for an interdisciplinary approach in developing effective treatments for visual impairments related to brain tumors, illustrating the significant implications for neurosurgical practices and the advancement of rehabilitation sciences.

Exploring the 'EarSwitch' concept: a novel ear based control method for assistive technology

BACKGROUND

Loss of communication with loved ones and carers is one of the most isolating and debilitating effects of many neurological disorders. Assistive technology (AT) supports individuals with communication, but the acceptability of AT solutions is highly variable. In this paper a novel ear based control method of AT, the concept of 'EarSwitch', is presented. This new approach is based on detecting ear rumbling, which is the voluntary contraction of the tensor tympani muscle (TTM), resulting in observable movement of the eardrum and a dull rumbling sound. 'EarSwitch' has the potential to be a discreet method that can complement existing AT control methods. However, only a subset of the population can ear rumble and little is known about the ability of rumbling in populations with neurological disorders.

METHODS

To explore the viability of the 'EarSwitch' concept as an AT control method we conducted in-depth online surveys with (N=1853) respondents from the general population and (N=170) respondents with self-declared neurological disorders including Motor Neurone Disease (MND) and Multiple Sclerosis (MS).This is the largest ever study to explore ear rumbling and the first to explore whether rumbling is preserved among individuals with neurological disorders. In addition, we validated rumbling, and investigated usability of the 'EarSwitch' concept as a control input, using in-person otoscopic examination with a subset of participants.

RESULTS

A significant proportion of the population with neurological disorders could benefit from 'EarSwitch' controllable AT. The upper bound prevalence of the ability to rumble without accompanying movements was 55% in the general population, 38% in the neurological population, and 20% of participants with MND (N=95) reported this ability. During the validation procedure, participants achieved high accuracy in self-reporting the ability to rumble (80%) and proved concept of using the 'EarSwitch' method to control a basic interface.

DISCUSSION

'EarSwitch' is a potential new AT control method control, either by itself or as a supplement to other existing methods. Results demonstrate self-reported ear rumbling is present among patients with different neurological disorders, including MND. Further research should explore how well the ability to rumble is preserved in different types and stages of neurological disorders.

Advances in brain-computer interface for decoding speech imagery from EEG signals: a systematic review

Numerous individuals encounter challenges in verbal communication due to various factors, including physical disabilities, neurological disorders, and strokes. In response to this pressing need, technology has actively pursued solutions to bridge the communication gap, recognizing the inherent difficulties faced in verbal communication, particularly in contexts where traditional methods may be inadequate. Electroencephalogram (EEG) has emerged as a primary non-invasive method for measuring brain activity, offering valuable insights from a cognitive neurodevelopmental perspective. It forms the basis for Brain-Computer Interfaces (BCIs) that provide a communication channel for individuals with neurological impairments, thereby empowering them to express themselves effectively. EEG-based BCIs, especially those adapted to decode imagined speech from EEG signals, represent a significant advancement in enabling individuals with speech disabilities to communicate through text or synthesized speech. By utilizing cognitive neurodevelopmental insights, researchers have been able to develop innovative approaches for interpreting EEG signals and translating them into meaningful communication outputs. To aid researchers in effectively addressing this complex challenge, this review article synthesizes key findings from state-of-the-art significant studies. It investigates into the methodologies employed by various researchers, including preprocessing techniques, feature extraction methods, and classification algorithms utilizing Deep Learning and Machine Learning approaches and their integration. Furthermore, the review outlines the potential avenues for future research, with the goal of advancing the practical implementation of EEG-based BCI systems for decoding imagined speech from a cognitive neurodevelopmental perspective.

Speech motor cortex enables BCI cursor control and click

Decoding neural activity from ventral (speech) motor cortex is known to enable high-performance speech brain-computer interface (BCI) control. It was previously unknown whether this brain area could also enable computer control via neural cursor and click, as is typically associated with dorsal (arm and hand) motor cortex. We recruited a clinical trial participant with ALS and implanted intracortical microelectrode arrays in ventral precentral gyrus (vPCG), which the participant used to operate a speech BCI in a prior study. We developed a cursor BCI driven by the participant's vPCG neural activity, and evaluated performance on a series of target selection tasks. The reported vPCG cursor BCI enabled rapidly-calibrating (40 seconds), accurate (2.90 bits per second) cursor control and click. The participant also used the BCI to control his own personal computer independently. These results suggest that placing electrodes in vPCG to optimize for speech decoding may also be a viable strategy for building a multi-modal BCI which enables both speech-based communication and computer control via cursor and click.

Using data from cue presentations results in grossly overestimating semantic BCI performance

Neuroimaging studies have reported the possibility of semantic neural decoding to identify specific semantic concepts from neural activity. This offers promise for brain-computer interfaces (BCIs) for communication. However, translating these findings into a BCI paradigm has proven challenging. Existing EEG-based semantic decoding studies often rely on neural activity recorded when a cue is present, raising concerns about decoding reliability. To address this, we investigate the effects of cue presentation on EEG-based semantic decoding. In an experiment with a clear separation between cue presentation and mental task periods, we attempt to differentiate between semantic categories of animals and tools in four mental tasks. By using state-of-the-art decoding analyses, we demonstrate significant mean classification accuracies up to 71.3% during cue presentation but not during mental tasks, even with adapted analyses from previous studies. These findings highlight a potential issue when using neural activity recorded during cue presentation periods for semantic decoding. Additionally, our results show that semantic decoding without external cues may be more challenging than current state-of-the-art research suggests. By bringing attention to these issues, we aim to stimulate discussion and drive advancements in the field toward more effective semantic BCI applications.

Neural interfaces: Bridging the brain to the world beyond healthcare

Neural interfaces, emerging at the intersection of neurotechnology and urban planning, promise to transform how we interact with our surroundings and communicate. By recording and decoding neural signals, these interfaces facilitate direct connections between the brain and external devices, enabling seamless information exchange and shared experiences. Nevertheless, their development is challenged by complexities in materials science, electrochemistry, and algorithmic design. Electrophysiological crosstalk and the mismatch between electrode rigidity and tissue flexibility further complicate signal fidelity and biocompatibility. Recent closed-loop brain-computer interfaces, while promising for mood regulation and cognitive enhancement, are limited by decoding accuracy and the adaptability of user interfaces. This perspective outlines these challenges and discusses the progress in neural interfaces, contrasting non-invasive and invasive approaches, and explores the dynamics between stimulation and direct interfacing. Emphasis is placed on applications beyond healthcare, highlighting the need for implantable interfaces with high-resolution recording and stimulation capabilities.

Real-time detection of spoken speech from unlabeled ECoG signals: A pilot study with an ALS participant

Objective

Brain-Computer Interfaces (BCIs) hold significant promise for restoring communication in individuals with partial or complete loss of the ability to speak due to paralysis from amyotrophic lateral sclerosis (ALS), brainstem stroke, and other neurological disorders. Many of the approaches to speech decoding reported in the BCI literature have required time-aligned target representations to allow successful training - a major challenge when translating such approaches to people who have already lost their voice.

Approach

In this pilot study, we made a first step toward scenarios in which no ground truth is available. We utilized a graph-based clustering approach to identify temporal segments of speech production from electrocorticographic (ECoG) signals alone. We then used the estimated speech segments to train a voice activity detection (VAD) model using only ECoG signals. We evaluated our approach using held-out open-loop recordings of a single dysarthric clinical trial participant living with ALS, and we compared the resulting performance to previous solutions trained with ground truth acoustic voice recordings.

Main results

Our approach achieves a median error rate of around 0.5 seconds with respect to the actual spoken speech. Embedded into a real-time BCI, our approach is capable of providing VAD results with a latency of only 10 ms.

Significance

To the best of our knowledge, our results show for the first time that speech activity can be predicted purely from unlabeled ECoG signals, a crucial step toward individuals who cannot provide this information anymore due to their neurological condition, such as patients with locked-in syndrome.

Clinical Trial Information

ClinicalTrials.gov, registration number NCT03567213.

Brain-computer interfaces: the innovative key to unlocking neurological conditions

Neurological disorders such as Parkinson's disease, stroke, and spinal cord injury can pose significant threats to human mortality, morbidity, and functional independence. Brain-Computer Interface (BCI) technology, which facilitates direct communication between the brain and external devices, emerges as an innovative key to unlocking neurological conditions, demonstrating significant promise in this context. This comprehensive review uniquely synthesizes the latest advancements in BCI research across multiple neurological disorders, offering an interdisciplinary perspective on both clinical applications and emerging technologies. We explore the progress in BCI research and its applications in addressing various neurological conditions, with a particular focus on recent clinical studies and prospective developments. Initially, the review provides an up-to-date overview of BCI technology, encompassing its classification, operational principles, and prevalent paradigms. It then critically examines specific BCI applications in movement disorders, disorders of consciousness, cognitive and mental disorders, as well as sensory disorders, highlighting novel approaches and their potential impact on patient care. This review reveals emerging trends in BCI applications, such as the integration of artificial intelligence and the development of closed-loop systems, which represent significant advancements over previous technologies. The review concludes by discussing the prospects and directions of BCI technology, underscoring the need for interdisciplinary collaboration and ethical considerations. It emphasizes the importance of prioritizing bidirectional and high-performance BCIs, areas that have been underexplored in previous reviews. Additionally, we identify crucial gaps in current research, particularly in long-term clinical efficacy and the need for standardized protocols. The role of neurosurgery in spearheading the clinical translation of BCI research is highlighted. Our comprehensive analysis presents BCI technology as an innovative key to unlocking neurological disorders, offering a transformative approach to diagnosing, treating, and rehabilitating neurological conditions, with substantial potential to enhance patients' quality of life and advance the field of neurotechnology.

Iterative alignment discovery of speech-associated neural activity

Objective. Brain-computer interfaces (BCIs) have the potential to preserve or restore speech in patients with neurological disorders that weaken the muscles involved in speech production. However, successful training of low-latency speech synthesis and recognition models requires alignment of neural activity with intended phonetic or acoustic output with high temporal precision. This is particularly challenging in patients who cannot produce audible speech, as ground truth with which to pinpoint neural activity synchronized with speech is not available.Approach. In this study, we present a new iterative algorithm for neural voice activity detection (nVAD) called iterative alignment discovery dynamic time warping (IAD-DTW) that integrates DTW into the loss function of a deep neural network (DNN). The algorithm is designed to discover the alignment between a patient's electrocorticographic (ECoG) neural responses and their attempts to speak during collection of data for training BCI decoders for speech synthesis and recognition.Main results. To demonstrate the effectiveness of the algorithm, we tested its accuracy in predicting the onset and duration of acoustic signals produced by able-bodied patients with intact speech undergoing short-term diagnostic ECoG recordings for epilepsy surgery. We simulated a lack of ground truth by randomly perturbing the temporal correspondence between neural activity and an initial single estimate for all speech onsets and durations. We examined the model's ability to overcome these perturbations to estimate ground truth. IAD-DTW showed no notable degradation (<1% absolute decrease in accuracy) in performance in these simulations, even in the case of maximal misalignments between speech and silence.Significance. IAD-DTW is computationally inexpensive and can be easily integrated into existing DNN-based nVAD approaches, as it pertains only to the final loss computation. This approach makes it possible to train speech BCI algorithms using ECoG data from patients who are unable to produce audible speech, including those with Locked-In Syndrome.

An Accurate and Rapidly Calibrating Speech Neuroprosthesis

BACKGROUND

Brain-computer interfaces can enable communication for people with paralysis by transforming cortical activity associated with attempted speech into text on a computer screen. Communication with brain-computer interfaces has been restricted by extensive training requirements and limited accuracy.

METHODS

A 45-year-old man with amyotrophic lateral sclerosis (ALS) with tetraparesis and severe dysarthria underwent surgical implantation of four microelectrode arrays into his left ventral precentral gyrus 5 years after the onset of the illness; these arrays recorded neural activity from 256 intracortical electrodes. We report the results of decoding his cortical neural activity as he attempted to speak in both prompted and unstructured conversational contexts. Decoded words were displayed on a screen and then vocalized with the use of text-to-speech software designed to sound like his pre-ALS voice.

RESULTS

On the first day of use (25 days after surgery), the neuroprosthesis achieved 99.6% accuracy with a 50-word vocabulary. Calibration of the neuroprosthesis required 30 minutes of cortical recordings while the participant attempted to speak, followed by subsequent processing. On the second day, after 1.4 additional hours of system training, the neuroprosthesis achieved 90.2% accuracy using a 125,000-word vocabulary. With further training data, the neuroprosthesis sustained 97.5% accuracy over a period of 8.4 months after surgical implantation, and the participant used it to communicate in self-paced conversations at a rate of approximately 32 words per minute for more than 248 cumulative hours.

CONCLUSIONS

In a person with ALS and severe dysarthria, an intracortical speech neuroprosthesis reached a level of performance suitable to restore conversational communication after brief training. (Funded by the Office of the Assistant Secretary of Defense for Health Affairs and others; BrainGate2 ClinicalTrials.gov number, NCT00912041.).

A bilingual speech neuroprosthesis driven by cortical articulatory representations shared between languages

Advancements in decoding speech from brain activity have focused on decoding a single language. Hence, the extent to which bilingual speech production relies on unique or shared cortical activity across languages has remained unclear. Here, we leveraged electrocorticography, along with deep-learning and statistical natural-language models of English and Spanish, to record and decode activity from speech-motor cortex of a Spanish-English bilingual with vocal-tract and limb paralysis into sentences in either language. This was achieved without requiring the participant to manually specify the target language. Decoding models relied on shared vocal-tract articulatory representations across languages, which allowed us to build a syllable classifier that generalized across a shared set of English and Spanish syllables. Transfer learning expedited training of the bilingual decoder by enabling neural data recorded in one language to improve decoding in the other language. Overall, our findings suggest shared cortical articulatory representations that persist after paralysis and enable the decoding of multiple languages without the need to train separate language-specific decoders.

Stability of ECoG high gamma signals during speech and implications for a speech BCI system in an individual with ALS: a year-long longitudinal study

Objective.Speech brain-computer interfaces (BCIs) have the potential to augment communication in individuals with impaired speech due to muscle weakness, for example in amyotrophic lateral sclerosis (ALS) and other neurological disorders. However, to achieve long-term, reliable use of a speech BCI, it is essential for speech-related neural signal changes to be stable over long periods of time. Here we study, for the first time, the stability of speech-related electrocorticographic (ECoG) signals recorded from a chronically implanted ECoG BCI over a 12 month period.Approach.ECoG signals were recorded by an ECoG array implanted over the ventral sensorimotor cortex in a clinical trial participant with ALS. Because ECoG-based speech decoding has most often relied on broadband high gamma (HG) signal changes relative to baseline (non-speech) conditions, we studied longitudinal changes of HG band power at baseline and during speech, and we compared these with residual high frequency noise levels at baseline. Stability was further assessed by longitudinal measurements of signal-to-noise ratio, activation ratio, and peak speech-related HG response magnitude (HG response peaks). Lastly, we analyzed the stability of the event-related HG power changes (HG responses) for individual syllables at each electrode.Main Results.We found that speech-related ECoG signal responses were stable over a range of syllables activating different articulators for the first year after implantation.Significance.Together, our results indicate that ECoG can be a stable recording modality for long-term speech BCI systems for those living with severe paralysis.Clinical Trial Information.ClinicalTrials.gov, registration number NCT03567213.

The speech neuroprosthesis

Loss of speech after paralysis is devastating, but circumventing motor-pathway injury by directly decoding speech from intact cortical activity has the potential to restore natural communication and self-expression. Recent discoveries have defined how key features of speech production are facilitated by the coordinated activity of vocal-tract articulatory and motor-planning cortical representations. In this Review, we highlight such progress and how it has led to successful speech decoding, first in individuals implanted with intracranial electrodes for clinical epilepsy monitoring and subsequently in individuals with paralysis as part of early feasibility clinical trials to restore speech. We discuss high-spatiotemporal-resolution neural interfaces and the adaptation of state-of-the-art speech computational algorithms that have driven rapid and substantial progress in decoding neural activity into text, audible speech, and facial movements. Although restoring natural speech is a long-term goal, speech neuroprostheses already have performance levels that surpass communication rates offered by current assistive-communication technology. Given this accelerated rate of progress in the field, we propose key evaluation metrics for speed and accuracy, among others, to help standardize across studies. We finish by highlighting several directions to more fully explore the multidimensional feature space of speech and language, which will continue to accelerate progress towards a clinically viable speech neuroprosthesis.

Online speech synthesis using a chronically implanted brain-computer interface in an individual with ALS

Brain-computer interfaces (BCIs) that reconstruct and synthesize speech using brain activity recorded with intracranial electrodes may pave the way toward novel communication interfaces for people who have lost their ability to speak, or who are at high risk of losing this ability, due to neurological disorders. Here, we report online synthesis of intelligible words using a chronically implanted brain-computer interface (BCI) in a man with impaired articulation due to ALS, participating in a clinical trial (ClinicalTrials.gov, NCT03567213) exploring different strategies for BCI communication. The 3-stage approach reported here relies on recurrent neural networks to identify, decode and synthesize speech from electrocorticographic (ECoG) signals acquired across motor, premotor and somatosensory cortices. We demonstrate a reliable BCI that synthesizes commands freely chosen and spoken by the participant from a vocabulary of 6 keywords previously used for decoding commands to control a communication board. Evaluation of the intelligibility of the synthesized speech indicates that 80% of the words can be correctly recognized by human listeners. Our results show that a speech-impaired individual with ALS can use a chronically implanted BCI to reliably produce synthesized words while preserving the participant's voice profile, and provide further evidence for the stability of ECoG for speech-based BCIs.

An accurate and rapidly calibrating speech neuroprosthesis

Brain-computer interfaces can enable rapid, intuitive communication for people with paralysis by transforming the cortical activity associated with attempted speech into text on a computer screen. Despite recent advances, communication with brain-computer interfaces has been restricted by extensive training data requirements and inaccurate word output. A man in his 40's with ALS with tetraparesis and severe dysarthria (ALSFRS-R = 23) was enrolled into the BrainGate2 clinical trial. He underwent surgical implantation of four microelectrode arrays into his left precentral gyrus, which recorded neural activity from 256 intracortical electrodes. We report a speech neuroprosthesis that decoded his neural activity as he attempted to speak in both prompted and unstructured conversational settings. Decoded words were displayed on a screen, then vocalized using text-to-speech software designed to sound like his pre-ALS voice. On the first day of system use, following 30 minutes of attempted speech training data, the neuroprosthesis achieved 99.6% accuracy with a 50-word vocabulary. On the second day, the size of the possible output vocabulary increased to 125,000 words, and, after 1.4 additional hours of training data, the neuroprosthesis achieved 90.2% accuracy. With further training data, the neuroprosthesis sustained 97.5% accuracy beyond eight months after surgical implantation. The participant has used the neuroprosthesis to communicate in self-paced conversations for over 248 hours. In an individual with ALS and severe dysarthria, an intracortical speech neuroprosthesis reached a level of performance suitable to restore naturalistic communication after a brief training period.

Don't put words in my mouth: Speech perception can generate False Positive activation of a speech BCI

Recent studies have demonstrated that speech can be decoded from brain activity and used for brain-computer interface (BCI)-based communication. It is however also known that the area often used as a signal source for speech decoding BCIs, the sensorimotor cortex (SMC), is also engaged when people perceive speech, thus making speech perception a potential source of false positive activation of the BCI. The current study investigated if and how speech perception may interfere with reliable speech BCI control. We recorded high-density electrocorticography (HD-ECoG) data from five subjects while they performed a speech perception and speech production task and trained a support-vector machine (SVM) on the produced speech data. Our results show that decoders that are highly reliable at detecting self-produced speech from brain signals also generate false positives during the perception of speech. We conclude that speech perception interferes with reliable BCI control, and that efforts to limit the occurrence of false positives during daily-life BCI use should be implemented in BCI design to increase the likelihood of successful adaptation by end users.

Neurosurgery and the Brain-Computer Interface

Brain-computer interfaces (BCIs) are devices that connect the human brain to an effector via a computer and electrode interface. BCIs may also transmit sensory data to the brain. We describe progress with the many types of surgically implanted BCIs, in which electrodes contact or penetrate the cerebral cortex. BCIs developed for restoration of movement in paralyzed limbs or control a robotic arm; restoration of somatic sensation, speech, vision, memory, hearing, and olfaction are also presented. Most devices remain experimental. Commercialization is costly, incurs financial risk, and is time consuming. There are many ethical principles that should be considered by neurosurgeons and by all those responsible for the care of people with serious neurological disability. These considerations are also paramount when the technology is used in for the purpose of enhancement of normal function and where commercial gain is a factor. A new regulatory and legislative framework is urgently required. The evolution of BCIs is occurring rapidly with advances in computer science, artificial intelligence, electronic engineering including wireless transmission, and materials science. The era of the brain-"cloud" interface is approaching.