Neuroprosthesis for Decoding Speech in a Paralyzed Person with Anarthria

The Case for Neurosurgical Intervention in Cancer Neuroscience

The emerging field of cancer neuroscience reshapes our understanding of the intricate relationship between the nervous system and cancer biology; this new paradigm is likely to fundamentally change and advance neuro-oncological care. The profound interplay between cancers and the nervous system is reciprocal: Cancer growth can be induced and regulated by the nervous system; conversely, tumors can themselves alter the nervous system. Such crosstalk between cancer cells and the nervous system is evident in both the peripheral and central nervous systems. Recent advances have uncovered numerous direct neuron-cancer interactions at glioma-neuronal synapses, paracrine mechanisms within the tumor microenvironment, and indirect neuroimmune interactions. Neurosurgeons have historically played a central role in neuro-oncological care, and as the field of cancer neuroscience is becoming increasingly established, the role of neurosurgical intervention is becoming clearer. Examples include peripheral denervation procedures, delineation of neuron-glioma networks, development of neuroprostheses, neuromodulatory procedures, and advanced local delivery systems. The present review seeks to highlight key cancer neuroscience mechanisms with neurosurgical implications and outline the future role of neurosurgical intervention in cancer neuroscience.

A Method to Extract Task-Related EEG Feature Based on Lightweight Convolutional Neural Network

Unlocking task-related EEG spectra is crucial for neuroscience. Traditional convolutional neural networks (CNNs) effectively extract these features but face limitations like overfitting due to small datasets. To address this issue, we propose a lightweight CNN and assess its interpretability through the fully connected layer (FCL). Initially tested with two tasks (Task 1: open vs closed eyes, Task 2: interictal vs ictal stage), the CNN demonstrated enhanced spectral features in the alpha band for Task 1 and the theta band for Task 2, aligning with established neurophysiological characteristics. Subsequent experiments on two brain-computer interface tasks revealed a correlation between delta activity (around 1.55 Hz) and hand movement, with consistent results across pericentral electroencephalogram (EEG) channels. Compared to recent research, our method stands out by delivering task-related spectral features through FCL, resulting in significantly fewer trainable parameters while maintaining comparable interpretability. This indicates its potential suitability for a wider array of EEG decoding scenarios.

A brain-to-text framework for decoding natural tonal sentences

Speech brain-computer interfaces (BCIs) directly translate brain activity into speech sound and text. Despite successful applications in non-tonal languages, the distinct syllabic structures and pivotal lexical information conveyed through tonal nuances present challenges in BCI decoding for tonal languages like Mandarin Chinese. Here, we designed a brain-to-text framework to decode Mandarin sentences from invasive neural recordings. Our framework dissects speech onset, base syllables, and lexical tones, integrating them with contextual information through Bayesian likelihood and a Viterbi decoder. The results demonstrate accurate tone and syllable decoding during naturalistic speech production. The overall word error rate (WER) for 10 offline-decoded tonal sentences with a vocabulary of 40 high-frequency Chinese characters is 21% (chance: 95.3%) averaged across five participants, and tone decoding accuracy reaches 93% (chance: 25%), surpassing previous intracranial Mandarin tonal syllable decoders. This study provides a robust and generalizable approach for brain-to-text decoding of continuous tonal speech sentences.

Direct-Print 3D Electrodes for Large-Scale, High-Density, and Customizable Neural Interfaces

Silicon-based microelectronics can scalably record and modulate neural activity at high spatiotemporal resolution, but their planar form factor poses challenges in targeting 3D neural structures. A method for fabricating tissue-penetrating 3D microelectrodes directly onto planar microelectronics using high-resolution 3D printing via 2-photon polymerization and scalable microfabrication technologies are presented. This approach enables customizable electrode shape, height, and positioning for precise targeting of neuron populations distributed in 3D. The effectiveness of this approach is demonstrated in tackling the critical challenge of interfacing with the retina-specifically, selectively targeting retinal ganglion cell (RGC) somas while avoiding the axon bundle layer. 6,600-microelectrode, 35 µm pitch, tissue-penetrating arrays are fabricated to obtain high-fidelity, high-resolution, and large-scale retinal recording that reveals little axonal interference, a capability previously undemonstrated. Confocal microscopy further confirms the precise placement of the microelectrodes. This technology can be a versatile solution for interfacing silicon microelectronics with neural structures at a large scale and cellular resolution.

Speech motor cortex enables BCI cursor control and click

Decoding neural activity from ventral (speech) motor cortex is known to enable high-performance speech brain-computer interface (BCI) control. It was previously unknown whether this brain area could also enable computer control via neural cursor and click, as is typically associated with dorsal (arm and hand) motor cortex. We recruited a clinical trial participant with ALS and implanted intracortical microelectrode arrays in ventral precentral gyrus (vPCG), which the participant used to operate a speech BCI in a prior study. We developed a cursor BCI driven by the participant's vPCG neural activity, and evaluated performance on a series of target selection tasks. The reported vPCG cursor BCI enabled rapidly-calibrating (40 seconds), accurate (2.90 bits per second) cursor control and click. The participant also used the BCI to control his own personal computer independently. These results suggest that placing electrodes in vPCG to optimize for speech decoding may also be a viable strategy for building a multi-modal BCI which enables both speech-based communication and computer control via cursor and click.

Superior cortical venous anatomy for endovascular device implantation: a systematic review

Endovascular electrode arrays provide a minimally invasive approach to access intracranial structures for neural recording and stimulation. These arrays are currently used as brain-computer interfaces (BCIs) and are deployed within the superior sagittal sinus (SSS), although cortical vein implantation could improve the quality and quantity of recorded signals. However, the anatomy of the superior cortical veins is heterogenous and poorly characterised. MEDLINE and Embase databases were systematically searched from inception to December 15, 2023 for studies describing the anatomy of the superior cortical veins. A total of 28 studies were included: 19 cross-sectional imaging studies, six cadaveric studies, one intraoperative anatomical study and one review. There was substantial variability in cortical vein diameter, length, confluence angle, and location relative to the underlying cortex. The mean number of SSS branches ranged from 11 to 45. The vein of Trolard was most often reported as the largest superior cortical vein, with a mean diameter ranging from 2.1 mm to 3.3 mm. The mean vein of Trolard was identified posterior to the central sulcus. One study found a significant age-related variability in cortical vein diameter and another identified myoendothelial sphincters at the base of the cortical veins. Cortical vein anatomical data are limited and inconsistent. The vein of Trolard is the largest tributary vein of the SSS; however, its relation to the underlying cortex is variable. Variability in cortical vein anatomy may necessitate individualized pre-procedural planning of training and neural decoding in endovascular BCI. Future focus on the relation to the underlying cortex, sulcal vessels, and vessel wall anatomy is required.

From Single Words to Sentence Production: Shared Cortical Representations but Distinct Temporal Dynamics

Sentence production is the uniquely human ability to transform complex thoughts into strings of words. Despite the importance of this process, language production research has primarily focused on single words. It remains an untested assumption that insights from this literature generalize to more naturalistic utterances like sentences. Here, we investigate this using high-resolution neurosurgical recordings (ECoG) and an overt production experiment where patients produce six words in isolation (picture naming) and in sentences (scene description). We trained machine learning models to identify the unique brain activity pattern for each word during picture naming, and used these patterns to decode which words patients were processing while they produced sentences. Our findings reveal that words share cortical representations across tasks. In sensorimotor cortex, words were consistently activated in the order in which they were said in the sentence. However, in inferior and middle frontal gyri (IFG and MFG), the order in which words were processed depended on the syntactic structure of the sentence. This dynamic interplay between sentence structure and word processing reveals that sentence production is not simply a sequence of single word production tasks, and highlights a regional division of labor within the language network. Finally, we argue that the dynamics of word processing in prefrontal cortex may impose a subtle pressure on language evolution, explaining why nearly all the world's languages position subjects before objects.

Using data from cue presentations results in grossly overestimating semantic BCI performance

Neuroimaging studies have reported the possibility of semantic neural decoding to identify specific semantic concepts from neural activity. This offers promise for brain-computer interfaces (BCIs) for communication. However, translating these findings into a BCI paradigm has proven challenging. Existing EEG-based semantic decoding studies often rely on neural activity recorded when a cue is present, raising concerns about decoding reliability. To address this, we investigate the effects of cue presentation on EEG-based semantic decoding. In an experiment with a clear separation between cue presentation and mental task periods, we attempt to differentiate between semantic categories of animals and tools in four mental tasks. By using state-of-the-art decoding analyses, we demonstrate significant mean classification accuracies up to 71.3% during cue presentation but not during mental tasks, even with adapted analyses from previous studies. These findings highlight a potential issue when using neural activity recorded during cue presentation periods for semantic decoding. Additionally, our results show that semantic decoding without external cues may be more challenging than current state-of-the-art research suggests. By bringing attention to these issues, we aim to stimulate discussion and drive advancements in the field toward more effective semantic BCI applications.

Neural Decoding of Spontaneous Overt and Intended Speech

PURPOSE

The aim of this study was to decode intended and overt speech from neuromagnetic signals while the participants performed spontaneous overt speech tasks without cues or prompts (stimuli).

METHOD

Magnetoencephalography (MEG), a noninvasive neuroimaging technique, was used to collect neural signals from seven healthy adult English speakers performing spontaneous, overt speech tasks. The participants randomly spoke the words yes or no at a self-paced rate without cues. Two machine learning models, namely, linear discriminant analysis (LDA) and one-dimensional convolutional neural network (1D CNN), were employed to classify the two words from the recorded MEG signals.

RESULTS

LDA and 1D CNN achieved average decoding accuracies of 79.02% and 90.40%, respectively, in decoding overt speech, significantly surpassing the chance level (50%). The accuracy for decoding intended speech was 67.19% using 1D CNN.

CONCLUSIONS

This study showcases the possibility of decoding spontaneous overt and intended speech directly from neural signals in the absence of perceptual interference. We believe that these findings make a steady step toward the future spontaneous speech-based brain-computer interface.

Reducing power requirements for high-accuracy decoding in iBCIs

Objective.Current intracortical brain-computer interfaces (iBCIs) rely predominantly on threshold crossings ('spikes') for decoding neural activity into a control signal for an external device. Spiking data can yield high accuracy online control during complex behaviors; however, its dependence on high-sampling-rate data collection can pose challenges. An alternative signal for iBCI decoding is the local field potential (LFP), a continuous-valued signal that can be acquired simultaneously with spiking activity. However, LFPs are seldom used alone for online iBCI control as their decoding performance has yet to achieve parity with spikes.Approach.Here, we present a strategy to improve the performance of LFP-based decoders by first training a neural dynamics model to use LFPs to reconstruct the firing rates underlying spiking data, and then decoding from the estimated rates. We test these models on previously-collected macaque data during center-out and random-target reaching tasks as well as data collected from a human iBCI participant during attempted speech.Main results.In all cases, training models from LFPs enables firing rate reconstruction with accuracy comparable to spiking-based dynamics models. In addition, LFP-based dynamics models enable decoding performance exceeding that of LFPs alone and approaching that of spiking-based models. In all applications except speech, LFP-based dynamics models also facilitate decoding accuracy exceeding that of direct decoding from spikes.Significance.Because LFP-based dynamics models operate on lower bandwidth and with lower sampling rate than spiking models, our findings indicate that iBCI devices can be designed to operate with lower power requirements than devices dependent on recorded spiking activity, without sacrificing high-accuracy decoding.

Brain-Controlled Augmented Hearing for Spatially Moving Conversations in Multi-Talker Environments

Focusing on a specific conversation amidst multiple interfering talkers is challenging, especially for those with hearing loss. Brain-controlled assistive hearing devices aim to alleviate this problem by enhancing the attended speech based on the listener's neural signals using auditory attention decoding (AAD). Departing from conventional AAD studies that relied on oversimplified scenarios with stationary talkers, a realistic AAD task that involves multiple talkers taking turns as they continuously move in space in background noise is presented. Invasive electroencephalography (iEEG) data are collected from three neurosurgical patients as they focused on one of the two moving conversations. An enhanced brain-controlled assistive hearing system that combines AAD and a binaural speaker-independent speech separation model is presented. The separation model unmixes talkers while preserving their spatial location and provides talker trajectories to the neural decoder to improve AAD accuracy. Subjective and objective evaluations show that the proposed system enhances speech intelligibility and facilitates conversation tracking while maintaining spatial cues and voice quality in challenging acoustic environments. This research demonstrates the potential of this approach in real-world scenarios and marks a significant step toward developing assistive hearing technologies that adapt to the intricate dynamics of everyday auditory experiences.

Implanted cortical neuroprosthetics for speech and movement restoration

Implanted cortical neuroprosthetics (ICNs) are medical devices developed to replace dysfunctional neural pathways by creating information exchange between the brain and a digital system which can facilitate interaction with the external world. Over the last decade, researchers have explored the application of ICNs for diverse conditions including blindness, aphasia, and paralysis. Both transcranial and endovascular approaches have been used to record neural activity in humans, and in a laboratory setting, high-performance decoding of the signals associated with speech intention has been demonstrated. Particular progress towards a device which can move into clinical practice has been made with ICNs focussed on the restoration of speech and movement. This article provides an overview of contemporary ICNs for speech and movement restoration, their mechanisms of action and the unique ethical challenges raised by the field.

Caregivers in implantable brain-computer interface research: a scoping review

Introduction

While the ethical significance of caregivers in neurological research has increasingly been recognized, the role of caregivers in brain-computer interface (BCI) research has received relatively less attention.

Objectives

This report investigates the extent to which caregivers are mentioned in publications describing implantable BCI (iBCI) research for individuals with motor dysfunction, communication impairment, and blindness.

Methods

The scoping review was conducted in June 2024 using the PubMed and Web of Science bibliographic databases. The articles were systematically searched using query terms for caregivers, family members, and guardians, and the results were quantitatively and qualitatively analyzed.

Results

Our search yielded 315 unique studies, 78 of which were included in this scoping review. Thirty-four (43.6%) of the 78 articles mentioned the study participant's caregivers. We sorted these into 5 categories: Twenty-two (64.7%) of the 34 articles thanked caregivers in the acknowledgement section, 6 (17.6%) articles described the caregiver's role with regard to the consent process, 12 (35.3%) described the caregiver's role in the technical maintenance and upkeep of the BCI system or in other procedural aspects of the study, 9 (26.5%) discussed how the BCI enhanced participant communication and goal-directed behavior with the help of a caregiver, and 3 (8.8%) articles included general comments that did not fit into the other categories but still related to the importance of caregivers in the lives of the research participants.

Discussion

Caregivers were mentioned in less than half of BCI studies in this review. The studies that offered more robust discussions of caregivers provide valuable insight into the integral role that caregivers play in supporting the study participants and the research process. Attention to the role of caregivers in successful BCI research studies can help guide the responsible development of future BCI study protocols.

A click-based electrocorticographic brain-computer interface enables long-term high-performance switch scan spelling

BACKGROUND

Brain-computer interfaces (BCIs) can restore communication for movement- and/or speech-impaired individuals by enabling neural control of computer typing applications. Single command click detectors provide a basic yet highly functional capability.

METHODS

We sought to test the performance and long-term stability of click decoding using a chronically implanted high density electrocorticographic (ECoG) BCI with coverage of the sensorimotor cortex in a human clinical trial participant (ClinicalTrials.gov, NCT03567213) with amyotrophic lateral sclerosis. We trained the participant's click detector using a small amount of training data (<44 min across 4 days) collected up to 21 days prior to BCI use, and then tested it over a period of 90 days without any retraining or updating.

RESULTS

Using a click detector to navigate a switch scanning speller interface, the study participant can maintain a median spelling rate of 10.2 characters per min. Though a transient reduction in signal power modulation can interrupt usage of a fixed model, a new click detector can achieve comparable performance despite being trained with even less data (<15 min, within 1 day).

CONCLUSIONS

These results demonstrate that a click detector can be trained with a small ECoG dataset while retaining robust performance for extended periods, providing functional text-based communication to BCI users.

Gustatory interface for operative assessment and taste decoding in patients with tongue cancer

Taste, a pivotal sense modality, plays a fundamental role in discerning flavors and evaluating the potential harm of food, thereby contributing to human survival, physical and mental health. Patients with tongue cancer may experience a loss of taste following extensive surgical resection with flap reconstruction. Here, we designed a gustatory interface that enables the non-invasive detection of tongue electrical activities for a comprehensive operative assessment. Moreover, it decodes gustatory information from the reconstructed tongue without taste buds. Our gustatory interface facilitates the recording and analysis of electrical activities on the tongue, yielding an electrical mapping across the entire tongue surface, which delineates the safe margin for surgical management and assesses flap viability for postoperative structure monitoring and prompt intervention. Furthermore, the gustatory interface helps patients discern tastes with an accuracy of 97.8%. Our invention offers a promising approach to clinical assessment and management and holds potential for improving the quality of life for individuals with tongue cancer.

Decoding the brain: From neural representations to mechanistic models

A central principle in neuroscience is that neurons within the brain act in concert to produce perception, cognition, and adaptive behavior. Neurons are organized into specialized brain areas, dedicated to different functions to varying extents, and their function relies on distributed circuits to continuously encode relevant environmental and body-state features, enabling other areas to decode (interpret) these representations for computing meaningful decisions and executing precise movements. Thus, the distributed brain can be thought of as a series of computations that act to encode and decode information. In this perspective, we detail important concepts of neural encoding and decoding and highlight the mathematical tools used to measure them, including deep learning methods. We provide case studies where decoding concepts enable foundational and translational science in motor, visual, and language processing.

DO NOT LOSE SLEEP OVER IT: IMPLANTED BRAIN-COMPUTER INTERFACE FUNCTIONALITY DURING NIGHTTIME IN LATE-STAGE AMYOTROPHIC LATERAL SCLEROSIS

Background and objectives

Brain-computer interfaces (BCIs) hold promise as augmentative and alternative communication technology for people with severe motor and speech impairment (locked-in syndrome) due to neural disease or injury. Although such BCIs should be available 24/7, to enable communication at all times, feasibility of nocturnal BCI use has not been investigated. Here, we addressed this question using data from an individual with amyotrophic lateral sclerosis (ALS) who was implanted with an electrocorticography-based BCI that enabled the generation of click-commands for spelling words and call-caregiver signals.

Methods

We investigated nocturnal dynamics of neural signal features used for BCI control, namely low (LFB: 10-30Hz) and high frequency band power (HFB: 65-95Hz). Additionally, we assessed the nocturnal performance of a BCI decoder that was trained on daytime data by quantifying the number of unintentional BCI activations at night. Finally, we developed and implemented a nightmode decoder that allowed the participant to call a caregiver at night, and assessed its performance.

Results

Power and variance in HFB and LFB were significantly higher at night than during the day in the majority of the nights, with HFB variance being higher in 88% of nights. Daytime decoders caused 245 unintended selection-clicks and 13 unintended caregiver-calls per hour when applied to night data. The developed nightmode decoder functioned error-free in 79% of nights over a period of ±1.5 years, allowing the user to reliably call the caregiver, with unintended activations occurring only once every 12 nights.

Discussion

Reliable nighttime use of a BCI requires decoders that are adjusted to sleep-related signal changes. This demonstration of a reliable BCI nightmode and its long-term use by an individual with advanced ALS underscores the importance of 24/7 BCI reliability.

Trial registration

This trial is registered in clinicaltrials.gov under number NCT02224469 (https://clinicaltrials.gov/study/NCT02224469?term=NCT02224469&rank=1). Date of submission to registry: August 21, 2014. Enrollment of first participant: September 7, 2015.

AI Applications in Adult Stroke Recovery and Rehabilitation: A Scoping Review Using AI

Stroke is a leading cause of long-term disability worldwide. With the advancements in sensor technologies and data availability, artificial intelligence (AI) holds the promise of improving the amount, quality and efficiency of care and enhancing the precision of stroke rehabilitation. We aimed to identify and characterize the existing research on AI applications in stroke recovery and rehabilitation of adults, including categories of application and progression of technologies over time. Data were collected from peer-reviewed articles across various electronic databases up to January 2024. Insights were extracted using AI-enhanced multi-method, data-driven techniques, including clustering of themes and topics. This scoping review summarizes outcomes from 704 studies. Four common themes (impairment, assisted intervention, prediction and imaging, and neuroscience) were identified, in which time-linked patterns emerged. The impairment theme revealed a focus on motor function, gait and mobility, while the assisted intervention theme included applications of robotic and brain-computer interface (BCI) techniques. AI applications progressed over time, starting from conceptualization and then expanding to a broader range of techniques in supervised learning, artificial neural networks (ANN), natural language processing (NLP) and more. Applications focused on upper limb rehabilitation were reviewed in more detail, with machine learning (ML), deep learning techniques and sensors such as inertial measurement units (IMU) used for upper limb and functional movement analysis. AI applications have potential to facilitate tailored therapeutic delivery, thereby contributing to the optimization of rehabilitation outcomes and promoting sustained recovery from rehabilitation to real-world settings.

Dissociative and prioritized modeling of behaviorally relevant neural dynamics using recurrent neural networks

Understanding the dynamical transformation of neural activity to behavior requires new capabilities to nonlinearly model, dissociate and prioritize behaviorally relevant neural dynamics and test hypotheses about the origin of nonlinearity. We present dissociative prioritized analysis of dynamics (DPAD), a nonlinear dynamical modeling approach that enables these capabilities with a multisection neural network architecture and training approach. Analyzing cortical spiking and local field potential activity across four movement tasks, we demonstrate five use-cases. DPAD enabled more accurate neural-behavioral prediction. It identified nonlinear dynamical transformations of local field potentials that were more behavior predictive than traditional power features. Further, DPAD achieved behavior-predictive nonlinear neural dimensionality reduction. It enabled hypothesis testing regarding nonlinearities in neural-behavioral transformation, revealing that, in our datasets, nonlinearities could largely be isolated to the mapping from latent cortical dynamics to behavior. Finally, DPAD extended across continuous, intermittently sampled and categorical behaviors. DPAD provides a powerful tool for nonlinear dynamical modeling and investigation of neural-behavioral data.

Subject-Agnostic Transformer-Based Neural Speech Decoding from Surface and Depth Electrode Signals

Objective

This study investigates speech decoding from neural signals captured by intracranial electrodes. Most prior works can only work with electrodes on a 2D grid (i.e., Electrocorticographic or ECoG array) and data from a single patient. We aim to design a deep-learning model architecture that can accommodate both surface (ECoG) and depth (stereotactic EEG or sEEG) electrodes. The architecture should allow training on data from multiple participants with large variability in electrode placements and the trained model should perform well on participants unseen during training.

Approach

We propose a novel transformer-based model architecture named SwinTW that can work with arbitrarily positioned electrodes by leveraging their 3D locations on the cortex rather than their positions on a 2D grid. We train subject-specific models using data from a single participant and multi-patient models exploiting data from multiple participants.

Main Results

The subject-specific models using only low-density 8x8 ECoG data achieved high decoding Pearson Correlation Coefficient with ground truth spectrogram (PCC=0.817), over N=43 participants, outperforming our prior convolutional ResNet model and the 3D Swin transformer model. Incorporating additional strip, depth, and grid electrodes available in each participant (N=39) led to further improvement (PCC=0.838). For participants with only sEEG electrodes (N=9), subject-specific models still enjoy comparable performance with an average PCC=0.798. The multi-subject models achieved high performance on unseen participants, with an average PCC=0.765 in leave-one-out cross-validation.

Significance

The proposed SwinTW decoder enables future speech neuroprostheses to utilize any electrode placement that is clinically optimal or feasible for a particular participant, including using only depth electrodes, which are more routinely implanted in chronic neurosurgical procedures. Importantly, the generalizability of the multi-patient models suggests that such a model can be applied to new patients that do not have paired acoustic and neural data, providing an advance in neuroprostheses for people with speech disability, where acoustic-neural training data is not feasible.

Real-time detection of spoken speech from unlabeled ECoG signals: A pilot study with an ALS participant

Objective

Brain-Computer Interfaces (BCIs) hold significant promise for restoring communication in individuals with partial or complete loss of the ability to speak due to paralysis from amyotrophic lateral sclerosis (ALS), brainstem stroke, and other neurological disorders. Many of the approaches to speech decoding reported in the BCI literature have required time-aligned target representations to allow successful training - a major challenge when translating such approaches to people who have already lost their voice.

Approach

In this pilot study, we made a first step toward scenarios in which no ground truth is available. We utilized a graph-based clustering approach to identify temporal segments of speech production from electrocorticographic (ECoG) signals alone. We then used the estimated speech segments to train a voice activity detection (VAD) model using only ECoG signals. We evaluated our approach using held-out open-loop recordings of a single dysarthric clinical trial participant living with ALS, and we compared the resulting performance to previous solutions trained with ground truth acoustic voice recordings.

Main results

Our approach achieves a median error rate of around 0.5 seconds with respect to the actual spoken speech. Embedded into a real-time BCI, our approach is capable of providing VAD results with a latency of only 10 ms.

Significance

To the best of our knowledge, our results show for the first time that speech activity can be predicted purely from unlabeled ECoG signals, a crucial step toward individuals who cannot provide this information anymore due to their neurological condition, such as patients with locked-in syndrome.

Clinical Trial Information

ClinicalTrials.gov, registration number NCT03567213.

An instantaneous voice synthesis neuroprosthesis

Brain computer interfaces (BCIs) have the potential to restore communication to people who have lost the ability to speak due to neurological disease or injury. BCIs have been used to translate the neural correlates of attempted speech into text1-3. However, text communication fails to capture the nuances of human speech such as prosody, intonation and immediately hearing one's own voice. Here, we demonstrate a "brain-to-voice" neuroprosthesis that instantaneously synthesizes voice with closed-loop audio feedback by decoding neural activity from 256 microelectrodes implanted into the ventral precentral gyrus of a man with amyotrophic lateral sclerosis and severe dysarthria. We overcame the challenge of lacking ground-truth speech for training the neural decoder and were able to accurately synthesize his voice. Along with phonemic content, we were also able to decode paralinguistic features from intracortical activity, enabling the participant to modulate his BCI-synthesized voice in real-time to change intonation, emphasize words, and sing short melodies. These results demonstrate the feasibility of enabling people with paralysis to speak intelligibly and expressively through a BCI.

Effective Phoneme Decoding With Hyperbolic Neural Networks for High-Performance Speech BCIs

OBJECTIVE

Speech brain-computer interfaces (speech BCIs), which convert brain signals into spoken words or sentences, have demonstrated great potential for high-performance BCI communication. Phonemes are the basic pronunciation units. For monosyllabic languages such as Chinese Mandarin, where a word usually contains less than three phonemes, accurate decoding of phonemes plays a vital role. We found that in the neural representation space, phonemes with similar pronunciations are often inseparable, leading to confusion in phoneme classification.

METHODS

We mapped the neural signals of phoneme pronunciation into a hyperbolic space for a more distinct phoneme representation. Critically, we proposed a hyperbolic hierarchical clustering approach to specifically learn a phoneme-level structure to guide the representation.

RESULTS

We found such representation facilitated greater distance between similar phonemes, effectively reducing confusion. In the phoneme decoding task, our approach demonstrated an average accuracy of 75.21% for 21 phonemes and outperformed existing methods across different experimental days.

CONCLUSION

Our approach showed high accuracy in phoneme classification. By learning the phoneme-level neural structure, the representations of neural signals were more discriminative and interpretable.

SIGNIFICANCE

Our approach can potentially facilitate high-performance speech BCIs for Chinese and other monosyllabic languages.

BELT: Bootstrapped EEG-to-Language Training by Natural Language Supervision

Decoding natural language from noninvasive brain signals has been an exciting topic with the potential to expand the applications of brain-computer interface (BCI) systems. However, current methods face limitations in decoding sentences from electroencephalography (EEG) signals. Improving decoding performance requires the development of a more effective encoder for the EEG modality. Nonetheless, learning generalizable EEG representations remains a challenge due to the relatively small scale of existing EEG datasets. In this paper, we propose enhancing the EEG encoder to improve subsequent decoding performance. Specifically, we introduce the discrete Conformer encoder (D-Conformer) to transform EEG signals into discrete representations and bootstrap the learning process by imposing EEG-language alignment from the early training stage. The D-Conformer captures both local and global patterns from EEG signals and discretizes the EEG representation, making the representation more resilient to variations, while early-stage EEG-language alignment mitigates the limitations of small EEG datasets and facilitates the learning of the semantic representations from EEG signals. These enhancements result in improved EEG representations and decoding performance. We conducted extensive experiments and ablation studies to thoroughly evaluate the proposed method. Utilizing the D-Conformer encoder and bootstrapping training strategy, our approach demonstrates superior decoding performance across various tasks, including word-level, sentence-level, and sentiment-level decoding from EEG signals. Specifically, in word-level classification, we show that our encoding method produces more distinctive representations and higher classification performance compared to the EEG encoders from existing methods. At the sentence level, our model outperformed the baseline by 5.45%, achieving a BLEU-1 score of 42.31%. Furthermore, in sentiment classification, our model exceeded the baseline by 14%, achieving a sentiment classification accuracy of 69.3%.

Diffusion model-based image generation from rat brain activity

Brain-computer interface (BCI) technology has gained recognition in various fields, including clinical applications, assistive technology, and human-computer interaction research. BCI enables communication, control, and monitoring of the affective/cognitive states of users. Recently, BCI has also found applications in the artistic field, enabling real-time art composition using brain activity signals, and engaging performers, spectators, or an entire audience with brain activity-based artistic environments. Existing techniques use specific features of brain activity, such as the P300 wave and SSVEPs, to control drawing tools, rather than directly reflecting brain activity in the output image. In this study, we present a novel approach that uses a latent diffusion model, a type of deep neural network, to generate images directly from continuous brain activity. We demonstrate this technology using local field potentials from the neocortex of freely moving rats. This system continuously converted the recorded brain activity into images. Our end-to-end method for generating images from brain activity opens new possibilities for creative expression and experimentation. Notably, our results show that the generated images successfully reflect the dynamic and stochastic nature of the underlying neural activity, providing a unique procedure for visualization of brain function.

Decoding Single and Paired Phonemes Using 7T Functional MRI

Several studies have shown that mouth movements related to the pronunciation of individual phonemes are represented in the sensorimotor cortex. This would theoretically allow for brain computer interfaces that are capable of decoding continuous speech by training classifiers based on the activity in the sensorimotor cortex related to the production of individual phonemes. To address this, we investigated the decodability of trials with individual and paired phonemes (pronounced consecutively with one second interval) using activity in the sensorimotor cortex. Fifteen participants pronounced 3 different phonemes and 3 combinations of two of the same phonemes in a 7T functional MRI experiment. We confirmed that support vector machine (SVM) classification of single and paired phonemes was possible. Importantly, by combining classifiers trained on single phonemes, we were able to classify paired phonemes with an accuracy of 53% (33% chance level), demonstrating that activity of isolated phonemes is present and distinguishable in combined phonemes. A SVM searchlight analysis showed that the phoneme representations are widely distributed in the ventral sensorimotor cortex. These findings provide insights about the neural representations of single and paired phonemes. Furthermore, it supports the notion that speech BCI may be feasible based on machine learning algorithms trained on individual phonemes using intracranial electrode grids.

Language follows a distinct mode of extra-genomic evolution

As one of the most specific, yet most diverse of human behaviors, language is shaped by both genomic and extra-genomic evolution. Sharing methods and models between these modes of evolution has significantly advanced our understanding of language and inspired generalized theories of its evolution. Progress is hampered, however, by the fact that the extra-genomic evolution of languages, i.e. linguistic evolution, maps only partially to other forms of evolution. Contrasting it with the biological evolution of eukaryotes and the cultural evolution of technology as the best understood models, we show that linguistic evolution is special by yielding a stationary dynamic rather than stable solutions, and that this dynamic allows the use of language change for social differentiation while maintaining its global adaptiveness. Linguistic evolution furthermore differs from technological evolution by requiring vertical transmission, allowing the reconstruction of phylogenies; and it differs from eukaryotic biological evolution by foregoing a genotype vs phenotype distinction, allowing deliberate and biased change. Recognising these differences will improve our empirical tools and open new avenues for analyzing how linguistic, cultural, and biological evolution interacted with each other when language emerged in the hominin lineage. Importantly, our framework will help to cope with unprecedented scientific and ethical challenges that presently arise from how rapid cultural evolution impacts language, most urgently from interventional clinical tools for language disorders, potential epigenetic effects of technology on language, artificial intelligence and linguistic communicators, and global losses of linguistic diversity and identity. Beyond language, the distinctions made here allow identifying variation in other forms of biological and cultural evolution, developing new perspectives for empirical research.

Iterative alignment discovery of speech-associated neural activity

Objective. Brain-computer interfaces (BCIs) have the potential to preserve or restore speech in patients with neurological disorders that weaken the muscles involved in speech production. However, successful training of low-latency speech synthesis and recognition models requires alignment of neural activity with intended phonetic or acoustic output with high temporal precision. This is particularly challenging in patients who cannot produce audible speech, as ground truth with which to pinpoint neural activity synchronized with speech is not available.Approach. In this study, we present a new iterative algorithm for neural voice activity detection (nVAD) called iterative alignment discovery dynamic time warping (IAD-DTW) that integrates DTW into the loss function of a deep neural network (DNN). The algorithm is designed to discover the alignment between a patient's electrocorticographic (ECoG) neural responses and their attempts to speak during collection of data for training BCI decoders for speech synthesis and recognition.Main results. To demonstrate the effectiveness of the algorithm, we tested its accuracy in predicting the onset and duration of acoustic signals produced by able-bodied patients with intact speech undergoing short-term diagnostic ECoG recordings for epilepsy surgery. We simulated a lack of ground truth by randomly perturbing the temporal correspondence between neural activity and an initial single estimate for all speech onsets and durations. We examined the model's ability to overcome these perturbations to estimate ground truth. IAD-DTW showed no notable degradation (<1% absolute decrease in accuracy) in performance in these simulations, even in the case of maximal misalignments between speech and silence.Significance. IAD-DTW is computationally inexpensive and can be easily integrated into existing DNN-based nVAD approaches, as it pertains only to the final loss computation. This approach makes it possible to train speech BCI algorithms using ECoG data from patients who are unable to produce audible speech, including those with Locked-In Syndrome.

Advanced rehabilitation in ischaemic stroke research

At present, due to the rapid progress of treatment technology in the acute phase of ischaemic stroke, the mortality of patients has been greatly reduced but the number of disabled survivors is increasing, and most of them are elderly patients. Physicians and rehabilitation therapists pay attention to develop all kinds of therapist techniques including physical therapy techniques, robot-assisted technology and artificial intelligence technology, and study the molecular, cellular or synergistic mechanisms of rehabilitation therapies to promote the effect of rehabilitation therapy. Here, we discussed different animal and in vitro models of ischaemic stroke for rehabilitation studies; the compound concept and technology of neurological rehabilitation; all kinds of biological mechanisms of physical therapy; the significance, assessment and efficacy of neurological rehabilitation; the application of brain-computer interface, rehabilitation robotic and non-invasive brain stimulation technology in stroke rehabilitation.

An Accurate and Rapidly Calibrating Speech Neuroprosthesis

BACKGROUND

Brain-computer interfaces can enable communication for people with paralysis by transforming cortical activity associated with attempted speech into text on a computer screen. Communication with brain-computer interfaces has been restricted by extensive training requirements and limited accuracy.

METHODS

A 45-year-old man with amyotrophic lateral sclerosis (ALS) with tetraparesis and severe dysarthria underwent surgical implantation of four microelectrode arrays into his left ventral precentral gyrus 5 years after the onset of the illness; these arrays recorded neural activity from 256 intracortical electrodes. We report the results of decoding his cortical neural activity as he attempted to speak in both prompted and unstructured conversational contexts. Decoded words were displayed on a screen and then vocalized with the use of text-to-speech software designed to sound like his pre-ALS voice.

RESULTS

On the first day of use (25 days after surgery), the neuroprosthesis achieved 99.6% accuracy with a 50-word vocabulary. Calibration of the neuroprosthesis required 30 minutes of cortical recordings while the participant attempted to speak, followed by subsequent processing. On the second day, after 1.4 additional hours of system training, the neuroprosthesis achieved 90.2% accuracy using a 125,000-word vocabulary. With further training data, the neuroprosthesis sustained 97.5% accuracy over a period of 8.4 months after surgical implantation, and the participant used it to communicate in self-paced conversations at a rate of approximately 32 words per minute for more than 248 cumulative hours.

CONCLUSIONS

In a person with ALS and severe dysarthria, an intracortical speech neuroprosthesis reached a level of performance suitable to restore conversational communication after brief training. (Funded by the Office of the Assistant Secretary of Defense for Health Affairs and others; BrainGate2 ClinicalTrials.gov number, NCT00912041.).

A bilingual speech neuroprosthesis driven by cortical articulatory representations shared between languages

Advancements in decoding speech from brain activity have focused on decoding a single language. Hence, the extent to which bilingual speech production relies on unique or shared cortical activity across languages has remained unclear. Here, we leveraged electrocorticography, along with deep-learning and statistical natural-language models of English and Spanish, to record and decode activity from speech-motor cortex of a Spanish-English bilingual with vocal-tract and limb paralysis into sentences in either language. This was achieved without requiring the participant to manually specify the target language. Decoding models relied on shared vocal-tract articulatory representations across languages, which allowed us to build a syllable classifier that generalized across a shared set of English and Spanish syllables. Transfer learning expedited training of the bilingual decoder by enabling neural data recorded in one language to improve decoding in the other language. Overall, our findings suggest shared cortical articulatory representations that persist after paralysis and enable the decoding of multiple languages without the need to train separate language-specific decoders.

Spatial transcriptomics at the brain-electrode interface in rat motor cortex and the relationship to recording quality

Study of the foreign body reaction to implanted electrodes in the brain is an important area of research for the future development of neuroprostheses and experimental electrophysiology. After electrode implantation in the brain, microglial activation, reactive astrogliosis, and neuronal cell death create an environment immediately surrounding the electrode that is significantly altered from its homeostatic state.Objective.To uncover physiological changes potentially affecting device function and longevity, spatial transcriptomics (ST) was implemented to identify changes in gene expression driven by electrode implantation and compare this differential gene expression to traditional metrics of glial reactivity, neuronal loss, and electrophysiological recording quality.Approach.For these experiments, rats were chronically implanted with functional Michigan-style microelectrode arrays, from which electrophysiological recordings (multi-unit activity, local field potential) were taken over a six-week time course. Brain tissue cryosections surrounding each electrode were then mounted for ST processing. The tissue was immunolabeled for neurons and astrocytes, which provided both a spatial reference for ST and a quantitative measure of glial fibrillary acidic protein and neuronal nuclei immunolabeling surrounding each implant.Main results. Results from rat motor cortex within 300µm of the implanted electrodes at 24 h, 1 week, and 6 weeks post-implantation showed up to 553 significantly differentially expressed (DE) genes between implanted and non-implanted tissue sections. Regression on the significant DE genes identified the 6-7 genes that had the strongest relationship to histological and electrophysiological metrics, revealing potential candidate biomarkers of recording quality and the tissue response to implanted electrodes.Significance. Our analysis has shed new light onto the potential mechanisms involved in the tissue response to implanted electrodes while generating hypotheses regarding potential biomarkers related to recorded signal quality. A new approach has been developed to understand the tissue response to electrodes implanted in the brain using genes identified through transcriptomics, and to screen those results for potential relationships with functional outcomes.

Approaches of wearable and implantable biosensor towards of developing in precision medicine

In the relentless pursuit of precision medicine, the intersection of cutting-edge technology and healthcare has given rise to a transformative era. At the forefront of this revolution stands the burgeoning field of wearable and implantable biosensors, promising a paradigm shift in how we monitor, analyze, and tailor medical interventions. As these miniature marvels seamlessly integrate with the human body, they weave a tapestry of real-time health data, offering unprecedented insights into individual physiological landscapes. This log embarks on a journey into the realm of wearable and implantable biosensors, where the convergence of biology and technology heralds a new dawn in personalized healthcare. Here, we explore the intricate web of innovations, challenges, and the immense potential these bioelectronics sentinels hold in sculpting the future of precision medicine.

Stability of ECoG high gamma signals during speech and implications for a speech BCI system in an individual with ALS: a year-long longitudinal study

Objective.Speech brain-computer interfaces (BCIs) have the potential to augment communication in individuals with impaired speech due to muscle weakness, for example in amyotrophic lateral sclerosis (ALS) and other neurological disorders. However, to achieve long-term, reliable use of a speech BCI, it is essential for speech-related neural signal changes to be stable over long periods of time. Here we study, for the first time, the stability of speech-related electrocorticographic (ECoG) signals recorded from a chronically implanted ECoG BCI over a 12 month period.Approach.ECoG signals were recorded by an ECoG array implanted over the ventral sensorimotor cortex in a clinical trial participant with ALS. Because ECoG-based speech decoding has most often relied on broadband high gamma (HG) signal changes relative to baseline (non-speech) conditions, we studied longitudinal changes of HG band power at baseline and during speech, and we compared these with residual high frequency noise levels at baseline. Stability was further assessed by longitudinal measurements of signal-to-noise ratio, activation ratio, and peak speech-related HG response magnitude (HG response peaks). Lastly, we analyzed the stability of the event-related HG power changes (HG responses) for individual syllables at each electrode.Main Results.We found that speech-related ECoG signal responses were stable over a range of syllables activating different articulators for the first year after implantation.Significance.Together, our results indicate that ECoG can be a stable recording modality for long-term speech BCI systems for those living with severe paralysis.Clinical Trial Information.ClinicalTrials.gov, registration number NCT03567213.

Imagined speech event detection from electrocorticography and its transfer between speech modes and subjects

Speech brain-computer interfaces aim to support communication-impaired patients by translating neural signals into speech. While impressive progress was achieved in decoding performed, perceived and attempted speech, imagined speech remains elusive, mainly due to the absence of behavioral output. Nevertheless, imagined speech is advantageous since it does not depend on any articulator movements that might become impaired or even lost throughout the stages of a neurodegenerative disease. In this study, we analyzed electrocortigraphy data recorded from 16 participants in response to 3 speech modes: performed, perceived (listening), and imagined speech. We used a linear model to detect speech events and examined the contributions of each frequency band, from delta to high gamma, given the speech mode and electrode location. For imagined speech detection, we observed a strong contribution of gamma bands in the motor cortex, whereas lower frequencies were more prominent in the temporal lobe, in particular of the left hemisphere. Based on the similarities in frequency patterns, we were able to transfer models between speech modes and participants with similar electrode locations.

The speech neuroprosthesis

Loss of speech after paralysis is devastating, but circumventing motor-pathway injury by directly decoding speech from intact cortical activity has the potential to restore natural communication and self-expression. Recent discoveries have defined how key features of speech production are facilitated by the coordinated activity of vocal-tract articulatory and motor-planning cortical representations. In this Review, we highlight such progress and how it has led to successful speech decoding, first in individuals implanted with intracranial electrodes for clinical epilepsy monitoring and subsequently in individuals with paralysis as part of early feasibility clinical trials to restore speech. We discuss high-spatiotemporal-resolution neural interfaces and the adaptation of state-of-the-art speech computational algorithms that have driven rapid and substantial progress in decoding neural activity into text, audible speech, and facial movements. Although restoring natural speech is a long-term goal, speech neuroprostheses already have performance levels that surpass communication rates offered by current assistive-communication technology. Given this accelerated rate of progress in the field, we propose key evaluation metrics for speed and accuracy, among others, to help standardize across studies. We finish by highlighting several directions to more fully explore the multidimensional feature space of speech and language, which will continue to accelerate progress towards a clinically viable speech neuroprosthesis.

The ethical significance of user-control in AI-driven speech-BCIs: a narrative review

AI-driven brain-computed interfaces aimed at restoring speech for individuals living with locked-in-syndrome are paired with ethical implications for user's autonomy, privacy and responsibility. Embedding options for sufficient levels of user-control in speech-BCI design has been proposed to mitigate these ethical challenges. However, how user-control in speech-BCIs is conceptualized and how it relates to these ethical challenges is underdetermined. In this narrative literature review, we aim to clarify and explicate the notion of user-control in speech-BCIs, to better understand in what way user-control could operationalize user's autonomy, privacy and responsibility and explore how such suggestions for increasing user-control can be translated to recommendations for the design or use of speech-BCIs. First, we identified types of user control, including executory control that can protect voluntariness of speech, and guidance control that can contribute to semantic accuracy. Second, we identified potential causes for a loss of user-control, including contributions of predictive language models, a lack of ability for neural control, or signal interference and external control. Such a loss of user control may have implications for semantic accuracy and mental privacy. Third we explored ways to design for user-control. While embedding initiation signals for users may increase executory control, they may conflict with other aims such as speed and continuity of speech. Design mechanisms for guidance control remain largely conceptual, similar trade-offs in design may be expected. We argue that preceding these trade-offs, the overarching aim of speech-BCIs needs to be defined, requiring input from current and potential users. Additionally, conceptual clarification of user-control and other (ethical) concepts in this debate has practical relevance for BCI researchers. For instance, different concepts of inner speech may have distinct ethical implications. Increased clarity of such concepts can improve anticipation of ethical implications of speech-BCIs and may help to steer design decisions.

A Speech Neuroprosthesis in the Frontal Lobe and Hippocampus: Decoding High-Frequency Activity into Phonemes

BACKGROUND AND OBJECTIVES

Loss of speech due to injury or disease is devastating. Here, we report a novel speech neuroprosthesis that artificially articulates building blocks of speech based on high-frequency activity in brain areas never harnessed for a neuroprosthesis before: anterior cingulate and orbitofrontal cortices, and hippocampus.

METHODS

A 37-year-old male neurosurgical epilepsy patient with intact speech, implanted with depth electrodes for clinical reasons only, silently controlled the neuroprosthesis almost immediately and in a natural way to voluntarily produce 2 vowel sounds.

RESULTS

During the first set of trials, the participant made the neuroprosthesis produce the different vowel sounds artificially with 85% accuracy. In the following trials, performance improved consistently, which may be attributed to neuroplasticity. We show that a neuroprosthesis trained on overt speech data may be controlled silently.

CONCLUSION

This may open the way for a novel strategy of neuroprosthesis implantation at earlier disease stages (eg, amyotrophic lateral sclerosis), while speech is intact, for improved training that still allows silent control at later stages. The results demonstrate clinical feasibility of direct decoding of high-frequency activity that includes spiking activity in the aforementioned areas for silent production of phonemes that may serve as a part of a neuroprosthesis for replacing lost speech control pathways.

Speech decoding from stereo-electroencephalography (sEEG) signals using advanced deep learning methods

Objective.Brain-computer interfaces (BCIs) are technologies that bypass damaged or disrupted neural pathways and directly decode brain signals to perform intended actions. BCIs for speech have the potential to restore communication by decoding the intended speech directly. Many studies have demonstrated promising results using invasive micro-electrode arrays and electrocorticography. However, the use of stereo-electroencephalography (sEEG) for speech decoding has not been fully recognized.Approach.In this research, recently released sEEG data were used to decode Dutch words spoken by epileptic participants. We decoded speech waveforms from sEEG data using advanced deep-learning methods. Three methods were implemented: a linear regression method, an recurrent neural network (RNN)-based sequence-to-sequence model (RNN), and a transformer model.Main results.Our RNN and transformer models outperformed the linear regression significantly, while no significant difference was found between the two deep-learning methods. Further investigation on individual electrodes showed that the same decoding result can be obtained using only a few of the electrodes.Significance.This study demonstrated that decoding speech from sEEG signals is possible, and the location of the electrodes is critical to the decoding performance.

Large-scale foundation models and generative AI for BigData neuroscience

Recent advances in machine learning have led to revolutionary breakthroughs in computer games, image and natural language understanding, and scientific discovery. Foundation models and large-scale language models (LLMs) have recently achieved human-like intelligence thanks to BigData. With the help of self-supervised learning (SSL) and transfer learning, these models may potentially reshape the landscapes of neuroscience research and make a significant impact on the future. Here we present a mini-review on recent advances in foundation models and generative AI models as well as their applications in neuroscience, including natural language and speech, semantic memory, brain-machine interfaces (BMIs), and data augmentation. We argue that this paradigm-shift framework will open new avenues for many neuroscience research directions and discuss the accompanying challenges and opportunities.

Transcranial focused ultrasound to V5 enhances human visual motion brain-computer interface by modulating feature-based attention

A brain-computer interface (BCI) enables users to control devices with their minds. Despite advancements, non-invasive BCIs still exhibit high error rates, prompting investigation into the potential reduction through concurrent targeted neuromodulation. Transcranial focused ultrasound (tFUS) is an emerging non-invasive neuromodulation technology with high spatiotemporal precision. This study examines whether tFUS neuromodulation can improve BCI outcomes, and explores the underlying mechanism of action using high-density electroencephalography (EEG) source imaging (ESI). As a result, V5-targeted tFUS significantly reduced the error in a BCI speller task. Source analyses revealed a significantly increase in theta and alpha activities in the tFUS condition at both V5 and downstream in the dorsal visual processing pathway. Correlation analysis indicated that the connection within the dorsal processing pathway was preserved during tFUS stimulation, while the ventral connection was weakened. These findings suggest that V5-targeted tFUS enhances feature-based attention to visual motion.

Representation of internal speech by single neurons in human supramarginal gyrus

Speech brain-machine interfaces (BMIs) translate brain signals into words or audio outputs, enabling communication for people having lost their speech abilities due to diseases or injury. While important advances in vocalized, attempted and mimed speech decoding have been achieved, results for internal speech decoding are sparse and have yet to achieve high functionality. Notably, it is still unclear from which brain areas internal speech can be decoded. Here two participants with tetraplegia with implanted microelectrode arrays located in the supramarginal gyrus (SMG) and primary somatosensory cortex (S1) performed internal and vocalized speech of six words and two pseudowords. In both participants, we found significant neural representation of internal and vocalized speech, at the single neuron and population level in the SMG. From recorded population activity in the SMG, the internally spoken and vocalized words were significantly decodable. In an offline analysis, we achieved average decoding accuracies of 55% and 24% for each participant, respectively (chance level 12.5%), and during an online internal speech BMI task, we averaged 79% and 23% accuracy, respectively. Evidence of shared neural representations between internal speech, word reading and vocalized speech processes was found in participant 1. SMG represented words as well as pseudowords, providing evidence for phonetic encoding. Furthermore, our decoder achieved high classification with multiple internal speech strategies (auditory imagination/visual imagination). Activity in S1 was modulated by vocalized but not internal speech in both participants, suggesting no articulator movements of the vocal tract occurred during internal speech production. This work represents a proof-of-concept for a high-performance internal speech BMI.

NeuroDots: From Single-Target to Brain-Network Modulation: Why and What Is Needed?

OBJECTIVES

Current techniques in brain stimulation are still largely based on a phrenologic approach that a single brain target can treat a brain disorder. Nevertheless, meta-analyses of brain implants indicate an overall success rate of 50% improvement in 50% of patients, irrespective of the brain-related disorder. Thus, there is still a large margin for improvement. The goal of this manuscript is to 1) develop a general theoretical framework of brain functioning that is amenable to surgical neuromodulation, and 2) describe the engineering requirements of the next generation of implantable brain stimulators that follow from this theoretic model.

MATERIALS AND METHODS

A neuroscience and engineering literature review was performed to develop a universal theoretical model of brain functioning and dysfunctioning amenable to surgical neuromodulation.

RESULTS

Even though a single target can modulate an entire network, research in network science reveals that many brain disorders are the consequence of maladaptive interactions among multiple networks rather than a single network. Consequently, targeting the main connector hubs of those multiple interacting networks involved in a brain disorder is theoretically more beneficial. We, thus, envision next-generation network implants that will rely on distributed, multisite neuromodulation targeting correlated and anticorrelated interacting brain networks, juxtaposing alternative implant configurations, and finally providing solid recommendations for the realization of such implants. In doing so, this study pinpoints the potential shortcomings of other similar efforts in the field, which somehow fall short of the requirements.

CONCLUSION

The concept of network stimulation holds great promise as a universal approach for treating neurologic and psychiatric disorders.

Consistent spectro-spatial features of human ECoG successfully decode naturalistic behavioral states

Objective

Understanding the neural correlates of naturalistic behavior is critical for extending and confirming the results obtained from trial-based experiments and designing generalizable brain-computer interfaces that can operate outside laboratory environments. In this study, we aimed to pinpoint consistent spectro-spatial features of neural activity in humans that can discriminate between naturalistic behavioral states.

Approach

We analyzed data from five participants using electrocorticography (ECoG) with broad spatial coverage. Spontaneous and naturalistic behaviors such as "Talking" and "Watching TV" were labeled from manually annotated videos. Linear discriminant analysis (LDA) was used to classify the two behavioral states. The parameters learned from the LDA were then used to determine whether the neural signatures driving classification performance are consistent across the participants.

Main results

Spectro-spatial feature values were consistently discriminative between the two labeled behavioral states across participants. Mainly, θ, α, and low and high γ in the postcentral gyrus, precentral gyrus, and temporal lobe showed significant classification performance and feature consistency across participants. Subject-specific performance exceeded 70%. Combining neural activity from multiple cortical regions generally does not improve decoding performance, suggesting that information regarding the behavioral state is non-additive as a function of the cortical region.

Significance

To the best of our knowledge, this is the first attempt to identify specific spectro-spatial neural correlates that consistently decode naturalistic and active behavioral states. The aim of this work is to serve as an initial starting point for developing brain-computer interfaces that can be generalized in a realistic setting and to further our understanding of the neural correlates of naturalistic behavior in humans.

Visual context affects the perceived timing of tactile sensations elicited through intra-cortical microstimulation

Intra-cortical microstimulation (ICMS) is a technique to provide tactile sensations for a somatosensory brain-machine interface (BMI). A viable BMI must function within the rich, multisensory environment of the real world, but how ICMS is integrated with other sensory modalities is poorly understood. To investigate how ICMS percepts are integrated with visual information, ICMS and visual stimuli were delivered at varying times relative to one another. Both visual context and ICMS current amplitude were found to bias the qualitative experience of ICMS. In two tetraplegic participants, ICMS and visual stimuli were more likely to be experienced as occurring simultaneously when visual stimuli were more realistic, demonstrating an effect of visual context on the temporal binding window. The peak of the temporal binding window varied but was consistently offset from zero, suggesting that multisensory integration with ICMS can suffer from temporal misalignment. Recordings from primary somatosensory cortex (S1) during catch trials where visual stimuli were delivered without ICMS demonstrated that S1 represents visual information related to ICMS across visual contexts.

Machine learning decoding of single neurons in the thalamus for speech brain-machine interfaces

Objective. Our goal is to decode firing patterns of single neurons in the left ventralis intermediate nucleus (Vim) of the thalamus, related to speech production, perception, and imagery. For realistic speech brain-machine interfaces (BMIs), we aim to characterize the amount of thalamic neurons necessary for high accuracy decoding.Approach. We intraoperatively recorded single neuron activity in the left Vim of eight neurosurgical patients undergoing implantation of deep brain stimulator or RF lesioning during production, perception and imagery of the five monophthongal vowel sounds. We utilized the Spade decoder, a machine learning algorithm that dynamically learns specific features of firing patterns and is based on sparse decomposition of the high dimensional feature space.Main results. Spade outperformed all algorithms compared with, for all three aspects of speech: production, perception and imagery, and obtained accuracies of 100%, 96%, and 92%, respectively (chance level: 20%) based on pooling together neurons across all patients. The accuracy was logarithmic in the amount of neurons for all three aspects of speech. Regardless of the amount of units employed, production gained highest accuracies, whereas perception and imagery equated with each other.Significance. Our research renders single neuron activity in the left Vim a promising source of inputs to BMIs for restoration of speech faculties for locked-in patients or patients with anarthria or dysarthria to allow them to communicate again. Our characterization of how many neurons are necessary to achieve a certain decoding accuracy is of utmost importance for planning BMI implantation.

Considerations for implanting speech brain computer interfaces based on functional magnetic resonance imaging

Objective.Brain-computer interfaces (BCIs) have the potential to reinstate lost communication faculties. Results from speech decoding studies indicate that a usable speech BCI based on activity in the sensorimotor cortex (SMC) can be achieved using subdurally implanted electrodes. However, the optimal characteristics for a successful speech implant are largely unknown. We address this topic in a high field blood oxygenation level dependent functional magnetic resonance imaging (fMRI) study, by assessing the decodability of spoken words as a function of hemisphere, gyrus, sulcal depth, and position along the ventral/dorsal-axis.Approach.Twelve subjects conducted a 7T fMRI experiment in which they pronounced 6 different pseudo-words over 6 runs. We divided the SMC by hemisphere, gyrus, sulcal depth, and position along the ventral/dorsal axis. Classification was performed on in these SMC areas using multiclass support vector machine (SVM).Main results.Significant classification was possible from the SMC, but no preference for the left or right hemisphere, nor for the precentral or postcentral gyrus for optimal word classification was detected. Classification while using information from the cortical surface was slightly better than when using information from deep in the central sulcus and was highest within the ventral 50% of SMC. Confusion matrices where highly similar across the entire SMC. An SVM-searchlight analysis revealed significant classification in the superior temporal gyrus and left planum temporale in addition to the SMC.Significance.The current results support a unilateral implant using surface electrodes, covering the ventral 50% of the SMC. The added value of depth electrodes is unclear. We did not observe evidence for variations in the qualitative nature of information across SMC. The current results need to be confirmed in paralyzed patients performing attempted speech.