Neural decoding of electrocorticographic signals using dynamic mode decomposition

Epileptic network identification: insights from dynamic mode decomposition of sEEG data

Objective.For medically-refractory epilepsy patients, stereoelectroencephalography (sEEG) is a surgical method using intracranial electrode recordings to identify brain networks participating in early seizure organization and propagation (i.e. the epileptogenic zone, EZ). If identified, surgical EZ treatment via resection, ablation or neuromodulation can lead to seizure-freedom. To date, quantification of sEEG data, including its visualization and interpretation, remains a clinical and computational challenge. Given elusiveness of physical laws or governing equations modelling complex brain dynamics, data science offers unique insight into identifying unknown patterns within high-dimensional sEEG data. We apply here an unsupervised data-driven algorithm, dynamic mode decomposition (DMD), to sEEG recordings from five focal epilepsy patients (three with temporal lobe, and two with cingulate epilepsy), who underwent subsequent resective or ablative surgery and became seizure free.Approach.DMD obtains a linear approximation of nonlinear data dynamics, generating coherent structures ('modes') defining important signal features, used to extract frequencies, growth rates and spatial structures. DMD was adapted to produce dynamic modal maps (DMMs) across frequency sub-bands, capturing onset and evolution of epileptiform dynamics in sEEG data. Additionally, we developed a static estimate of EZ-localized electrode contacts, termed the higher-frequency mode-based norm index (MNI). DMM and MNI maps for representative patient seizures were validated against clinical sEEG results and seizure-free outcomes following surgery.Main results.DMD was most informative at higher frequencies, i.e. gamma (including high-gamma) and beta range, successfully identifying EZ contacts. Combined interpretation of DMM/MNI plots best identified spatiotemporal evolution of mode-specific network changes, with strong concordance to sEEG results and outcomes across all five patients. The method identified network attenuation in other contacts not implicated in the EZ.Significance.This is the first application of DMD to sEEG data analysis, supporting integration of neuroengineering, mathematical and machine learning methods into traditional workflows for sEEG review and epilepsy surgical decision-making.

Text and image generation from intracranial electroencephalography using an embedding space for text and images

Objective.Invasive brain-computer interfaces (BCIs) are promising communication devices for severely paralyzed patients. Recent advances in intracranial electroencephalography (iEEG) coupled with natural language processing have enhanced communication speed and accuracy. It should be noted that such a speech BCI uses signals from the motor cortex. However, BCIs based on motor cortical activities may experience signal deterioration in users with motor cortical degenerative diseases such as amyotrophic lateral sclerosis. An alternative approach to using iEEG of the motor cortex is necessary to support patients with such conditions.Approach. In this study, a multimodal embedding of text and images was used to decode visual semantic information from iEEG signals of the visual cortex to generate text and images. We used contrastive language-image pretraining (CLIP) embedding to represent images presented to 17 patients implanted with electrodes in the occipital and temporal cortices. A CLIP image vector was inferred from the high-γpower of the iEEG signals recorded while viewing the images.Main results.Text was generated by CLIPCAP from the inferred CLIP vector with better-than-chance accuracy. Then, an image was created from the generated text using StableDiffusion with significant accuracy.Significance.The text and images generated from iEEG through the CLIP embedding vector can be used for improved communication.

Fast, accurate, and interpretable decoding of electrocorticographic signals using dynamic mode decomposition

Dynamic mode (DM) decomposition decomposes spatiotemporal signals into basic oscillatory components (DMs). DMs can improve the accuracy of neural decoding when used with the nonlinear Grassmann kernel, compared to conventional power features. However, such kernel-based machine learning algorithms have three limitations: large computational time preventing real-time application, incompatibility with non-kernel algorithms, and low interpretability. Here, we propose a mapping function corresponding to the Grassmann kernel that explicitly transforms DMs into spatial DM (sDM) features, which can be used in any machine learning algorithm. Using electrocorticographic signals recorded during various movement and visual perception tasks, the sDM features were shown to improve the decoding accuracy and computational time compared to conventional methods. Furthermore, the components of the sDM features informative for decoding showed similar characteristics to the high-γ power of the signals, but with higher trial-to-trial reproducibility. The proposed sDM features enable fast, accurate, and interpretable neural decoding.

Manifold alteration between major depressive disorder and healthy control subjects using dynamic mode decomposition in resting-state fMRI data

Background

The World Health Organization has reported that approximately 300 million individuals suffer from the mood disorder known as MDD. Non-invasive measurement techniques have been utilized to reveal the mechanism of MDD, with rsfMRI being the predominant method. The previous functional connectivity and energy landscape studies have shown the difference in the coactivation patterns between MDD and HCs. However, these studies did not consider oscillatory temporal dynamics.

Methods

In this study, the dynamic mode decomposition, a method to compute a set of coherent spatial patterns associated with the oscillation frequency and temporal decay rate, was employed to investigate the alteration of the occurrence of dynamic modes between MDD and HCs. Specifically, The BOLD signals of each subject were transformed into dynamic modes representing coherent spatial patterns and discrete-time eigenvalues to capture temporal variations using dynamic mode decomposition. All the dynamic modes were disentangled into a two-dimensional manifold using t-SNE. Density estimation and density ratio estimation were applied to the two-dimensional manifolds after the two-dimensional manifold was split based on HCs and MDD.

Results

The dynamic modes that uniquely emerged in the MDD were not observed. Instead, we have found some dynamic modes that have shown increased or reduced occurrence in MDD compared with HCs. The reduced dynamic modes were associated with the visual and saliency networks while the increased dynamic modes were associated with the default mode and sensory-motor networks.

Conclusion

To the best of our knowledge, this study showed initial evidence of the alteration of occurrence of the dynamic modes between MDD and HCs. To deepen understanding of how the alteration of the dynamic modes emerges from the structure, it is vital to investigate the relationship between the dynamic modes, cortical thickness, and surface areas.

Implementation of artificial intelligence and machine learning-based methods in brain-computer interaction

Brain-computer interfaces are used for direct two-way communication between the human brain and the computer. Brain signals contain valuable information about the mental state and brain activity of the examined subject. However, due to their non-stationarity and susceptibility to various types of interference, their processing, analysis and interpretation are challenging. For these reasons, the research in the field of brain-computer interfaces is focused on the implementation of artificial intelligence, especially in five main areas: calibration, noise suppression, communication, mental condition estimation, and motor imagery. The use of algorithms based on artificial intelligence and machine learning has proven to be very promising in these application domains, especially due to their ability to predict and learn from previous experience. Therefore, their implementation within medical technologies can contribute to more accurate information about the mental state of subjects, alleviate the consequences of serious diseases or improve the quality of life of disabled patients.

Nine decades of electrocorticography: A comparison between epidural and subdural recordings

In recent years, electrocorticography (ECoG) has arisen as a neural signal recording tool in the development of clinically viable neural interfaces. ECoG electrodes are generally placed below the dura mater (subdural) but can also be placed on top of the dura (epidural). In deciding which of these modalities best suits long-term implants, complications and signal quality are important considerations. Conceptually, epidural placement may present a lower risk of complications as the dura is left intact but also a lower signal quality due to the dura acting as a signal attenuator. The extent to which complications and signal quality are affected by the dura, however, has been a matter of debate. To improve our understanding of the effects of the dura on complications and signal quality, we conducted a literature review. We inventorized the effect of the dura on signal quality, decodability and longevity of acute and chronic ECoG recordings in humans and non-human primates. Also, we compared the incidence and nature of serious complications in studies that employed epidural and subdural ECoG. Overall, we found that, even though epidural recordings exhibit attenuated signal amplitude over subdural recordings, particularly for high-density grids, the decodability of epidural recorded signals does not seem to be markedly affected. Additionally, we found that the nature of serious complications was comparable between epidural and subdural recordings. These results indicate that both epidural and subdural ECoG may be suited for long-term neural signal recordings, at least for current generations of clinical and high-density ECoG grids.

Decoding trajectories of imagined hand movement using electrocorticograms for brain-machine interface

OBJECTIVE

Reaching hand movement is an important motor skill actively examined in brain-computer interface (BCI). Among various components of movement analyzed is the hand's trajectory, which describes the hand's continuous positions in three-dimensional space. While a large body of studies have investigated the decoding of real movements and the reconstruction of real hand movement trajectories from neural signals, fewer studies have attempted to decode the trajectory of imagined hand movement. To develop BCI systems for patients with hand motor dysfunctions, the systems essentially require to achieve movement-free control of external devices, which is only possible through successful decoding of purely imagined hand movement.

APPROACH

To achieve this goal, this study used a machine learning technique (i.e., the variational Bayesian least square) to analyze the electrocorticogram (ECoG) of eighteen epilepsy patients obtained from when they performed movement execution (ME) and kinesthetic movement imagination (KMI) of the reach-and-grasp hand action.

MAIN RESULTS

The variational Bayesian decoding model was able to successfully predict the imagined trajectories of hand movement significantly above chance level. The Pearson's correlation coefficient between imagined and predicted trajectories was 0.3393 and 0.4936 for the KMI (KMI trials only) and MEKMI paradigm (alternating trials of ME and KMI) respectively.

SIGNIFICANCE

This study demonstrated a high accuracy of prediction for trajectories of imagined hand movement, and more importantly, higher decoding accuracy of imagined trajectories in the MEKMI paradigm than in the KMI paradigm solely.

A fast and accurate learning-based decoding algorithm for the classification of cardiovascular and respiratory challenges using intraneural electrodes in the pig vagus nerve

Bioelectronic medicine is a new approach for developing closed-loop neuromodulation protocols on the peripheral nervous system (PNS) to treat a wide range of disorders currently treated with pharmacological approaches. Algorithms need to have low computational cost in order to acquire, process and model data for the modulation of the PNS in real time. Here, we present a fast learning-based decoding algorithm for the classification of cardiovascular and respiratory functional alterations (i.e., challenges) by using neural signals recorded from intraneural electrodes implanted in the vagus nerve of 5 pigs. Our algorithm relies on 9 handcrafted features, extracted following signal temporal windowing, and a multi-layer perceptron (MLP) for feature classification. We achieved fast and accurate classification of the challenges, with a computational time for feature extraction and prediction lower than 1.5 ms. The MLP achieved a balanced accuracy higher than 80 % for all recordings. Our algorithm could represent a step towards the development of a closed-loop system based on a single intraneural interface with both the potential of real time classification and selective modulation of the PNS.

Predicting behavior through dynamic modes in resting-state fMRI data

Dynamic properties of resting-state functional connectivity (FC) provide rich information on brain-behavior relationships. Dynamic mode decomposition (DMD) has been used as a method to characterize FC dynamics. However, it remains unclear whether dynamic modes (DMs), spatial-temporal coherent patterns computed by DMD, provide information about individual behavioral differences. This study established a methodological approach to predict individual differences in behavior using DMs. Furthermore, we investigated the contribution of DMs within each of seven specific frequency bands (0-0.1,...,0.6-0.7 Hz) for prediction. To validate our approach, we tested whether each of 59 behavioral measures could be predicted by performing multivariate pattern analysis on a Gram matrix, which was created using subject-specific DMs computed from resting-state functional magnetic resonance imaging (rs-fMRI) data of individuals. DMD successfully predicted behavior and outperformed temporal and spatial independent component analysis, which is the conventional data decomposition method for extracting spatial activity patterns. Most of the behavioral measures that were predicted with significant accuracy in a permutation test were related to cognition. We found that DMs within frequency bands <0.2 Hz primarily contributed to prediction and had spatial structures similar to several common resting-state networks. Our results indicate that DMD is efficient in extracting spatiotemporal features from rs-fMRI data.

Delineating between-subject heterogeneity in alpha networks with Spatio-Spectral Eigenmodes

•

A data-driven modal decomposition describes oscillations by their resonant frequency, damping time and network structure.

•

We show that the full multivariate transfer function can be rewritten as a linear superposition of these modes.

•

These modal coordinates factorise oscillatory systems without pre-specification of frequency bands or regions of interest.

•

Using these modes, we find a spatial gradient in alpha peak frequency between Occipital and Parietal cortex .

•

This gradient is highly variable between participants, showing shifts in spatial structure and peak frequency.

Between subject variability in the spatial and spectral structure of oscillatory networks can be highly informative but poses a considerable analytic challenge. Here, we describe a data-driven modal decomposition of a multivariate autoregressive model that simultaneously identifies oscillations by their peak frequency, damping time and network structure. We use this decomposition to define a set of Spatio-Spectral Eigenmodes (SSEs) providing a parsimonious description of oscillatory networks. We show that the multivariate system transfer function can be rewritten in these modal coordinates, and that the full transfer function is a linear superposition of all modes in the decomposition. The modal transfer function is a linear summation and therefore allows for single oscillatory signals to be isolated and analysed in terms of their spectral content, spatial distribution and network structure. We validate the method on simulated data and explore the structure of whole brain oscillatory networks in eyes-open resting state MEG data from the Human Connectome Project. We are able to show a wide between participant variability in peak frequency and network structure of alpha oscillations and show a distinction between occipital ’high-frequency alpha’ and parietal ’low-frequency alpha’. The frequency difference between occipital and parietal alpha components is present within individual participants but is partially masked by larger between subject variability; a 10Hz oscillation may represent the high-frequency occipital component in one participant and the low-frequency parietal component in another. This rich characterisation of individual neural phenotypes has the potential to enhance analyses into the relationship between neural dynamics and a person’s behavioural, cognitive or clinical state.

Classification of visuomotor tasks based on electroencephalographic data depends on age-related differences in brain activity patterns

Classification of physiological data provides a data driven approach to study central aspects of motor control, which changes with age. To implement such results in real-life applications for elderly it is important to identify age-specific characteristics of movement classification. We compared task-classification based on EEG derived activity patterns related to brain network characteristics between older and younger adults performing force tracking with two task characteristics (sinusoidal; constant) with the right or left hand. We extracted brain network patterns with dynamic mode decomposition (DMD) and classified the tasks on an individual level using linear discriminant analysis (LDA). Next, we compared the models' performance between the groups. Studying brain activity patterns, we identified signatures of altered motor network function reflecting dedifferentiated and compensational brain activation in older adults. We found that the classification performance of the body side was lower in older adults. However, classification performance with respect to task characteristics was better in older adults. This may indicate a higher susceptibility of brain network mechanisms to task difficulty in elderly. Signatures of dedifferentiation and compensation refer to an age-related reorganization of functional brain networks, which suggests that classification of visuomotor tracking tasks is influenced by age-specific characteristics of brain activity patterns. In addition to insights into central aspects of fine motor control, the results presented here are relevant in application-oriented areas such as brain computer interfaces.

Mining naturalistic human behaviors in long-term video and neural recordings

BACKGROUND

Recent technological advances in brain recording and machine learning algorithms are enabling the study of neural activity underlying spontaneous human behaviors, beyond the confines of cued, repeated trials. However, analyzing such unstructured data lacking a priori experimental design remains a significant challenge, especially when the data is multi-modal and long-term.

NEW METHOD

Here we describe an automated, behavior-first approach for analyzing simultaneously recorded long-term, naturalistic electrocorticography (ECoG) and behavior video data. We identify and characterize spontaneous human upper-limb movements by combining computer vision, discrete latent-variable modeling, and string pattern-matching on the video.

RESULTS

Our pipeline discovers and annotates over 40,000 instances of naturalistic arm movements in long term (7-9 day) behavioral videos, across 12 subjects. Analysis of the simultaneously recorded brain data reveals neural signatures of movement that corroborate previous findings. Our pipeline produces large training datasets for brain-computer interfacing applications, and we show decoding results from a movement initiation detection task.

COMPARISON WITH EXISTING METHODS

Spontaneous movements capture real-world neural and behavior variability that is missing from traditional cued tasks. Building beyond window-based movement detection metrics, our unsupervised discretization scheme produces a queryable pose representation, allowing localization of movements with finer temporal resolution.

CONCLUSIONS

Our work addresses the unique analytic challenges of studying naturalistic human behaviors and contributes methods that may generalize to other neural recording modalities beyond ECoG. We publish our curated dataset and believe that it will be a valuable resource for future studies of naturalistic movements.