BCIAUT-P300: A Multi-Session and Multi-Subject Benchmark Dataset on Autism for P300-Based Brain-Computer-Interfaces

The STOIC2021 COVID-19 AI challenge: Applying reusable training methodologies to private data

Challenges drive the state-of-the-art of automated medical image analysis. The quantity of public training data that they provide can limit the performance of their solutions. Public access to the training methodology for these solutions remains absent. This study implements the Type Three (T3) challenge format, which allows for training solutions on private data and guarantees reusable training methodologies. With T3, challenge organizers train a codebase provided by the participants on sequestered training data. T3 was implemented in the STOIC2021 challenge, with the goal of predicting from a computed tomography (CT) scan whether subjects had a severe COVID-19 infection, defined as intubation or death within one month. STOIC2021 consisted of a Qualification phase, where participants developed challenge solutions using 2000 publicly available CT scans, and a Final phase, where participants submitted their training methodologies with which solutions were trained on CT scans of 9724 subjects. The organizers successfully trained six of the eight Final phase submissions. The submitted codebases for training and running inference were released publicly. The winning solution obtained an area under the receiver operating characteristic curve for discerning between severe and non-severe COVID-19 of 0.815. The Final phase solutions of all finalists improved upon their Qualification phase solutions.

Convolutional neural networks reveal properties of reach-to-grasp encoding in posterior parietal cortex

Deep neural networks (DNNs) are widely adopted to decode motor states from both non-invasively and invasively recorded neural signals, e.g., for realizing brain-computer interfaces. However, the neurophysiological interpretation of how DNNs make the decision based on the input neural activity is limitedly addressed, especially when applied to invasively recorded data. This reduces decoder reliability and transparency, and prevents the exploitation of decoders to better comprehend motor neural encoding. Here, we adopted an explainable artificial intelligence approach - based on a convolutional neural network and an explanation technique - to reveal spatial and temporal neural properties of reach-to-grasping from single-neuron recordings of the posterior parietal area V6A. The network was able to accurately decode 5 different grip types, and the explanation technique automatically identified the cells and temporal samples that most influenced the network prediction. Grip encoding in V6A neurons already started at movement preparation, peaking during movement execution. A difference was found within V6A: dorsal V6A neurons progressively encoded more for increasingly advanced grips, while ventral V6A neurons for increasingly rudimentary grips, with both subareas following a linear trend between the amount of grip encoding and the level of grip skills. By revealing the elements of the neural activity most relevant for each grip with no a priori assumptions, our approach supports and advances current knowledge about reach-to-grasp encoding in V6A, and it may represent a general tool able to investigate neural correlates of motor or cognitive tasks (e.g., attention and memory tasks) from single-neuron recordings.

A brain functional network feature extraction method based on directed transfer function and graph theory for MI-BCI decoding tasks

Background

The development of Brain-Computer Interface (BCI) technology has brought tremendous potential to various fields. In recent years, prominent research has focused on enhancing the accuracy of BCI decoding algorithms by effectively utilizing meaningful features extracted from electroencephalographic (EEG) signals.

Objective

This paper proposes a method for extracting brain functional network features based on directed transfer function (DTF) and graph theory. The method incorporates the extracted brain network features with common spatial pattern (CSP) to enhance the performance of motor imagery (MI) classification task.

Methods

The signals from each electrode of the EEG, utilizing a total of 32 channels, are used as input signals for the network nodes. In this study, 26 healthy participants were recruited to provide EEG data. The brain functional network is constructed in Alpha and Beta bands using the DTF method. The node degree (ND), clustering coefficient (CC), and global efficiency (GE) of the brain functional network are obtained using graph theory. The DTF network features and graph theory are combined with the traditional signal processing method, the CSP algorithm. The redundant network features are filtered out using the Lasso method, and finally, the fused features are classified using a support vector machine (SVM), culminating in a novel approach we have termed CDGL.

Results

For Beta frequency band, with 8 electrodes, the proposed CDGL method achieved an accuracy of 89.13%, a sensitivity of 90.15%, and a specificity of 88.10%, which are 14.10, 16.69, and 11.50% percentage higher than the traditional CSP method (75.03, 73.46, and 76.60%), respectively. Furthermore, the results obtained with 8 channels were superior to those with 4 channels (82.31, 83.35, and 81.74%), and the result for the Beta frequency band were better than those for the Alpha frequency band (87.42, 87.48, and 87.36%). Similar results were also obtained on two public datasets, where the CDGL algorithm's performance was found to be optimal.

Conclusion

The feature fusion of DTF network and graph theory features enhanced CSP algorithm's performance in MI task classification. Increasing the number of channels allows for more EEG signal feature information, enhancing the model's sensitivity and discriminative ability toward specific activities in brain regions. It should be noted that the functional brain network features in the Beta band exhibit superior performance improvement for the algorithm compared to those in the Alpha band.

A toolbox for decoding BCI commands based on event-related potentials

Commands in brain-computer interface (BCI) applications often rely on the decoding of event-related potentials (ERP). For instance, the P300 potential is frequently used as a marker of attention to an oddball event. Error-related potentials and the N2pc signal are further examples of ERPs used for BCI control. One challenge in decoding brain activity from the electroencephalogram (EEG) is the selection of the most suitable channels and appropriate features for a particular classification approach. Here we introduce a toolbox that enables ERP-based decoding using the full set of channels, while automatically extracting informative components from relevant channels. The strength of our approach is that it handles sequences of stimuli that encode multiple items using binary classification, such as target vs. nontarget events typically used in ERP-based spellers. We demonstrate examples of application scenarios and evaluate the performance of four openly available datasets: a P300-based matrix speller, a P300-based rapid serial visual presentation (RSVP) speller, a binary BCI based on the N2pc, and a dataset capturing error potentials. We show that our approach achieves performances comparable to those in the original papers, with the advantage that only conventional preprocessing is required by the user, while channel weighting and decoding algorithms are internally performed. Thus, we provide a tool to reliably decode ERPs for BCI use with minimal programming requirements.

Improving the performance of P300-based BCIs by mitigating the effects of stimuli-related evoked potentials through regularized spatial filtering

Objective.the P300-based brain-computer interface (BCI) establishes a communication channel between the mind and a computer by translating brain signals into commands. These systems typically employ a visual oddball paradigm, where different objects (linked to specific commands) are randomly and frequently intensified. Upon observing the target object, users experience an elicitation of a P300 event-related potential in their electroencephalography (EEG). However, detecting the P300 signal can be challenging due to its very low signal-to-noise ratio (SNR), often compromised by the sequence of visual evoked potentials (VEPs) generated in the occipital regions of the brain in response to periodic visual stimuli. While various approaches have been explored to enhance the SNR of P300 signals, the impact of VEPs has been largely overlooked. The main objective of this study is to investigate how VEPs impact P300-based BCIs. Subsequently, the study aims to propose a method for EEG spatial filtering to alleviate the effect of VEPs and enhance the overall performance of these BCIs.Approach.our approach entails analyzing recorded EEG signals from visual P300-based BCIs through temporal, spectral, and spatial analysis techniques to identify the impact of VEPs. Subsequently, we introduce a regularized version of the xDAWN algorithm, a well-established spatial filter known for enhancing single-trial P300s. This aims to simultaneously enhance P300 signals and suppress VEPs, contributing to an improved overall signal quality.Main results.analyzing EEG signals shows that VEPs can significantly contaminate P300 signals, resulting in a decrease in the overall performance of P300-based BCIs. However, our proposed method for simultaneous enhancement of P300 and suppression of VEPs demonstrates improved performance in P300-based BCIs. This improvement is verified through several experiments conducted with real P300 data.Significance.this study focuses on the effects of VEPs on the performance of P300-based BCIs, a problem that has not been adequately addressed in previous studies. It opens up a new path for investigating these BCIs. Moreover, the proposed spatial filtering technique has the potential to further enhance the performance of these systems.

Noise-Factorized Disentangled Representation Learning for Generalizable Motor Imagery EEG Classification

Motor Imagery (MI) Electroencephalography (EEG) is one of the most common Brain-Computer Interface (BCI) paradigms that has been widely used in neural rehabilitation and gaming. Although considerable research efforts have been dedicated to developing MI EEG classification algorithms, they are mostly limited in handling scenarios where the training and testing data are not from the same subject or session. Such poor generalization capability significantly limits the realization of BCI in real-world applications. In this paper, we proposed a novel framework to disentangle the representation of raw EEG data into three components, subject/session-specific, MI-task-specific, and random noises, so that the subject/session-specific feature extends the generalization capability of the system. This is realized by a joint discriminative and generative framework, supported by a series of fundamental training losses and training strategies. We evaluated our framework on three public MI EEG datasets, and detailed experimental results show that our method can achieve superior performance by a large margin compared to current state-of-the-art benchmark algorithms.

Deep learning on independent spatial EEG activity patterns delineates time windows relevant for response inhibition

Inhibitory control processes are an important aspect of executive functions and goal-directed behavior. However, the mostly correlative nature of neurophysiological studies was not able to provide insights which aspects of neural dynamics can best predict whether an individual is confronted with a situation requiring the inhibition of a response. This is particularly the case when considering the complex spatio-temporal nature of neural processes captured by EEG data. In the current study, we ask whether independent spatial activity profiles in the EEG data are useful to predict whether an individual is confronted with a situation requiring response inhibition. We combine independent component analysis (ICA) with explainable artificial intelligence approaches (EEG-based deep learning) using data from a Go/Nogo task (N = 255 participants). We show that there are four dissociable spatial activity profiles important to classify Go and Nogo trials as revealed by deep learning. Of note, for all of these four independent activity profiles, neural activity in the time period between 300 and 550 ms after stimulus presentation was most informative. Source localization analyses further revealed regions in the pre-central gyrus (BA6), the middle frontal gyrus (BA10), the inferior frontal gyrus (BA46), and the insular cortex (BA13) were associated with the isolated spatial activity profiles. The data suggest concomitant processes being reflected in the identified time window. This has implications for the ongoing debate on the functional significance of event-related potential correlates of inhibitory control.

Decoding movement kinematics from EEG using an interpretable convolutional neural network

Continuous decoding of hand kinematics has been recently explored for the intuitive control of electroencephalography (EEG)-based Brain-Computer Interfaces (BCIs). Deep neural networks (DNNs) are emerging as powerful decoders, for their ability to automatically learn features from lightly pre-processed signals. However, DNNs for kinematics decoding lack in the interpretability of the learned features and are only used to realize within-subject decoders without testing other training approaches potentially beneficial for reducing calibration time, such as transfer learning. Here, we aim to overcome these limitations by using an interpretable convolutional neural network (ICNN) to decode 2-D hand kinematics (position and velocity) from EEG in a pursuit tracking task performed by 13 participants. The ICNN is trained using both within-subject and cross-subject strategies, and also testing the feasibility of transferring the knowledge learned on other subjects on a new one. Moreover, the network eases the interpretation of learned spectral and spatial EEG features. Our ICNN outperformed most of the other state-of-the-art decoders, showing the best trade-off between performance, size, and training time. Furthermore, transfer learning improved kinematics prediction in the low data regime. The network attributed the highest relevance for decoding to the delta-band across all subjects, and to higher frequencies (alpha, beta, low-gamma) for a cluster of them; contralateral central and parieto-occipital sites were the most relevant, reflecting the involvement of sensorimotor, visual and visuo-motor processing. The approach improved the quality of kinematics prediction from the EEG, at the same time allowing interpretation of the most relevant spectral and spatial features.

Deep learning applied to EEG source-data reveals both ventral and dorsal visual stream involvement in holistic processing of social stimuli

Perception of social stimuli (faces and bodies) relies on "holistic" (i.e., global) mechanisms, as supported by picture-plane inversion: perceiving inverted faces/bodies is harder than perceiving their upright counterpart. Albeit neuroimaging evidence suggested involvement of face-specific brain areas in holistic processing, their spatiotemporal dynamics and selectivity for social stimuli is still debated. Here, we investigate the spatiotemporal dynamics of holistic processing for faces, bodies and houses (adopted as control non-social category), by applying deep learning to high-density electroencephalographic signals (EEG) at source-level. Convolutional neural networks were trained to classify cortical EEG responses to stimulus orientation (upright/inverted), separately for each stimulus type (faces, bodies, houses), resulting to perform well above chance for faces and bodies, and close to chance for houses. By explaining network decision, the 150-200 ms time interval and few visual ventral-stream regions were identified as mostly relevant for discriminating face and body orientation (lateral occipital cortex, and for face only, precuneus cortex, fusiform and lingual gyri), together with two additional dorsal-stream areas (superior and inferior parietal cortices). Overall, the proposed approach is sensitive in detecting cortical activity underlying perceptual phenomena, and by maximally exploiting discriminant information contained in data, may reveal spatiotemporal features previously undisclosed, stimulating novel investigations.

Machine Learning Enabled P300 Classifier for Autism Spectrum Disorder Using Adaptive Signal Decomposition

Joint attention skills deficiency in Autism spectrum disorder (ASD) hinders individuals from communicating effectively. The P300 Electroencephalogram (EEG) signal-based brain-computer interface (BCI) helps these individuals in neurorehabilitation training to overcome this deficiency. The detection of the P300 signal is more challenging in ASD as it is noisy, has less amplitude, and has a higher latency than in other individuals. This paper presents a novel application of the variational mode decomposition (VMD) technique in a BCI system involving ASD subjects for P300 signal identification. The EEG signal is decomposed into five modes using VMD. Thirty linear and non-linear time and frequency domain features are extracted for each mode. Synthetic minority oversampling technique data augmentation is performed to overcome the class imbalance problem in the chosen dataset. Then, a comparative analysis of three popular machine learning classifiers is performed for this application. VMD's fifth mode with a support vector machine (fine Gaussian kernel) classifier gave the best performance parameters, namely accuracy, F1-score, and the area under the curve, as 91.12%, 91.18%, and 96.6%, respectively. These results are better when compared to other state-of-the-art methods.

A new error-monitoring brain-computer interface based on reinforcement learning for people with autism spectrum disorders

Objective.Brain-computer interfaces (BCIs) are emerging as promising cognitive training tools in neurodevelopmental disorders, as they combine the advantages of traditional computerized interventions with real-time tailored feedback. We propose a gamified BCI based on non-volitional neurofeedback for cognitive training, aiming at reaching a neurorehabilitation tool for application in autism spectrum disorders (ASDs).Approach.The BCI consists of an emotional facial expression paradigm controlled by an intelligent agent that makes correct and wrong actions, while the user observes and judges the agent's actions. The agent learns through reinforcement learning (RL) an optimal strategy if the participant generates error-related potentials (ErrPs) upon incorrect agent actions. We hypothesize that this training approach will allow not only the agent to learn but also the BCI user, by participating through implicit error scrutiny in the process of learning through operant conditioning, making it of particular interest for disorders where error monitoring processes are altered/compromised such as in ASD. In this paper, the main goal is to validate the whole methodological BCI approach and assess whether it is feasible enough to move on to clinical experiments. A control group of ten neurotypical participants and one participant with ASD tested the proposed BCI approach.Main results.We achieved an online balanced-accuracy in ErrPs detection of 81.6% and 77.1%, respectively for two different game modes. Additionally, all participants achieved an optimal RL strategy for the agent at least in one of the test sessions.Significance.The ErrP classification results and the possibility of successfully achieving an optimal learning strategy, show the feasibility of the proposed methodology, which allows to move towards clinical experimentation with ASD participants to assess the effectiveness of the approach as hypothesized.

A Bayesian-optimized design for an interpretable convolutional neural network to decode and analyze the P300 response in autism

OBJECTIVE

P300 can be analyzed in autism spectrum disorder (ASD) to derive biomarkers and can be decoded in BCIs to reinforce ASD impaired skills. Convolutional neural networks (CNNs) have been proposed for P300 decoding, outperforming traditional algorithms but they i) do not investigate optimal designs in different training conditions; ii) lack in interpretability. To overcome these limitations, an interpretable CNN (ICNN), that we recently proposed for motor decoding, has been modified and adopted here, with its optimal design searched via Bayesian optimization.

APPROACH

The ICNN provides a straightforward interpretation of spectral and spatial features learned to decode P300. The Bayesian-optimized (BO) ICNN design was investigated separately for different training strategies (within-subject, within-session, and cross-subject) and BO models were used for the subsequent analyses. Specifically, transfer learning (TL) potentialities were investigated by assessing how pretrained cross-subject BO models performed on a new subject vs. random-initialized models. Furthermore, within-subject BO-derived models were combined with an Explanation Technique (ICNN+ET) to analyze P300 spectral and spatial features.

MAIN RESULTS

The ICNN resulted comparable or even outperformed existing CNNs, at the same time being lighter. Bayesian-optimized ICNN designs differed depending on the training strategy, needing more capacity as the training set variability increased. Furthermore, TL provided higher performance than networks trained from scratch. The ICNN+ET analysis suggested the frequency range [2, 5.8] Hz as the most relevant, and spatial features showed a right-hemispheric parietal asymmetry. The ICNN+ET-derived features, but not ERP-derived features, resulted significantly and highly correlated to ADOS clinical scores.

SIGNIFICANCE

This study substantiates the idea that a CNN can be designed both accurate and interpretable for P300 decoding, with an optimized design depending on the training condition. The novel ICNN-based analysis tool was able to better capture ASD neural signatures than traditional ERP analysis, possibly paving the way for identifying novel biomarkers.

Decoding sensorimotor information from superior parietal lobule of macaque via Convolutional Neural Networks

Despite the well-recognized role of the posterior parietal cortex (PPC) in processing sensory information to guide action, the differential encoding properties of this dynamic processing, as operated by different PPC brain areas, are scarcely known. Within the monkey's PPC, the superior parietal lobule hosts areas V6A, PEc, and PE included in the dorso-medial visual stream that is specialized in planning and guiding reaching movements. Here, a Convolutional Neural Network (CNN) approach is used to investigate how the information is processed in these areas. We trained two macaque monkeys to perform a delayed reaching task towards 9 positions (distributed on 3 different depth and direction levels) in the 3D peripersonal space. The activity of single cells was recorded from V6A, PEc, PE and fed to convolutional neural networks that were designed and trained to exploit the temporal structure of neuronal activation patterns, to decode the target positions reached by the monkey. Bayesian Optimization was used to define the main CNN hyper-parameters. In addition to discrete positions in space, we used the same network architecture to decode plausible reaching trajectories. We found that data from the most caudal V6A and PEc areas outperformed PE area in the spatial position decoding. In all areas, decoding accuracies started to increase at the time the target to reach was instructed to the monkey, and reached a plateau at movement onset. The results support a dynamic encoding of the different phases and properties of the reaching movement differentially distributed over a network of interconnected areas. This study highlights the usefulness of neurons' firing rate decoding via CNNs to improve our understanding of how sensorimotor information is encoded in PPC to perform reaching movements. The obtained results may have implications in the perspective of novel neuroprosthetic devices based on the decoding of these rich signals for faithfully carrying out patient's intentions.

EEG as a marker of brain plasticity in clinical applications

Neural networks are dynamic, and the brain has the capacity to reorganize itself. This capacity is named neuroplasticity and is fundamental for many processes ranging from learning and adaptation to new environments to the response to brain injuries. Measures of brain plasticity involve several techniques, including neuroimaging and neurophysiology. Electroencephalography, often used together with other techniques, is a common tool for prognostic and diagnostic purposes, and cortical reorganization is reflected by EEG measurements. Changes of power bands in different cortical areas occur with fatigue and in response to training stimuli leading to learning processes. Sleep has a fundamental role in brain plasticity, restoring EEG bands alterations and promoting consolidation of learning. Exercise and physical inactivity have been extensively studied as both strongly impact brain plasticity. Indeed, EEG studies showed the importance of the physical activity to promote learning and the effects of inactivity or microgravity on cortical reorganization to cope with absent or altered sensorimotor stimuli. Finally, this chapter will describe some of the EEG changes as markers of neural plasticity in neurologic conditions, focusing on cerebrovascular and neurodegenerative diseases. In conclusion, neuroplasticity is the fundamental mechanism necessary to ensure adaptation to new stimuli and situations, as part of the dynamicity of life.

A Lightweight Multi-Scale Convolutional Neural Network for P300 Decoding: Analysis of Training Strategies and Uncovering of Network Decision

Convolutional neural networks (CNNs), which automatically learn features from raw data to approximate functions, are being increasingly applied to the end-to-end analysis of electroencephalographic (EEG) signals, especially for decoding brain states in brain-computer interfaces (BCIs). Nevertheless, CNNs introduce a large number of trainable parameters, may require long training times, and lack in interpretability of learned features. The aim of this study is to propose a CNN design for P300 decoding with emphasis on its lightweight design while guaranteeing high performance, on the effects of different training strategies, and on the use of post-hoc techniques to explain network decisions. The proposed design, named MS-EEGNet, learned temporal features in two different timescales (i.e., multi-scale, MS) in an efficient and optimized (in terms of trainable parameters) way, and was validated on three P300 datasets. The CNN was trained using different strategies (within-participant and within-session, within-participant and cross-session, leave-one-subject-out, transfer learning) and was compared with several state-of-the-art (SOA) algorithms. Furthermore, variants of the baseline MS-EEGNet were analyzed to evaluate the impact of different hyper-parameters on performance. Lastly, saliency maps were used to derive representations of the relevant spatio-temporal features that drove CNN decisions. MS-EEGNet was the lightest CNN compared with the tested SOA CNNs, despite its multiple timescales, and significantly outperformed the SOA algorithms. Post-hoc hyper-parameter analysis confirmed the benefits of the innovative aspects of MS-EEGNet. Furthermore, MS-EEGNet did benefit from transfer learning, especially using a low number of training examples, suggesting that the proposed approach could be used in BCIs to accurately decode the P300 event while reducing calibration times. Representations derived from the saliency maps matched the P300 spatio-temporal distribution, further validating the proposed decoding approach. This study, by specifically addressing the aspects of lightweight design, transfer learning, and interpretability, can contribute to advance the development of deep learning algorithms for P300-based BCIs.

EEG-Inception: A Novel Deep Convolutional Neural Network for Assistive ERP-Based Brain-Computer Interfaces

In recent years, deep-learning models gained attention for electroencephalography (EEG) classification tasks due to their excellent performance and ability to extract complex features from raw data. In particular, convolutional neural networks (CNN) showed adequate results in brain-computer interfaces (BCI) based on different control signals, including event-related potentials (ERP). In this study, we propose a novel CNN, called EEG-Inception, that improves the accuracy and calibration time of assistive ERP-based BCIs. To the best of our knowledge, EEG-Inception is the first model to integrate Inception modules for ERP detection, which combined efficiently with other structures in a light architecture, improved the performance of our approach. The model was validated in a population of 73 subjects, of which 31 present motor disabilities. Results show that EEG-Inception outperforms 5 previous approaches, yielding significant improvements for command decoding accuracy up to 16.0%, 10.7%, 7.2%, 5.7% and 5.1% in comparison to rLDA, xDAWN + Riemannian geometry, CNN-BLSTM, DeepConvNet and EEGNet, respectively. Moreover, EEG-Inception requires very few calibration trials to achieve state-of-the-art performances taking advantage of a novel training strategy that combines cross-subject transfer learning and fine-tuning to increase the feasibility of this approach for practical use in assistive applications.