Inter-individual single-trial classification of MEG data using M-CCA

Neuroscientific studies often involve some form of group analysis over multiple participants. This requires alignment of recordings across participants. A naive solution is to assume that participants' recordings can be aligned anatomically in sensor space. However, this assumption is likely violated due to anatomical and functional differences between individual brains. In magnetoencephalography (MEG) recordings the problem of inter-subject alignment is exacerbated by the susceptibility of MEG to individual cortical folding patterns as well as the inter-subject variability of sensor locations over the brain due to the use of a fixed helmet. Hence, an approach to combine MEG data over individual brains should relax the assumptions that a) brain anatomy and function are tightly linked and b) that the same sensors capture functionally comparable brain activation across individuals. Here we use multiset canonical correlation analysis (M-CCA) to find a common representation of MEG activations recorded from 15 participants performing a grasping task. The M-CCA algorithm was applied to transform the data of a set of multiple participants into a common space with maximum correlation between participants. Importantly, we derive a method to transform data from a new, previously unseen participant into this common representation. This makes it useful for applications that require transfer of models derived from a group of individuals to new individuals. We demonstrate the usefulness and superiority of the approach with respect to previously used approaches. Finally, we show that our approach requires only a small number of labeled data from the new participant. The proposed method demonstrates that functionally motivated common spaces have potential applications in reducing training time of online brain-computer interfaces, where models can be pre-trained on previous participants/sessions. Moreover, inter-subject alignment via M-CCA has the potential for combining data of different participants and could become helpful in future endeavors on large open datasets.

Classification of Electrophysiological Signatures With Explainable Artificial Intelligence: The Case of Alarm Detection in Flight Simulator

Relevant sounds such as alarms are sometimes involuntarily ignored, a phenomenon called inattentional deafness. This phenomenon occurs under specific conditions including high workload (i.e., multitasking) and/or cognitive fatigue. In the context of aviation, such an error can have drastic consequences on flight safety. This study uses an oddball paradigm in which participants had to detect rare sounds in an ecological context of simulated flight. Cognitive fatigue and cognitive load were manipulated to trigger inattentional deafness, and brain activity was recorded via electroencephalography (EEG). Our results showed that alarm omission and alarm detection can be classified based on time-frequency analysis of brain activity. We reached a maximum accuracy of 76.4% when the algorithm was trained on all participants and a maximum of 90.5%, on one participant, when the algorithm was trained individually. This method can benefit from explainable artificial intelligence to develop efficient and understandable passive brain-computer interfaces, improve flight safety by detecting such attentional failures in real time, and give appropriate feedback to pilots, according to our ambitious goal, providing them with reliable and rich human/machine interactions.

Tracking the Effect of Therapy With Single-Trial Based Classification After Stroke

Stroke is a debilitating disease that leads, in the 50% of cases, to permanent motor or cognitive impairments. The effectiveness of therapies that promote recovery after stroke depends on indicators of the disease state that can measure the degree of recovery or predict treatment response or both. Here, we propose to use single-trial classification of task dependent neural activity to assess the disease state and track recovery after stroke. We tested this idea on calcium imaging data of the dorsal cortex of healthy, spontaneously recovered and rehabilitated mice while performing a forelimb retraction task. Results show that, at a single-trial level for the three experimental groups, neural activation during the reward pull can be detected with high accuracy with respect to the background activity in all cortical areas of the field of view and this activation is quite stable across trials and subjects of the same group. Moreover, single-trial responses during the reward pull can be used to discriminate between healthy and stroke subjects with areas closer to the injury site displaying higher discrimination capability than areas closer to this site. Finally, a classifier built to discriminate between controls and stroke at the single-trial level can be used to generate an index of the disease state, the therapeutic score, which is validated on the group of rehabilitated mice. In conclusion, task-related neural activity can be used as an indicator of disease state and track recovery without selecting a peculiar feature of the neural responses. This novel method can be used in both the development and assessment of different therapeutic strategies.

Investigating the Single Trial Detectability of Cognitive Face Processing by a Passive Brain-Computer Interface

An automated recognition of faces enables machines to visually identify a person and to gain access to non-verbal communication, including mimicry. Different approaches in lab settings or controlled realistic environments provided evidence that automated face detection and recognition can work in principle, although applications in complex real-world scenarios pose a different kind of problem that could not be solved yet. Specifically, in autonomous driving-it would be beneficial if the car could identify non-verbal communication of pedestrians or other drivers, as it is a common way of communication in daily traffic. Automated identification from observation whether pedestrians or other drivers communicate through subtle cues in mimicry is an unsolved problem so far, as intent and other cognitive factors are hard to derive from observation. In contrast, communicating persons usually have clear understanding whether they communicate or not, and such information is represented in their mindsets. This work investigates whether the mental processing of faces can be identified through means of a Passive Brain-Computer Interface (pBCI). This then could be used to support the cars' autonomous interpretation of facial mimicry of pedestrians to identify non-verbal communication. Furthermore, the attentive driver can be utilized as a sensor to improve the context awareness of the car in partly automated driving. This work presents a laboratory study in which a pBCI is calibrated to detect responses of the fusiform gyrus in the electroencephalogram (EEG), reflecting face recognition. Participants were shown pictures from three different categories: faces, abstracts, and houses evoking different responses used to calibrate the pBCI. The resulting classifier could distinguish responses to faces from that evoked by other stimuli with accuracy above 70%, in a single trial. Further analysis of the classification approach and the underlying data identified activation patterns in the EEG that corresponds to face recognition in the fusiform gyrus. The resulting pBCI approach is promising as it shows better-than-random accuracy and is based on relevant and intended brain responses. Future research has to investigate whether it can be transferred from the laboratory to the real world and how it can be implemented into artificial intelligences, as used in autonomous driving.

Classification of error-related potentials evoked during stroke rehabilitation training

Objective.Error-related potentials (ErrPs) are elicited in the human brain following an error's perception. Recently, ErrPs have been observed in a novel task situation, i.e. when stroke patients perform upper-limb rehabilitation exercises. These ErrPs can be used to developassist-as-needed(AAN) robotic stroke rehabilitation systems. However, to date, there is no reported research on assessing the feasibility of using the ErrPs to implement the AAN approach. Hence, in this study, we evaluated and compared the single-trial classification of novel ErrPs using various classical machine learning and deep learning approaches.Approach.Electroencephalogram data of 13 stroke patients recorded while performing an upper-limb physical rehabilitation exercise were used. Two classification approaches, one combining the xDAWN spatial filtering and support vector machines, and the other using a convolutional neural network-based double transfer learning, were utilized.Main results.Results showed that the ErrPs could be detected with a mean area under the receiver operating characteristics curve of 0.838, and a mean accuracy of 0.842, 0.257 above the chance level (p< 0.05), for a within-subject classification. The results indicated the feasibility of using ErrP signals in real-time AAN robot therapy with evidence from the conducted latency analysis, cross-subject classification, and three-class asynchronous classification.Significance.The findings presented support our proposed approach of using ErrPs as a measure to trigger and/or modulate as required the robotic assistance in a real-timehuman-in-the-looprobotic stroke rehabilitation system.

Visual epidural field potentials possess high functional specificity in single trials

Recordings of epidural field potentials (EFPs) allow neuronal activity to be acquired over a large region of cortical tissue with minimal invasiveness. Because electrodes are placed on top of the dura and do not enter the neuronal tissue, EFPs offer intriguing options for both clinical and basic science research. On the other hand, EFPs represent the integrated activity of larger neuronal populations and possess a higher trial-by-trial variability and a reduced signal-to-noise ratio due the additional barrier of the dura. It is thus unclear whether and to what extent EFPs have sufficient spatial selectivity to allow for conclusions about the underlying functional cortical architecture, and whether single EFP trials provide enough information on the short timescales relevant for many clinical and basic neuroscience purposes. We used the high spatial resolution of primary visual cortex to address these issues and investigated the extent to which very short EFP traces allow reliable decoding of spatial information. We briefly presented different visual objects at one of nine closely adjacent locations and recorded neuronal activity with a high-density epidural multielectrode array in three macaque monkeys. With the use of receiver operating characteristics (ROC) to identify the most informative data, machine-learning algorithms provided close-to-perfect classification rates for all 27 stimulus conditions. A binary classifier applying a simple max function on ROC-selected data further showed that single trials might be classified with 100% performance even without advanced offline classifiers. Thus, although highly variable, EFPs constitute an extremely valuable source of information and offer new perspectives for minimally invasive recording of large-scale networks.NEW & NOTEWORTHY Epidural field potential (EFP) recordings provide a minimally invasive approach to investigate large-scale neural networks, but little is known about whether they possess the required specificity for basic and clinical neuroscience. By making use of the spatial selectivity of primary visual cortex, we show that single-trial information can be decoded with close-to-perfect performance, even without using advanced classifiers and based on very few data. This labels EFPs as a highly attractive and widely usable signal.

Multi-Tasking and Choice of Training Data Influencing Parietal ERP Expression and Single-Trial Detection-Relevance for Neuroscience and Clinical Applications

Event-related potentials (ERPs) are often used in brain-computer interfaces (BCIs) for communication or system control for enhancing or regaining control for motor-disabled persons. Especially results from single-trial EEG classification approaches for BCIs support correlations between single-trial ERP detection performance and ERP expression. Hence, BCIs can be considered as a paradigm shift contributing to new methods with strong influence on both neuroscience and clinical applications. Here, we investigate the relevance of the choice of training data and classifier transfer for the interpretability of results from single-trial ERP detection. In our experiments, subjects performed a visual-motor oddball task with motor-task relevant infrequent (targets), motor-task irrelevant infrequent (deviants), and motor-task irrelevant frequent (standards) stimuli. Under dual-task condition, a secondary senso-motor task was performed, compared to the simple-task condition. For evaluation, average ERP analysis and single-trial detection analysis with different numbers of electrodes were performed. Further, classifier transfer was investigated between simple and dual task. Parietal positive ERPs evoked by target stimuli (but not by deviants) were expressed stronger under dual-task condition, which is discussed as an increase of task emphasis and brain processes involved in task coordination and change of task set. Highest classification performance was found for targets irrespective whether all 62, 6 or 2 parietal electrodes were used. Further, higher detection performance of targets compared to standards was achieved under dual-task compared to simple-task condition in case of training on data from 2 parietal electrodes corresponding to results of ERP average analysis. Classifier transfer between tasks improves classification performance in case that training took place on more varying examples (from dual task). In summary, we showed that P300 and overlaying parietal positive ERPs can successfully be detected while subjects are performing additional ongoing motor activity. This supports single-trial detection of ERPs evoked by target events to, e.g., infer a patient's attentional state during therapeutic intervention.

Assessing the Depth of Cognitive Processing as the Basis for Potential User-State Adaptation

Objective: Decoding neurocognitive processes on a single-trial basis with Brain-Computer Interface (BCI) techniques can reveal the user's internal interpretation of the current situation. Such information can potentially be exploited to make devices and interfaces more user aware. In this line of research, we took a further step by studying neural correlates of different levels of cognitive processes and developing a method that allows to quantify how deeply presented information is processed in the brain.

Methods/Approach: Seventeen participants took part in an EEG study in which we evaluated different levels of cognitive processing (no processing, shallow, and deep processing) within three distinct domains (memory, language, and visual imagination). Our investigations showed gradual differences in the amplitudes of event-related potentials (ERPs) and in the extend and duration of event-related desynchronization (ERD) which both correlate with task difficulty. We performed multi-modal classification to map the measured correlates of neurocognitive processing to the corresponding level of processing.

Results: Successful classification of the neural components was achieved, which reflects the level of cognitive processing performed by the participants. The results show performances above chance level for each participant and a mean performance of 70–90% for all conditions and classification pairs.

Significance: The successful estimation of the level of cognition on a single-trial basis supports the feasibility of user-state adaptation based on ongoing neural activity. There is a variety of potential use cases such as: a user-friendly adaptive design of an interface or the development of assistance systems in safety critical workplaces.

Classification of Eye Fixation Related Potentials for Variable Stimulus Saliency

OBJECTIVE

Electroencephalography (EEG) and eye tracking can possibly provide information about which items displayed on the screen are relevant for a person. Exploiting this implicit information promises to enhance various software applications. The specific problem addressed by the present study is that items shown in real applications are typically diverse. Accordingly, the saliency of information, which allows to discriminate between relevant and irrelevant items, varies. As a consequence, recognition can happen in foveal or in peripheral vision, i.e., either before or after the saccade to the item. Accordingly, neural processes related to recognition are expected to occur with a variable latency with respect to the eye movements. The aim was to investigate if relevance estimation based on EEG and eye tracking data is possible despite of the aforementioned variability.

APPROACH

Sixteen subjects performed a search task where the target saliency was varied while the EEG was recorded and the unrestrained eye movements were tracked. Based on the acquired data, it was estimated which of the items displayed were targets and which were distractors in the search task.

RESULTS

Target prediction was possible also when the stimulus saliencies were mixed. Information contained in EEG and eye tracking data was found to be complementary and neural signals were captured despite of the unrestricted eye movements. The classification algorithm was able to cope with the experimentally induced variable timing of neural activity related to target recognition.

SIGNIFICANCE

It was demonstrated how EEG and eye tracking data can provide implicit information about the relevance of items on the screen for potential use in online applications.

Quantifying attentional modulation of auditory-evoked cortical responses from single-trial electroencephalography

Selective auditory attention is essential for human listeners to be able to communicate in multi-source environments. Selective attention is known to modulate the neural representation of the auditory scene, boosting the representation of a target sound relative to the background, but the strength of this modulation, and the mechanisms contributing to it, are not well understood. Here, listeners performed a behavioral experiment demanding sustained, focused spatial auditory attention while we measured cortical responses using electroencephalography (EEG). We presented three concurrent melodic streams; listeners were asked to attend and analyze the melodic contour of one of the streams, randomly selected from trial to trial. In a control task, listeners heard the same sound mixtures, but performed the contour judgment task on a series of visual arrows, ignoring all auditory streams. We found that the cortical responses could be fit as weighted sum of event-related potentials evoked by the stimulus onsets in the competing streams. The weighting to a given stream was roughly 10 dB higher when it was attended compared to when another auditory stream was attended; during the visual task, the auditory gains were intermediate. We then used a template-matching classification scheme to classify single-trial EEG results. We found that in all subjects, we could determine which stream the subject was attending significantly better than by chance. By directly quantifying the effect of selective attention on auditory cortical responses, these results reveal that focused auditory attention both suppresses the response to an unattended stream and enhances the response to an attended stream. The single-trial classification results add to the growing body of literature suggesting that auditory attentional modulation is sufficiently robust that it could be used as a control mechanism in brain-computer interfaces (BCIs).

Single-trial classification of motor imagery differing in task complexity: a functional near-infrared spectroscopy study

BACKGROUND

For brain computer interfaces (BCIs), which may be valuable in neurorehabilitation, brain signals derived from mental activation can be monitored by non-invasive methods, such as functional near-infrared spectroscopy (fNIRS). Single-trial classification is important for this purpose and this was the aim of the presented study. In particular, we aimed to investigate a combined approach: 1) offline single-trial classification of brain signals derived from a novel wireless fNIRS instrument; 2) to use motor imagery (MI) as mental task thereby discriminating between MI signals in response to different tasks complexities, i.e. simple and complex MI tasks.

METHODS

12 subjects were asked to imagine either a simple finger-tapping task using their right thumb or a complex sequential finger-tapping task using all fingers of their right hand. fNIRS was recorded over secondary motor areas of the contralateral hemisphere. Using Fisher's linear discriminant analysis (FLDA) and cross validation, we selected for each subject a best-performing feature combination consisting of 1) one out of three channel, 2) an analysis time interval ranging from 5-15 s after stimulation onset and 3) up to four Δ[O2Hb] signal features (Δ[O2Hb] mean signal amplitudes, variance, skewness and kurtosis).

RESULTS

The results of our single-trial classification showed that using the simple combination set of channels, time intervals and up to four Δ[O2Hb] signal features comprising Δ[O2Hb] mean signal amplitudes, variance, skewness and kurtosis, it was possible to discriminate single-trials of MI tasks differing in complexity, i.e. simple versus complex tasks (inter-task paired t-test p ≤ 0.001), over secondary motor areas with an average classification accuracy of 81%.

CONCLUSIONS

Although the classification accuracies look promising they are nevertheless subject of considerable subject-to-subject variability. In the discussion we address each of these aspects, their limitations for future approaches in single-trial classification and their relevance for neurorehabilitation.