Decoding of top-down cognitive processing for SSVEP-controlled BMI

Alpha and SSVEP power outperform gamma power in capturing attentional modulation in human EEG

Attention typically reduces power in the alpha (8-12 Hz) band and increases power in gamma (>30 Hz) band in brain signals, as reported in macaque local field potential (LFP) and human electro/magneto-encephalogram (EEG/MEG) studies. In addition, EEG studies often use flickering stimuli that produce a specific measure called steady-state-visually-evoked-potential (SSVEP), whose power also increases with attention. However, effectiveness of these neural measures in capturing attentional modulation is unknown since stimuli and task paradigms vary widely across studies. In a recent macaque study, attentional modulation was more salient in the gamma band of the LFP, compared to alpha or SSVEP. To compare this with human EEG, we designed an orientation change detection task where we presented both static and counterphasing stimuli of matched difficulty levels to 26 subjects and compared attentional modulation of various measures under similar conditions. We report two main results. First, attentional modulation was comparable for SSVEP and alpha. Second, non-foveal stimuli produced weak gamma despite various stimulus optimizations and showed negligible attentional modulation although full-screen gratings showed robust gamma activity. Our results are useful for brain-machine-interfacing studies where suitable features are used for decoding attention, and also provide clues about spatial scales of neural mechanisms underlying attention.

Decoding of the neural representation of the visual RGB color model

RGB color is a basic visual feature. Here we use machine learning and visual evoked potential (VEP) of electroencephalogram (EEG) data to investigate the decoding features of the time courses and space location that extract it, and whether they depend on a common brain cortex channel. We show that RGB color information can be decoded from EEG data and, with the task-irrelevant paradigm, features can be decoded across fast changes in VEP stimuli. These results are consistent with the theory of both event-related potential (ERP) and P300 mechanisms. The latency on time course is shorter and more temporally precise for RGB color stimuli than P300, a result that does not depend on a task-relevant paradigm, suggesting that RGB color is an updating signal that separates visual events. Meanwhile, distribution features are evident for the brain cortex of EEG signal, providing a space correlate of RGB color in classification accuracy and channel location. Finally, space decoding of RGB color depends on the channel classification accuracy and location obtained through training and testing EEG data. The result is consistent with channel power value distribution discharged by both VEP and electrophysiological stimuli mechanisms.

Evidence of Serial Dependence from Decoding of Visual Evoked Potentials

It is well known that recent sensory experience influences perception, recently demonstrated by a phenomenon termed "serial dependence." However, its underlying neural mechanisms are poorly understood. We measured ERP responses to pairs of stimuli presented randomly to the left or right hemifield. Seventeen male and female adults judged whether the upper or lower half of the grating had higher spatial frequency, independent of the horizontal position of the grating. This design allowed us to trace the memory signal modulating task performance and also the implicit memory signal associated with hemispheric position. Using classification techniques, we decoded the position of the current and previous stimuli and the response from voltage scalp distributions of the current trial. Classification of previous responses reached full significance only 700 ms after presentation of the current stimulus, consistent with retrieval of an activity-silent memory trace. Cross-condition classification accuracy of past responses (trained on current responses) correlated with the strength of serial dependence effects of individual participants. Overall, our data provide evidence for a silent memory signal that can be decoded from the EEG potential, which interacts with the neural processing of the current stimulus. This silent memory signal could be the physiological substrate subserving at least one type of serial dependence.SIGNIFICANCE STATEMENT The neurophysiological underpinnings of how past perceptual experience affects current perception are poorly understood. Here, we show that recent experience is reactivated when a new stimulus is presented and that the strength of this reactivation correlates with serial biases in individual participants, suggesting that serial dependence is established on the basis of a silent memory signal.

Optimal flickering light stimulation for entraining gamma rhythms in older adults

With aging, optimal parameters of flickering light stimulation (FLS) for gamma entrainment may change in the eyes and brain. We investigated the optimal FLS parameters for gamma entrainment in 35 cognitively normal old adults by comparing event-related synchronization (ERS) and spectral Granger causality (sGC) of entrained gamma rhythms between different luminance intensities, colors, and flickering frequencies of FLSs. ERS entrained by 700 cd/m² FLS and 32 Hz or 34 Hz FLSs was stronger than that entrained by 400 cd/m² at Pz (p < 0.01) and 38 Hz or 40 Hz FLSs, respectively, at both Pz (p < 0.05) and Fz (p < 0.01). Parieto-occipital-to-frontotemporal connectivities of gamma rhythm entrained by 700 cd/m² FLS and 32 Hz or 34 Hz FLSs were also stronger than those entrained by 400 cd/m² at Pz (p < 0.01) and 38 Hz or 40 Hz FLSs, respectively (p < 0.001). ERS and parieto-occipital-to-frontotemporal connectivities of entrained gamma rhythms did not show significant difference between white and red lights. Adverse effects were comparable between different parameters. In older adults, 700 cd/m² FLS at 32 Hz or 34 Hz can entrain a strong gamma rhythm in the whole brain with tolerable adverse effects.

Deep-Learning-Based Automatic Selection of Fewest Channels for Brain-Machine Interfaces

Due to the development of convenient brain-machine interfaces (BMIs), the automatic selection of a minimum channel (electrode) set has attracted increasing interest because the decrease in the number of channels increases the efficiency of BMIs. This study proposes a deep-learning-based technique to automatically search for the minimum number of channels applicable to general BMI paradigms using a compact convolutional neural network for electroencephalography (EEG)-based BMIs. For verification, three types of BMI paradigms are assessed: 1) the typical P300 auditory oddball; 2) the new top-down steady-state visually evoked potential; and 3) the endogenous motor imagery. We observe that the optimized minimal EEG-channel sets are automatically selected in all three cases. Their decoding accuracies using the minimal channels are statistically equivalent to (or even higher than) those based on all channels. The brain areas of the selected channel set are neurophysiologically interpretable for all of these cognitive task paradigms. This study shows that the minimal EEG channel set can be automatically selected, irrespective of the types of BMI paradigms or EEG input features using a deep-learning approach, which also contributes to their portability.

Automatic Speech Discrimination Assessment Methods Based on Event-Related Potentials (ERP)

Speech discrimination is used by audiologists in diagnosing and determining treatment for hearing loss patients. Usually, assessing speech discrimination requires subjective responses. Using electroencephalography (EEG), a method that is based on event-related potentials (ERPs), could provide objective speech discrimination. In this work we proposed a visual-ERP-based method to assess speech discrimination using pictures that represent word meaning. The proposed method was implemented with three strategies, each with different number of pictures and test sequences. Machine learning was adopted to classify between the task conditions based on features that were extracted from EEG signals. The results from the proposed method were compared to that of a similar visual-ERP-based method using letters and a method that is based on the auditory mismatch negativity (MMN) component. The P3 component and the late positive potential (LPP) component were observed in the two visual-ERP-based methods while MMN was observed during the MMN-based method. A total of two out of three strategies of the proposed method, along with the MMN-based method, achieved approximately 80% average classification accuracy by a combination of support vector machine (SVM) and common spatial pattern (CSP). Potentially, these methods could serve as a pre-screening tool to make speech discrimination assessment more accessible, particularly in areas with a shortage of audiologists.

Review of brain encoding and decoding mechanisms for EEG-based brain-computer interface

A brain-computer interface (BCI) can connect humans and machines directly and has achieved successful applications in the past few decades. Many new BCI paradigms and algorithms have been developed in recent years. Therefore, it is necessary to review new progress in BCIs. This paper summarizes progress for EEG-based BCIs from the perspective of encoding paradigms and decoding algorithms, which are two key elements of BCI systems. Encoding paradigms are grouped by their underlying neural meachanisms, namely sensory- and motor-related, vision-related, cognition-related and hybrid paradigms. Decoding algorithms are reviewed in four categories, namely decomposition algorithms, Riemannian geometry, deep learning and transfer learning. This review will provide a comprehensive overview of both modern primary paradigms and algorithms, making it helpful for those who are developing BCI systems.

Sensorimotor Functional Connectivity: A Neurophysiological Factor Related to BCI Performance

Brain-Computer Interfaces (BCIs) are systems that allow users to control devices using brain activity alone. However, the ability of participants to command BCIs varies from subject to subject. About 20% of potential users of sensorimotor BCIs do not gain reliable control of the system. The inefficiency to decode user's intentions requires the identification of neurophysiological factors determining “good” and “poor” BCI performers. One of the important neurophysiological aspects in BCI research is that the neuronal oscillations, used to control these systems, show a rich repertoire of spatial sensorimotor interactions. Considering this, we hypothesized that neuronal connectivity in sensorimotor areas would define BCI performance. Analyses for this study were performed on a large dataset of 80 inexperienced participants. They took part in a calibration and an online feedback session recorded on the same day. Undirected functional connectivity was computed over sensorimotor areas by means of the imaginary part of coherency. The results show that post- as well as pre-stimulus connectivity in the calibration recording is significantly correlated to online feedback performance in μ and feedback frequency bands. Importantly, the significance of the correlation between connectivity and BCI feedback accuracy was not due to the signal-to-noise ratio of the oscillations in the corresponding post and pre-stimulus intervals. Thus, this study demonstrates that BCI performance is not only dependent on the amplitude of sensorimotor oscillations as shown previously, but that it also relates to sensorimotor connectivity measured during the preceding training session. The presence of such connectivity between motor and somatosensory systems is likely to facilitate motor imagery, which in turn is associated with the generation of a more pronounced modulation of sensorimotor oscillations (manifested in ERD/ERS) required for the adequate BCI performance. We also discuss strategies for the up-regulation of such connectivity in order to enhance BCI performance.

Electro-Encephalography and Electro-Oculography in Aeronautics: A Review Over the Last Decade (2010-2020)

Electro-encephalography (EEG) and electro-oculography (EOG) are methods of electrophysiological monitoring that have potentially fruitful applications in neuroscience, clinical exploration, the aeronautical industry, and other sectors. These methods are often the most straightforward way of evaluating brain oscillations and eye movements, as they use standard laboratory or mobile techniques. This review describes the potential of EEG and EOG systems and the application of these methods in aeronautics. For example, EEG and EOG signals can be used to design brain-computer interfaces (BCI) and to interpret brain activity, such as monitoring the mental state of a pilot in determining their workload. The main objectives of this review are to, (i) offer an in-depth review of literature on the basics of EEG and EOG and their application in aeronautics; (ii) to explore the methodology and trends of research in combined EEG-EOG studies over the last decade; and (iii) to provide methodological guidelines for beginners and experts when applying these methods in environments outside the laboratory, with a particular focus on human factors and aeronautics. The study used databases from scientific, clinical, and neural engineering fields. The review first introduces the characteristics and the application of both EEG and EOG in aeronautics, undertaking a large review of relevant literature, from early to more recent studies. We then built a novel taxonomy model that includes 150 combined EEG-EOG papers published in peer-reviewed scientific journals and conferences from January 2010 to March 2020. Several data elements were reviewed for each study (e.g., pre-processing, extracted features and performance metrics), which were then examined to uncover trends in aeronautics and summarize interesting methods from this important body of literature. Finally, the review considers the advantages and limitations of these methods as well as future challenges.

A Hybrid Speller Design Using Eye Tracking and SSVEP Brain-Computer Interface

Steady-state visual evoked potentials (SSVEPs) have been extensively utilized to develop brain-computer interfaces (BCIs) due to the advantages of robustness, large number of commands, high classification accuracies, and information transfer rates (ITRs). However, the use of several simultaneous flickering stimuli often causes high levels of user discomfort, tiredness, annoyingness, and fatigue. Here we propose to design a stimuli-responsive hybrid speller by using electroencephalography (EEG) and video-based eye-tracking to increase user comfortability levels when presented with large numbers of simultaneously flickering stimuli. Interestingly, a canonical correlation analysis (CCA)-based framework was useful to identify target frequency with a 1 s duration of flickering signal. Our proposed BCI-speller uses only six frequencies to classify forty-eight targets, thus achieve greatly increased ITR, whereas basic SSVEP BCI-spellers use an equal number of frequencies to the number of targets. Using this speller, we obtained an average classification accuracy of 90.35 ± 3.597% with an average ITR of 184.06 ± 12.761 bits per minute in a cued-spelling task and an ITR of 190.73 ± 17.849 bits per minute in a free-spelling task. Consequently, our proposed speller is superior to the other spellers in terms of targets classified, classification accuracy, and ITR, while producing less fatigue, annoyingness, tiredness and discomfort. Together, our proposed hybrid eye tracking and SSVEP BCI-based system will ultimately enable a truly high-speed communication channel.

Development of an electrooculogram-based human-computer interface using involuntary eye movement by spatially rotating sound for communication of locked-in patients

Individuals who have lost normal pathways for communication need augmentative and alternative communication (AAC) devices. In this study, we propose a new electrooculogram (EOG)-based human-computer interface (HCI) paradigm for AAC that does not require a user’s voluntary eye movement for binary yes/no communication by patients in locked-in state (LIS). The proposed HCI uses a horizontal EOG elicited by involuntary auditory oculogyric reflex, in response to a rotating sound source. In the proposed HCI paradigm, a user was asked to selectively attend to one of two sound sources rotating in directions opposite to each other, based on the user’s intention. The user’s intentions could then be recognised by quantifying EOGs. To validate its performance, a series of experiments was conducted with ten healthy subjects, and two patients with amyotrophic lateral sclerosis (ALS). The online experimental results exhibited high-classification accuracies of 94% in both healthy subjects and ALS patients in cases where decisions were made every six seconds. The ALS patients also participated in a practical yes/no communication experiment with 26 or 30 questions with known answers. The accuracy of the experiments with questionnaires was 94%, demonstrating that our paradigm could constitute an auxiliary AAC system for some LIS patients.

Simultaneous acquisition of EEG and NIRS during cognitive tasks for an open access dataset

We provide an open access multimodal brain-imaging dataset of simultaneous electroencephalography (EEG) and near-infrared spectroscopy (NIRS) recordings. Twenty-six healthy participants performed three cognitive tasks: 1) n-back (0-, 2- and 3-back), 2) discrimination/selection response task (DSR) and 3) word generation (WG) tasks. The data provided includes: 1) measured data, 2) demographic data, and 3) basic analysis results. For n-back (dataset A) and DSR tasks (dataset B), event-related potential (ERP) analysis was performed, and spatiotemporal characteristics and classification results for 'target' versus 'non-target' (dataset A) and symbol 'O' versus symbol 'X' (dataset B) are provided. Time-frequency analysis was performed to show the EEG spectral power to differentiate the task-relevant activations. Spatiotemporal characteristics of hemodynamic responses are also shown. For the WG task (dataset C), the EEG spectral power and spatiotemporal characteristics of hemodynamic responses are analyzed, and the potential merit of hybrid EEG-NIRS BCIs was validated with respect to classification accuracy. We expect that the dataset provided will facilitate performance evaluation and comparison of many neuroimaging analysis techniques.

Motion-Based Rapid Serial Visual Presentation for Gaze-Independent Brain-Computer Interfaces

Most event-related potential (ERP)-based brain-computer interface (BCI) spellers primarily use matrix layouts and generally require moderate eye movement for successful operation. The fundamental objective of this paper is to enhance the perceptibility of target characters by introducing motion stimuli to classical rapid serial visual presentation (RSVP) spellers that do not require any eye movement, thereby applying them to paralyzed patients with oculomotor dysfunctions. To test the feasibility of the proposed motion-based RSVP paradigm, we implemented three RSVP spellers: 1) fixed-direction motion (FM-RSVP); 2) random-direction motion (RM-RSVP); and 3) (the conventional) non-motion stimulation (NM-RSVP), and evaluated the effect of the three different stimulation methods on spelling performance. The two motion-based stimulation methods, FM- and RM-RSVP, showed shorter P300 latency and higher P300 amplitudes (i.e., 360.4-379.6 ms; 5.5867- ) than the NM-RSVP (i.e., 480.4 ms; ). This led to higher and more stable performances for FM- and RM-RSVP spellers than NM-RSVP speller (i.e., 79.06±6.45% for NM-RSVP, 90.60±2.98% for RM-RSVP, and 92.74±2.55% for FM-RSVP). In particular, the proposed motion-based RSVP paradigm was significantly beneficial for about half of the subjects who might not accurately perceive rapidly presented static stimuli. These results indicate that the use of proposed motion-based RSVP paradigm is more beneficial for target recognition when developing BCI applications for severely paralyzed patients with complex ocular dysfunctions.