Development of a hybrid mental spelling system combining SSVEP-based brain–computer interface and webcam-based eye tracking

Improving the quality of Persian clinical text with a novel spelling correction system

BACKGROUND

The accuracy of spelling in Electronic Health Records (EHRs) is a critical factor for efficient clinical care, research, and ensuring patient safety. The Persian language, with its abundant vocabulary and complex characteristics, poses unique challenges for real-word error correction. This research aimed to develop an innovative approach for detecting and correcting spelling errors in Persian clinical text.

METHODS

Our strategy employs a state-of-the-art pre-trained model that has been meticulously fine-tuned specifically for the task of spelling correction in the Persian clinical domain. This model is complemented by an innovative orthographic similarity matching algorithm, PERTO, which uses visual similarity of characters for ranking correction candidates.

RESULTS

The evaluation of our approach demonstrated its robustness and precision in detecting and rectifying word errors in Persian clinical text. In terms of non-word error correction, our model achieved an F1-Score of 90.0% when the PERTO algorithm was employed. For real-word error detection, our model demonstrated its highest performance, achieving an F1-Score of 90.6%. Furthermore, the model reached its highest F1-Score of 91.5% for real-word error correction when the PERTO algorithm was employed.

CONCLUSIONS

Despite certain limitations, our method represents a substantial advancement in the field of spelling error detection and correction for Persian clinical text. By effectively addressing the unique challenges posed by the Persian language, our approach paves the way for more accurate and efficient clinical documentation, contributing to improved patient care and safety. Future research could explore its use in other areas of the Persian medical domain, enhancing its impact and utility.

A robotic arm control system with simultaneous and sequential modes combining eye-tracking with steady-state visual evoked potential in virtual reality environment

At present, single-modal brain-computer interface (BCI) still has limitations in practical application, such as low flexibility, poor autonomy, and easy fatigue for subjects. This study developed an asynchronous robotic arm control system based on steady-state visual evoked potentials (SSVEP) and eye-tracking in virtual reality (VR) environment, including simultaneous and sequential modes. For simultaneous mode, target classification was realized by decision-level fusion of electroencephalography (EEG) and eye-gaze. The stimulus duration for each subject was non-fixed, which was determined by an adjustable window method. Subjects could autonomously control the start and stop of the system using triple blink and eye closure, respectively. For sequential mode, no calibration was conducted before operation. First, subjects’ gaze area was obtained through eye-gaze, and then only few stimulus blocks began to flicker. Next, target classification was determined using EEG. Additionally, subjects could reject false triggering commands using eye closure. In this study, the system effectiveness was verified through offline experiment and online robotic-arm grasping experiment. Twenty subjects participated in offline experiment. For simultaneous mode, average ACC and ITR at the stimulus duration of 0.9 s were 90.50% and 60.02 bits/min, respectively. For sequential mode, average ACC and ITR at the stimulus duration of 1.4 s were 90.47% and 45.38 bits/min, respectively. Fifteen subjects successfully completed the online tasks of grabbing balls in both modes, and most subjects preferred the sequential mode. The proposed hybrid brain-computer interface (h-BCI) system could increase autonomy, reduce visual fatigue, meet individual needs, and improve the efficiency of the system.

Independent bilateral-eye stimulation for gaze pattern recognition based on steady-state pupil light reflex

Objective:recently, pupil oscillations synchronized with steady visual stimuli were used as input for an interface. The proposed system, inspired by a brain-computer interface based on steady-state visual evoked potentials, does not require contact with the participant. However, the pupil oscillation mechanism limits the stimulus frequency to 2.5 Hz or less, making it hard to enhance the information transfer rate (ITR).Approach:here, we compared multiple conditions for stimulation to increase the ITR of the pupil vibration-based interface, which were called monocular-single, monocular-superposed, and binocular-independent conditions. The binocular-independent condition stimulates each eye at different frequencies respectively and mixes them by using the visual stereoscopic perception of users. The monocular-superposed condition stimulates both eyes by a mixed signal of two different frequencies. We selected the shape of the stimulation signal, evaluated the amount of spectral leakage in the monocular-superposed and binocular-independent conditions, and compared the power spectrum density at the stimulation frequency. Moreover, 5, 10, and 15 patterns of stimuli were classified in each condition.Main results:a square wave, which causes an efficient pupil response, was used as the stimulus. Spectral leakage at the beat frequency was higher in the monocular-superposed condition than in the binocular-independent one. The power spectral density of stimulus frequencies was greatest in the monocular-single condition. Finally, we could classify the 15-stimulus pattern, with ITRs of 14.4 (binocular-independent, using five frequencies), 14.5 (monocular-superimposed, using five frequencies), and 23.7 bits min^-1(monocular-single, using 15 frequencies). There were no significant differences for the binocular-independent and monocular-superposed conditions.Significance:this paper shows a way to increase the number of stimuli that can be simultaneously displayed without decreasing ITR, even when only a small number of frequencies are available. This could lead to the provision of an interface based on pupil oscillation to a wider range of users.

Evaluation of Real-Time Endogenous Brain-Computer Interface Developed Using Ear-Electroencephalography

While previous studies have demonstrated the feasibility of using ear-electroencephalography (ear-EEG) for the development of brain-computer interfaces (BCIs), most of them have been performed using exogenous paradigms in offline environments. To verify the reliable feasibility of constructing ear-EEG-based BCIs, the feasibility of using ear-EEG should be further demonstrated using another BCI paradigm, namely the endogenous paradigm, in real-time online environments. Exogenous and endogenous BCIs are to use the EEG evoked by external stimuli and induced by self-modulation, respectively. In this study, we investigated whether an endogenous ear-EEG-based BCI with reasonable performance can be implemented in online environments that mimic real-world scenarios. To this end, we used three different mental tasks, i.e., mental arithmetic, word association, and mental singing, and performed BCI experiments with fourteen subjects on three different days to investigate not only the reliability of a real-time endogenous ear-EEG-based BCI, but also its test-retest reliability. The mean online classification accuracy was almost 70%, which was equivalent to a marginal accuracy for a practical two-class BCI (70%), demonstrating the feasibility of using ear-EEG for the development of real-time endogenous BCIs, but further studies should follow to improve its performance enough to be used for practical ear-EEG-based BCI applications.

An autonomous hybrid brain-computer interface system combined with eye-tracking in virtual environment

BACKGROUND

Brain-computer interface (BCI) has become an effective human-machine interactive way. However, the performance of the traditional BCI system needs to be further improved, such as flexibility, robustness, and accuracy. We aim to develop an autonomous hybrid BCI system combined with eye-tracking for the control tasks in the virtual environment.

NEW METHOD

This work developed an autonomous control strategy and proposed an effective fusion method for electroencephalogram (EEG) and eye tracking. For the autonomous control, the sliding window method was adopted to analyze the user's eye-gaze data. When the variance of eye-gaze data was less than the threshold, target recognition was triggered. EEG and eye-gaze data were synchronously collected and fused for classification. In addition, a fusion method based on particle swarm optimization (PSO) was proposed, which can find the best fusion weights to adapt to the differences of single modalities.

RESULTS

EEG data and eye-gaze data of 15 subjects in steady-state visual evoked potentials (SSVEP) tasks were collected to evaluate the effectiveness of the hybrid BCI system. The results showed that the PSO fusion method performed best in all fusion methods. And the proposed hybrid BCI system obtained higher accuracy and information transfer rate (ITR) than the single-modality.

COMPARISON WITH EXISTING METHODS

The PSO fusion method was compared with average weighting fusion, prior weighting fusion, support vector machine, decision tree, random forest, and extreme random tree.

CONCLUSION

The proposed methods of autonomous control and dual-modal fusion can improve the flexibility, robustness and classification performance of the hybrid BCI system.

Influence of the Number of Channels and Classification Algorithm on the Performance Robustness to Electrode Shift in Steady-State Visual Evoked Potential-Based Brain-Computer Interfaces

There remains an active investigation on elevating the classification accuracy and information transfer rate of brain-computer interfaces based on steady-state visual evoked potential. However, it has often been ignored that the performance of steady-state visual evoked potential (SSVEP)-based brain-computer interfaces (BCIs) can be affected through the minor displacement of the electrodes from their optimal locations in practical applications because of the mislocation of electrodes and/or concurrent use of electroencephalography (EEG) devices with external devices, such as virtual reality headsets. In this study, we evaluated the performance robustness of SSVEP-based BCIs with respect to the changes in electrode locations for various channel configurations and classification algorithms. Our experiments involved 21 participants, where EEG signals were recorded from the scalp electrodes densely attached to the occipital area of the participants. The classification accuracies for all the possible cases of electrode location shifts for various channel configurations (1–3 channels) were calculated using five training-free SSVEP classification algorithms, i.e., the canonical correlation analysis (CCA), extended CCA, filter bank CCA, multivariate synchronization index (MSI), and extended MSI (EMSI). Then, the performances of the BCIs were evaluated using two measures, i.e., the average classification accuracy (ACA) across the electrode shifts and robustness to the electrode shift (RES). Our results showed that the ACA increased with an increase in the number of channels regardless of the algorithm. However, the RES was enhanced with an increase in the number of channels only when MSI and EMSI were employed. While both ACA and RES values for the five algorithms were similar under the single-channel condition, both ACA and RES values for MSI and EMSI were higher than those of the other algorithms under the multichannel (i.e., two or three electrodes) conditions. In addition, EMSI outperformed MSI when comparing the ACA and RES values under the multichannel conditions. In conclusion, our results suggested that the use of multichannel configuration and employment of EMSI could make the performance of SSVEP-based BCIs more robust to the electrode shift from the optimal locations.

EEG-based hybrid QWERTY mental speller with high information transfer rate

BACKGROUND

Brain-computer interface (BCI) spellers detect variations in brain waves to help subjects communicate with the world. This study introduces a P300-SSVEP hybrid BCI-based QWERTY speller.

METHODS

The proposed hybrid speller, combines SSVEP and P300 features using a hybrid paradigm. P300 was used as time division multiplexing index which results in the use of lesser number of assumed frequencies for SSVEP elicitation. Each flickering frequency was also assigned a unique colour, to enhance system accuracy.

RESULTS

On the basis of 20 subjects, an average accuracy of classification of 96.42% and a mean information transfer rate (ITR) of 131.0 bits per min. (BPM) was achieved during the free spelling trial (trial-F).

COMPARISON

The t test results revealed that the hybrid QWERTY speller performed significantly better (on the basis of mean classification accuracy and ITR) as compared to the traditional P300 speller) and the QWERTY SSVEP speller. Also, the amount of time taken to spell a word was significantly lesser in the case of hybrid QWERTY speller in contrast to traditional P300 speller while it was almost the same as compared to QWERTY SSVEP speller.

CONCLUSION

QWERTY speller outperformed the stereotypical P300 speller as well as QWERTY SSVEP speller.

Non-invasive Neurophysiology in Learning and Training: Mechanisms and a SWOT Analysis

Although many scholars deem non-invasive measures of neurophysiology to have promise in assessing learning, these measures are currently not widely applied, neither in educational settings nor in training. How can non-invasive neurophysiology provide insight into learning and how should research on this topic move forward to ensure valid applications? The current article addresses these questions by discussing the mechanisms underlying neurophysiological changes during learning followed by a SWOT (strengths, weaknesses, opportunities, and threats) analysis of non-invasive neurophysiology in learning and training. This type of analysis can provide a structured examination of factors relevant to the current state and future of a field. The findings of the SWOT analysis indicate that the field of neurophysiology in learning and training is developing rapidly. By leveraging the opportunities of neurophysiology in learning and training (while bearing in mind weaknesses, threats, and strengths) the field can move forward in promising directions. Suggestions for opportunities for future work are provided to ensure valid and effective application of non-invasive neurophysiology in a wide range of learning and training settings.

Brain-Switches for Asynchronous Brain–Computer Interfaces: A Systematic Review

A brain–computer interface (BCI) has been extensively studied to develop a novel communication system for disabled people using their brain activities. An asynchronous BCI system is more realistic and practical than a synchronous BCI system, in that, BCI commands can be generated whenever the user wants. However, the relatively low performance of an asynchronous BCI system is problematic because redundant BCI commands are required to correct false-positive operations. To significantly reduce the number of false-positive operations of an asynchronous BCI system, a two-step approach has been proposed using a brain-switch that first determines whether the user wants to use an asynchronous BCI system before the operation of the asynchronous BCI system. This study presents a systematic review of the state-of-the-art brain-switch techniques and future research directions. To this end, we reviewed brain-switch research articles published from 2000 to 2019 in terms of their (a) neuroimaging modality, (b) paradigm, (c) operation algorithm, and (d) performance.

A Hybrid Asynchronous Brain-Computer Interface Combining SSVEP and EOG Signals

OBJECTIVE

A challenging task for an electroencephalography (EEG)-based asynchronous brain-computer interface (BCI) is to effectively distinguish between the idle state and the control state while maintaining a short response time and a high accuracy when commands are issued in the control state. This study proposes a novel hybrid asynchronous BCI system based on a combination of steady-state visual evoked potentials (SSVEPs) in the EEG signal and blink-related electrooculography (EOG) signals.

METHODS

Twelve buttons corresponding to 12 characters are included in the graphical user interface (GUI). These buttons flicker at different fixed frequencies and phases to evoke SSVEPs and are simultaneously highlighted by changing their sizes. The user can select a character by focusing on its frequency-phase stimulus and simultaneously blinking his/her eyes in accordance with its highlighting as his/her EEG and EOG signals are recorded. A multifrequency band-based canonical correlation analysis (CCA) method is applied to the EEG data to detect the evoked SSVEPs, whereas the EOG data are analyzed to identify the user's blinks. Finally, the target character is identified based on the SSVEP and blink detection results.

RESULTS

Ten healthy subjects participated in our experiments and achieved an average information transfer rate (ITR) of 105.52 bits/min, an average accuracy of 95.42%, an average response time of 1.34 s and an average false-positive rate (FPR) of 0.8%.

CONCLUSION

The proposed BCI generates multiple commands with a high ITR and low FPR.

SIGNIFICANCE

The hybrid asynchronous BCI has great potential for practical applications in communication and control.

A Hybrid Speller Design Using Eye Tracking and SSVEP Brain-Computer Interface

Steady-state visual evoked potentials (SSVEPs) have been extensively utilized to develop brain-computer interfaces (BCIs) due to the advantages of robustness, large number of commands, high classification accuracies, and information transfer rates (ITRs). However, the use of several simultaneous flickering stimuli often causes high levels of user discomfort, tiredness, annoyingness, and fatigue. Here we propose to design a stimuli-responsive hybrid speller by using electroencephalography (EEG) and video-based eye-tracking to increase user comfortability levels when presented with large numbers of simultaneously flickering stimuli. Interestingly, a canonical correlation analysis (CCA)-based framework was useful to identify target frequency with a 1 s duration of flickering signal. Our proposed BCI-speller uses only six frequencies to classify forty-eight targets, thus achieve greatly increased ITR, whereas basic SSVEP BCI-spellers use an equal number of frequencies to the number of targets. Using this speller, we obtained an average classification accuracy of 90.35 ± 3.597% with an average ITR of 184.06 ± 12.761 bits per minute in a cued-spelling task and an ITR of 190.73 ± 17.849 bits per minute in a free-spelling task. Consequently, our proposed speller is superior to the other spellers in terms of targets classified, classification accuracy, and ITR, while producing less fatigue, annoyingness, tiredness and discomfort. Together, our proposed hybrid eye tracking and SSVEP BCI-based system will ultimately enable a truly high-speed communication channel.

A multi-day and multi-band dataset for a steady-state visual-evoked potential-based brain-computer interface

BACKGROUND

A steady-state visual-evoked potential (SSVEP) is a brain response to visual stimuli modulated at certain frequencies; it has been widely used in electroencephalography (EEG)-based brain-computer interface research. However, there are few published SSVEP datasets for brain-computer interface. In this study, we obtained a new SSVEP dataset based on measurements from 30 participants, performed on 2 days; our dataset complements existing SSVEP datasets: (i) multi-band SSVEP datasets are provided, and all 3 possible frequency bands (low, middle, and high) were used for SSVEP stimulation; (ii) multi-day datasets are included; and (iii) the EEG datasets include simultaneously obtained physiological measurements, such as respiration, electrocardiography, electromyography, and head motion (accelerator).

FINDINGS

To validate our dataset, we estimated the spectral powers and classification performance for the EEG (SSVEP) datasets and created an example plot to visualize the physiological time-series data. Strong SSVEP responses were observed at stimulation frequencies, and the mean classification performance of the middle frequency band was significantly higher than the low- and high-frequency bands. Other physiological data also showed reasonable results.

CONCLUSIONS

Our multi-band, multi-day SSVEP datasets can be used to optimize stimulation frequencies because they enable simultaneous investigation of the characteristics of the SSVEPs evoked in each of the 3 frequency bands, and solve session-to-session (day-to-day) transfer problems by enabling investigation of the non-stationarity of SSVEPs measured on different days. Additionally, auxiliary physiological data can be used to explore the relationship between SSVEP characteristics and physiological conditions, providing useful information for optimizing experimental paradigms to achieve high performance.

Real-time decoding of question-and-answer speech dialogue using human cortical activity

Natural communication often occurs in dialogue, differentially engaging auditory and sensorimotor brain regions during listening and speaking. However, previous attempts to decode speech directly from the human brain typically consider listening or speaking tasks in isolation. Here, human participants listened to questions and responded aloud with answers while we used high-density electrocorticography (ECoG) recordings to detect when they heard or said an utterance and to then decode the utterance's identity. Because certain answers were only plausible responses to certain questions, we could dynamically update the prior probabilities of each answer using the decoded question likelihoods as context. We decode produced and perceived utterances with accuracy rates as high as 61% and 76%, respectively (chance is 7% and 20%). Contextual integration of decoded question likelihoods significantly improves answer decoding. These results demonstrate real-time decoding of speech in an interactive, conversational setting, which has important implications for patients who are unable to communicate.

On the Relative Contribution of Deep Convolutional Neural Networks for SSVEP-Based Bio-Signal Decoding in BCI Speller Applications

Brain-computer interfaces (BCI) harnessing steady state visual evoked potentials (SSVEPs) manipulate the frequency and phase of visual stimuli to generate predictable oscillations in neural activity. For BCI spellers, oscillations are matched with alphanumeric characters allowing users to select target numbers and letters. Advances in BCI spellers can, in part, be accredited to subject-specific optimization, including; 1) custom electrode arrangements; 2) filter sub-band assessments; and 3) stimulus parameter tuning. Here, we apply deep convolutional neural networks (DCNNs) demonstrating cross-subject functionality for the classification of frequency and phase encoded SSVEP. Electroencephalogram (EEG) data are collected and classified using the same parameters across subjects. Subjects fixate forty randomly cued flickering characters ( 5 ×8 keyboard array) during concurrent wet-EEG acquisition. These data are provided by an open source SSVEP dataset. Our proposed DCNN, PodNet, achieves 86% and 77% offline accuracy of classification across-subjects for two data capture periods, respectively, 6-seconds (information transfer rate = 40 bpm) and 2-seconds (information transfer rate = 101 bpm). Subjects demonstrating sub-optimal (<70%) performance are classified to similar levels after a short subject-specific training period. PodNet outperforms filter-bank canonical correlation analysis for a low volume (3-channel) clinically feasible occipital electrode configuration. The networks defined in this study achieve functional performance for the largest number of SSVEP classes decoded via DCNN to date. Our results demonstrate PodNet achieves cross-subject, calibrationless classification and adaptability to sub-optimal subject data, and low-volume EEG electrode arrangements.

Comparing eye trackers by correlating their eye-metric data

Up to now, the potential of eye tracking in science as well as in everyday life has not been fully realized because of the high acquisition cost of trackers. Recently, manufacturers have introduced low-cost devices, preparing the way for wider use of this underutilized technology. As soon as scientists show independently of the manufacturers that low-cost devices are accurate enough for application and research, the real advent of eye trackers will have arrived. To facilitate this development, we propose a simple approach for comparing two eye trackers by adopting a method that psychologists have been practicing in diagnostics for decades: correlating constructs to show reliability and validity. In a laboratory study, we ran the newer, low-cost EyeTribe eye tracker and an established SensoMotoric Instruments eye tracker at the same time, positioning one above the other. This design allowed us to directly correlate the eye-tracking metrics of the two devices over time. The experiment was embedded in a research project on memory where 26 participants viewed pictures or words and had to make cognitive judgments afterwards. The outputs of both trackers, that is, the pupil size and point of regard, were highly correlated, as estimated in a mixed effects model. Furthermore, calibration quality explained a substantial amount of individual differences for gaze, but not pupil size. Since data quality is not compromised, we conclude that low-cost eye trackers, in many cases, may be reliable alternatives to established devices.

Robust Fusion of c-VEP and Gaze

Brain computer interfaces (BCIs) are one of the developing technologies, serving as a communication interface for people with neuromuscular disorders. Electroencephalography (EEG) and gaze signals are among the commonly used inputs for the user intent classification problem arising in BCIs. Fusing different types of input modalities, i.e. EEG and gaze, is an obvious but effective solution for achieving high performance on this problem. Even though there are some simplistic approaches for fusing these two evidences, a more effective method is required for classification performances and speeds suitable for real-life scenarios. One of the main problems that is left unrecognized is highly noisy real-life data. In the context of the BCI framework utilized in this work, noisy data stem from user error in the form of tracking a nontarget stimuli, which in turn results in misleading EEG and gaze signals. We propose a method for fusing aforementioned evidences in a probabilistic manner that is highly robust against noisy data. We show the performance of the proposed method on real EEG and gaze data for different configurations of noise control variables. Compared to the regular fusion method, robust method achieves up to 15% higher classification accuracy.

Hybrid Brain-Computer Interface Techniques for Improved Classification Accuracy and Increased Number of Commands: A Review

In this article, non-invasive hybrid brain-computer interface (hBCI) technologies for improving classification accuracy and increasing the number of commands are reviewed. Hybridization combining more than two modalities is a new trend in brain imaging and prosthesis control. Electroencephalography (EEG), due to its easy use and fast temporal resolution, is most widely utilized in combination with other brain/non-brain signal acquisition modalities, for instance, functional near infrared spectroscopy (fNIRS), electromyography (EMG), electrooculography (EOG), and eye tracker. Three main purposes of hybridization are to increase the number of control commands, improve classification accuracy and reduce the signal detection time. Currently, such combinations of EEG + fNIRS and EEG + EOG are most commonly employed. Four principal components (i.e., hardware, paradigm, classifiers, and features) relevant to accuracy improvement are discussed. In the case of brain signals, motor imagination/movement tasks are combined with cognitive tasks to increase active brain-computer interface (BCI) accuracy. Active and reactive tasks sometimes are combined: motor imagination with steady-state evoked visual potentials (SSVEP) and motor imagination with P300. In the case of reactive tasks, SSVEP is most widely combined with P300 to increase the number of commands. Passive BCIs, however, are rare. After discussing the hardware and strategies involved in the development of hBCI, the second part examines the approaches used to increase the number of control commands and to enhance classification accuracy. The future prospects and the extension of hBCI in real-time applications for daily life scenarios are provided.

Fusion of P300 and eye-tracker data for spelling using BCI2000

OBJECTIVE

Various augmentative and alternative communication (AAC) devices have been developed in order to aid communication for individuals with communication disorders. Recently, there has been interest in combining EEG data and eye-gaze data with the goal of developing a hybrid (or 'fused') BCI (hBCI) AAC system. This work explores the effectiveness of a speller that fuses data from an eye-tracker and the P300 speller in order to create a hybrid P300 speller.

APPROACH

This hybrid speller collects both eye-tracking and EEG data in parallel, and the user spells characters on the screen in the same way that they would if they were only using the P300 speller. Online and offline experiments were performed. The online experiments measured the performance of the speller for sixteen non-disabled participants, while the offline simulations were used to assess the robustness of the hybrid system.

MAIN RESULTS

Online results showed that for fifteen non-disabled participants, using eye-gaze in a Bayesian framework with EEG data from the P300 speller improved accuracy ([Formula: see text], [Formula: see text], [Formula: see text] for estimated, medium and high variance configurations) and reduced the average number of flashes required to spell a character compared to the standard P300 speller that relies solely on EEG data ([Formula: see text], [Formula: see text], [Formula: see text] for estimated, medium and high variance configurations). Offline simulations indicate that the system provides more robust performance than a standalone eye gaze system.

SIGNIFICANCE

The results of this work on non-disabled participants shows the potential efficacy of hybrid P300 and eye-tracker speller. Further validation on the amyotrophic lateral sceloris population is needed to assess the benefit of this hybrid system.

A systematic review of hybrid brain-computer interfaces: Taxonomy and usability perspectives

A new Brain-Computer Interface (BCI) technique, which is called a hybrid BCI, has recently been proposed to address the limitations of conventional single BCI system. Although some hybrid BCI studies have shown promising results, the field of hybrid BCI is still in its infancy and there is much to be done. Especially, since the hybrid BCI systems are so complicated and complex, it is difficult to understand the constituent and role of a hybrid BCI system at a glance. Also, the complicated and complex systems make it difficult to evaluate the usability of the systems. We systematically reviewed and analyzed the current state-of-the-art hybrid BCI studies, and proposed a systematic taxonomy for classifying the types of hybrid BCIs with multiple taxonomic criteria. After reviewing 74 journal articles, hybrid BCIs could be categorized with respect to 1) the source of brain signals, 2) the characteristics of the brain signal, and 3) the characteristics of operation in each system. In addition, we exhaustively reviewed recent literature on usability of BCIs. To identify the key evaluation dimensions of usability, we focused on task and measurement characteristics of BCI usability. We classified and summarized 31 BCI usability journal articles according to task characteristics (type and description of task) and measurement characteristics (subjective and objective measures). Afterwards, we proposed usability dimensions for BCI and hybrid BCI systems according to three core-constructs: Satisfaction, effectiveness, and efficiency with recommendations for further research. This paper can help BCI researchers, even those who are new to the field, can easily understand the complex structure of the hybrid systems at a glance. Recommendations for future research can also be helpful in establishing research directions and gaining insight in how to solve ergonomics and HCI design issues surrounding BCI and hybrid BCI systems by usability evaluation.

A Novel Hybrid Mental Spelling Application Based on Eye Tracking and SSVEP-Based BCI

Steady state visual evoked potentials (SSVEPs)-based Brain-Computer interfaces (BCIs), as well as eyetracking devices, provide a pathway for re-establishing communication for people with severe disabilities. We fused these control techniques into a novel eyetracking/SSVEP hybrid system, which utilizes eye tracking for initial rough selection and the SSVEP technology for fine target activation. Based on our previous studies, only four stimuli were used for the SSVEP aspect, granting sufficient control for most BCI users. As Eye tracking data is not used for activation of letters, false positives due to inappropriate dwell times are avoided. This novel approach combines the high speed of eye tracking systems and the high classification accuracies of low target SSVEP-based BCIs, leading to an optimal combination of both methods. We evaluated accuracy and speed of the proposed hybrid system with a 30-target spelling application implementing all three control approaches (pure eye tracking, SSVEP and the hybrid system) with 32 participants. Although the highest information transfer rates (ITRs) were achieved with pure eye tracking, a considerable amount of subjects was not able to gain sufficient control over the stand-alone eye-tracking device or the pure SSVEP system (78.13% and 75% of the participants reached reliable control, respectively). In this respect, the proposed hybrid was most universal (over 90% of users achieved reliable control), and outperformed the pure SSVEP system in terms of speed and user friendliness. The presented hybrid system might offer communication to a wider range of users in comparison to the standard techniques.

Clinical feasibility of brain-computer interface based on steady-state visual evoked potential in patients with locked-in syndrome: Case studies

Although the feasibility of brain-computer interface (BCI) systems based on steady-state visual evoked potential (SSVEP) has been extensively investigated, only a few studies have evaluated its clinical feasibility in patients with locked-in syndrome (LIS), who are the main targets of BCI technology. The main objective of this case report was to share our experiences of SSVEP-based BCI experiments involving five patients with LIS, thereby providing researchers with useful information that can potentially help them to design BCI experiments for patients with LIS. In our experiments, a four-class online SSVEP-based BCI system was implemented and applied to four of five patients repeatedly on multiple days to investigate its test-retest reliability. In the last experiments with two of the four patients, the practical usability of our BCI system was tested using a questionnaire survey. All five patients showed clear and distinct SSVEP responses at all four fundamental stimulation frequencies (6, 6.66, 7.5, 10 Hz), and responses at harmonic frequencies were also observed in three patients. Mean classification accuracy was 76.99% (chance level = 25%). The test-retest reliability experiments demonstrated stable performance of our BCI system over different days even when the initial experimental settings (e.g., electrode configuration, fixation time, visual angle) used in the first experiment were used without significant modifications. Our results suggest that SSVEP-based BCI paradigms might be successfully used to implement clinically feasible BCI systems for severely paralyzed patients.