Combining BCI with Virtual Reality: Towards New Applications and Improved BCI

Influence of Auditory Cues on the Neuronal Response to Naturalistic Visual Stimuli in a Virtual Reality Setting

Virtual reality environments offer great opportunities to study the performance of brain-computer interfaces (BCIs) in real-world contexts. As real-world stimuli are typically multimodal, their neuronal integration elicits complex response patterns. To investigate the effect of additional auditory cues on the processing of visual information, we used virtual reality to mimic safety-related events in an industrial environment while we concomitantly recorded electroencephalography (EEG) signals. We simulated a box traveling on a conveyor belt system where two types of stimuli – an exploding and a burning box – interrupt regular operation. The recordings from 16 subjects were divided into two subsets, a visual-only and an audio-visual experiment. In the visual-only experiment, the response patterns for both stimuli elicited a similar pattern – a visual evoked potential (VEP) followed by an event-related potential (ERP) over the occipital-parietal lobe. Moreover, we found the perceived severity of the event to be reflected in the signal amplitude. Interestingly, the additional auditory cues had a twofold effect on the previous findings: The P1 component was significantly suppressed in the case of the exploding box stimulus, whereas the N2c showed an enhancement for the burning box stimulus. This result highlights the impact of multisensory integration on the performance of realistic BCI applications. Indeed, we observed alterations in the offline classification accuracy for a detection task based on a mixed feature extraction (variance, power spectral density, and discrete wavelet transform) and a support vector machine classifier. In the case of the explosion, the accuracy slightly decreased by –1.64% p. in an audio-visual experiment compared to the visual-only. Contrarily, the classification accuracy for the burning box increased by 5.58% p. when additional auditory cues were present. Hence, we conclude, that especially in challenging detection tasks, it is favorable to consider the potential of multisensory integration when BCIs are supposed to operate under (multimodal) real-world conditions.

Cybernetic Hive Minds: A Review

Insect swarms and migratory birds are known to exhibit something known as a hive mind, collective consciousness, and herd mentality, among others. This has inspired a whole new stream of robotics known as swarm intelligence, where small-sized robots perform tasks in coordination. The social media and smartphone revolution have helped people collectively work together and organize in their day-to-day jobs or activism. This revolution has also led to the massive spread of disinformation amplified during the COVID-19 pandemic by alt-right Neo Nazi Cults like QAnon and their counterparts from across the globe, causing increases in the spread of infection and deaths. This paper presents the case for a theoretical cybernetic hive mind to explain how existing cults like QAnon weaponize group think and carry out crimes using social media-based alternate reality games. We also showcase a framework on how cybernetic hive minds have come into existence and how the hive mind might evolve in the future. We also discuss the implications of these hive minds for the future of free will and how different malfeasant entities have utilized these technologies to cause problems and inflict harm by various forms of cyber-crimes and predict how these crimes can evolve in the future.

BCI-Based Control for Ankle Exoskeleton T-FLEX: Comparison of Visual and Haptic Stimuli with Stroke Survivors

Brain–computer interface (BCI) remains an emerging tool that seeks to improve the patient interaction with the therapeutic mechanisms and to generate neuroplasticity progressively through neuromotor abilities. Motor imagery (MI) analysis is the most used paradigm based on the motor cortex’s electrical activity to detect movement intention. It has been shown that motor imagery mental practice with movement-associated stimuli may offer an effective strategy to facilitate motor recovery in brain injury patients. In this sense, this study aims to present the BCI associated with visual and haptic stimuli to facilitate MI generation and control the T-FLEX ankle exoskeleton. To achieve this, five post-stroke patients (55–63 years) were subjected to three different strategies using T-FLEX: stationary therapy (ST) without motor imagination, motor imagination with visual stimulation (MIV), and motor imagination with visual-haptic inducement (MIVH). The quantitative characterization of both BCI stimuli strategies was made through the motor imagery accuracy rate, the electroencephalographic (EEG) analysis during the MI active periods, the statistical analysis, and a subjective patient’s perception. The preliminary results demonstrated the viability of the BCI-controlled ankle exoskeleton system with the beta rebound, in terms of patient’s performance during MI active periods and satisfaction outcomes. Accuracy differences employing haptic stimulus were detected with an average of 68% compared with the 50.7% over only visual stimulus. However, the power spectral density (PSD) did not present changes in prominent activation of the MI band but presented significant variations in terms of laterality. In this way, visual and haptic stimuli improved the subject’s MI accuracy but did not generate differential brain activity over the affected hemisphere. Hence, long-term sessions with a more extensive sample and a more robust algorithm should be carried out to evaluate the impact of the proposed system on neuronal and motor evolution after stroke.

Affective Neurofeedback Under Naturalistic Conditions: A Mini-Review of Current Achievements and Open Challenges

Affective neurofeedback training allows for the self-regulation of the putative circuits of emotion regulation. This approach has recently been studied as a possible additional treatment for psychiatric disorders, presenting positive effects in symptoms and behaviors. After neurofeedback training, a critical aspect is the transference of the learned self-regulation strategies to outside the laboratory and how to continue reinforcing these strategies in non-controlled environments. In this mini-review, we discuss the current achievements of affective neurofeedback under naturalistic setups. For this, we first provide a brief overview of the state-of-the-art for affective neurofeedback protocols. We then discuss virtual reality as a transitional step toward the final goal of "in-the-wild" protocols and current advances using mobile neurotechnology. Finally, we provide a discussion of open challenges for affective neurofeedback protocols in-the-wild, including topics such as convenience and reliability, environmental effects in attention and workload, among others.

Control Modification of Grasp Force Covaries Agency and Performance on Rigid and Compliant Surfaces

This study investigated how modifications in the display of a computer trace under user control of grasp forces can co-modulate agency (perception of control) and performance of grasp on rigid and compliant surfaces. We observed positive correlation (p < 0.01) between implicit agency, measured from time-interval estimation for intentional binding, and grasp performance, measured by force-tracking error, across varying control modes for each surface type. The implications of this work are design directives for cognition-centered device interfaces for rehabilitation of grasp after neurotraumas such as spinal cord and brain injuries while considering if grasp interaction is rigid or compliant. These device interfaces should increase user integration to virtual reality training and powered assistive devices such as exoskeletons and prostheses. The modifications in control modes for this study included changes in force magnitude, addition of mild noise, and a measure of automation. Significant differences (p < 0.001) were observed for each surface type across control modes with metrics for implicit agency, performance, and grasp control efficiency. Explicit agency, measured from user survey responses, did not exhibit significant variations in this study, suggesting implicit measures of agency are needed for identifying co-modulation with grasp performance. Grasp on the compliant surface resulted in greater dependence of performance on agency and increases in agency and performance with the addition of mild noise. Noise in conjunction with perceived freedom at a flexible surface may have amplified visual feedback responses. Introducing automation in control decreased agency and performance for both surfaces, suggesting the value in continuous user control of grasp. In conclusion, agency and performance of grasp can be co-modulated across varying modes of control, especially for compliant grasp actions. Future studies should consider reliable measures of implicit agency, including physiological recordings, to automatically adapt rehabilitation interfaces for better cognitive engagement and to accelerate functional outcomes.

Covariation Informed Graph Slepians for Motor Imagery Decoding

Graph signal processing (GSP) provides signal analytic tools for data defined in irregular domains, as is the case of non-invasive electroencephalography (EEG). In this work, the recently introduced technique of Graph Slepian functions is exploited for the robust decoding of motor imagery (MI) brain activity. The particular technique builds over the concept of graph Fourier transform (GFT) and provides additional flexibility in the subsequent data analysis by incorporating domain knowledge. Based on contrastive learning, we introduce an algorithmic pipeline that attains a data driven and subject specific design of Graph Slepian functions. These functions, by incorporating both the topology of the sensor array and the empirical evidence about the differential functional covariation, act as spatial filters that enhance the information conveyed by the multichannel signal and specifically relates to the participant's intention. The proposed technique for crafting Graph Slepians is incorporated in a MI-decoding scheme, in which the informed projections are fed to a support vector machine (SVM) that casts a prediction regarding the type of intended movement. The employed MI-decoder is evaluated based on two publicly available datasets and its superiority against popular alternatives in the field is established. Computational efficiency is listed among its main advantages, since it involves only simple matrix operations, allowing to consider its use in real-time implementations.

Passive Brain-Computer Interfaces for Enhanced Human-Robot Interaction

Brain-computer interfaces (BCIs) have long been seen as control interfaces that translate changes in brain activity, produced either by means of a volitional modulation or in response to an external stimulation. However, recent trends in the BCI and neurofeedback research highlight passive monitoring of a user's brain activity in order to estimate cognitive load, attention level, perceived errors and emotions. Extraction of such higher order information from brain signals is seen as a gateway for facilitation of interaction between humans and intelligent systems. Particularly in the field of robotics, passive BCIs provide a promising channel for prediction of user's cognitive and affective state for development of a user-adaptive interaction. In this paper, we first illustrate the state of the art in passive BCI technology and then provide examples of BCI employment in human-robot interaction (HRI). We finally discuss the prospects and challenges in integration of passive BCIs in socially demanding HRI settings. This work intends to inform HRI community of the opportunities offered by passive BCI systems for enhancement of human-robot interaction while recognizing potential pitfalls.

Stereoscopic Motion Perception Research Based on Steady-state Visual Motion Evoked Potential

The combination of BCI technology and stereoscopic three-dimensional (3D) display has gradually become a trend, while stereo-based BCI has been used in rehabilitation training and medical testing. However, present stereo-based BCI research mainly stays in the static stereo environment, and the main method of visual stimulation is flickering, which does not effectively utilize the characteristic of the stereoscopic display technology. Therefore, we proposed a novel stimulation method based on stereoscopic motion. It utilized the stereo reciprocating motion of the plane of intensive line to elicit steady-state visual motion evoked potential (SSMVEP). The results shown that the correlation canonical analysis (CCA) coefficients of the EEG signal of stereoscopic motion (4.3 Hz-6.3 Hz) was significant higher than the non-stereoscopic motion, and more brain areas were activated. This stimulation method can induce significant visual response and has a great potential in the application of virtual reality stereo-based BCI system.

Passive Brain-Computer Interfaces for Enhanced Human-Robot Interaction

Brain-computer interfaces (BCIs) have long been seen as control interfaces that translate changes in brain activity, produced either by means of a volitional modulation or in response to an external stimulation. However, recent trends in the BCI and neurofeedback research highlight passive monitoring of a user's brain activity in order to estimate cognitive load, attention level, perceived errors and emotions. Extraction of such higher order information from brain signals is seen as a gateway for facilitation of interaction between humans and intelligent systems. Particularly in the field of robotics, passive BCIs provide a promising channel for prediction of user's cognitive and affective state for development of a user-adaptive interaction. In this paper, we first illustrate the state of the art in passive BCI technology and then provide examples of BCI employment in human-robot interaction (HRI). We finally discuss the prospects and challenges in integration of passive BCIs in socially demanding HRI settings. This work intends to inform HRI community of the opportunities offered by passive BCI systems for enhancement of human-robot interaction while recognizing potential pitfalls.

Brain-computer interfaces and virtual reality for neurorehabilitation

Brain-computer interfaces (BCIs) and virtual reality (VR) are two technologic advances that are changing our way of interacting with the world. BCIs can be used to influence and can serve as a control mechanism in navigation tasks, communication, or other assistive functions. VR can create ad hoc interactive scenarios that involve all our senses, stimulate the brain in a multisensory fashion, and increase the motivation and fun with game-like environments. VR and motion tracking enable natural human-computer interaction at cognitive and physical levels. This includes both brain and body in the design of meaningful VR experiences; these cases in which participants feel naturally present could help augment the benefits of BCIs for assistive and neurorehabilitation applications for the relearning of motor and cognitive skills. VR technology is now available at the consumer level thanks to the proliferation of affordable head-mounted displays (HMDs). Merging both technologies into simplified, practical devices may help democratize these technologies.

Co-adaptive Training Improves Efficacy of a Multi-Day EEG-Based Motor Imagery BCI Training

Motor imagery (MI) based brain computer interfaces (BCI) detect changes in brain activity associated with imaginary limb movements, and translate them into device commands. MI based BCIs require training, during which the user gradually learns how to control his or her brain activity with the help of feedback. Additionally, machine learning techniques are frequently used to boost BCI performance and to adapt the decoding algorithm to the user's brain. Thus, both the brain and the machine need to adapt in order to improve performance. To study the utility of co-adaptive training in the BCI paradigm and the time scales involved, we investigated the performance of two groups of subjects, in a 4-day MI experiment using EEG recordings. One group (control, n = 9 subjects) performed the BCI task using a fixed classifier based on MI data from day 1. In the second group (experimental, n = 9 subjects), the classifier was regularly adapted based on brain activity patterns during the experiment days. We found that the experimental group showed a significantly larger change in performance following training compared to the control group. Specifically, although the experimental group exhibited a decrease in performance between days, it showed an increase in performance within each day, which compensated for the decrease. The control group showed decreases both within and between days. A correlation analysis in subjects who had a notable improvement in performance following training showed that performance was mainly associated with modulation of power in the α frequency band. To conclude, continuous updating of the classification algorithm improves the performance of subjects in longitudinal BCI training.

Heading for new shores! Overcoming pitfalls in BCI design

Research in brain-computer interfaces has achieved impressive progress towards implementing assistive technologies for restoration or substitution of lost motor capabilities, as well as supporting technologies for able-bodied subjects. Notwithstanding this progress, effective translation of these interfaces from proof-of concept prototypes into reliable applications remains elusive. As a matter of fact, most of the current BCI systems cannot be used independently for long periods of time by their intended end-users. Multiple factors that impair achieving this goal have already been identified. However, it is not clear how do they affect the overall BCI performance or how they should be tackled. This is worsened by the publication bias where only positive results are disseminated, preventing the research community from learning from its errors. This paper is the result of a workshop held at the 6th International BCI meeting in Asilomar. We summarize here the discussion on concrete research avenues and guidelines that may help overcoming common pitfalls and make BCIs become a useful alternative communication device.