Controlling an avatar by thought using real-time fMRI

A multi-modal modified feedback self-paced BCI to control the gait of an avatar

Brain-computer interfaces (BCI) have been used to control the gait of a virtual self-avatar with a proposed application in the field of gait rehabilitation.

OBJECTIVE

to develop a high performance multi-modal BCI to control single steps and forward walking of an immersive virtual reality avatar. This system will overcome the limitation of existing systems.

APPROACH

This system used MI of these actions, in cue-paced and self-paced modes. Twenty healthy participants participated in this 4 sessions study across 4 different days. They were cued to imagine a single step forward with their right or left foot, or to imagine walking forward. They were instructed to reach a target by using the MI of multiple steps (self-paced switch-control mode) or by maintaining MI of forward walking (continuous-control mode). The movement of the avatar was controlled by two calibrated RLDA classifiers that used the µ power spectral density (PSD) over the foot area of the motor cortex as a feature. The classifiers were retrained after every session. For a subset of the trials, positive modified feedback was presented to half of the participants.

MAIN RESULTS

All participants were able to operate the BCI. Their average offline performance, after retraining the classifiers was 86.0 ± 6.1%, showing that the recalibration of the classifiers enhanced the offline performance of the BCI (p < 0.01). The average online performance was 85.9 ± 8.4% showing that modified feedback enhanced BCI performance (p =0.001). The average performance was 83% at self-paced switch control and 92% at continuous control mode.

SIGNIFICANCE

This study reports on the first novel integration of different design approaches, different control modes and different performance enhancement techniques, all in parallel in one single high performance and multi-modal BCI system, to control single steps and forward walking of an immersive virtual reality avatar.

Toward Enhanced Teleoperation Through Embodiment

Telerobotics aims to transfer human manipulation skills and dexterity over an arbitrary distance and at an arbitrary scale to a remote workplace. A telerobotic system that is transparent enables a natural and intuitive interaction. We postulate that embodiment (with three sub-components: sense of ownership, agency, and self-location) of the robotic system leads to optimal perceptual transparency and increases task performance. However, this has not yet been investigated directly. We reason along four premises and present findings from the literature that substantiate each of them: (1) the brain can embody non-bodily objects (e.g., robotic hands), (2) embodiment can be elicited with mediated sensorimotor interaction, (3) embodiment is robust against inconsistencies between the robotic system and the operator's body, and (4) embodiment positively correlates to dexterous task performance. We use the predictive encoding theory as a framework to interpret and discuss the results reported in the literature. Numerous previous studies have shown that it is possible to induce embodiment over a wide range of virtual and real extracorporeal objects (including artificial limbs, avatars, and android robots) through mediated sensorimotor interaction. Also, embodiment can occur for non-human morphologies including for elongated arms and a tail. In accordance with the predictive encoding theory, none of the sensory modalities is critical in establishing ownership, and discrepancies in multisensory signals do not necessarily lead to loss of embodiment. However, large discrepancies in terms of multisensory synchrony or visual likeness can prohibit embodiment from occurring. The literature provides less extensive support for the link between embodiment and (dexterous) task performance. However, data gathered with prosthetic hands do indicate a positive correlation. We conclude that all four premises are supported by direct or indirect evidence in the literature, suggesting that embodiment of a remote manipulator may improve dexterous performance in telerobotics. This warrants further implementation testing of embodiment in telerobotics. We formulate a first set of guidelines to apply embodiment in telerobotics and identify some important research topics.

Trends in research participant categories and descriptions in abstracts from the International BCI Meeting series, 1999 to 2016

Much brain-computer interface (BCI) research is intended to benefit people with disabilities (PWD), but inclusion of these individuals as study participants remains relatively rare. When participants with disabilities are included, they are described with a range of clinical and non-clinical terms with varying degrees of specificity, often leading to difficulty in interpreting or replicating results. This study examined trends in inclusion and description of study participants with disabilities across six International BCI Meetings from 1999 to 2016. Abstracts from each Meeting were analyzed by two trained independent reviewers. Results suggested a decline in participation by PWD across Meetings until the 2016 Meeting. Increased diagnostic specificity was noted at the 2013 and 2016 Meetings. Fifty-eight percent of the abstracts identified PWD as being the target beneficiaries of BCI research, though only twenty-two percent included participants with disabilities, suggesting evidence of a persistent translational gap. Participants with disabilities were most commonly described as having physical and/or communication impairments compared to impairments in other areas. Implementing participatory action research principles and user-centered design strategies continues to be necessary within BCI research to bridge the translational gap and facilitate use of BCI systems within functional environments for PWD.

Altered resting-state dynamics in autism spectrum disorder: Causal to the social impairment?

Autism spectrum disorder (ASD) is characterized by profound impairments in social abilities and by restricted interests and repetitive behaviors. Much work in the past decade has been dedicated to understanding the brain-bases of ASD, and in the context of resting-state functional connectivity fMRI in high-functioning adolescents and adults, the field has established a set of reliable findings: decreased cortico-cortical interactions among brain regions thought to be engaged in social processing, along with a simultaneous increase in thalamo-cortical and striato-cortical interactions. However, few studies have attempted to manipulate these altered patterns, leading to the question of whether such patterns are actually causally involved in producing the corresponding behavioral impairments. We discuss a few such recent attempts in the domains of fMRI neurofeedback and overt social interaction during scanning, and we conclude that the evidence of causal involvement is somewhat mixed. We highlight the potential role of the thalamus and striatum in ASD and emphasize the need for studies that directly compare scanning during multiple cognitive states in addition to the resting-state.

A Decoding Scheme for Incomplete Motor Imagery EEG With Deep Belief Network

High accuracy decoding of electroencephalogram (EEG) signal is still a major challenge that can hardly be solved in the design of an effective motor imagery-based brain-computer interface (BCI), especially when the signal contains various extreme artifacts and outliers arose from data loss. The conventional process to avoid such cases is to directly reject the entire severely contaminated EEG segments, which leads to a drawback that the BCI has no decoding results during that certain period. In this study, a novel decoding scheme based on the combination of Lomb-Scargle periodogram (LSP) and deep belief network (DBN) was proposed to recognize the incomplete motor imagery EEG. Particularly, instead of discarding the entire segment, two forms of data removal were adopted to eliminate the EEG portions with extreme artifacts and data loss. The LSP was utilized to steadily extract the power spectral density (PSD) features from the incomplete EEG constructed by the remaining portions. A DBN structure based on the restricted Boltzmann machine (RBM) was exploited and optimized to perform the classification task. Various comparative experiments were conducted and evaluated on simulated signal and real incomplete motor imagery EEG, including the comparison of three PSD extraction methods (fast Fourier transform, Welch and LSP) and two classifiers (DBN and support vector machine, SVM). The results demonstrate that the LSP can estimate relative robust PSD features and the proposed scheme can significantly improve the decoding performance for the incomplete motor imagery EEG. This scheme can provide an alternative decoding solution for the motor imagery EEG contaminated by extreme artifacts and data loss. It can be beneficial to promote the stability, smoothness and maintain consecutive outputs without interruption for a BCI system that is suitable for the online and long-term application.

Why Is Virtual Reality Interesting for Philosophers?

This article explores promising points of contact between philosophy and the expanding field of virtual reality research. Aiming at an interdisciplinary audience, it proposes a series of new research targets by presenting a range of concrete examples characterized by high theoretical relevance and heuristic fecundity. Among these examples are conscious experience itself, “Bayesian” and social VR, amnestic re-embodiment, merging human-controlled avatars and virtual agents, virtual ego-dissolution, controlling the reality/virtuality continuum, the confluence of VR and artificial intelligence (AI) as well as of VR and functional magnetic resonance imaging (fMRI), VR-based social hallucinations and the emergence of a virtual Lebenswelt, religious faith and practical phenomenology. Hopefully, these examples can serve as first proposals for intensified future interaction and mark out some potential new directions for research.

Brain-Machine Interfaces: From Basic Science to Neuroprostheses and Neurorehabilitation

Brain-machine interfaces (BMIs) combine methods, approaches, and concepts derived from neurophysiology, computer science, and engineering in an effort to establish real-time bidirectional links between living brains and artificial actuators. Although theoretical propositions and some proof of concept experiments on directly linking the brains with machines date back to the early 1960s, BMI research only took off in earnest at the end of the 1990s, when this approach became intimately linked to new neurophysiological methods for sampling large-scale brain activity. The classic goals of BMIs are 1) to unveil and utilize principles of operation and plastic properties of the distributed and dynamic circuits of the brain and 2) to create new therapies to restore mobility and sensations to severely disabled patients. Over the past decade, a wide range of BMI applications have emerged, which considerably expanded these original goals. BMI studies have shown neural control over the movements of robotic and virtual actuators that enact both upper and lower limb functions. Furthermore, BMIs have also incorporated ways to deliver sensory feedback, generated from external actuators, back to the brain. BMI research has been at the forefront of many neurophysiological discoveries, including the demonstration that, through continuous use, artificial tools can be assimilated by the primate brain's body schema. Work on BMIs has also led to the introduction of novel neurorehabilitation strategies. As a result of these efforts, long-term continuous BMI use has been recently implicated with the induction of partial neurological recovery in spinal cord injury patients.

Support vector machines to detect physiological patterns for EEG and EMG-based human-computer interaction: a review

Support vector machines (SVMs) are widely used classifiers for detecting physiological patterns in human-computer interaction (HCI). Their success is due to their versatility, robustness and large availability of free dedicated toolboxes. Frequently in the literature, insufficient details about the SVM implementation and/or parameters selection are reported, making it impossible to reproduce study analysis and results. In order to perform an optimized classification and report a proper description of the results, it is necessary to have a comprehensive critical overview of the applications of SVM. The aim of this paper is to provide a review of the usage of SVM in the determination of brain and muscle patterns for HCI, by focusing on electroencephalography (EEG) and electromyography (EMG) techniques. In particular, an overview of the basic principles of SVM theory is outlined, together with a description of several relevant literature implementations. Furthermore, details concerning reviewed papers are listed in tables and statistics of SVM use in the literature are presented. Suitability of SVM for HCI is discussed and critical comparisons with other classifiers are reported.

The Importance of Visual Feedback Design in BCIs; from Embodiment to Motor Imagery Learning

Brain computer interfaces (BCIs) have been developed and implemented in many areas as a new communication channel between the human brain and external devices. Despite their rapid growth and broad popularity, the inaccurate performance and cost of user-training are yet the main issues that prevent their application out of the research and clinical environment. We previously introduced a BCI system for the control of a very humanlike android that could raise a sense of embodiment and agency in the operators only by imagining a movement (motor imagery) and watching the robot perform it. Also using the same setup, we further discovered that the positive bias of subjects' performance both increased their sensation of embodiment and improved their motor imagery skills in a short period. In this work, we studied the shared mechanism between the experience of embodiment and motor imagery. We compared the trend of motor imagery learning when two groups of subjects BCI-operated different looking robots, a very humanlike android's hands and a pair of metallic gripper. Although our experiments did not show a significant change of learning between the two groups immediately during one session, the android group revealed better motor imagery skills in the follow up session when both groups repeated the task using the non-humanlike gripper. This result shows that motor imagery skills learnt during the BCI-operation of humanlike hands are more robust to time and visual feedback changes. We discuss the role of embodiment and mirror neuron system in such outcome and propose the application of androids for efficient BCI training.

The Role of Audio-Visual Feedback in a Thought-Based Control of a Humanoid Robot: A BCI Study in Healthy and Spinal Cord Injured People

The efficient control of our body and successful interaction with the environment are possible through the integration of multisensory information. Brain-computer interface (BCI) may allow people with sensorimotor disorders to actively interact in the world. In this study, visual information was paired with auditory feedback to improve the BCI control of a humanoid surrogate. Healthy and spinal cord injured (SCI) people were asked to embody a humanoid robot and complete a pick-and-place task by means of a visual evoked potentials BCI system. Participants observed the remote environment from the robot's perspective through a head mounted display. Human-footsteps and computer-beep sounds were used as synchronous/asynchronous auditory feedback. Healthy participants achieved better placing accuracy when listening to human footstep sounds relative to a computer-generated sound. SCI people demonstrated more difficulty in steering the robot during asynchronous auditory feedback conditions. Importantly, subjective reports highlighted that the BCI mask overlaying the display did not limit the observation of the scenario and the feeling of being in control of the robot. Overall, the data seem to suggest that sensorimotor-related information may improve the control of external devices. Further studies are required to understand how the contribution of residual sensory channels could improve the reliability of BCI systems.

Covert neurofeedback without awareness shapes cortical network spontaneous connectivity

Recent advances in blood oxygen level-dependent-functional MRI (BOLD-fMRI)-based neurofeedback reveal that participants can modulate neuronal properties. However, it is unknown whether such training effects can be introduced in the absence of participants' awareness that they are being trained. Here, we show unconscious neurofeedback training, which consequently produced changes in functional connectivity, introduced in participants who received positive and negative rewards that were covertly coupled to activity in two category-selective visual cortex regions. The results indicate that brain networks can be modified even in the complete absence of intention and awareness of the learning situation, raising intriguing possibilities for clinical interventions.

Optimal hemodynamic response model for functional near-infrared spectroscopy

Functional near-infrared spectroscopy (fNIRS) is an emerging non-invasive brain imaging technique and measures brain activities by means of near-infrared light of 650-950 nm wavelengths. The cortical hemodynamic response (HR) differs in attributes at different brain regions and on repetition of trials, even if the experimental paradigm is kept exactly the same. Therefore, an HR model that can estimate such variations in the response is the objective of this research. The canonical hemodynamic response function (cHRF) is modeled by two Gamma functions with six unknown parameters (four of them to model the shape and other two to scale and baseline respectively). The HRF model is supposed to be a linear combination of HRF, baseline, and physiological noises (amplitudes and frequencies of physiological noises are supposed to be unknown). An objective function is developed as a square of the residuals with constraints on 12 free parameters. The formulated problem is solved by using an iterative optimization algorithm to estimate the unknown parameters in the model. Inter-subject variations in HRF and physiological noises have been estimated for better cortical functional maps. The accuracy of the algorithm has been verified using 10 real and 15 simulated data sets. Ten healthy subjects participated in the experiment and their HRF for finger-tapping tasks have been estimated and analyzed. The statistical significance of the estimated activity strength parameters has been verified by employing statistical analysis (i.e., t-value > t critical and p-value < 0.05).

Differential magnetic resonance neurofeedback modulations across extrinsic (visual) and intrinsic (default-mode) nodes of the human cortex

Previous advances in magnetic resonance imaging allow the analysis of blood oxygen level-dependent signals in real time, thus opening the possibility of feeding an index of these signals back to scanned human participants. However, it is still not known to what extent different cortical networks may differ in their sensitivity to such internally generated neurofeedback (NF). Here, we compare NF efficacy across six cortical regions including: early and high-order visual areas and the posterior parietal lobe, a prominent node of the default mode network (DMN). Our results reveal a consistent difference in NF activation across these areas. Sham controls ruled out a role of attention/arousal in these effects. These differences are suggestive of a relationship to the relative reliance on intrinsic information, moving from early visual cortex (lowest) to the DMN (highest). Interestingly, the visual parahippocampal place area showed NF activation closer to the DMN node. The results are compatible with the notion of the DMN as an intrinsically oriented system.

FRIEND Engine Framework: a real time neurofeedback client-server system for neuroimaging studies

In this methods article, we present a new implementation of a recently reported FSL-integrated neurofeedback tool, the standalone version of “Functional Real-time Interactive Endogenous Neuromodulation and Decoding” (FRIEND). We will refer to this new implementation as the FRIEND Engine Framework. The framework comprises a client-server cross-platform solution for real time fMRI and fMRI/EEG neurofeedback studies, enabling flexible customization or integration of graphical interfaces, devices, and data processing. This implementation allows a fast setup of novel plug-ins and frontends, which can be shared with the user community at large. The FRIEND Engine Framework is freely distributed for non-commercial, research purposes.