Real-time fMRI-based brain–computer interfacing for neurofeedback therapy and compensation of lost motor functions

A Library for fMRI Real-Time Processing Systems in Python (RTPSpy) With Comprehensive Online Noise Reduction, Fast and Accurate Anatomical Image Processing, and Online Processing Simulation

Real-time fMRI (rtfMRI) has enormous potential for both mechanistic brain imaging studies or treatment-oriented neuromodulation. However, the adaption of rtfMRI has been limited due to technical difficulties in implementing an efficient computational framework. Here, we introduce a python library for real-time fMRI (rtfMRI) data processing systems, Real-Time Processing System in python (RTPSpy), to provide building blocks for a custom rtfMRI application with extensive and advanced functionalities. RTPSpy is a library package including (1) a fast, comprehensive, and flexible online fMRI image processing modules comparable to offline denoising, (2) utilities for fast and accurate anatomical image processing to define an anatomical target region, (3) a simulation system of online fMRI processing to optimize a pipeline and target signal calculation, (4) simple interface to an external application for feedback presentation, and (5) a boilerplate graphical user interface (GUI) integrating operations with RTPSpy library. The fast and accurate anatomical image processing utility wraps external tools, including FastSurfer, ANTs, and AFNI, to make tissue segmentation and region of interest masks. We confirmed that the quality of the output masks was comparable with FreeSurfer, and the anatomical image processing could complete in a few minutes. The modular nature of RTPSpy provides the ability to use it for a simulation analysis to optimize a processing pipeline and target signal calculation. We present a sample script for building a real-time processing pipeline and running a simulation using RTPSpy. The library also offers a simple signal exchange mechanism with an external application using a TCP/IP socket. While the main components of the RTPSpy are the library modules, we also provide a GUI class for easy access to the RTPSpy functions. The boilerplate GUI application provided with the package allows users to develop a customized rtfMRI application with minimum scripting labor. The limitations of the package as it relates to environment-specific implementations are discussed. These library components can be customized and can be used in parts. Taken together, RTPSpy is an efficient and adaptable option for developing rtfMRI applications.

Code available at:https://github.com/mamisaki/RTPSpy

The impact of real-time fMRI denoising on online evaluation of brain activity and functional connectivity

Objective. Comprehensive denoising is imperative in functional magnetic resonance imaging (fMRI) analysis to reliably evaluate neural activity from the blood oxygenation level dependent signal. In real-time fMRI, however, only a minimal denoising process has been applied and the impact of insufficient denoising on online brain activity estimation has not been assessed comprehensively. This study evaluated the noise reduction performance of online fMRI processes in a real-time estimation of regional brain activity and functional connectivity.Approach.We performed a series of real-time processing simulations of online fMRI processing, including slice-timing correction, motion correction, spatial smoothing, signal scaling, and noise regression with high-pass filtering, motion parameters, motion derivatives, global signal, white matter/ventricle average signals, and physiological noise models with image-based retrospective correction of physiological motion effects (RETROICOR) and respiration volume per time (RVT).Main results.All the processing was completed in less than 400 ms for whole-brain voxels. Most processing had a benefit for noise reduction except for RVT that did not work due to the limitation of the online peak detection. The global signal regression, white matter/ventricle signal regression, and RETROICOR had a distinctive noise reduction effect, depending on the target signal, and could not substitute for each other. Global signal regression could eliminate the noise-associated bias in the mean dynamic functional connectivity across time.Significance.The results indicate that extensive real-time denoising is possible and highly recommended for real-time fMRI applications.

fMRI volume classification using a 3D convolutional neural network robust to shifted and scaled neuronal activations

Deep-learning methods based on deep neural networks (DNNs) have recently been successfully utilized in the analysis of neuroimaging data. A convolutional neural network (CNN) is a type of DNN that employs a convolution kernel that covers a local area of the input sample and moves across the sample to provide a feature map for the subsequent layers. In our study, we hypothesized that a 3D-CNN model with down-sampling operations such as pooling and/or stride would have the ability to extract robust feature maps from the shifted and scaled neuronal activations in a single functional MRI (fMRI) volume for the classification of task information associated with that volume. Thus, the 3D-CNN model would be able to ameliorate the potential misalignment of neuronal activations and over-/under-activation in local brain regions caused by imperfections in spatial alignment algorithms, confounded by variability in blood-oxygenation-level-dependent (BOLD) responses across sessions and/or subjects. To this end, the fMRI volumes acquired from four sensorimotor tasks (left-hand clenching, right-hand clenching, auditory attention, and visual stimulation) were used as input for our 3D-CNN model to classify task information using a single fMRI volume. The classification performance of the 3D-CNN was systematically evaluated using fMRI volumes obtained from various minimal preprocessing scenarios applied to raw fMRI volumes that excluded spatial normalization to a template and those obtained from full preprocessing that included spatial normalization. Alternative classifier models such as the 1D fully connected DNN (1D-fcDNN) and support vector machine (SVM) were also used for comparison. The classification performance was also assessed for several k-fold cross-validation (CV) schemes, including leave-one-subject-out CV (LOOCV). Overall, the classification results of the 3D-CNN model were superior to that of the 1D-fcDNN and SVM models. When using the fully-processed fMRI volumes with LOOCV, the mean error rates (± the standard error of the mean) for the 3D-CNN, 1D-fcDNN, and SVM models were 2.1% (± 0.9), 3.1% (± 1.2), and 4.1% (± 1.5), respectively (p = 0.041 from a one-way ANOVA). The error rates for 3-fold CV were higher (2.4% ± 1.0, 4.2% ± 1.3, and 10.1% ± 2.0; p < 0.0003 from a one-way ANOVA). The mean error rates also increased considerably using the raw fMRI 3D volume data without preprocessing (26.2% for the 3D-CNN, 75.0% for the 1D-fcDNN, and 75.0% for the SVM). Furthermore, the ability of the pre-trained 3D-CNN model to handle shifted and scaled neuronal activations was demonstrated in an online scenario for five-class classification (i.e., four sensorimotor tasks and the resting state) using the real-time fMRI of three participants. The resulting classification accuracy was 78.5% (± 1.4), 26.7% (± 5.9), and 21.5% (± 3.1) for the 3D-CNN, 1D-fcDNN, and SVM models, respectively. The superior performance of the 3D-CNN compared to the 1D-fcDNN was verified by analyzing the resulting feature maps and convolution filters that handled the shifted and scaled neuronal activations and by utilizing an independent public dataset from the Human Connectome Project.

Effects of Motor Imagery and Visual Neurofeedback on Activation in the Swallowing Network: A Real-Time fMRI Study

Motor imagery of movements is used as mental strategy in neurofeedback applications to gain voluntary control over activity in motor areas of the brain. In the present functional magnetic resonance imaging (fMRI) study, we first addressed the question whether motor imagery and execution of swallowing activate comparable brain areas, which has been already proven for hand and foot movements. Prior near-infrared spectroscopy (NIRS) studies provide evidence that this is the case in the outer layer of the cortex. With the present fMRI study, we want to expand these prior NIRS findings to the whole brain. Second, we used motor imagery of swallowing as mental strategy during visual neurofeedback to investigate whether one can learn to modulate voluntarily activity in brain regions, which are associated with active swallowing, using real-time fMRI. Eleven healthy adults performed one offline session, in which they executed swallowing movements and imagined swallowing on command during fMRI scanning. Based on this functional localizer task, we identified brain areas active during both tasks and defined individually regions for feedback. During the second session, participants performed two real-time fMRI neurofeedback runs (each run comprised 10 motor imagery trials), in which they should increase voluntarily the activity in the left precentral gyrus by means of motor imagery of swallowing while receiving visual feedback (the visual feedback depicted one's own fMRI signal changes in real-time). Motor execution and imagery of swallowing activated a comparable network of brain areas including the bilateral pre- and postcentral gyrus, inferior frontal gyrus, basal ganglia, insula, SMA, and the cerebellum compared to a resting condition. During neurofeedback training, participants were able to increase the activity in the feedback region (left lateral precentral gyrus) but also in other brain regions, which are generally active during swallowing, compared to the motor imagery offline task. Our results indicate that motor imagery of swallowing is an adequate mental strategy to activate the swallowing network of the whole brain, which might be useful for future treatments of swallowing disorders.

fMRI Neurofeedback Training for Increasing Anterior Cingulate Cortex Activation in Adult Attention Deficit Hyperactivity Disorder. An Exploratory Randomized, Single-Blinded Study

Attention Deficit Hyperactivity Disorder (ADHD) is characterized by poor cognitive control/attention and hypofunctioning of the dorsal anterior cingulate cortex (dACC). In the current study, we investigated for the first time whether real-time fMRI neurofeedback (rt-fMRI) training targeted at increasing activation levels within dACC in adults with ADHD leads to a reduction of clinical symptoms and improved cognitive functioning. An exploratory randomized controlled treatment study with blinding of the participants was conducted. Participants with ADHD (n = 7 in the neurofeedback group, and n = 6 in the control group) attended four weekly MRI training sessions (60-min training time/session), during which they performed a mental calculation task at varying levels of difficulty, in order to learn how to up-regulate dACC activation. Only neurofeedback participants received continuous feedback information on actual brain activation levels within dACC. Before and after the training, ADHD symptoms and relevant cognitive functioning was assessed. Results showed that both groups achieved a significant increase in dACC activation levels over sessions. While there was no significant difference between the neurofeedback and control group in clinical outcome, neurofeedback participants showed stronger improvement on cognitive functioning. The current study demonstrates the general feasibility of the suggested rt-fMRI neurofeedback training approach as a potential novel treatment option for ADHD patients. Due to the study’s small sample size, potential clinical benefits need to be further investigated in future studies.

Trial Registration: ISRCTN12390961

When the Brain Takes 'BOLD' Steps: Real-Time fMRI Neurofeedback Can Further Enhance the Ability to Gradually Self-regulate Regional Brain Activation

•

Humans are able to gradually self-regulate regional brain activation by applying cognitive strategies.

•

Providing rtfMRI neurofeedback can enhance the gradual self-regulation ability.

•

Findings are generalizable to various mental tasks and clinical MR field strengths.

•

Novel parametric activation paradigm enriches spectrum of rtfMRI-neurofeedback and BCI methodology.

Brain-computer interfaces (BCIs) based on real-time functional magnetic resonance imaging (rtfMRI) are currently explored in the context of developing alternative (motor-independent) communication and control means for the severely disabled. In such BCI systems, the user encodes a particular intention (e.g., an answer to a question or an intended action) by evoking specific mental activity resulting in a distinct brain state that can be decoded from fMRI activation. One goal in this context is to increase the degrees of freedom in encoding different intentions, i.e., to allow the BCI user to choose from as many options as possible. Recently, the ability to voluntarily modulate spatial and/or temporal blood oxygenation level-dependent (BOLD)-signal features has been explored implementing different mental tasks and/or different encoding time intervals, respectively. Our two-session fMRI feasibility study systematically investigated for the first time the possibility of using magnitudinal BOLD-signal features for intention encoding. Particularly, in our novel paradigm, participants (n = 10) were asked to alternately self-regulate their regional brain-activation level to 30%, 60% or 90% of their maximal capacity by applying a selected activation strategy (i.e., performing a mental task, e.g., inner speech) and modulation strategies (e.g., using different speech rates) suggested by the experimenters. In a second step, we tested the hypothesis that the additional availability of feedback information on the current BOLD-signal level within a region of interest improves the gradual-self regulation performance. Therefore, participants were provided with neurofeedback in one of the two fMRI sessions. Our results show that the majority of the participants were able to gradually self-regulate regional brain activation to at least two different target levels even in the absence of neurofeedback. When provided with continuous feedback on their current BOLD-signal level, most participants further enhanced their gradual self-regulation ability. Our findings were observed across a wide variety of mental tasks and across clinical MR field strengths (i.e., at 1.5 T and 3 T), indicating that these findings are robust and can be generalized across mental tasks and scanner types. The suggested novel parametric activation paradigm enriches the spectrum of current rtfMRI-neurofeedback and BCI methodology and has considerable potential for fundamental and clinical neuroscience applications.

Decoding the direction of imagined visual motion using 7T ultra-high field fMRI

There is a long-standing debate about the neurocognitive implementation of mental imagery. One form of mental imagery is the imagery of visual motion, which is of interest due to its naturalistic and dynamic character. However, so far only the mere occurrence rather than the specific content of motion imagery was shown to be detectable. In the current study, the application of multi-voxel pattern analysis to high-resolution functional data of 12 subjects acquired with ultra-high field 7T functional magnetic resonance imaging allowed us to show that imagery of visual motion can indeed activate the earliest levels of the visual hierarchy, but the extent thereof varies highly between subjects. Our approach enabled classification not only of complex imagery, but also of its actual contents, in that the direction of imagined motion out of four options was successfully identified in two thirds of the subjects and with accuracies of up to 91.3% in individual subjects. A searchlight analysis confirmed the local origin of decodable information in striate and extra-striate cortex. These high-accuracy findings not only shed new light on a central question in vision science on the constituents of mental imagery, but also show for the first time that the specific sub-categorical content of visual motion imagery is reliably decodable from brain imaging data on a single-subject level.

The Inclusion of Functional Connectivity Information into fMRI-based Neurofeedback Improves Its Efficacy in the Reduction of Cigarette Cravings

Real-time fMRI (rtfMRI) neurofeedback (NF) facilitates volitional control over brain activity and the modulation of associated mental functions. The NF signals of traditional rtfMRI-NF studies predominantly reflect neuronal activity within ROIs. In this study, we describe a novel rtfMRI-NF approach that includes a functional connectivity (FC) component in the NF signal (FC-added rtfMRI-NF). We estimated the efficacy of the FC-added rtfMRI-NF method by applying it to nicotine-dependent heavy smokers in an effort to reduce cigarette craving. ACC and medial pFC as well as the posterior cingulate cortex and precuneus are associated with cigarette craving and were chosen as ROIs. Fourteen heavy smokers were randomly assigned to receive one of two types of NF: traditional activity-based rtfMRI-NF or FC-added rtfMRI-NF. Participants received rtfMRI-NF training during two separate visits after overnight smoking cessation, and cigarette craving score was assessed. The FC-added rtfMRI-NF resulted in greater neuronal activity and increased FC between the targeted ROIs than the traditional activity-based rtfMRI-NF and resulted in lower craving score. In the FC-added rtfMRI-NF condition, the average of neuronal activity and FC was tightly associated with craving score (Bonferroni-corrected p = .028). However, in the activity-based rtfMRI-NF condition, no association was detected (uncorrected p &gt; .081). Non-rtfMRI data analysis also showed enhanced neuronal activity and FC with FC-added NF than with activity-based NF. These results demonstrate that FC-added rtfMRI-NF facilitates greater volitional control over brain activity and connectivity and greater modulation of mental function than activity-based rtfMRI-NF.

fMRI neurofeedback facilitates anxiety regulation in females with spider phobia

Background: Spider phobics show an exaggerated fear response when encountering spiders. This fear response is aggravated by negative and irrational beliefs about the feared object. Cognitive reappraisal can target these beliefs, and therefore has a fear regulating effect. The presented study investigated if neurofeedback derived from functional magnetic resonance imaging (fMRI) would facilitate anxiety regulation by cognitive reappraisal, using spider phobia as a model of anxiety disorders. Feedback was provided based on activation in left dorsolateral prefrontal cortex and right insula, as indicators of engagement and regulation success, respectively.

Methods: Eighteen female spider phobics participated in a randomized, controlled, single-blinded study. All participants completed a training session in the MRI scanner. Participants assigned to the neurofeedback condition were instructed to shape their regulatory strategy based on the provided feedback. Participants assigned to the control condition were asked to adapt their strategy intuitively.

Results: Neurofeedback participants exhibited lower anxiety levels than the control group at the end of the training. In addition, only neurofeedback participants achieved down-regulation of insula activation levels by cognitive reappraisal. Group differences became more pronounced over time, supporting learning as a mechanism behind this effect. Importantly, within the neurofeedback group, achieved changes in insula activation levels during training predicted long-term anxiety reduction.

Conclusions: The conducted study provides first evidence that fMRI neurofeedback has a facilitating effect on anxiety regulation in spider phobia.

Decoding the individual finger movements from single-trial functional magnetic resonance imaging recordings of human brain activity

Multivariate pattern classification analysis (MVPA) has been applied to functional magnetic resonance imaging (fMRI) data to decode brain states from spatially distributed activation patterns. Decoding upper limb movements from non-invasively recorded human brain activation is crucial for implementing a brain-machine interface that directly harnesses an individual's thoughts to control external devices or computers. The aim of this study was to decode the individual finger movements from fMRI single-trial data. Thirteen healthy human subjects participated in a visually cued delayed finger movement task, and only one slight button press was performed in each trial. Using MVPA, the decoding accuracy (DA) was computed separately for the different motor-related regions of interest. For the construction of feature vectors, the feature vectors from two successive volumes in the image series for a trial were concatenated. With these spatial-temporal feature vectors, we obtained a 63.1% average DA (84.7% for the best subject) for the contralateral primary somatosensory cortex and a 46.0% average DA (71.0% for the best subject) for the contralateral primary motor cortex; both of these values were significantly above the chance level (20%). In addition, we implemented searchlight MVPA to search for informative regions in an unbiased manner across the whole brain. Furthermore, by applying searchlight MVPA to each volume of a trial, we visually demonstrated the information for decoding, both spatially and temporally. The results suggest that the non-invasive fMRI technique may provide informative features for decoding individual finger movements and the potential of developing an fMRI-based brain-machine interface for finger movement.

A graphics processing unit accelerated motion correction algorithm and modular system for real-time fMRI

Real-time functional magnetic resonance imaging (rt-fMRI) has recently gained interest as a possible means to facilitate the learning of certain behaviors. However, rt-fMRI is limited by processing speed and available software, and continued development is needed for rt-fMRI to progress further and become feasible for clinical use. In this work, we present an open-source rt-fMRI system for biofeedback powered by a novel Graphics Processing Unit (GPU) accelerated motion correction strategy as part of the BioImage Suite project ( www.bioimagesuite.org ). Our system contributes to the development of rt-fMRI by presenting a motion correction algorithm that provides an estimate of motion with essentially no processing delay as well as a modular rt-fMRI system design. Using empirical data from rt-fMRI scans, we assessed the quality of motion correction in this new system. The present algorithm performed comparably to standard (non real-time) offline methods and outperformed other real-time methods based on zero order interpolation of motion parameters. The modular approach to the rt-fMRI system allows the system to be flexible to the experiment and feedback design, a valuable feature for many applications. We illustrate the flexibility of the system by describing several of our ongoing studies. Our hope is that continuing development of open-source rt-fMRI algorithms and software will make this new technology more accessible and adaptable, and will thereby accelerate its application in the clinical and cognitive neurosciences.

Windowed correlation: a suitable tool for providing dynamic fMRI-based functional connectivity neurofeedback on task difficulty

The goal of neurofeedback training is to provide participants with relevant information on their ongoing brain processes in order to enable them to change these processes in a meaningful way. Under the assumption of an intrinsic brain-behavior link, neurofeedback can be a tool to guide a participant towards a desired behavioral state, such as a healthier state in the case of patients. Current research in clinical neuroscience regarding the most robust indicators of pathological brain processes in psychiatric and neurological disorders indicates that fMRI-based functional connectivity measures may be among the most important biomarkers of disease. The present study therefore investigated the general potential of providing fMRI neurofeedback based on functional correlations, computed from short-window time course data at the level of single task periods. The ability to detect subtle changes in task performance with block-wise functional connectivity measures was evaluated based on imaging data from healthy participants performing a simple motor task, which was systematically varied along two task dimensions representing two different aspects of task difficulty. The results demonstrate that fMRI-based functional connectivity measures may provide a better indicator for an increase in overall (motor) task difficulty than activation level-based measures. Windowed functional correlations thus seem to provide relevant and unique information regarding ongoing brain processes, which is not captured equally well by standard activation level-based neurofeedback measures. Functional connectivity markers, therefore, may indeed provide a valuable tool to enhance and monitor learning within an fMRI neurofeedback setup.