Brain–machine interfaces using functional near-infrared spectroscopy: a review

Three-dimensional-printed headcap with embedded microdrive system for customizable multi-region brain recordings with neural probes

Electrophysiological recordings from single neurons are crucial for understanding the complex functioning of the brain and for developing eventual therapeutic interventions. For electrophysiology, the accuracy and fidelity of invasive implantations of small devices remains unmatched. This study introduces an innovative, cost-efficient, 3D-printed headcap with embedded microdrive (THEM) system designed to streamline the manual labor-intensive in-vivo electrode implantation process for efficient and precise multi-region brain neural probe implantations. A custom bregma-referenced headcap design and fabrication, embedded microdrive integration, and upper support structure for probe packaging are described. With the Sprague Dawley rat as test species and medial prefrontal cortex and CA1 of the dorsal hippocampus as targets, surgeries and electrophysiological recordings were conducted to test the capability of the THEM system as compared to conventional surgical methods. By shifting manual stereotaxic alignment work to pre-surgical preparation of a fully assembled headcap system, incorporating fully preassembled upper support framework for packaging management, and easy customization for specific experiment designs and probe types, our system significantly reduces the surgical time, simplifies multi-implant procedures, and enhances procedural accuracy and repeatability. The THEM system demonstrates a significant improvement over conventional surgical implantation methods and offers a promising tool for future neuroscience research.

Real-time motion artifact suppression using convolution neural networks with penalty in fNIRS

Introduction

Removing motion artifacts (MAs) from functional near-infrared spectroscopy (fNIRS) signals is crucial in practical applications, but a standard procedure is not available yet. Artificial neural networks have found applications in diverse domains, such as voice and image processing, while their utility in signal processing remains limited.

Method

In this work, we introduce an innovative neural network-based approach for online fNIRS signals processing, tailored to individual subjects and requiring minimal prior experimental data. Specifically, this approach employs one-dimensional convolutional neural networks with a penalty network (1DCNNwP), incorporating a moving window and an input data augmentation procedure. In the training process, the neural network is fed with simulated data derived from the balloon model for simulation validation and semi-simulated data for experimental validation, respectively.

Results

Visual validation underscores 1DCNNwP's capacity to effectively suppress MAs. Quantitative analysis reveals a remarkable improvement in signal-to-noise ratio by over 11.08 dB, surpassing the existing methods, including the spline-interpolation, wavelet-based, temporal derivative distribution repair with a 1 s moving window, and spline Savitzky-Goaly methods. Contrast-to-noise ratio (CNR) analysis further demonstrated 1DCNNwP's ability to restore or enhance CNRs for motionless signals. In the experiments of eight subjects, our method significantly outperformed the other approaches (except offline TDDR, t < -3.82, p < 0.01). With an average signal processing time of 0.53 ms per sample, 1DCNNwP exhibited strong potential for real-time fNIRS data processing.

Discussion

This novel univariate approach for fNIRS signal processing presents a promising avenue that requires minimal prior experimental data and adapts seamlessly to varying experimental paradigms.

Classification Algorithm for fNIRS-based Brain Signals Using Convolutional Neural Network with Spatiotemporal Feature Extraction Mechanism

Brain Computer Interface (BCI) is a highly promising human-computer interaction method that can utilize brain signals to control external devices. BCI based on functional near-infrared spectroscopy (fNIRS) is considered a relatively new and promising paradigm. fNIRS is a technique of measuring functional changes in cerebral hemodynamics. It detects changes in the hemodynamic activity of the cerebral cortex by measuring oxyhemoglobin and deoxyhemoglobin (HbR) concentrations and inversely predicts the neural activity of the brain. At the present time, Deep learning (DL) methods have not been widely used in fNIRS decoding, and there are fewer studies considering both spatial and temporal dimensions for fNIRS classification. To solve these problems, we proposed an end-to-end hybrid neural network for feature extraction of fNIRS. The method utilizes a spatial-temporal convolutional layer for automatic extraction of temporally valid information and uses a spatial attention mechanism to extract spatially localized information. A temporal convolutional network (TCN) is used to further utilize the temporal information of fNIRS before the fully connected layer. We validated our approach on a publicly available dataset including 29 subjects, including left-hand and right-hand motor imagery (MI), mental arithmetic (MA), and a baseline task. The results show that the method has few training parameters and high accuracy, providing a meaningful reference for BCI development.

Effectiveness assessment of repetitive transcranial alternating current stimulation with concurrent EEG and fNIRS measurement

Transcranial alternating current stimulation (tACS) exhibits the capability to interact with endogenous brain oscillations using an external low-intensity sinusoidal current and influences cerebral function. Despite its potential benefits, the physiological mechanisms and effectiveness of tACS are currently a subject of debate and disagreement. The aims of our study are to (i) evaluate the neurological and behavioral impact of tACS by conducting repetitive sham-controlled experiments and (ii) propose criteria to evaluate effectiveness, which can serve as a benchmark to determine optimal individual-based tACS protocols. In this study, 15 healthy adults participated in the experiment over two visiting: sham and tACS (i.e., 5 Hz, 1 mA). During each visit, we used multimodal recordings of the participants' brain, including simultaneous electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS), along with a working memory (WM) score to quantify neurological effects and cognitive changes immediately after each repetitive sham/tACS session. Our results indicate increased WM scores, hemodynamic response strength, and EEG power in theta and delta bands both during and after the tACS period. Additionally, the observed effects do not increase with prolonged stimulation time, as the effects plateau towards the end of the experiment. In conclusion, our proposed closed-loop scheme offers a promising advance for evaluating the effectiveness of tACS during the stimulation session. Specifically, the assessment criteria use participant-specific brain-based signals along with a behavioral output. Moreover, we propose a feedback efficacy score that can aid in determining the optimal stimulation duration based on a participant-specific brain state, thereby preventing the risk of overstimulation.

The Emergence of Functional Ultrasound for Noninvasive Brain-Computer Interface

A noninvasive brain-computer interface is a central task in the comprehensive analysis and understanding of the brain and is an important challenge in international brain-science research. Current implanted brain-computer interfaces are cranial and invasive, which considerably limits their applications. The development of new noninvasive reading and writing technologies will advance substantial innovations and breakthroughs in the field of brain-computer interfaces. Here, we review the theory and development of the ultrasound brain functional imaging and its applications. Furthermore, we introduce latest advancements in ultrasound brain modulation and its applications in rodents, primates, and human; its mechanism and closed-loop ultrasound neuromodulation based on electroencephalograph are also presented. Finally, high-frequency acoustic noninvasive brain-computer interface is prospected based on ultrasound super-resolution imaging and acoustic tweezers.

CGAN-rIRN: a data-augmented deep learning approach to accurate classification of mental tasks for a fNIRS-based brain-computer interface

Functional near-infrared spectroscopy (fNIRS) is increasingly used to investigate different mental tasks for brain-computer interface (BCI) control due to its excellent environmental and motion robustness. Feature extraction and classification strategy for fNIRS signal are essential to enhance the classification accuracy of voluntarily controlled BCI systems. The limitation of traditional machine learning classifiers (MLCs) lies in manual feature engineering, which is considered as one of the drawbacks that reduce accuracy. Since the fNIRS signal is a typical multivariate time series with multi-dimensionality and complexity, it makes the deep learning classifier (DLC) ideal for classifying neural activation patterns. However, the inherent bottleneck of DLCs is the requirement of substantial-scale, high-quality labeled training data and expensive computational resources to train deep networks. The existing DLCs for classifying mental tasks do not fully consider the temporal and spatial properties of fNIRS signals. Therefore, a specifically-designed DLC is desired to classify multi-tasks with high accuracy in fNIRS-BCI. To this end, we herein propose a novel data-augmented DLC to accurately classify mental tasks, which employs a convolution-based conditional generative adversarial network (CGAN) for data augmentation and a revised Inception-ResNet (rIRN) based DLC. The CGAN is utilized to generate class-specific synthetic fNIRS signals to augment the training dataset. The network architecture of rIRN is elaborately designed in accordance with the characteristics of the fNIRS signal, with serial multiple spatial and temporal feature extraction modules (FEMs), where each FEM performs deep and multi-scale feature extraction and fusion. The results of the paradigm experiments show that the proposed CGAN-rIRN approach improves the single-trial accuracy for mental arithmetic and mental singing tasks in both the data augmentation and classifier, as compared to the traditional MLCs and the commonly used DLCs. The proposed fully data-driven hybrid deep learning approach paves a promising way to improve the classification performance of volitional control fNIRS-BCI.

Translational opportunities and challenges of invasive electrodes for neural interfaces

Invasive brain-machine interfaces can restore motor, sensory and cognitive functions. However, their clinical adoption has been hindered by the surgical risk of implantation and by suboptimal long-term reliability. In this Review, we highlight the opportunities and challenges of invasive technology for clinically relevant electrophysiology. Specifically, we discuss the characteristics of neural probes that are most likely to facilitate the clinical translation of invasive neural interfaces, describe the neural signals that can be acquired or produced by intracranial electrodes, the abiotic and biotic factors that contribute to their failure, and emerging neural-interface architectures.

Impacts of simplifying articulation movements imagery to speech imagery BCI performance

Objective.Speech imagery (SI) can be used as a reliable, natural, and user-friendly activation task for the development of brain-computer interface (BCI), which empowers individuals with severe disabilities to interact with their environment. The functional near-infrared spectroscopy (fNIRS) is advanced as one of the most suitable brain imaging methods for developing BCI systems owing to its advantages of being non-invasive, portable, insensitive to motion artifacts, and having relatively high spatial resolution.Approach.To improve the classification performance of SI BCI based on fNIRS, a novel paradigm was developed in this work by simplifying the articulation movements in SI to make the articulation movement differences clearer between different words imagery tasks. A SI BCI was proposed to directly answer questions by covertly rehearsing the word '' or '' ('yes' or 'no' in English), and an unconstrained rest task also was contained in this BCI. The articulation movements of SI were simplified by retaining only the movements of the jaw and lips of vowels in Chinese Pinyin for words '' and ''.Main results.Compared with conventional speech imagery, simplifying the articulation movements in SI could generate more different brain activities among different tasks, which led to more differentiable temporal features and significantly higher classification performance. The average 3-class classification accuracies of the proposed paradigm across all 20 participants reached 69.6% and 60.2% which were about 10.8% and 5.6% significantly higher than those of the conventional SI paradigm operated in the 0-10 s and 0-2.5 s time windows, respectively.Significance.These results suggested that simplifying the articulation movements in SI is promising for improving the classification performance of intuitive BCIs based on speech imagery.

Lower limb exoskeleton robot and its cooperative control: A review, trends, and challenges for future research

Effective control of an exoskeleton robot (ER) using a human-robot interface is crucial for assessing the robot's movements and the force they produce to generate efficient control signals. Interestingly, certain surveys were done to show off cutting-edge exoskeleton robots. The review papers that were previously published have not thoroughly examined the control strategy, which is a crucial component of automating exoskeleton systems. As a result, this review focuses on examining the most recent developments and problems associated with exoskeleton control systems, particularly during the last few years (2017-2022). In addition, the trends and challenges of cooperative control, particularly multi-information fusion, are discussed.

Forecasting Unplanned Purchase Behavior under Buy-One Get-One-Free Promotions Using Functional Near-Infrared Spectroscopy

It is very important for consumers to recognize their wrong shopping habits such as unplanned purchase behavior (UPB). The traditional methods used for measuring the UPB in qualitative and quantitative studies have some drawbacks because of human perception and memory. We proposed a UPB identification methodology applied with the brain-computer interface technique using a support vector machine (SVM) along with a functional near-infrared spectroscopy (fNIRS). Hemodynamic signals and behavioral data were collected from 33 subjects by performing Task 1 which included the Buy-One-Get-One-Free (BOGOF) and Task 2 which excluded the BOGOF condition. The acquired data were calculated with 6 time-domain features and then classified them using SVM with 10-cross validations. Thereafter, we evaluated whether the results were reliable using the area under the receiver operating characteristic curve (AUC). As a result, we achieved average accuracy greater than 94%, which is reliable because of the AUC values above 0.97. We found that the UPB brain activity was more relevant to Task 1 with the BOGOF condition than with Task 2 in the prefrontal cortex. UPBs were sufficiently derived from self-reported measurement, indicating that the subjects perceived increased impulsivity in the BOGOF condition. Therefore, this study improves the detection and understanding of UPB as a path for a computer-aided detection perspective for rating the severity of UPBs.

A comparative analysis of masking empirical mode decomposition and a neural network with feed-forward and back propagation along with masking empirical mode decomposition to improve the classification performance for a reliable brain-computer interface

In general, extraction and classification are used in various fields like image processing, pattern recognition, signal processing, and so on. Extracting effective characteristics from raw electroencephalogram (EEG) signals is a crucial role of the brain-computer interface for motor imagery. Recently, there has been a great deal of focus on motor imagery in the EEG signals since they encode a person's intent to do an action. Researchers have been using MI signals to assist paralyzed people and even move them on their own with certain equipment, like wheelchairs. As a result, proper decoding is an important step required for the interconnection of the brain and the computer. EEG decoding is a challenging process because of poor SNR, complexity, and other reasons. However, choosing an appropriate method to extract the features to improve the performance of motor imagery recognition is still a research hotspot. To extract the features of the EEG signal in the classification task, this paper proposes a Masking Empirical Mode Decomposition (MEMD) based Feed Forward Back Propagation Neural Network (MEMD-FFBPNN). The dataset consists of EEG signals which are first normalized using the minimax method and given as input to the MEMD to extract the features and then given to the FFBPNN to classify the tasks. The accuracy of the proposed method MEMD-FFBPNN has been measured using the confusion matrix, mean square error and which has been recorded up to 99.9%. Thus, the proposed method gives better accuracy than the other conventional methods.

Combining robust level extraction and unsupervised adaptive classification for high-accuracy fNIRS-BCI: An evidence on single-trial differentiation between mentally arithmetic- and singing-tasks

Functional near-infrared spectroscopy (fNIRS) is a safe and non-invasive optical imaging technique that is being increasingly used in brain-computer interfaces (BCIs) to recognize mental tasks. Unlike electroencephalography (EEG) which directly measures neural activation, fNIRS signals reflect neurovascular-coupling inducing hemodynamic response that can be slow in time and varying in the pattern. The established classifiers extend the EEG-ones by mostly employing the feature based supervised models such as the support vector machine (SVM) and linear discriminant analysis (LDA), and fail to timely characterize the level-sensitive hemodynamic pattern. A dedicated classifier is desired for intentional activity recognition of fNIRS-BCI, including the adaptive acquisition of response relevant features and accurate discrimination of implied ideas. To this end, we herein propose a specifically-designed joint adaptive classification method that combines a Kalman filtering (KF) for robust level extraction and an adaptive Gaussian mixture model (a-GMM) for enhanced pattern recognition. The simulative investigations and paradigm experiments have shown that the proposed KF/a-GMM classification method can effectively track the random variations of task-evoked brain activation patterns, and improve the accuracy of single-trial classification task of mental arithmetic vs. mental singing, as compared to the conventional methods, e.g., those that employ combinations of the band-pass filtering (BPF) based feature extractors (mean, slope, and variance, etc.) and the classical recognizers (GMM, SVM, and LDA). The proposed approach paves a promising way for developing the real-time fNIRS-BCI technique.

EEG-fNIRS-based hybrid image construction and classification using CNN-LSTM

The constantly evolving human–machine interaction and advancement in sociotechnical systems have made it essential to analyze vital human factors such as mental workload, vigilance, fatigue, and stress by monitoring brain states for optimum performance and human safety. Similarly, brain signals have become paramount for rehabilitation and assistive purposes in fields such as brain–computer interface (BCI) and closed-loop neuromodulation for neurological disorders and motor disabilities. The complexity, non-stationary nature, and low signal-to-noise ratio of brain signals pose significant challenges for researchers to design robust and reliable BCI systems to accurately detect meaningful changes in brain states outside the laboratory environment. Different neuroimaging modalities are used in hybrid settings to enhance accuracy, increase control commands, and decrease the time required for brain activity detection. Functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) measure the hemodynamic and electrical activity of the brain with a good spatial and temporal resolution, respectively. However, in hybrid settings, where both modalities enhance the output performance of BCI, their data compatibility due to the huge discrepancy between their sampling rate and the number of channels remains a challenge for real-time BCI applications. Traditional methods, such as downsampling and channel selection, result in important information loss while making both modalities compatible. In this study, we present a novel recurrence plot (RP)-based time-distributed convolutional neural network and long short-term memory (CNN-LSTM) algorithm for the integrated classification of fNIRS EEG for hybrid BCI applications. The acquired brain signals are first projected into a non-linear dimension with RPs and fed into the CNN to extract essential features without performing any downsampling. Then, LSTM is used to learn the chronological features and time-dependence relation to detect brain activity. The average accuracies achieved with the proposed model were 78.44% for fNIRS, 86.24% for EEG, and 88.41% for hybrid EEG-fNIRS BCI. Moreover, the maximum accuracies achieved were 85.9, 88.1, and 92.4%, respectively. The results confirm the viability of the RP-based deep-learning algorithm for successful BCI systems.

Non-invasive transcranial electrical brain stimulation guided by functional near-infrared spectroscopy for targeted neuromodulation: A review

One of the primary goals in cognitive neuroscience is to understand the neural mechanisms on which cognition is based. Researchers are trying to find how cognitive mechanisms are related to oscillations generated due to brain activity. The research focused on this topic has been considerably aided by developing non-invasive brain stimulation techniques. The dynamics of brain networks and the resultant behavior can be affected by non-invasive brain stimulation techniques, which make their use a focus of interest in many experiments and clinical fields. One essential non-invasive brain stimulation technique is transcranial electrical stimulation (tES), subdivided into transcranial direct and alternating current stimulation. tES has recently become more well-known because of the effective results achieved in treating chronic conditions. In addition, there has been exceptional progress in the interpretation and feasibility of tES techniques. Summarizing the beneficial effects of tES, this article provides an updated depiction of what has been accomplished to date, brief history, and the open questions that need to be addressed in the future. An essential issue in the field of tES is stimulation duration. This review briefly covers the stimulation durations that have been utilized in the field while monitoring the brain using functional-near infrared spectroscopy-based brain imaging.

Kernel Temporal Differences for EEG-based Reinforcement Learning Brain Machine Interfaces

Kernel temporal differences (KTD) (λ) algorithm integrated in Q-learning (Q-KTD) has shown its applicability and feasibility for reinforcement learning brain machine interfaces (RLBMIs). RLBMI with its unique learning strategy based on trial-error allows continuous learning and adaptation in BMIs. Q-KTD has shown good performance in both open and closed-loop experiments for finding a proper mapping from neural intention to control commands of an external device. However, previous studies have been limited to intracortical BMIs where monkey's firing rates from primary motor cortex were used as inputs to the neural decoder. This study provides the first attempt to investigate Q-KTD algorithm's applicability in EEG-based RLBMIs. Two different publicly available EEG data sets are considered, we refer to them as Data set A and Data set B. EEG motor imagery tasks are integrated in a single step center-out reaching task, and we observe the open-loop RLBMI experiments reach 100% average success rates after sufficient learning experience. Data set A converges after approximately 20 epochs for raw features and Data set B shows convergence after approximately 40 epochs for both raw and Fourier transform features. Although there still exist challenges to overcome in EEG-based RLBMI using Q-KTD, including increasing the learning speed, and optimization of a continuously growing number of kernel units, the results encourage further investigation of Q-KTD in closed-loop RLBMIs using EEG. Clinical Relevance- This study supports feasibility of noninvasive EEG-based RLBMI implementations and addresses benefits and challenges of RLBMI using EEG.

Intelligent Classification Technique of Hand Motor Imagery Using EEG Beta Rebound Follow-Up Pattern

To apply EEG-based brain-machine interfaces during rehabilitation, separating various tasks during motor imagery (MI) and assimilating MI into motor execution (ME) are needed. Previous studies were focusing on classifying different MI tasks based on complex algorithms. In this paper, we implement intelligent, straightforward, comprehensible, time-efficient, and channel-reduced methods to classify ME versus MI and left- versus right-hand MI. EEG of 30 healthy participants undertaking motional tasks is recorded to investigate two classification tasks. For the first task, we first propose a “follow-up” pattern based on the beta rebound. This method achieves an average classification accuracy of 59.77% ± 11.95% and can be up to 89.47% for finger-crossing. Aside from time-domain information, we map EEG signals to feature space using extraction methods including statistics, wavelet coefficients, average power, sample entropy, and common spatial patterns. To evaluate their practicability, we adopt a support vector machine as an intelligent classifier model and sparse logistic regression as a feature selection technique and achieve 79.51% accuracy. Similar approaches are taken for the second classification reaching 75.22% accuracy. The classifiers we propose show high accuracy and intelligence. The achieved results make our approach highly suitable to be applied to the rehabilitation of paralyzed limbs.

Unmanned Aerial Vehicle for Laser Based Biomedical Sensor Development and Examination of Device Trajectory

Controller design and signal processing for the control of air-vehicles have gained extreme importance while interacting with humans to form a brain–computer interface. This is because fewer commands need to be mapped into multiple controls. For our anticipated biomedical sensor for breath analysis, it is mandatory to provide medication to the patients on an urgent basis. To address this increasingly tense situation in terms of emergencies, we plan to design an unmanned vehicle that can aid spontaneously to monitor the person’s health, and help the physician spontaneously during the rescue mission. Simultaneously, that must be done in such a computationally efficient algorithm that the minimum amount of energy resources are consumed. For this purpose, we resort to an unmanned logistic air-vehicle which flies from the medical centre to the affected person. After obtaining restricted permission from the regional administration, numerous challenges are identified for this design. The device is able to lift a weight of 2 kg successfully which is required for most emergency medications, while choosing the smallest distance to the destination with the GPS. By recording the movement of the vehicle in numerous directions, the results deviate to a maximum of 2% from theoretical investigations. In this way, our biomedical sensor provides critical information to the physician, who is able to provide medication to the patient urgently. On account of reasonable supply of medicines to the destination in terms of weight and time, this experimentation has been rendered satisfactory by the relevant physicians in the vicinity.

An Energy-Efficient Wearable Functional Near-infrared Spectroscopy System Employing Dual-level Adaptive Sampling Technique

Functional near-infrared spectroscopy (fNIRS) is a powerful medical imaging tool in brain science and psychology, it can also be employed in brain-computer interface (BCI) due to its noninvasive and artifact-less-sensitive characteristics. Conventional ways to detect large-area brain activity using near-infrared (NIR) technology are based on Time-division or Frequency-division modulation technique, which traverses all physical sensory channels in a specific period. To achieve higher imaging resolution or brain-tasks classification accuracy, the NIRS system require higher density and more channels, which conflict with the limited battery capacity. Inspired by the functional atlas of the human brain, this paper proposes a spatial adaptive sampling (SAS) method. It can change the active channel pattern of the fNIRS system to match with the real-time brain activity, to increase the energy efficiency without significant reduction on the brain imaging quality or the accuracy of brain activity classification. Therefore, the number of the averaging enabled channels will be dramatically reduced in practice. To verify the proposed SAS technique, a wearable and flexible NIRS system has been implemented, in which each channel of light-emitting diode (LED) drive circuits and photodiode (PD) detection circuits can be power gated independently. Brain task experiments have been conducted to validate the proposed method, the power consumption of the LED drive module is reduced by 46.58% compared to that without SAS technology while maintaining an average brain imaging PSNR (Peak Signal to Noise Ratio) of 35 dB. The brain-task classification accuracy is 80.47%, which has a 2.67% reduction compared to that without the SAS technique.

Acupuncture enhances brain function in patients with mild cognitive impairment: evidence from a functional-near infrared spectroscopy study

Mild cognitive impairment (MCI) is a precursor to Alzheimer’s disease. It is imperative to develop a proper treatment for this neurological disease in the aging society. This observational study investigated the effects of acupuncture therapy on MCI patients. Eleven healthy individuals and eleven MCI patients were recruited for this study. Oxy- and deoxy-hemoglobin signals in the prefrontal cortex during working-memory tasks were monitored using functional near-infrared spectroscopy. Before acupuncture treatment, working-memory experiments were conducted for healthy control (HC) and MCI groups (MCI-0), followed by 24 sessions of acupuncture for the MCI group. The acupuncture sessions were initially carried out for 6 weeks (two sessions per week), after which experiments were performed again on the MCI group (MCI-1). This was followed by another set of acupuncture sessions that also lasted for 6 weeks, after which the experiments were repeated on the MCI group (MCI-2). Statistical analyses of the signals and classifications based on activation maps as well as temporal features were performed. The highest classification accuracies obtained using binary connectivity maps were 85.7% HC vs. MCI-0, 69.5% HC vs. MCI-1, and 61.69% HC vs. MCI-2. The classification accuracies using the temporal features mean from 5 seconds to 28 seconds and maximum (i.e, max(5:28 seconds)) values were 60.6% HC vs. MCI-0, 56.9% HC vs. MCI-1, and 56.4% HC vs. MCI-2. The results reveal that there was a change in the temporal characteristics of the hemodynamic response of MCI patients due to acupuncture. This was reflected by a reduction in the classification accuracy after the therapy, indicating that the patients’ brain responses improved and became comparable to those of healthy subjects. A similar trend was reflected in the classification using the image feature. These results indicate that acupuncture can be used for the treatment of MCI patients.

Review on Patient-Cooperative Control Strategies for Upper-Limb Rehabilitation Exoskeletons

Technology-supported rehabilitation therapy for neurological patients has gained increasing interest since the last decades. The literature agrees that the goal of robots should be to induce motor plasticity in subjects undergoing rehabilitation treatment by providing the patients with repetitive, intensive, and task-oriented treatment. As a key element, robot controllers should adapt to patients’ status and recovery stage. Thus, the design of effective training modalities and their hardware implementation play a crucial role in robot-assisted rehabilitation and strongly influence the treatment outcome. The objective of this paper is to provide a multi-disciplinary vision of patient-cooperative control strategies for upper-limb rehabilitation exoskeletons to help researchers bridge the gap between human motor control aspects, desired rehabilitation training modalities, and their hardware implementations. To this aim, we propose a three-level classification based on 1) “high-level” training modalities, 2) “low-level” control strategies, and 3) “hardware-level” implementation. Then, we provide examples of literature upper-limb exoskeletons to show how the three levels of implementation have been combined to obtain a given high-level behavior, which is specifically designed to promote motor relearning during the rehabilitation treatment. Finally, we emphasize the need for the development of compliant control strategies, based on the collaboration between the exoskeleton and the wearer, we report the key findings to promote the desired physical human-robot interaction for neurorehabilitation, and we provide insights and suggestions for future works.

Most favorable stimulation duration in the sensorimotor cortex for fNIRS-based BCI

One of the primary objectives of the brain-computer interface (BCI) is to obtain a command with higher classification accuracy within the shortest possible time duration. Therefore, this study evaluates several stimulation durations to propose a duration that can yield the highest classification accuracy. Furthermore, this study aims to address the inherent delay in the hemodynamic responses (HRs) for the command generation time. To this end, HRs in the sensorimotor cortex were evaluated for the functional near-infrared spectroscopy (fNIRS)-based BCI. To evoke brain activity, right-hand-index finger poking and tapping tasks were used. In this study, six different stimulation durations (i.e., 1, 3, 5, 7, 10, and 15 s) were tested on 10 healthy male subjects. Upon stimulation, different temporal features and multiple time windows were utilized to extract temporal features. The extracted features were then classified using linear discriminant analysis. The classification results using the main HR showed that a 5 s stimulation duration could yield the highest classification accuracy, i.e., 74%, with a combination of the mean and maximum value features. However, the results were not significantly different from the classification accuracy obtained using the 15 s stimulation. To further validate the results, a classification using the initial dip was performed. The results obtained endorsed the finding with an average classification accuracy of 73.5% using the features of minimum peak and skewness in the 5 s window. The results based on classification using the initial dip for 5 s were significantly different from all other tested stimulation durations (p < 0.05) for all feature combinations. Moreover, from the visual inspection of the HRs, it is observed that the initial dip occurred as soon as the task started, but the main HR had a delay of more than 2 s. Another interesting finding is that impulsive stimulation in the sensorimotor cortex can result in the generation of a clearer initial dip phenomenon. The results reveal that the command for the fNIRS-based BCI can be generated using the 5 s stimulation duration. In conclusion, the use of the initial dip can reduce the time taken for the generation of commands and can be used to achieve a higher classification accuracy for the fNIRS-BCI within a 5 s task duration rather than relying on longer durations.

Recognition of Flexion and Extension Imagery Involving the Right and Left Arms Based on Deep Belief Network and Functional Near-Infrared Spectroscopy

Brain-computer interaction based on motor imagery (MI) is an important brain-computer interface (BCI). Most methods for MI classification are based on electroencephalogram (EEG), and few studies have investigated signal processing based on MI-Functional Near-Infrared Spectroscopy (fNIRS). In addition, there is a need to improve the classification accuracy for MI fNIRS methods. In this study, a deep belief network (DBN) based on a restricted Boltzmann machine (RBM) was used to classify fNIRS signals of flexion and extension imagery involving the left and right arms. fNIRS signals from 16 channels covering the motor cortex area were recorded for each of 10 subjects executing or imagining flexion and extension involving the left and right arms. Oxygenated hemoglobin (HbO) concentration was used as a feature to train two RBMs that were subsequently stacked with an additional softmax regression output layer to construct DBN. We also explored the DBN model classification accuracy for the test dataset from one subject using training dataset from other subjects. The average DBN classification accuracy for flexion and extension movement and imagery involving the left and right arms was 84.35 ± 3.86% and 78.19 ± 3.73%, respectively. For a given DBN model, better classification results are obtained for test datasets for a given subject when the model is trained using dataset from the same subject than when the model is trained using datasets from other subjects. The results show that the DBN algorithm can effectively identify flexion and extension imagery involving the right and left arms using fNIRS. This study is expected to serve as a reference for constructing online MI-BCI systems based on DBN and fNIRS.

Systemic Review on Transcranial Electrical Stimulation Parameters and EEG/fNIRS Features for Brain Diseases

Background

Brain disorders are gradually becoming the leading cause of death worldwide. However, the lack of knowledge of brain disease’s underlying mechanisms and ineffective neuropharmacological therapy have led to further exploration of optimal treatments and brain monitoring techniques.

Objective

This study aims to review the current state of brain disorders, which utilize transcranial electrical stimulation (tES) and daily usable noninvasive neuroimaging techniques. Furthermore, the second goal of this study is to highlight available gaps and provide a comprehensive guideline for further investigation.

Method

A systematic search was conducted of the PubMed and Web of Science databases from January 2000 to October 2020 using relevant keywords. Electroencephalography (EEG) and functional near-infrared spectroscopy were selected as noninvasive neuroimaging modalities. Nine brain disorders were investigated in this study, including Alzheimer’s disease, depression, autism spectrum disorder, attention-deficit hyperactivity disorder, epilepsy, Parkinson’s disease, stroke, schizophrenia, and traumatic brain injury.

Results

Sixty-seven studies (1,385 participants) were included for quantitative analysis. Most of the articles (82.6%) employed transcranial direct current stimulation as an intervention method with modulation parameters of 1 mA intensity (47.2%) for 16–20 min (69.0%) duration of stimulation in a single session (36.8%). The frontal cortex (46.4%) and the cerebral cortex (47.8%) were used as a neuroimaging modality, with the power spectrum (45.7%) commonly extracted as a quantitative EEG feature.

Conclusion

An appropriate stimulation protocol applying tES as a therapy could be an effective treatment for cognitive and neurological brain disorders. However, the optimal tES criteria have not been defined; they vary across persons and disease types. Therefore, future work needs to investigate a closed-loop tES with monitoring by neuroimaging techniques to achieve personalized therapy for brain disorders.

Motor Training Using Mental Workload (MWL) With an Assistive Soft Exoskeleton System: A Functional Near-Infrared Spectroscopy (fNIRS) Study for Brain-Machine Interface (BMI)

Mental workload is a neuroergonomic human factor, which is widely used in planning a system's safety and areas like brain-machine interface (BMI), neurofeedback, and assistive technologies. Robotic prosthetics methodologies are employed for assisting hemiplegic patients in performing routine activities. Assistive technologies' design and operation are required to have an easy interface with the brain with fewer protocols, in an attempt to optimize mobility and autonomy. The possible answer to these design questions may lie in neuroergonomics coupled with BMI systems. In this study, two human factors are addressed: designing a lightweight wearable robotic exoskeleton hand that is used to assist the potential stroke patients with an integrated portable brain interface using mental workload (MWL) signals acquired with portable functional near-infrared spectroscopy (fNIRS) system. The system may generate command signals for operating a wearable robotic exoskeleton hand using two-state MWL signals. The fNIRS system is used to record optical signals in the form of change in concentration of oxy and deoxygenated hemoglobin (HbO and HbR) from the pre-frontal cortex (PFC) region of the brain. Fifteen participants participated in this study and were given hand-grasping tasks. Two-state MWL signals acquired from the PFC region of the participant's brain are segregated using machine learning classifier-support vector machines (SVM) to utilize in operating a robotic exoskeleton hand. The maximum classification accuracy is 91.31%, using a combination of mean-slope features with an average information transfer rate (ITR) of 1.43. These results show the feasibility of a two-state MWL (fNIRS-based) robotic exoskeleton hand (BMI system) for hemiplegic patients assisting in the physical grasping tasks.

Subject-Independent Functional Near-Infrared Spectroscopy-Based Brain-Computer Interfaces Based on Convolutional Neural Networks

Functional near-infrared spectroscopy (fNIRS) has attracted increasing attention in the field of brain–computer interfaces (BCIs) owing to their advantages such as non-invasiveness, user safety, affordability, and portability. However, fNIRS signals are highly subject-specific and have low test-retest reliability. Therefore, individual calibration sessions need to be employed before each use of fNIRS-based BCI to achieve a sufficiently high performance for practical BCI applications. In this study, we propose a novel deep convolutional neural network (CNN)-based approach for implementing a subject-independent fNIRS-based BCI. A total of 18 participants performed the fNIRS-based BCI experiments, where the main goal of the experiments was to distinguish a mental arithmetic task from an idle state task. Leave-one-subject-out cross-validation was employed to evaluate the average classification accuracy of the proposed subject-independent fNIRS-based BCI. As a result, the average classification accuracy of the proposed method was reported to be 71.20 ± 8.74%, which was higher than the threshold accuracy for effective BCI communication (70%) as well as that obtained using conventional shrinkage linear discriminant analysis (65.74 ± 7.68%). To achieve a classification accuracy comparable to that of the proposed subject-independent fNIRS-based BCI, 24 training trials (of approximately 12 min) were necessary for the traditional subject-dependent fNIRS-based BCI. It is expected that our CNN-based approach would reduce the necessity of long-term individual calibration sessions, thereby enhancing the practicality of fNIRS-based BCIs significantly.

Decoding of Walking Imagery and Idle State Using Sparse Representation Based on fNIRS

Objectives

Brain-computer interface (BCI) based on functional near-infrared spectroscopy (fNIRS) is expected to provide an optional active rehabilitation training method for patients with walking dysfunction, which will affect their quality of life seriously. Sparse representation classification (SRC) oxyhemoglobin (HbO) concentration was used to decode walking imagery and idle state to construct fNIRS-BCI based on walking imagery.

Methods

15 subjects were recruited and fNIRS signals were collected during walking imagery and idle state. Firstly, band-pass filtering and baseline drift correction for HbO signal were carried out, and then the mean value, peak value, and root mean square (RMS) of HbO and their combinations were extracted as classification features; SRC was used to identify the extracted features and the result of SRC was compared with those of support vector machine (SVM), K-Nearest Neighbor (KNN), linear discriminant analysis (LDA), and logistic regression (LR).

Results

The experimental results showed that the average classification accuracy for walking imagery and idle state by SRC using three features combination was 91.55±3.30%, which was significantly higher than those of SVM, KNN, LDA, and LR (86.37±4.42%, 85.65±5.01%, 86.43±4.41%, and 76.14±5.32%, respectively), and the classification accuracy of other combined features was higher than that of single feature.

Conclusions

The study showed that introducing SRC into fNIRS-BCI can effectively identify walking imagery and idle state. It also showed that different time windows for feature extraction have an impact on the classification results, and the time window of 2–8 s achieved a better classification accuracy (94.33±2.60%) than other time windows. Significance. The study was expected to provide a new and optional active rehabilitation training method for patients with walking dysfunction. In addition, the experiment was also a rare study based on fNIRS-BCI using SRC to decode walking imagery and idle state.

Quantitative Assessment of Resting-State for Mild Cognitive Impairment Detection: A Functional Near-Infrared Spectroscopy and Deep Learning Approach

BACKGROUND

Mild cognitive impairment (MCI) is considered a prodromal stage of Alzheimer's disease. Early diagnosis of MCI can allow for treatment to improve cognitive function and reduce modifiable risk factors.

OBJECTIVE

This study aims to investigate the feasibility of individual MCI detection from healthy control (HC) using a minimum duration of resting-state functional near-infrared spectroscopy (fNIRS) signals.

METHODS

In this study, nine different measurement durations (i.e., 30, 60, 90, 120, 150, 180, 210, 240, and 270 s) were evaluated for MCI detection via the graph theory analysis and traditional machine learning approach, such as linear discriminant analysis, support vector machine, and K-nearest neighbor algorithms. Moreover, feature representation- and classification-based transfer learning (TL) methods were applied to identify MCI from HC through the input of connectivity maps with 30 and 90 s duration.

RESULTS

There was no significant difference among the nine various time windows in the machine learning and graph theory analysis. The feature representation-based TL showed improved accuracy in both 30 and 90 s cases (i.e., 30 s: 81.27% and 90 s: 76.73%). Notably, the classification-based TL method achieved the highest accuracy of 95.81% using the pre-trained convolutional neural network (CNN) model with the 30 s interval functional connectivity map input.

CONCLUSION

The results indicate that a 30 s measurement of the resting-state with fNIRS could be used to detect MCI. Moreover, the combination of neuroimaging (e.g., functional connectivity maps) and deep learning methods (e.g., CNN and TL) can be considered as novel biomarkers for clinical computer-assisted MCI diagnosis.

Domain adaptation for robust workload level alignment between sessions and subjects using fNIRS

SIGNIFICANCE

We demonstrated the potential of using domain adaptation on functional near-infrared spectroscopy (fNIRS) data to classify different levels of n-back tasks that involve working memory.

AIM

Domain shift in fNIRS data is a challenge in the workload level alignment across different experiment sessions and subjects. To address this problem, two domain adaptation approaches-Gromov-Wasserstein (G-W) and fused Gromov-Wasserstein (FG-W) were used.

APPROACH

Specifically, we used labeled data from one session or one subject to classify trials in another session (within the same subject) or another subject. We applied G-W for session-by-session alignment and FG-W for subject-by-subject alignment to fNIRS data acquired during different n-back task levels. We compared these approaches with three supervised methods: multiclass support vector machine (SVM), convolutional neural network (CNN), and recurrent neural network (RNN).

RESULTS

In a sample of six subjects, G-W resulted in an alignment accuracy of 68  %    ±  4  %   (weighted mean ± standard error) for session-by-session alignment, FG-W resulted in an alignment accuracy of 55  %    ±  2  %   for subject-by-subject alignment. In each of these cases, 25% accuracy represents chance. Alignment accuracy results from both G-W and FG-W are significantly greater than those from SVM, CNN, and RNN. We also showed that removal of motion artifacts from the fNIRS data plays an important role in improving alignment performance.

CONCLUSIONS

Domain adaptation has potential for session-by-session and subject-by-subject alignment of mental workload by using fNIRS data.

Prefrontal Asymmetry BCI Neurofeedback Datasets

Prefrontal cortex (PFC) asymmetry is an important marker in affective neuroscience and has attracted significant interest, having been associated with studies of motivation, eating behavior, empathy, risk propensity, and clinical depression. The data presented in this paper are the result of three different experiments using PFC asymmetry neurofeedback (NF) as a Brain-Computer Interface (BCI) paradigm, rather than a therapeutic mechanism aiming at long-term effects, using functional near-infrared spectroscopy (fNIRS) which is known to be particularly well-suited to the study of PFC asymmetry and is less sensitive to artifacts. From an experimental perspective the BCI context brings more emphasis on individual subjects' baselines, successful and sustained activation during epochs, and minimal training. The subject pool is also drawn from the general population, with less bias toward specific behavioral patterns, and no inclusion of any patient data. We accompany our datasets with a detailed description of data formats, experiment and protocol designs, as well as analysis of the individualized metrics for definitions of success scores based on baseline thresholds as well as reference tasks. The work presented in this paper is the result of several experiments in the domain of BCI where participants are interacting with continuous visual feedback following a real-time NF paradigm, arising from our long-standing research in the field of affective computing. We offer the community access to our fNIRS datasets from these experiments. We specifically provide data drawn from our empirical studies in the field of affective interactions with computer-generated narratives as well as interfacing with algorithms, such as heuristic search, which all provide a mechanism to improve the ability of the participants to engage in active BCI due to their realistic visual feedback. Beyond providing details of the methodologies used where participants received real-time NF of left-asymmetric increase in activation in their dorsolateral prefrontal cortex (DLPFC), we re-establish the need for carefully designing protocols to ensure the benefits of NF paradigm in BCI are enhanced by the ability of the real-time visual feedback to adapt to the individual responses of the participants. Individualized feedback is paramount to the success of NF in BCIs.

Immediate effects of visual-motor illusion on resting-state functional connectivity

Visual-motor illusion (VMI) is to evoke a kinesthetic sensation by viewing images of oneself performing physical exercise while the body is at rest. Previous studies demonstrated that VMI activates the motor association brain areas; however, it is unclear whether VMI immediately alters the resting-state functional connectivity (RSFC). This study is aimed to verify whether the VMI induction changed the RSFC using functional near-infrared spectroscopy (fNIRS). The right hands of 13 healthy adults underwent illusion and observation conditions for 20 min each. Before and after each condition, RSFC was measured using fNIRS. After each condition, degree of kinesthetic illusion and a sense of body ownership measured using the Likert scale. Our results indicated that, compared with the observation condition, the degree of kinesthetic illusion and the sense of body ownership were significantly higher after the illusion condition. Compared with the observation condition, RSFC after the illusion condition significantly increased brain areas associated with kinesthetic illusion, a sense of body ownership, and motor execution. In conclusion, RSFC has become a biomarker that shows changes in brain function occurring due to VMI. VMI may be applied to the treatment of patients with stroke or orthopedic diseases.

Detection of Mild Cognitive Impairment Using Convolutional Neural Network: Temporal-Feature Maps of Functional Near-Infrared Spectroscopy

Mild cognitive impairment (MCI) is the clinical precursor of Alzheimer's disease (AD), which is considered the most common neurodegenerative disease in the elderly. Some MCI patients tend to remain stable over time and do not evolve to AD. It is essential to diagnose MCI in its early stages and provide timely treatment to the patient. In this study, we propose a neuroimaging approach to identify MCI using a deep learning method and functional near-infrared spectroscopy (fNIRS). For this purpose, fifteen MCI subjects and nine healthy controls (HCs) were asked to perform three mental tasks: N-back, Stroop, and verbal fluency (VF) tasks. Besides examining the oxygenated hemoglobin changes (ΔHbO) in the region of interest, ΔHbO maps at 13 specific time points (i.e., 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, and 65 s) during the tasks and seven temporal feature maps (i.e., two types of mean, three types of slope, kurtosis, and skewness) in the prefrontal cortex were investigated. A four-layer convolutional neural network (CNN) was applied to identify the subjects into either MCI or HC, individually, after training the CNN model with ΔHbO maps and temporal feature maps above. Finally, we used the 5-fold cross-validation approach to evaluate the performance of the CNN. The results of temporal feature maps exhibited high classification accuracies: The average accuracies for the N-back task, Stroop task, and VFT, respectively, were 89.46, 87.80, and 90.37%. Notably, the highest accuracy of 98.61% was achieved from the ΔHbO slope map during 20-60 s interval of N-back tasks. Our results indicate that the fNIRS imaging approach based on temporal feature maps is a promising diagnostic method for early detection of MCI and can be used as a tool for clinical doctors to identify MCI from their patients.