fNIRS-based brain-computer interfaces: a review

Random Subspace Ensemble Learning for Functional Near-Infrared Spectroscopy Brain-Computer Interfaces

The feasibility of the random subspace ensemble learning method was explored to improve the performance of functional near-infrared spectroscopy-based brain-computer interfaces (fNIRS-BCIs). Feature vectors have been constructed using the temporal characteristics of concentration changes in fNIRS chromophores such as mean, slope, and variance to implement fNIRS-BCIs systems. The mean and slope, which are the most popular features in fNIRS-BCIs, were adopted. Linear support vector machine and linear discriminant analysis were employed, respectively, as a single strong learner and multiple weak learners. All features in every channel and available time window were employed to train the strong learner, and the feature subsets were selected at random to train multiple weak learners. It was determined that random subspace ensemble learning is beneficial to enhance the performance of fNIRS-BCIs.

Hemodynamic responses during standing and sitting activities: a study toward fNIRS-BCI

In this paper, we utilized functional near-infrared spectroscopy (fNIRS) technology to examine the hemodynamic responses in the motor cortex for two conditions, namely standing and sitting tasks. Nine subjects performed five trials of standing and sitting (SAS) tasks with both real movements and imagery thinking of SAS. A group level of statistical parametric mapping (SPM) analysis during these tasks showed bilateral activation of oxy-hemoglobin for both real movements and imagery experiments. Interestingly, the SPM analysis clearly revealed that the sitting tasks induced a higher oxy-hemoglobin level activation compared to the standing task. Remarkably, this finding is persistent across the 22 measured channels at the individual and group levels for both experiments. Furthermore, six features were extracted from pre-processed HbO signals and the performance of four different classifiers was examined in order to test the viability of using SAS tasks in future fNIRS-brain-computer interface (fNIRS-BCI) systems. In particular, two features-combination tests revealed that the signal slope with signal variance represents one of the three best two-combined features for its consistency in providing high accuracy results for both real and imagery experiments. This study shows the potential of implementing such tasks into the fNIRS-BCI system. In the future, the results of this work could pave the way towards the application of fNIRS-BCI in lower limb rehabilitation.

fNIRS Complexity Analysis for the Assessment of Motor Imagery and Mental Arithmetic Tasks

Conventional methods for analyzing functional near-infrared spectroscopy (fNIRS) signals primarily focus on characterizing linear dynamics of the underlying metabolic processes. Nevertheless, linear analysis may underrepresent the true physiological processes that fully characterizes the complex and nonlinear metabolic activity sustaining brain function. Although there have been recent attempts to characterize nonlinearities in fNIRS signals in various experimental protocols, to our knowledge there has yet to be a study that evaluates the utility of complex characterizations of fNIRS in comparison to standard methods, such as the mean value of hemoglobin. Thus, the aim of this study was to investigate the entropy of hemoglobin concentration time series obtained from fNIRS signals and perform a comparitive analysis with standard mean hemoglobin analysis of functional activation. Publicly available data from 29 subjects performing motor imagery and mental arithmetics tasks were exploited for the purpose of this study. The experimental results show that entropy analysis on fNIRS signals may potentially uncover meaningful activation areas that enrich and complement the set identified through a traditional linear analysis.

Design of a wearable four-channel near-infrared spectroscopy system for the measurement of brain hemodynamic responses

Objectives This work describes the design and development of a four-channel near-infrared spectroscopy system to detect the oxygenated and deoxygenated hemoglobin concentration changes in the brain during various motor tasks. Methods The system uses light-emitting diodes corresponding to two wavelengths of 760 nm and 850 nm sensitive to deoxygenated and oxygenated hemoglobin concentration changes, respectively. The response is detected using a photodetector with an integrated transimpedance amplifier. The system is designed with four channels for functional near-infrared spectroscopy (fNIRS) signals acquisition. Two experiments were conducted to demonstrate the ability of the system to detect the changes in hemodynamic responses of different tasks. In the first experiment, the hemodynamic changes during motor execution and imagery of right- and left-fist clenching tasks were acquired by the developed system and validated against a standard multichannel NIRS system. In another experiment, the fNIRS signals during rest and motor execution of right-fist clenching task were acquired using the system and classified. Results The results demonstrate the ability of the designed system to detect the brain hemodynamic changes during various tasks. Also, the activation patterns obtained by the developed system with a minimum number of channels are on par with those obtained by the commercial system. Conclusions The developed four-channel NIRS system is user-friendly and has been designed with inexpensive components, unlike the commercially available NIRS instruments that are cumbersome and expensive.

Complexity Analysis on Functional-Near Infrared Spectroscopy Time Series: a Preliminary Study on Mental Arithmetic

It is well known that physiological systems show complex and nonlinear behaviours. In spite of that, functional near-infrared spectroscopy (fNIRS) is usually analyzed in the time and frequency domains with the assumption that metabolic activity is generated from a linear system. To leverage the full information provided by fNIRS signals, in this study we investigate topological entropy in fNIRS series collected from 10 healthy subjects during mental mental arithmetic task. While sample entropy and fuzzy entropy were used to estimate time series irregularity, distribution entropy was used to estimate time series complexity. Our findings show that entropy estimates may provide complementary characterization of fNIRS dynamics with respect to reference time domain measurements. This finding paves the way to further investigate functional activation in fNIRS in different case studies using nonlinear and complexity system theory.

Enhanced Accuracy for Multiclass Mental Workload Detection Using Long Short-Term Memory for Brain-Computer Interface

Cognitive workload is one of the widely invoked human factors in the areas of human-machine interaction (HMI) and neuroergonomics. The precise assessment of cognitive and mental workload (MWL) is vital and requires accurate neuroimaging to monitor and evaluate the cognitive states of the brain. In this study, we have decoded four classes of MWL using long short-term memory (LSTM) with 89.31% average accuracy for brain-computer interface (BCI). The brain activity signals are acquired using functional near-infrared spectroscopy (fNIRS) from the prefrontal cortex (PFC) region of the brain. We performed a supervised MWL experimentation with four varying MWL levels on 15 participants (both male and female) and 10 trials of each MWL per participant. Real-time four-level MWL states are assessed using fNIRS system, and initial classification is performed using three strong machine learning (ML) techniques, support vector machine (SVM), k-nearest neighbor (k-NN), and artificial neural network (ANN) with obtained average accuracies of 54.33, 54.31, and 69.36%, respectively. In this study, novel deep learning (DL) frameworks are proposed, which utilizes convolutional neural network (CNN) and LSTM with 87.45 and 89.31% average accuracies, respectively, to solve high-dimensional four-level cognitive states classification problem. Statistical analysis, t-test, and one-way F-test (ANOVA) are also performed on accuracies obtained through ML and DL algorithms. Results show that the proposed DL (LSTM and CNN) algorithms significantly improve classification performance as compared with ML (SVM, ANN, and k-NN) algorithms.

Cortical Tasks-Based Optimal Filter Selection: An fNIRS Study

Functional near-infrared spectroscopy (fNIRS) is one of the latest noninvasive brain function measuring technique that has been used for the purpose of brain-computer interfacing (BCI). In this paper, we compare and analyze the effect of six most commonly used filtering techniques (i.e., Gaussian, Butterworth, Kalman, hemodynamic response filter (hrf), Wiener, and finite impulse response) on classification accuracies of fNIRS-BCI. To conclude with the best optimal filter for a specific cortical task owing to a specific cortical region, we divided our experimental tasks according to the three main cortical regions: prefrontal, motor, and visual cortex. Three different experiments were performed for prefrontal and motor execution tasks while one for visual stimuli. The tasks performed for prefrontal include rest (R) vs mental arithmetic (MA), R vs object rotation (OB), and OB vs MA. Similarly, for motor execution, R vs left finger tapping (LFT), R vs right finger tapping (RFT), and LFT vs RFT. Likewise, for the visual cortex, R vs visual stimuli (VS) task. These experiments were performed for ten trials with five subjects. For consistency among extracted data, six statistical features were evaluated using oxygenated hemoglobin, namely, slope, mean, peak, kurtosis, skewness, and variance. Combination of these six features was used to classify data by the nonlinear support vector machine (SVM). The classification accuracies obtained from SVM by using hrf and Gaussian were significantly higher for R vs MA, R vs OB, R vs RFT, and R vs VS and Wiener filter for OB vs MA. Similarly, for R vs LFT and LFT vs RFT, hrf was found to be significant p<0.05. These results show the feasibility of using hrf for effective removal of noises from fNIRS data.

Detection of Mild Cognitive Impairment Using Convolutional Neural Network: Temporal-Feature Maps of Functional Near-Infrared Spectroscopy

Mild cognitive impairment (MCI) is the clinical precursor of Alzheimer's disease (AD), which is considered the most common neurodegenerative disease in the elderly. Some MCI patients tend to remain stable over time and do not evolve to AD. It is essential to diagnose MCI in its early stages and provide timely treatment to the patient. In this study, we propose a neuroimaging approach to identify MCI using a deep learning method and functional near-infrared spectroscopy (fNIRS). For this purpose, fifteen MCI subjects and nine healthy controls (HCs) were asked to perform three mental tasks: N-back, Stroop, and verbal fluency (VF) tasks. Besides examining the oxygenated hemoglobin changes (ΔHbO) in the region of interest, ΔHbO maps at 13 specific time points (i.e., 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, and 65 s) during the tasks and seven temporal feature maps (i.e., two types of mean, three types of slope, kurtosis, and skewness) in the prefrontal cortex were investigated. A four-layer convolutional neural network (CNN) was applied to identify the subjects into either MCI or HC, individually, after training the CNN model with ΔHbO maps and temporal feature maps above. Finally, we used the 5-fold cross-validation approach to evaluate the performance of the CNN. The results of temporal feature maps exhibited high classification accuracies: The average accuracies for the N-back task, Stroop task, and VFT, respectively, were 89.46, 87.80, and 90.37%. Notably, the highest accuracy of 98.61% was achieved from the ΔHbO slope map during 20-60 s interval of N-back tasks. Our results indicate that the fNIRS imaging approach based on temporal feature maps is a promising diagnostic method for early detection of MCI and can be used as a tool for clinical doctors to identify MCI from their patients.

Hemodynamics Analysis of Patients With Mild Cognitive Impairment During Working Memory Tasks

Diagnosis of dementia in early stage is important to prevent progression of dementia in the aging society. Mild cognitive impairment (MCI) denotes an early stage of Alzheimer disease (AD). In this paper, we aim to classify MCI patients from healthy controls (HC) during working memory tasks using functional near-infrared spectroscopy (fNIRS). To achieve this objective, t-values and correlation coefficients are calculated to find the region of interest (ROI) channels and brain connectivity. From the ROI channels averaged over subjects, features (mean and slope) of hemodynamic responses were extracted for classification. Extracted features were labelled as two classes and classified via two classifiers, linear discriminant analysis (LDA) and support vector machine (SVM). The classification accuracies were 73.08 % with LDA and 71.15 % with SVM. The results show that there are significant differences in the hemodynamic responses (HR) between MCI patients and healthy controls. Therefore, these results suggest a possibility of using fNIRS as a diagnostic tool for MCI patients.

An Augmented-Reality fNIRS-Based Brain-Computer Interface: A Proof-of-Concept Study

Augmented reality (AR) enhances the user's environment by projecting virtual objects into the real world in real-time. Brain-computer interfaces (BCIs) are systems that enable users to control external devices with their brain signals. BCIs can exploit AR technology to interact with the physical and virtual world and to explore new ways of displaying feedback. This is important for users to perceive and regulate their brain activity or shape their communication intentions while operating in the physical world. In this study, twelve healthy participants were introduced to and asked to choose between two motor-imagery tasks: mental drawing and interacting with a virtual cube. Participants first performed a functional localizer run, which was used to select a single fNIRS channel for decoding their intentions in eight subsequent choice-encoding runs. In each run participants were asked to select one choice of a six-item list. A rotating AR cube was displayed on a computer screen as the main stimulus, where each face of the cube was presented for 6 s and represented one choice of the six-item list. For five consecutive trials, participants were instructed to perform the motor-imagery task when the face of the cube that represented their choice was facing them (therewith temporally encoding the selected choice). In the end of each run, participants were provided with the decoded choice based on a joint analysis of all five trials. If the decoded choice was incorrect, an active error-correction procedure was applied by the participant. The choice list provided in each run was based on the decoded choice of the previous run. The experimental design allowed participants to navigate twice through a virtual menu that consisted of four levels if all choices were correctly decoded. Here we demonstrate for the first time that by using AR feedback and flexible choice encoding in form of search trees, we can increase the degrees of freedom of a BCI system. We also show that participants can successfully navigate through a nested menu and achieve a mean accuracy of 74% using a single motor-imagery task and a single fNIRS channel.

Mental Workload Classification Method Based on EEG Independent Component Features

Excessive mental workload will reduce work efficiency, but low mental workload will cause a waste of human resources. It is very significant to study the mental workload status of operators. The existing mental workload classification method is based on electroencephalogram (EEG) features, and its classification accuracy is often low because the channel signals recorded by the EEG electrodes are a group of mixed brain signals, which are similar to multi-source mixed speech signals. It is not wise to directly analyze the mixed signals in order to distinguish the feature of EEG signals. In this study, we propose a mental workload classification method based on EEG independent components (ICs) features, which borrows from the blind source separation (BSS) idea of mixed speech signals. This presented method uses independent component analysis (ICA) to obtain pure signals, i.e., ICs. The energy features of ICs are directly extracted for classifying the mental workload, since this method directly uses ICs energy features for feature extraction. Compared with the existing solution, the proposed method can obtain better classification results. The presented method might provide a way to realize a fast, accurate, and automatic mental workload classification.

Resting-State NIRS-EEG in Unresponsive Patients with Acute Brain Injury: A Proof-of-Concept Study

BACKGROUND

Neurovascular-based imaging techniques such as functional MRI (fMRI) may reveal signs of consciousness in clinically unresponsive patients but are often subject to logistical challenges in the intensive care unit (ICU). Near-infrared spectroscopy (NIRS) is another neurovascular imaging technique but low cost, can be performed serially at the bedside, and may be combined with electroencephalography (EEG), which are important advantages compared to fMRI. Combined NIRS-EEG, however, has never been evaluated for the assessment of neurovascular coupling and consciousness in acute brain injury.

METHODS

We explored resting-state oscillations in eight-channel NIRS oxyhemoglobin and eight-channel EEG band-power signals to assess neurovascular coupling, the prerequisite for neurovascular-based imaging detection of consciousness, in patients with acute brain injury in the ICU (n = 9). Conscious neurological patients from step-down units and wards served as controls (n = 14). Unsupervised adaptive mixture-independent component analysis (AMICA) was used to correlate NIRS-EEG data with levels of consciousness and clinical outcome.

RESULTS

Neurovascular coupling between NIRS oxyhemoglobin (0.07-0.13 Hz) and EEG band-power (1-12 Hz) signals at frontal areas was sensitive and prognostic to changing consciousness levels. AMICA revealed a mixture of five models from EEG data, with the relative probabilities of these models reflecting levels of consciousness over multiple days, although the accuracy was less than 85%. However, when combined with two channels of bilateral frontal neurovascular coupling, weighted k-nearest neighbor classification of AMICA probabilities distinguished unresponsive patients from conscious controls with > 90% accuracy (positive predictive value 93%, false discovery rate 7%) and, additionally, identified patients who subsequently failed to recover consciousness with > 99% accuracy.

DISCUSSION

We suggest that NIRS-EEG for monitoring of acute brain injury in the ICU is worthy of further exploration. Normalization of neurovascular coupling may herald recovery of consciousness after acute brain injury.

Functional Near-Infrared Spectroscopy for the Classification of Motor-Related Brain Activity on the Sensor-Level

Sensor-level human brain activity is studied during real and imaginary motor execution using functional near-infrared spectroscopy (fNIRS). Blood oxygenation and deoxygenation spatial dynamics exhibit pronounced hemispheric lateralization when performing motor tasks with the left and right hands. This fact allowed us to reveal biomarkers of hemodynamical response of the motor cortex on the motor execution, and use them for designing a sensing method for classification of the type of movement. The recognition accuracy of real movements is close to 100%, while the classification accuracy of imaginary movements is lower but quite high (at the level of 90%). The advantage of the proposed method is its ability to classify real and imaginary movements with sufficiently high efficiency without the need for recalculating parameters. The proposed system can serve as a sensor of motor activity to be used for neurorehabilitation after severe brain injuries, including traumas and strokes.

Brain-Based Binary Communication Using Spatiotemporal Features of fNIRS Responses

“Locked-in” patients lose their ability to communicate naturally due to motor system dysfunction. Brain-computer interfacing offers a solution for their inability to communicate by enabling motor-independent communication. Straightforward and convenient in-session communication is essential in clinical environments. The present study introduces a functional near-infrared spectroscopy (fNIRS)-based binary communication paradigm that requires limited preparation time and merely nine optodes. Eighteen healthy participants performed two mental imagery tasks, mental drawing and spatial navigation, to answer yes/no questions during one of two auditorily cued time windows. Each of the six questions was answered five times, resulting in five trials per answer. This communication paradigm thus combines both spatial (two different mental imagery tasks, here mental drawing for “yes” and spatial navigation for “no”) and temporal (distinct time windows for encoding a “yes” and “no” answer) fNIRS signal features for information encoding. Participants’ answers were decoded in simulated real-time using general linear model analysis. Joint analysis of all five encoding trials resulted in an average accuracy of 66.67 and 58.33% using the oxygenated (HbO) and deoxygenated (HbR) hemoglobin signal respectively. For half of the participants, an accuracy of 83.33% or higher was reached using either the HbO signal or the HbR signal. For four participants, effective communication with 100% accuracy was achieved using either the HbO or HbR signal. An explorative analysis investigated the differentiability of the two mental tasks based solely on spatial fNIRS signal features. Using multivariate pattern analysis (MVPA) group single-trial accuracies of 58.33% (using 20 training trials per task) and 60.56% (using 40 training trials per task) could be obtained. Combining the five trials per run using a majority voting approach heightened these MVPA accuracies to 62.04 and 75%. Additionally, an fNIRS suitability questionnaire capturing participants’ physical features was administered to explore its predictive value for evaluating general data quality. Obtained questionnaire scores correlated significantly (r = -0.499) with the signal-to-noise of the raw light intensities. While more work is needed to further increase decoding accuracy, this study shows the potential of answer encoding using spatiotemporal fNIRS signal features or spatial fNIRS signal features only.

On fractality of functional near-infrared spectroscopy signals: analysis and applications

Significance: The human brain is a highly complex system with nonlinear, dynamic behavior. A majority of brain imaging studies employing functional near-infrared spectroscopy (fNIRS), however, have considered only the spatial domain and have ignored the temporal properties of fNIRS recordings. Methods capable of revealing nonlinearities in fNIRS recordings can provide new insights about how the brain functions. Aim: The temporal characteristics of fNIRS signals are explored by comprehensively investigating their fractal properties. Approach: Fractality of fNIRS signals is analyzed using scaled windowed variance (SWV), as well as using visibility graph (VG), a method which converts a given time series into a graph. Additionally, the fractality of fNIRS signals obtained under resting-state and task-based conditions is compared, and the application of fractality in differentiating brain states is demonstrated for the first time via various classification approaches. Results: Results from SWV analysis show the existence of high fractality in fNIRS recordings. It is shown that differences in the temporal characteristics of fNIRS signals related to task-based and resting-state conditions can be revealed via the VGs constructed for each case. Conclusions: fNIRS recordings, regardless of the experimental conditions, exhibit high fractality. Furthermore, VG-based metrics can be employed to differentiate rest and task-execution brain states.

Performance assessment of high-density diffuse optical topography regarding source-detector array topology

Recent advances in optical neuroimaging systems as a functional interface enhance our understanding of neuronal activity in the brain. High density diffuse optical topography (HD-DOT) uses multi-distance overlapped channels to improve the spatial resolution of images comparable to functional magnetic resonance imaging (fMRI). The topology of the source and detector (SD) array directly impacts the quality of the hemodynamic reconstruction in HD-DOT imaging modality. In this work, the effect of different SD configurations on the quality of cerebral hemodynamic recovery is investigated by presenting a simulation setup based on the analytical approach. Given that the SD arrangement determines the elements of the Jacobian matrix, we conclude that the more individual components in this matrix, the better the retrieval quality. The results demonstrate that the multi-distance multi-directional (MDMD) arrangement produces more unique elements in the Jacobian array. Consequently, the inverse problem can accurately retrieve the brain activity of diffuse optical topography data.

Differences in Net Information Flow and Dynamic Connectivity Metrics Between Physically Active and Inactive Subjects Measured by Functional Near-Infrared Spectroscopy (fNIRS) During a Fatiguing Handgrip Task

Twenty-three young adults (4 Females, 25.13 ± 3.72 years) performed an intermittent maximal handgrip force task using their dominant hand for 20 min (3.5 s squeeze/6.5 s release, 120 blocks) with concurrent cortical activity imaging by functional Near-Infrared Spectroscopy (fNRIS; OMM-3000, Shimadzu Corp., 111 channels). Subjects were grouped as physically active (n = 10) or inactive (n = 12) based on a questionnaire (active-exercise at least four times a week, inactive- exercise less than two times a week). We explored how motor task fatigue affected the vasomotion-induced oscillations in ΔHbO as measured by fNIRS at each hemodynamic frequency band: endothelial component (0.003–0.02 Hz) associated to microvascular activity, neurogenic component (0.02–0.04 Hz) related to intrinsic neuronal activity, and myogenic component (0.04–0.15 Hz) linked to activity of smooth muscles of arterioles. To help understand how these three neurovascular regulatory mechanisms relate to handgrip task performance we quantified several dynamic fNIRS metrics, including directional phase transfer entropy (dPTE), a computationally efficient and data-driven method used as a marker of information flow between cortical regions, and directional connectivity (DC), a means to compute directionality of information flow between two cortical regions. The relationship between static functional connectivity (SFC) and functional connectivity variability (FCV) was also explored to understand their mutual dependence for each frequency band in the context of handgrip performance as fatigued increased. Our findings ultimately showed differences between subject groups across all fNIRS metrics and hemodynamic frequency bands. These findings imply that physical activity modulates neurovascular control mechanisms at the endogenic, neurogenic, and myogenic frequency bands resulting in delayed fatigue onset and enhanced performance. The dynamic cortical network metrics quantified in this work for young, healthy subjects provides baseline measurements to guide future work on older individuals and persons with impaired cardiovascular health.

Performance Improvement of Near-Infrared Spectroscopy-Based Brain-Computer Interface Using Regularized Linear Discriminant Analysis Ensemble Classifier Based on Bootstrap Aggregating

Ensemble classifiers have been proven to result in better classification accuracy than that of a single strong learner in many machine learning studies. Although many studies on electroencephalography-brain-computer interface (BCI) used ensemble classifiers to enhance the BCI performance, ensemble classifiers have hardly been employed for near-infrared spectroscopy (NIRS)-BCIs. In addition, since there has not been any systematic and comparative study, the efficacy of ensemble classifiers for NIRS-BCIs remains unknown. In this study, four NIRS-BCI datasets were employed to evaluate the efficacy of linear discriminant analysis ensemble classifiers based on the bootstrap aggregating. From the analysis results, significant (or marginally significant) increases in the bitrate as well as the classification accuracy were found for all four NIRS-BCI datasets employed in this study. Moreover, significant bitrate improvements were found in two of the four datasets.

The present and future use of functional near-infrared spectroscopy (fNIRS) for cognitive neuroscience

The past few decades have seen a rapid increase in the use of functional near-infrared spectroscopy (fNIRS) in cognitive neuroscience. This fast growth is due to the several advances that fNIRS offers over the other neuroimaging modalities such as functional magnetic resonance imaging and electroencephalography/magnetoencephalography. In particular, fNIRS is harmless, tolerant to bodily movements, and highly portable, being suitable for all possible participant populations, from newborns to the elderly and experimental settings, both inside and outside the laboratory. In this review we aim to provide a comprehensive and state-of-the-art review of fNIRS basics, technical developments, and applications. In particular, we discuss some of the open challenges and the potential of fNIRS for cognitive neuroscience research, with a particular focus on neuroimaging in naturalistic environments and social cognitive neuroscience.

Reduction of Onset Delay in Functional Near-Infrared Spectroscopy: Prediction of HbO/HbR Signals

An intrinsic problem when using hemodynamic responses for the brain-machine interface is the slow nature of the physiological process. In this paper, a novel method that estimates the oxyhemoglobin changes caused by neuronal activations is proposed and validated. In monitoring the time responses of blood-oxygen-level-dependent signals with functional near-infrared spectroscopy (fNIRS), the early trajectories of both oxy- and deoxy-hemoglobins in their phase space are scrutinized. Furthermore, to reduce the detection time, a prediction method based upon a kernel-based recursive least squares (KRLS) algorithm is implemented. In validating the proposed approach, the fNIRS signals of finger tapping tasks measured from the left motor cortex are examined. The results show that the KRLS algorithm using the Gaussian kernel yields the best fitting for both ΔHbO (i.e., 87.5%) and ΔHbR (i.e., 85.2%) at q = 15 steps ahead (i.e., 1.63 s ahead at a sampling frequency of 9.19 Hz). This concludes that a neuronal activation can be concluded in about 0.1 s with fNIRS using prediction, which enables an almost real-time practice if combined with EEG.

Using the General Linear Model to Improve Performance in fNIRS Single Trial Analysis and Classification: A Perspective

Within a decade, single trial analysis of functional Near Infrared Spectroscopy (fNIRS) signals has gained significant momentum, and fNIRS joined the set of modalities frequently used for active and passive Brain Computer Interfaces (BCI). A great variety of methods for feature extraction and classification have been explored using state-of-the-art Machine Learning methods. In contrast, signal preprocessing and cleaning pipelines for fNIRS often follow simple recipes and so far rarely incorporate the available state-of-the-art in adjacent fields. In neuroscience, where fMRI and fNIRS are established neuroimaging tools, evoked hemodynamic brain activity is typically estimated across multiple trials using a General Linear Model (GLM). With the help of the GLM, subject, channel, and task specific evoked hemodynamic responses are estimated, and the evoked brain activity is more robustly separated from systemic physiological interference using independent measures of nuisance regressors, such as short-separation fNIRS measurements. When correctly applied in single trial analysis, e.g., in BCI, this approach can significantly enhance contrast to noise ratio of the brain signal, improve feature separability and ultimately lead to better classification accuracy. In this manuscript, we provide a brief introduction into the GLM and show how to incorporate it into a typical BCI preprocessing pipeline and cross-validation. Using a resting state fNIRS data set augmented with synthetic hemodynamic responses that provide ground truth brain activity, we compare the quality of commonly used fNIRS features for BCI that are extracted from (1) conventionally preprocessed signals, and (2) signals preprocessed with the GLM and physiological nuisance regressors. We show that the GLM-based approach can provide better single trial estimates of brain activity as well as a new feature type, i.e., the weight of the individual and channel-specific hemodynamic response function (HRF) regressor. The improved estimates yield features with higher separability, that significantly enhance accuracy in a binary classification task when compared to conventional preprocessing—on average +7.4% across subjects and feature types. We propose to adapt this well-established approach from neuroscience to the domain of single-trial analysis and preprocessing wherever the classification of evoked brain activity is of concern, for instance in BCI.

Assessing Time-Resolved fNIRS for Brain-Computer Interface Applications of Mental Communication

Brain-computer interfaces (BCIs) are becoming increasingly popular as a tool to improve the quality of life of patients with disabilities. Recently, time-resolved functional near-infrared spectroscopy (TR-fNIRS) based BCIs are gaining traction because of their enhanced depth sensitivity leading to lower signal contamination from the extracerebral layers. This study presents the first account of TR-fNIRS based BCI for “mental communication” on healthy participants. Twenty-one (21) participants were recruited and were repeatedly asked a series of questions where they were instructed to imagine playing tennis for “yes” and to stay relaxed for “no.” The change in the mean time-of-flight of photons was used to calculate the change in concentrations of oxy- and deoxyhemoglobin since it provides a good compromise between depth sensitivity and signal-to-noise ratio. Features were extracted from the average oxyhemoglobin signals to classify them as “yes” or “no” responses. Linear-discriminant analysis (LDA) and support vector machine (SVM) classifiers were used to classify the responses using the leave-one-out cross-validation method. The overall accuracies achieved for all participants were 75% and 76%, using LDA and SVM, respectively. The results also reveal that there is no significant difference in accuracy between questions. In addition, physiological parameters [heart rate (HR) and mean arterial pressure (MAP)] were recorded on seven of the 21 participants during motor imagery (MI) and rest to investigate changes in these parameters between conditions. No significant difference in these parameters was found between conditions. These findings suggest that TR-fNIRS could be suitable as a BCI for patients with brain injuries.

Effects of Processing Methods on fNIRS Signals Assessed During Active Walking Tasks in Older Adults

Functional near infrared spectroscopy (fNIRS) is a noninvasive optics-based neuroimaging modality successfully applied to real-life settings. The technology uses light in the near infrared range (650-950nm) to track changes in oxygenated (HbO2) and deoxygenated hemoglobin (Hb) obtained from measured light intensity using light-tissue interaction principles. fNIRS data processing involves artifact removal and hemodynamic signal conversion using modified Beer-Lambert law (MBLL) to obtain Hb and HbO2, reliably. fNIRS signals can get contaminated by various noise sources of physiological and non-physiological origins. Various algorithms have been proposed for the elimination of artifacts from frequency selective filters to blind source separation methods. Hemodynamic signal extraction using raw intensity measurements at multiple wavelengths based on MBLL usually involves apriori knowledge of certain conversion parameters such as molar extinction coefficients ( ε ) and differential path length factor (DPF). Different sets of conversion parameters dependent upon wavelength, chromophores, and age have been reported. Variation in processing algorithms and parameters can cause differences in Hb and HbO2 extraction which can in turn change study outcomes. Using fNIRS, we have previously shown significant increases in oxygenation in the prefrontal cortex from Single-Task-Walking (STW) to Dual-task-Walking (DTW) conditions in older adults due to greater cognitive demands inherent in the latter condition. In the current study, we re-analyzed our data and determined that although using different conversion parameters i.e. ε and age dependent DPF and filter cut-off frequencies at 0.14 and 0.08Hz including those designed to remove confounding effects of Mayer waves had caused some linear increases or decreases on the extracted Hb and HbO2 signals, those effects were minimal in task related comparisons and hence, the overall study outcomes.

Indoor Simulated Training Environment for Brain-Controlled Wheelchair Based on Steady-State Visual Evoked Potentials

Brain-controlled wheelchair (BCW) has the potential to improve the quality of life for people with motor disabilities. A lot of training is necessary for users to learn and improve BCW control ability and the performances of BCW control are crucial for patients in daily use. In consideration of safety and efficiency, an indoor simulated training environment is built up in this paper to improve the performance of BCW control. The indoor simulated environment mainly realizes BCW implementation, simulated training scenario setup, path planning and recommendation, simulated operation, and scoring. And the BCW is based on steady-state visual evoked potentials (SSVEP) and the filter bank canonical correlation analysis (FBCCA) is used to analyze the electroencephalography (EEG). Five tasks include individual accuracy, simple linear path, obstacles avoidance, comprehensive steering scenarios, and evaluation task are designed, 10 healthy subjects were recruited and carried out the 7-days training experiment to assess the performance of the training environment. Scoring and command-consuming are conducted to evaluate the improvement before and after training. The results indicate that the average accuracy is 93.55% and improves from 91.05% in the first stage to 96.05% in the second stage (p = 0.001). Meanwhile, the average score increases from 79.88 in the first session to 96.66 in the last session and tend to be stable (p < 0.001). The average number of commands and collisions to complete the tasks decreases significantly with or without the approximate shortest path (p < 0.001). These results show that the performance of subjects in BCW control achieves improvement and verify the feasibility and effectiveness of the proposed simulated training environment.

Development of a Brain-Computer Interface Toggle Switch with Low False-Positive Rate Using Respiration-Modulated Photoplethysmography

Asynchronous brain-computer interfaces (BCIs) based on electroencephalography (EEG) generally suffer from poor performance in terms of classification accuracy and false-positive rate (FPR). Thus, BCI toggle switches based on electrooculogram (EOG) signals were developed to toggle on/off synchronous BCI systems. The conventional BCI toggle switches exhibit fast responses with high accuracy; however, they have a high FPR or cannot be applied to patients with oculomotor impairments. To circumvent these issues, we developed a novel BCI toggle switch that users can employ to toggle on or off synchronous BCIs by holding their breath for a few seconds. Two states-normal breath and breath holding-were classified using a linear discriminant analysis with features extracted from the respiration-modulated photoplethysmography (PPG) signals. A real-time BCI toggle switch was implemented with calibration data trained with only 1-min PPG data. We evaluated the performance of our PPG switch by combining it with a steady-state visual evoked potential-based BCI system that was designed to control four external devices, with regard to the true-positive rate and FPR. The parameters of the PPG switch were optimized through an offline experiment with five subjects, and the performance of the switch system was evaluated in an online experiment with seven subjects. All the participants successfully turned on the BCI by holding their breath for approximately 10 s (100% accuracy), and the switch system exhibited a very low FPR of 0.02 false operations per minute, which is the lowest FPR reported thus far. All participants could successfully control external devices in the synchronous BCI mode. Our results demonstrated that the proposed PPG-based BCI toggle switch can be used to implement practical BCIs.

Channel Projection-Based CCA Target Identification Method for an SSVEP-Based BCI System of Quadrotor Helicopter Control

The brain-computer interface (BCI) plays an important role in assisting patients with amyotrophic lateral sclerosis (ALS) to enable them to participate in communication and entertainment. In this study, a novel channel projection-based canonical correlation analysis (CP-CCA) target identification method for steady-state visual evoked potential- (SSVEP-) based BCI system was proposed. The single-channel electroencephalography (EEG) signals of multiple trials were recorded when the subject is under the same stimulus frequency. The CCAs between single-channel EEG signals of multiple trials and sine-cosine reference signals were obtained. Then, the optimal reference signal of each channel was utilized to estimate the test EEG signal. To validate the proposed method, we acquired the training dataset with two testing conditions including the optimal time window length and the number of the trial of training data. The offline experiments conducted a comparison of the proposed method with the traditional canonical correlation analysis (CCA) and power spectrum density analysis (PSDA) method using a 5-class SSVEP dataset that was recorded from 10 subjects. Based on the training dataset, the online 3D-helicopter control experiment was carried out. The offline experimental results showed that the proposed method outperformed the CCA and the PSDA methods in terms of classification accuracy and information transfer rate (ITR). Furthermore, the online experiments of 3-DOF helicopter control achieved an average accuracy of 87.94 ± 5.93% with an ITR of 21.07 ± 4.42 bit/min.

Open-Access fNIRS Dataset for Classification of Unilateral Finger- and Foot-Tapping

Numerous open-access electroencephalography (EEG) datasets have been released and widely employed by EEG researchers. However, not many functional near-infrared spectroscopy (fNIRS) datasets are publicly available. More fNIRS datasets need to be freely accessible in order to facilitate fNIRS studies. Toward this end, we introduce an open-access fNIRS dataset for three-class classification. The concentration changes of oxygenated and reduced hemoglobin were measured, while 30 volunteers repeated each of the three types of overt movements (i.e., left- and right-hand unilateral complex finger-tapping, foot-tapping) for 25 times. The ternary support vector machine (SVM) classification accuracy obtained using leave-one-out cross-validation was estimated at 70.4% ± 18.4% on average. A total of 21 out of 30 volunteers scored a superior binary SVM classification accuracy (left-hand vs. right-hand finger-tapping) of over 80.0%. We believe that the introduced fNIRS dataset can facilitate future fNIRS studies.

An exploration of neural dynamics of motor imagery for people with amyotrophic lateral sclerosis

OBJECTIVE

Studies of the neuropathological effects of amyotrophic lateral sclerosis (ALS) on the underlying motor system have investigated abnormalities in the magnitude and timing of the event-related desynchronization (ERD) and synchronization (ERS) during motor execution (ME). However, the spatio-spectral-temporal dynamics of these sensorimotor oscillations during motor imagery (MI) have not been fully explored for these patients. This study explores the neural dynamics of sensorimotor oscillations for ALS patients during MI by quantifying ERD/ERS features in frequency, time, and space.

APPROACH

Electroencephalogram (EEG) data were recorded from six patients with ALS and 11 age-matched healthy controls (HC) while performing a MI task. ERD/ERS features were extracted using wavelet-based time-frequency analysis and compared between the two groups to quantify the abnormal neural dynamics of ALS in terms of both time and frequency. Topographic correlation analysis was conducted to compare the localization of MI activity between groups and to identify subject-specific frequencies in the µ and β frequency bands.

MAIN RESULTS

Overall, reduced and delayed ERD was observed for ALS patients, particularly during right-hand MI. ERD features were also correlated with ALS clinical scores, specifically disease duration, bulbar, and cognitive functions.

SIGNIFICANCE

The analyses in this study quantify abnormalities in the magnitude and timing of sensorimotor oscillations for ALS patients during MI tasks. Our findings reveal notable differences between MI and existing results on ME in ALS. The observed alterations are speculated to reflect disruptions in the underlying cortical networks involved in MI functions. Quantifying the neural dynamics of MI plays an important role in the study of EEG-based cortical markers for ALS.

Effects of Motor Imagery and Visual Neurofeedback on Activation in the Swallowing Network: A Real-Time fMRI Study

Motor imagery of movements is used as mental strategy in neurofeedback applications to gain voluntary control over activity in motor areas of the brain. In the present functional magnetic resonance imaging (fMRI) study, we first addressed the question whether motor imagery and execution of swallowing activate comparable brain areas, which has been already proven for hand and foot movements. Prior near-infrared spectroscopy (NIRS) studies provide evidence that this is the case in the outer layer of the cortex. With the present fMRI study, we want to expand these prior NIRS findings to the whole brain. Second, we used motor imagery of swallowing as mental strategy during visual neurofeedback to investigate whether one can learn to modulate voluntarily activity in brain regions, which are associated with active swallowing, using real-time fMRI. Eleven healthy adults performed one offline session, in which they executed swallowing movements and imagined swallowing on command during fMRI scanning. Based on this functional localizer task, we identified brain areas active during both tasks and defined individually regions for feedback. During the second session, participants performed two real-time fMRI neurofeedback runs (each run comprised 10 motor imagery trials), in which they should increase voluntarily the activity in the left precentral gyrus by means of motor imagery of swallowing while receiving visual feedback (the visual feedback depicted one's own fMRI signal changes in real-time). Motor execution and imagery of swallowing activated a comparable network of brain areas including the bilateral pre- and postcentral gyrus, inferior frontal gyrus, basal ganglia, insula, SMA, and the cerebellum compared to a resting condition. During neurofeedback training, participants were able to increase the activity in the feedback region (left lateral precentral gyrus) but also in other brain regions, which are generally active during swallowing, compared to the motor imagery offline task. Our results indicate that motor imagery of swallowing is an adequate mental strategy to activate the swallowing network of the whole brain, which might be useful for future treatments of swallowing disorders.

Prediction in Autism by Deep Learning Short-Time Spontaneous Hemodynamic Fluctuations

This study aims to explore the possibility of using a multilayer artificial neural network for the classification between children with autism spectrum disorder (ASD) and typically developing (TD) children based on short-time spontaneous hemodynamic fluctuations. Spontaneous hemodynamic fluctuations were collected by a functional near-infrared spectroscopy setup from bilateral inferior frontal gyrus and temporal cortex in 25 children with ASD and 22 TD children. To perform feature extraction and classification, a multilayer neural network called CGRNN was used which combined a convolution neural network (CNN) and a gate recurrent unit (GRU), since CGRNN has a strong ability in finding characteristic features and acquiring intrinsic relationship in time series. For the training and predicting, short-time (7 s) time-series raw functional near-infrared spectroscopy (fNIRS) signals were used as the input of the network. To avoid the over-fitting problem and effectively extract useful differentiation features from a sample with a very limited size (e.g., 25 ASDs and 22 TDs), a sliding window approach was utilized in which the initially recorded long-time (e.g., 480 s) time-series was divided into many partially overlapped short-time (7 s) sequences. By using this combined deep-learning network, a high accurate classification between ASD and TD could be achieved even with a single optical channel, e.g., 92.2% accuracy, 85.0% sensitivity, and 99.4% specificity. This result implies that the multilayer neural network CGRNN can identify characteristic features associated with ASD even in a short-time spontaneous hemodynamic fluctuation from a single optical channel, and second, the CGRNN can provide highly accurate prediction in ASD.

Multimodal exploration of non-motor neural functions in ALS patients using simultaneous EEG-fNIRS recording

OBJECTIVE

Despite the high prevalence of non-motor impairments reported in patients with amyotrophic lateral sclerosis (ALS), little is known about the functional neural markers underlying such dysfunctions. In this study, a new dual-task multimodal framework relying on simultaneous electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) recordings was developed to characterize integrative non-motor neural functions in people with ALS.

APPROACH

Simultaneous EEG-fNIRS data were recorded from six subjects with ALS and twelve healthy controls. Through a proposed visuo-mental paradigm, subjects performed a set of visuo-mental arithmetic operations. The data recorded were analyzed with respect to event-related changes both in the time and frequency domains for EEG and de/oxygen-hemoglobin level (HbR/HbO) changes for fNIRS. The correlation of EEG spectral features with fNIRS HbO/HbR features were then evaluated to assess the mechanisms of ALS on the electrical (EEG)-vascular (fNIRS) interrelationships.

MAIN RESULTS

We observed overall smaller increases in EEG delta and theta power, decreases in beta power, reductions in HbO responses, and distortions both in early and later EEG event-related potentials in ALS subjects compared to healthy controls. While significant correlations between EEG features and HbO responses were observed in healthy controls, these patterns were absent in ALS patients. Distortions in both electrical and hemodynamic responses are speculated to be associated with cognitive deficits in ALS that center primarily on attentional and working memory processing.

SIGNIFICANCE

Our results highlight the important role of ALS non-motor dysfunctions in electrical and hemodynamic neural dynamics as well as their interrelationships. The insights obtained through this study can enhance our understanding of the underlying non-motor neural processes in ALS and enrich future diagnostic and prognostic techniques.

A Newcomer's Guide to Functional Near Infrared Spectroscopy Experiments

This review presents a practical primer for functional near-infrared spectroscopy (fNIRS) with respect to technology, experimentation, and analysis software. Its purpose is to jump-start interested practitioners considering utilizing a non-invasive, versatile, nevertheless challenging window into the brain using optical methods. We briefly recapitulate relevant anatomical and optical foundations and give a short historical overview. We describe competing types of illumination (trans-illumination, reflectance, and differential reflectance) and data collection methods (continuous wave, time domain and frequency domain). Basic components (light sources, detection, and recording components) of fNIRS systems are presented. Advantages and limitations of fNIRS techniques are offered, followed by a list of very practical recommendations for its use. A variety of experimental and clinical studies with fNIRS are sampled, shedding light on many brain-related ailments. Finally, we describe and discuss a number of freely available analysis and presentation packages suited for data analysis. In conclusion, we recommend fNIRS due to its ever-growing body of clinical applications, state-of-the-art neuroimaging technique and manageable hardware requirements. It can be safely concluded that fNIRS adds a new arrow to the quiver of neuro-medical examinations due to both its great versatility and limited costs.

Mapping cortical network effects of fatigue during a handgrip task by functional near-infrared spectroscopy in physically active and inactive subjects

The temporal evolution of cortical activation and connectivity patterns during a fatiguing handgrip task were studied by functional near-infrared spectroscopy (fNIRS). Twenty-three young adults (18 to 35 years old) were recruited to use a handheld force sensor to perform intermittent handgrip contractions with their dominant hand at their personal maximum voluntary contraction force level for 3.5 s followed by 6.5 s of rest for 120 blocks. Subjects were divided into self-reported physically active and inactive groups, and their hemodynamic activity over the prefrontal and sensory-motor cortices (111 channels) was mapped while they performed this task. Using this fNIRS setup, a more detailed time sequence of cortical activation and connectivity patterns was observed compared to prior studies. A temporal evolution sequence of hemodynamic activation patterns was noted, which was different between the active and the inactive groups. Physically active subjects demonstrated delayed fatigue onset and significantly longer-lasting and more spatially extended functional connectivity (FC) patterns, compared to inactive subjects. The observed differences in activation and FC suggested differences in cortical network adaptation patterns as fatigue set in, which were dependent on subjects' physical activity. The findings of this study suggest that physical activity increases FC with regions involved in motor task control and correlates to extended fatigue onset and enhanced performance.

Fusing Near-Infrared Spectroscopy with Wearable Hemodynamic Measurements Improves Classification of Mental Stress

Human-computer interaction (HCI) technology, and the automatic classification of a person's mental state, are of interest to multiple industries. In this work, the fusion of sensing modalities that monitor the oxygenation of the human prefrontal cortex (PFC) and cardiovascular physiology was evaluated to differentiate between rest, mental arithmetic and N-back memory tasks. A flexible headband to measure near-infrared spectroscopy (NIRS) for quantifying PFC oxygenation, and forehead photoplethysmography (PPG) for assessing peripheral cardiovascular activity was designed. Physiological signals such as the electrocardiogram (ECG) and seismocardiogram (SCG) were collected, along with the measurements obtained using the headband. The setup was tested and validated with a total of 16 human subjects performing a series of arithmetic and N-back memory tasks. Features extracted were related to cardiac and peripheral sympathetic activity, vasomotor tone, pulse wave propagation, and oxygenation. Machine learning techniques were utilized to classify rest, arithmetic, and N-back tasks, using leave-one-subject-out cross validation. Macro-averaged accuracy of 85%, precision of 84%, recall rate of 83%, and F1 score of 80% were obtained from the classification of the three states. Statistical analyses on the subject-based results demonstrate that the fusion of NIRS and peripheral cardiovascular sensing significantly improves the accuracy, precision, recall, and F1 scores, compared to using NIRS sensing alone. Moreover, the fusion significantly improves the precision compared to peripheral cardiovascular sensing alone. The results of this work can be used in the future to design a multi-modal wearable sensing system for classifying mental state for applications such as acute stress detection.

A Graph-Based Hierarchical Attention Model for Movement Intention Detection from EEG Signals

An EEG-based Brain-Computer Interface (BCI) is a system that enables a user to communicate with and intuitively control external devices solely using the user's intentions. Current EEG-based BCI research usually involves a subject-specific adaptation step before a BCI system is ready to be employed by a new user. However, the subject-independent scenario, in which a well-trained model can be directly applied to new users without pre-calibration, is particularly desirable yet rarely explored. Considering this critical gap, our focus in this paper is the subject-independent scenario of EEG-based human intention recognition. We present a G raph-based H ierarchical A ttention M odel (G-HAM) that utilizes the graph structure to represent the spatial information of EEG sensors and the hierarchical attention mechanism to focus on both the most discriminative temporal periods and EEG nodes. Extensive experiments on a large EEG dataset containing 105 subjects indicate that our model is capable of exploiting the underlying invariant EEG patterns across different subjects and generalizing the patterns to new subjects with better performance than a series of state-of-the-art and baseline approaches.

Signal Processing in fNIRS: A Case for the Removal of Systemic Activity for Single Trial Data

Researchers using functional near infrared spectroscopy (fNIRS) are increasingly aware of the problem that conventional filtering methods do not eliminate systemic noise at frequencies overlapping with the task frequency. This is a problem when signals are averaged for analysis, even more so when single trial data are used as in online neurofeedback or BCI applications where insufficiently preprocessed data means feeding back noise instead of brain activity or when looking for brain-behavior relationships on a trial-by-trial basis. For removing this task-related noise statistical approaches have been proposed. Yet as evidence is lacking on how these approaches perform on independent data, choosing one approach over another can be difficult. Here signal quality at the single trial level was considered together with statistical effects to inform this choice. Compared were conventional band-pass filtering and wavelet minimum description length detrending and the combination of both with a more elaborate, published preprocessing approach for a motor execution—motor imagery data set. Temporal consistency between Δ[HbO] and Δ[HbR] and two measures of the spatial specificity of signals that are proposed here served as measures of data quality. Both improved strongly for the combinationed preprocessing approaches. Statistical effects showed a strong tendency toward getting smaller for the combined approaches. This underlines the importance to adequately deal with noise in fNIRS recordings and demonstrates how the quality of statistical correction approaches can be estimated.

Comparison of Feature Vector Compositions to Enhance the Performance of NIRS-BCI-Triggered Robotic Hand Orthosis for Post-Stroke Motor Recovery

Recently, brain–computer interfaces, combined with feedback systems and goal-oriented training, have been investigated for their capacity to promote functional recovery after stroke. Accordingly, we developed a brain–computer interface-triggered robotic hand orthosis that assists hand-closing and hand-opening for post-stroke patients without sufficient motor output. In this system, near-infrared spectroscopy is used to monitor the affected motor cortex, and a linear discriminant analysis-based binary classifier estimates hand posture. The estimated posture then wirelessly triggers the robotic hand orthosis. For better performance of the brain–computer interface, we tested feature windows of different lengths and varying feature vector compositions with motor execution data from seven neurologically intact participants. The interaction between a feature window and a delay in the hemodynamic response significantly affected both classification accuracy (Matthew Correlation Coefficient) and detection latency. The ‘preserving channels’ feature vector was able to increase accuracy by 13.14% and decrease latency by 29.48%, relative to averaging. Oxyhemoglobin combined with deoxyhemoglobin improved accuracy by 3.71% and decreased latency by 6.01% relative to oxyhemoglobin alone. Thus, the best classification performance resulted in an accuracy of 0.7154 and a latency of 2.8515 s. The hand rehabilitation system was successfully implemented using this feature vector composition, which yielded better classification performance.

Evaluation of Neural Degeneration Biomarkers in the Prefrontal Cortex for Early Identification of Patients With Mild Cognitive Impairment: An fNIRS Study

Mild cognitive impairment (MCI), a condition characterizing poor cognition, is associated with aging and depicts early symptoms of severe cognitive impairment, known as Alzheimer's disease (AD). Meanwhile, early detection of MCI can prevent progression to AD. A great deal of research has been performed in the past decade on MCI detection. However, availability of biomarkers for MCI detection requires greater attention. In our study, we evaluated putative and reliable biomarkers for diagnosing MCI by performing different mental tasks (i.e., N-back task, Stroop task, and verbal fluency task) using functional near-infrared spectroscopy (fNIRS) signals on a group of 15 MCI patients and 9 healthy control (HC). The 15 digital biomarkers (i.e., five means, seven slopes, peak, skewness, and kurtosis) and two image biomarkers (t-map, correlation map) in the prefrontal cortex (PFC) (i.e., left PFC, middle PFC, and right PFC) between the MCI and HC groups were investigated by the statistical analysis, linear discriminant analysis (LDA), and convolutional neural network (CNN) individually. The results reveal that the statistical analysis using digital biomarkers (with a p-value < 0.05) could not distinguish the MCI patients from the HC over 60% accuracy. Therefore, the current statistical analysis needs to be improved to be used for diagnosing the MCI patients. The best accuracy with LDA was 76.67% with the N-back and Stroop tasks. However, the CNN classification results trained by image biomarkers showed a high accuracy. In particular, the CNN results trained via t-maps revealed the best accuracy (90.62%) with the N-back task, whereas the CNN result trained by the correlation maps was 85.58% with the N-back task. Also, the results illustrated that investigating the sub-regions (i.e., right, middle, left) of the PFC for detecting MCI would be better than examining the whole PFC. The t-map (or/and the correlation map) is conclusively recommended as an image biomarker for early detection of AD. The combination of CNN and image biomarkers can provide a reliable clinical tool for diagnosing MCI patients.

Effects of Acupuncture Therapy on MCI Patients Using Functional Near-Infrared Spectroscopy

Acupuncture therapy (AT) is a non-pharmacological method of treatment that has been applied to various neurological diseases. However, studies on its longitudinal effect on the neural mechanisms of patients with mild cognitive impairment (MCI) for treatment purposes are still lacking in the literature. In this clinical study, we assess the longitudinal effects of ATs on MCI patients using two methods: (i) Montreal Cognitive Assessment test (MoCA-K, Korean version), and (ii) the hemodynamic response (HR) analyses using functional near-infrared spectroscopy (fNIRS). fNIRS signals of a working memory (WM) task were acquired from the prefrontal cortex. Twelve elderly MCI patients and 12 healthy people were recruited as target and healthy control (HC) groups, respectively. Each group went through an fNIRS scanning procedure three times: The initial data were obtained without any ATs, and subsequently a total of 24 AT sessions were conducted for MCI patients (i.e., MCI-0: the data prior to ATs, MCI-1: after 12 sessions of ATs for 6 weeks, MCI-2: another 12 sessions of ATs for 6 weeks). The mean HR responses of all MCI-0–2 cases were lower than those of HCs. To compare the effects of AT on MCI patients, MoCA-K results, temporal HR data, and spatial activation patterns (i.e., t-maps) were examined. In addition, analyses of functional connectivity (FC) and graph theory upon WM tasks were conducted. With ATs, (i) the averaged MoCA-K test scores were improved (MCI-1, p = 0.002; MCI-2, p = 2.9e^–4); (ii) the mean HR response of WM tasks was increased (p < 0.001); and (iii) the t-maps of MCI-1 and MCI-2 were enhanced. Furthermore, an increased FC in the prefrontal cortex in both MCI-1/MCI-2 cases in comparison to MCI-0 was obtained (p < 0.01), and an increasing trend in the graph theory parameters was observed. All these findings reveal that ATs have a positive impact on improving the cognitive function of MCI patients. In conclusion, ATs can be used as a therapeutic tool for MCI patients as a non-pharmacological method (Clinical trial registration number: KCT 0002451 https://cris.nih.go.kr/cris/en/).

Training -Free Steady-State Visual Evoked Potential Brain-Computer Interface Based on Filter Bank Canonical Correlation Analysis and Spatiotemporal Beamforming Decoding

A brain-computer interface (BCI) provides a novel non-muscular communication pathway for individuals with severe neuromuscular diseases. BCI systems based on steady-state visual evoked potentials (SSVEPs) have high classification accuracy, information transfer rate, and signal-to-noise ratio, giving them high research and application value. However, SSVEP-based BCI has several limitations in real-world applications. The main challenge is how to reduce or eliminate the need for a dedicated training process while maintaining high classification accuracy. Filter bank canonical correlation analysis (FBCCA) is a powerful and widely used feature extraction method for SSVEP-based BCI systems. However, the reference signals of FBCCA are fixed-frequency sine-cosine waves, which makes it difficult to accurately describe the complex, mutative, and individually different physiological SSVEPs. Therefore, there is huge room for improvement in classification performance based on the FBCCA method. In contrast, although spatiotemporal beamforming (BF) detects SSVEPs with high accuracy, it needs an additional training process, which limits its application. In this study, we propose a bimodal decoding algorithm (FBCCA+BF), which combines the advantages of the training-free classification of FBCCA and the data-driven and adaptive features of BF. Six-channel SSVEP data corresponding to eight targets measured from 15 subjects were used to test the effectiveness of three different CCA-based methods, BF, and our proposed FBCCA+BF methods. It was found that the classification accuracies for BF and FBCCA+BF are 95.6% and 92.2%, respectively, which are significantly higher than the other CCA-based methods. Notably, both BF and FBCCA+BF obtain state-of-the-art performance, but FBCCA+BF does this without the need for a dedicated training process. Therefore, we conclude that our proposed FBCCA+BF method provides a training-free and high-accuracy approach for SSVEP-based BCIs.

A Mini-Review on Functional Near-Infrared Spectroscopy (fNIRS): Where Do We Stand, and Where Should We Go?

This mini-review is aimed at briefly summarizing the present status of functional near-infrared spectroscopy (fNIRS) and predicting where the technique should go in the next decade. This mini-review quotes 33 articles on the different fNIRS basics and technical developments and 44 reviews on the fNIRS applications published in the last eight years. The huge number of review articles about a wide spectrum of topics in the field of cognitive and social sciences, functional neuroimaging research, and medicine testifies to the maturity achieved by this non-invasive optical vascular-based functional neuroimaging technique. Today, fNIRS has started to be utilized on healthy subjects while moving freely in different naturalistic settings. Further instrumental developments are expected to be done in the near future to fully satisfy this latter important aspect. In addition, fNIRS procedures, including correction methods for the strong extracranial interferences, need to be standardized before using fNIRS as a clinical tool in individual patients. New research avenues such as interactive neurosciences, cortical activation modulated by different type of sport performance, and cortical activation during neurofeedback training are highlighted.

Use of fNIRS to Characterize the Neural Mechanism of Inter-Individual Rhythmic Movement Coordination

Background: Inter-individual rhythmic movement coordination plays an important role in daily life, particularly in competitive sports. Behaviorally, it is more challenging to coordinate alternating movements than symmetrical movements. The neural activity underlying these different movement coordination modes remains to be clarified, particularly considering complex inter-individual coordination differences.

Methods: To further test the neural basis of inter-individual rhythmic movement coordination, a revised experimental paradigm of inter-individual coordination was adopted. Participants were asked to perform symmetric, alternate, or single movements (swinging the lower part of the leg) in the same rhythm. A multi-channel, continuous wave, functional near-infrared spectral (fNIRS) imaging instrument was used to monitor hemodynamic activity while 40 volunteers (9 male pairs and 11 female pairs) performed the task. Multivariate analyses of variance were conducted to compare mean oxy-hemoglobin concentration ([HbO]) across experimental conditions.

Results: A significant three-way interaction (leg-swing condition × ROI × laterality) on mean [HbO] was observed. Post hoc analysis revealed a significant main effect of leg-swing condition only in brain regions of interest [right inferior parietal lobule (IPL)] contralateral to movement execution. Activation in brain regions of interest [right inferior parietal lobule (IPL)] was much stronger in alternate mode compared with symmetric or single modes, and the differences between symmetric and single mode were not statistically significant. This result suggests that the alternate mode of movement coordination was more likely to be supported by the IPL region than the other modes.

Conclusion: The present findings provide neural evidence relevant to the theory of self-organization of movement coordination, in which an alternating movement mode appeared to be a more demanding condition than symmetrical movement.

Feasibility of a Real-Time Clinical Augmented Reality and Artificial Intelligence Framework for Pain Detection and Localization From the Brain

Background

For many years, clinicians have been seeking for objective pain assessment solutions via neuroimaging techniques, focusing on the brain to detect human pain. Unfortunately, most of those techniques are not applicable in the clinical environment or lack accuracy.

Objective

This study aimed to test the feasibility of a mobile neuroimaging-based clinical augmented reality (AR) and artificial intelligence (AI) framework, CLARAi, for objective pain detection and also localization direct from the patient’s brain in real time.

Methods

Clinical dental pain was triggered in 21 patients by hypersensitive tooth stimulation with 20 consecutive descending cold stimulations (32°C-0°C). We used a portable optical neuroimaging technology, functional near-infrared spectroscopy, to gauge their cortical activity during evoked acute clinical pain. The data were decoded using a neural network (NN)–based AI algorithm to classify hemodynamic response data into pain and no-pain brain states in real time. We tested the performance of several networks (NN with 7 layers, 6 layers, 5 layers, 3 layers, recurrent NN, and long short-term memory network) upon reorganized data features on pain diction and localization in a simulated real-time environment. In addition, we also tested the feasibility of transmitting the neuroimaging data to an AR device, HoloLens, in the same simulated environment, allowing visualization of the ongoing cortical activity on a 3-dimensional brain template virtually plotted on the patients’ head during clinical consult.

Results

The artificial neutral network (3-layer NN) achieved an optimal classification accuracy at 80.37% (126,000/156,680) for pain and no pain discrimination, with positive likelihood ratio (PLR) at 2.35. We further explored a 3-class localization task of left/right side pain and no-pain states, and convolutional NN-6 (6-layer NN) achieved highest classification accuracy at 74.23% (1040/1401) with PLR at 2.02.

Conclusions

Additional studies are needed to optimize and validate our prototype CLARAi framework for other pains and neurologic disorders. However, we presented an innovative and feasible neuroimaging-based AR/AI concept that can potentially transform the human brain into an objective target to visualize and precisely measure and localize pain in real time where it is most needed: in the doctor’s office.

International Registered Report Identifier (IRRID)

RR1-10.2196/13594

Towards Real-Time, Continuous Decoding of Gripping Force From Deep Brain Local Field Potentials

Lack of force information and longevity issues are impediments to the successful translation of brain-computer interface systems for prosthetic control from experimental settings to widespread clinical application. The ability to decode force using deep brain stimulation electrodes in the subthalamic nucleus (STN) of the basal ganglia provides an opportunity to address these limitations. This paper explores the use of various classes of algorithms (Wiener filter, Wiener-Cascade model, Kalman filter, and dynamic neural networks) and recommends the use of a Wiener-Cascade model for decoding force from STN. This recommendation is influenced by a combination of accuracy and practical considerations to enable real-time, continuous operation. This paper demonstrates an ability to decode a continuous signal (force) from the STN in real time, allowing the possibility of decoding more than two states from the brain at low latency.

Intersession Instability in fNIRS-Based Emotion Recognition

Emotion recognition based on neural signals is a promising technique for the detection of patients' emotions for enhancing healthcare. However, emotion-related neural signals, such as from functional near infrared spectroscopy (fNIRS), can be affected by various psychophysiological and environmental factors. There is a paucity of literature regarding data instability and classification instability in fNIRS-based emotion recognition systems, phenomenon which may lead to user dissatisfaction and abandonment. We collected data in an fNIRS-based 2-class emotion recognition test-retest experiment (3 week interval) with visual stimuli emotion induction to examine data instability and its impact on classification accuracy. We found a 22.2% average deterioration of emotion classification accuracy between the two sessions, suggesting that classification instability is a serious problem. We found that the changes in the distributions of the selected neural signal features, as evaluated by Kullback-Leibler (KL) divergence, were a likely cause of the accuracy decline. We analyzed the data instability and our results showed that instability of spatial activation patterns and instability of the hemodynamic response in the most activated region are correlated with accuracy decline. Finally, we propose a method for mitigating classification instability in fNIRS-based emotion recognition based on feature selection for stable features, the first such method to our knowledge. This new feature selection criterion considers not only the separability of features (evaluated by Fisher Score) but also their stability over time (evaluated by KL divergence between feature distributions at different time points). Testing showed that this method led to an approximately 5% improvement in cross-session generalization accuracy.