Simultaneous EEG-fNIRS Data Classification Through Selective Channel Representation and Spectrogram Imaging

The integration of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) can facilitate the advancement of brain-computer interfaces (BCIs). However, existing research in this domain has grappled with the challenge of the efficient selection of features, resulting in the underutilization of the temporal richness of EEG and the spatial specificity of fNIRS data.To effectively address this challenge, this study proposed a deep learning architecture called the multimodal DenseNet fusion (MDNF) model that was trained on two-dimensional (2D) EEG data images, leveraging advanced feature extraction techniques. The model transformed EEG data into 2D images using a short-time Fourier transform, applied transfer learning to extract discriminative features, and consequently integrated them with fNIRS-derived spectral entropy features. This approach aimed to bridge existing gaps in EEG-fNIRS-based BCI research by enhancing classification accuracy and versatility across various cognitive and motor imagery tasks.Experimental results on two public datasets demonstrated the superiority of our model over existing state-of-the-art methods.Thus, the high accuracy and precise feature utilization of the MDNF model demonstrates the potential in clinical applications for neurodiagnostics and rehabilitation, thereby paving the method for patient-specific therapeutic strategies.

Simplification of Mobility Tests and Data Processing to Increase Applicability of Wearable Sensors as Diagnostic Tools for Parkinson's Disease

Quantitative mobility analysis using wearable sensors, while promising as a diagnostic tool for Parkinson's disease (PD), is not commonly applied in clinical settings. Major obstacles include uncertainty regarding the best protocol for instrumented mobility testing and subsequent data processing, as well as the added workload and complexity of this multi-step process. To simplify sensor-based mobility testing in diagnosing PD, we analyzed data from 262 PD participants and 50 controls performing several motor tasks wearing a sensor on their lower back containing a triaxial accelerometer and a triaxial gyroscope. Using ensembles of heterogeneous machine learning models incorporating a range of classifiers trained on a set of sensor features, we show that our models effectively differentiate between participants with PD and controls, both for mixed-stage PD (92.6% accuracy) and a group selected for mild PD only (89.4% accuracy). Omitting algorithmic segmentation of complex mobility tasks decreased the diagnostic accuracy of our models, as did the inclusion of kinesiological features. Feature importance analysis revealed that Timed Up and Go (TUG) tasks to contribute the highest-yield predictive features, with only minor decreases in accuracy for models based on cognitive TUG as a single mobility task. Our machine learning approach facilitates major simplification of instrumented mobility testing without compromising predictive performance.

Connecting the dots: Sensory cueing enhances functional connectivity between pre-motor and supplementary motor areas in Parkinson's disease

People with Parkinson's disease often exhibit improvements in motor tasks when exposed to external sensory cues. While the effects of different types of sensory cues on motor functions in Parkinson's disease have been widely studied, the underlying neural mechanism of these effects and the potential of sensory cues to alter the motor cortical activity patterns and functional connectivity of cortical motor areas are still unclear. This study aims to compare changes in oxygenated haemoglobin, deoxygenated haemoglobin and correlations among different cortical regions of interest during wrist movement under different external stimulus conditions between people with Parkinson's disease and controls. Ten Parkinson's disease patients and 10 age- and sex-matched neurologically healthy individuals participated, performing repetitive wrist flexion and extension tasks under auditory and visual cues. Changes in oxygenated and deoxygenated haemoglobin in motor areas were measured using functional near-infrared spectroscopy, along with electromyograms from wrist muscles and wrist movement kinematics. The functional near-infrared spectroscopy data revealed significantly higher neural activity changes in the Parkinson's disease group's pre-motor area compared to controls (p = 0.006), and functional connectivity between the supplementary motor area and pre-motor area was also significantly higher in the Parkinson's disease group when external sensory cues were present (p = 0.016). These results indicate that external sensory cues' beneficial effects on motor tasks are linked to changes in the functional connectivity between motor areas responsible for planning and preparation of movements.

Diagnostic machine learning applications on clinical populations using functional near infrared spectroscopy: a review

Functional near-infrared spectroscopy (fNIRS) and its interaction with machine learning (ML) is a popular research topic for the diagnostic classification of clinical disorders due to the lack of robust and objective biomarkers. This review provides an overview of research on psychiatric diseases by using fNIRS and ML. Article search was carried out and 45 studies were evaluated by considering their sample sizes, used features, ML methodology, and reported accuracy. To our best knowledge, this is the first review that reports diagnostic ML applications using fNIRS. We found that there has been an increasing trend to perform ML applications on fNIRS-based biomarker research since 2010. The most studied populations are schizophrenia (n = 12), attention deficit and hyperactivity disorder (n = 7), and autism spectrum disorder (n = 6) are the most studied populations. There is a significant negative correlation between sample size (>21) and accuracy values. Support vector machine (SVM) and deep learning (DL) approaches were the most popular classifier approaches (SVM = 20) (DL = 10). Eight of these studies recruited a number of participants more than 100 for classification. Concentration changes in oxy-hemoglobin (ΔHbO) based features were used more than concentration changes in deoxy-hemoglobin (ΔHb) based ones and the most popular ΔHbO-based features were mean ΔHbO (n = 11) and ΔHbO-based functional connections (n = 11). Using ML on fNIRS data might be a promising approach to reveal specific biomarkers for diagnostic classification.

Neural dynamics of delayed feedback in robot teleoperation: insights from fNIRS analysis

Introduction

As robot teleoperation increasingly becomes integral in executing tasks in distant, hazardous, or inaccessible environments, operational delays remain a significant obstacle. These delays, inherent in signal transmission and processing, adversely affect operator performance, particularly in tasks requiring precision and timeliness. While current research has made strides in mitigating these delays through advanced control strategies and training methods, a crucial gap persists in understanding the neurofunctional impacts of these delays and the efficacy of countermeasures from a cognitive perspective.

Methods

This study addresses the gap by leveraging functional Near-Infrared Spectroscopy (fNIRS) to examine the neurofunctional implications of simulated haptic feedback on cognitive activity and motor coordination under delayed conditions. In a human-subject experiment (N = 41), sensory feedback was manipulated to observe its influences on various brain regions of interest (ROIs) during teleoperation tasks. The fNIRS data provided a detailed assessment of cerebral activity, particularly in ROIs implicated in time perception and the execution of precise movements.

Results

Our results reveal that the anchoring condition, which provided immediate simulated haptic feedback with a delayed visual cue, significantly optimized neural functions related to time perception and motor coordination. This condition also improved motor performance compared to the asynchronous condition, where visual and haptic feedback were misaligned.

Discussion

These findings provide empirical evidence about the neurofunctional basis of the enhanced motor performance with simulated synthetic force feedback in the presence of teleoperation delays. The study highlights the potential for immediate haptic feedback to mitigate the adverse effects of operational delays, thereby improving the efficacy of teleoperation in critical applications.

Effects of Transcranial Direct Current Stimulation on Clinical Features of Dizziness and Cortical Activation in a Patient with Vestibular Migraine

BACKGROUND

Vestibular migraine (VM) is common migraine that occurs in patients with dizziness. Vestibular rehabilitation for managing VM generally remains unclear. Recently, it has been reported that transcranial direct current stimulation (tDCS) has positive effects in alleviating dizziness. This study investigated the effects of tDCS on dizziness and cortical activation in a patient with VM.

METHODS

We recruited a male patient aged 31 years with no dizziness. The patient watched a video to induce dizziness using a virtual reality device. The study applied the intervention using tDCS for 4 weeks and measured 4 assessments: functional near-infrared spectroscopy (fNIRS), quantitative electroencephalography (qEEG), dizziness handicap inventory, and visual vertigo analog scale.

RESULTS

We showed the activation in the middle temporal gyrus and inferior temporal gyrus (ITG) of the left hemisphere and in the superior temporal gyrus and ITG of the right hemisphere in the pre-intervention. After the intervention, the activation of these areas decreased. In the results of qEEG, excessive activation of C3, P3, and T5 in the left hemisphere and C4 in the right hemisphere before intervention disappeared after the intervention.

CONCLUSIONS

This study indicated that tDCS-based intervention could be considered a viable approach to treating patients with vestibular dysfunction and dizziness caused by VM.

Investigation of electro-vascular phase-amplitude coupling during an auditory task

Multimodal neuroimaging using electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) provides complementary views of cortical processes, including those related to auditory processing. However, current multimodal approaches often overlook potential insights that can be gained from nonlinear interactions between electrical and hemodynamic signals. Here, we explore electro-vascular phase-amplitude coupling (PAC) between low-frequency hemodynamic and high-frequency electrical oscillations during an auditory task. We further apply a temporally embedded canonical correlation analysis (tCCA)-general linear model (GLM)-based correction approach to reduce the possible effect of systemic physiology on fNIRS recordings. Before correction, we observed significant PAC between fNIRS and broadband EEG in the frontal region (p ≪ 0.05), β (p ≪ 0.05) and γ (p = 0.010) in the left temporal/temporoparietal (left auditory; LA) region, and γ (p = 0.032) in the right temporal/temporoparietal (right auditory; RA) region across the entire dataset. Significant differences in PAC across conditions (task versus silence) were observed in LA (p = 0.023) and RA (p = 0.049) γ sub-bands and in lower frequency (5-20 Hz) frontal activity (p = 0.005). After correction, significant fNIRS-γ-band PAC was observed in the frontal (p = 0.021) and LA (p = 0.025) regions, while fNIRS-α (p = 0.003) and fNIRS-β (p = 0.041) PAC were observed in RA. Decreased frontal γ-band (p = 0.008) and increased β-band (p ≪ 0.05) PAC were observed during the task. These outcomes represent the first characterization of electro-vascular PAC between fNIRS and EEG signals during an auditory task, providing insights into electro-vascular coupling in auditory processing.

Physiological Recordings of the Cerebellum in Movement Disorders

The cerebellum plays an important role in movement disorders, specifically in symptoms of ataxia, tremor, and dystonia. Understanding the physiological signals of the cerebellum contributes to insights into the pathophysiology of these movement disorders and holds promise in advancing therapeutic development. Non-invasive techniques such as electroencephalogram and magnetoencephalogram can record neural signals with high temporal resolution at the millisecond level, which is uniquely suitable to interrogate cerebellar physiology. These techniques have recently been implemented to study cerebellar physiology in healthy subjects as well as individuals with movement disorders. In the present review, we focus on the current understanding of cerebellar physiology using these techniques to study movement disorders.

Toward a Multi-Modal Brain-Body Assessment in Parkinson's Disease: A Systematic Review in fNIRS

Parkinson's disease (PD) causes impairments in cortical structures leading to motor and cognitive symptoms. While common disease management and treatment strategies mainly depend on the subjective assessment of clinical scales and patients' diaries, research in recent years has focused on advances in automatic and objective tools to help with diagnosing PD and determining its severity. Due to the link between brain structure deficits and physical symptoms in PD, objective brain activity and body motion assessment of patients have been studied in the literature. This study aimed to explore the relationship between brain activity and body motion measures of people with PD to look at the feasibility of diagnosis or assessment of PD using these measures. In this study, we summarised the findings of 24 selected papers from the complete literature review using the Scopus database. Selected studies used both brain activity recording using functional near-infrared spectroscopy (fNIRS) and motion assessment using sensors for people with PD in their experiments. Results include 1) the most common study protocol is a combination of single tasks. 2) Prefrontal cortex is mostly studied region of interest in the literature. 3) Oxygenated haemoglobin (HbO 2) concentration is the predominant metric utilised in fNIRS, compared to deoxygenated haemoglobin (HHb). 4) Motion assessment in people with PD is mostly done with inertial measurement units (IMUs) and electronic walkway. 5) The relationship between brain activity and body motion measures is an important factor that has been neglected in the literature.

A Narrative Review on Multi-Domain Instrumental Approaches to Evaluate Neuromotor Function in Rehabilitation

In clinical scenarios, the use of biomedical sensors, devices and multi-parameter assessments is fundamental to provide a comprehensive portrait of patients' state, in order to adapt and personalize rehabilitation interventions and support clinical decision-making. However, there is a huge gap between the potential of the multidomain techniques available and the limited practical use that is made in the clinical scenario. This paper reviews the current state-of-the-art and provides insights into future directions of multi-domain instrumental approaches in the clinical assessment of patients involved in neuromotor rehabilitation. We also summarize the main achievements and challenges of using multi-domain approaches in the assessment of rehabilitation for various neurological disorders affecting motor functions. Our results showed that multi-domain approaches combine information and measurements from different tools and biological signals, such as kinematics, electromyography (EMG), electroencephalography (EEG), near-infrared spectroscopy (NIRS), and clinical scales, to provide a comprehensive and objective evaluation of patients' state and recovery. This multi-domain approach permits the progress of research in clinical and rehabilitative practice and the understanding of the pathophysiological changes occurring during and after rehabilitation. We discuss the potential benefits and limitations of multi-domain approaches for clinical decision-making, personalized therapy, and prognosis. We conclude by highlighting the need for more standardized methods, validation studies, and the integration of multi-domain approaches in clinical practice and research.

An EEG-fNIRS neurovascular coupling analysis method to investigate cognitive-motor interference

BACKGROUND AND OBJECTIVE

The simultaneous execution of a motor and cognitive dual task may lead to the deterioration of task performance in one or both tasks due to cognitive-motor interference (CMI). Neuroimaging techniques are promising ways to reveal the underlying neural mechanism of CMI. However, existing studies have only explored CMI from a single neuroimaging modality, which lack built-in validation and comparison of analysis results. This work is aimed to establish an effective analysis framework to comprehensively investigate the CMI by exploring the electrophysiological and hemodynamic activities as well as their neurovascular coupling.

METHODS

Experiments including an upper limb single motor task, single cognitive task, and cognitive-motor dual task were designed and performed with 16 healthy young participants. Bimodal signals of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) were recorded simultaneously during the experiments. A novel bimodal signal analysis framework was proposed to extract the task-related components for EEG and fNIRS signals respectively and analyze their correlation. Indicators including within-class similarity and between-class distance were utilized to validate the effectiveness of the proposed analysis framework compared to the canonical channel-averaged method. Statistical analysis was performed to investigate the difference in the behavior and neural correlates between the single and dual tasks.

RESULTS

Our results revealed that the extra cognitive interference caused divided attention in the dual task, which led to the decreased neurovascular coupling between fNIRS and EEG in all theta, alpha, and beta rhythms. The proposed framework was demonstrated to have a better ability in characterizing the neural patterns than the canonical channel-averaged method with significantly higher within-class similarity and between-class distance indicators.

CONCLUSIONS

This study proposed a method to investigate CMI by exploring the task-related electrophysiological and hemodynamic activities as well as their neurovascular coupling. Our concurrent EEG-fNIRS study provides new insight into the EEG-fNIRS correlation analysis and novel evidence for the mechanism of neurovascular coupling in the CMI.

Cerebral Hemodynamics and Microvasculature Changes in Relation to White Matter Microstructure After Pediatric Mild Traumatic Brain Injury: An A-CAP Pilot Study

Advanced neuroimaging techniques show promise as a biomarker for mild traumatic brain injury (mTBI). However, little research has evaluated cerebral hemodynamics or its relation to white matter microstructure post-mTBI in children. This novel pilot study examined differences in cerebral hemodynamics, as measured using functional near-infrared spectroscopy (fNIRS), and its association with diffusion tensor imaging (DTI) metrics in children with mTBI or mild orthopedic injury (OI) to address these gaps. Children 8.00-16.99 years of age with mTBI (n = 9) or OI (n = 6) were recruited in a pediatric emergency department, where acute injury characteristics were assessed. Participants completed DTI twice, post-acutely (2-33 days) and chronically (3 or 6 months), and fNIRS ∼1 month post-injury. Automated deterministic tractography was used to compute DTI metrics. There was reduced absolute phase globally and coherence in the dorsolateral pre-frontal cortex (DLPFC) after mTBI compared to the OI group. Coherence in the DLPFC and absolute phase globally showed distinct associations with fractional anisotropy in interhemispheric white matter pathways. Two fNIRS metrics (coherence and absolute phase) differentiated mTBI from OI in children. Variability in cerebral hemodynamics related to white matter microstructure. The results provide initial evidence that fNIRS may have utility as a clinical biomarker of pediatric mTBI.

Early-stage fusion of EEG and fNIRS improves classification of motor imagery

Introduction

Many research papers have reported successful implementation of hybrid brain-computer interfaces by complementarily combining EEG and fNIRS, to improve classification performance. However, modality or feature fusion of EEG and fNIRS was usually designed for specific user cases, which were generally customized and hard to be generalized. How to effectively utilize information from the two modalities was still unclear.

Methods

In this paper, we conducted a study to investigate the stage of bi-modal fusion based on EEG and fNIRS. A Y-shaped neural network was proposed and evaluated on an open dataset, which fuses the bimodal information in different stages.

Results

The results suggests that the early-stage fusion of EEG and fNIRS have significantly higher performance compared to middle-stage and late-stage fusion network configuration (N = 57, P < 0.05). With the proposed framework, the average accuracy of 29 participants reaches 76.21% in the left-or-right hand motor imagery task in leave-one-out cross-validation, using bi-modal data as network inputs respectively, which is in the same level as the state-of-the-art hybrid BCI methods based on EEG and fNIRS data.

Experiment protocols for brain-body imaging of locomotion: A systematic review

Introduction

Human locomotion is affected by several factors, such as growth and aging, health conditions, and physical activity levels for maintaining overall health and well-being. Notably, impaired locomotion is a prevalent cause of disability, significantly impacting the quality of life of individuals. The uniqueness and high prevalence of human locomotion have led to a surge of research to develop experimental protocols for studying the brain substrates, muscle responses, and motion signatures associated with locomotion. However, from a technical perspective, reproducing locomotion experiments has been challenging due to the lack of standardized protocols and benchmarking tools, which impairs the evaluation of research quality and the validation of previous findings.

Methods

This paper addresses the challenges by conducting a systematic review of existing neuroimaging studies on human locomotion, focusing on the settings of experimental protocols, such as locomotion intensity, duration, distance, adopted brain imaging technologies, and corresponding brain activation patterns. Also, this study provides practical recommendations for future experiment protocols.

Results

The findings indicate that EEG is the preferred neuroimaging sensor for detecting brain activity patterns, compared to fMRI, fNIRS, and PET. Walking is the most studied human locomotion task, likely due to its fundamental nature and status as a reference task. In contrast, running has received little attention in research. Additionally, cycling on an ergometer at a speed of 60 rpm using fNIRS has provided some research basis. Dual-task walking tasks are typically used to observe changes in cognitive function. Moreover, research on locomotion has primarily focused on healthy individuals, as this is the scenario most closely resembling free-living activity in real-world environments.

Discussion

Finally, the paper outlines the standards and recommendations for setting up future experiment protocols based on the review findings. It discusses the impact of neurological and musculoskeletal factors, as well as the cognitive and locomotive demands, on the experiment design. It also considers the limitations imposed by the sensing techniques used, including the acceptable level of motion artifacts in brain-body imaging experiments and the effects of spatial and temporal resolutions on brain sensor performance. Additionally, various experiment protocol constraints that need to be addressed and analyzed are explained.

Effects of acute high-altitude exposure on working memory: A functional near-infrared spectroscopy study

INTRODUCTION

Inadequate oxygen availability may lead to impairment of neurocognitive functions. The aim of the present study was to investigate the effect of acute high-altitude exposure on the cerebral hemodynamic response and working memory.

METHODS

The same subjects performed working memory exercises with forward and backward digit span tasks both under normal oxygen conditions and in large simulated hypobaric hypoxia chambers, and a series of physiological parameters were evaluated. Functional near-infrared spectroscopy was used to measure cerebral blood flow changes in the dorsolateral prefrontal cortex (DLPFC) during the tasks.

RESULTS

Compared with normoxic conditions, under hypoxic conditions, the heart rate and blood pressure increased, blood oxygen saturation decreased significantly, and the forward task had similar accuracy and response time, while the backward task had lower accuracy and longer response time. Neuroimaging analysis showed increased activation in the DLPFC during the forward task and deactivation during the backward task under hypobaric hypoxia conditions.

CONCLUSION

Acute high-altitude exposure leads to physiological adaptations. The abnormal hemodynamic responses of the DLPFC to hypoxia at low pressure reveal the disruption of neurocognitive function by acute high-altitude exposure, which compromises complex cognitive functions, and provides a promising application for functional near infrared spectroscopy in the exploration of neural mechanisms in the brain during high-altitude exposure.

Diagnosis of Parkinson's disease from hand drawing utilizing hybrid models

Parkinson's disease is a nervous system abnormality marked by decreased dopamine levels in the brain. Parkinson's disease inhibits one's ability to move. Speech difficulty, changes in movement and handwriting, and other symptoms are common with Parkinson's disease. A collection of hand drawings is employed to predict Parkinson's disease. There are 102 spiral images in the hand drawing dataset. Due to the minimal size of the dataset, augmentation is utilized to increase it. After that, the augmented images are utilized to train several machine learning and deep learning models, as well as pre-trained networks like RESNET50, VGG16, AlexNet, and VGG19. The performance metrics of hybrid models of deep learning with machine learning and hybrid models of deep learning (for feature extraction) with deep learning (for classification) are then compared. It was observed that the hybrid model of RESNET-50 and SVM performed well with better performance measures compared to other Machine Learning, Deep Learning and Hybrid Models with an accuracy score of 98.45%, sensitivity score of 0.99 and specificity score of 0.98.

A Two-stream Convolutional Network for Musculoskeletal and Neurological Disorders Prediction

Musculoskeletal and neurological disorders are the most common causes of walking problems among older people, and they often lead to diminished quality of life. Analyzing walking motion data manually requires trained professionals and the evaluations may not always be objective. To facilitate early diagnosis, recent deep learning-based methods have shown promising results for automated analysis, which can discover patterns that have not been found in traditional machine learning methods. We observe that existing work mostly applies deep learning on individual joint features such as the time series of joint positions. Due to the challenge of discovering inter-joint features such as the distance between feet (i.e. the stride width) from generally smaller-scale medical datasets, these methods usually perform sub-optimally. As a result, we propose a solution that explicitly takes both individual joint features and inter-joint features as input, relieving the system from the need of discovering more complicated features from small data. Due to the distinctive nature of the two types of features, we introduce a two-stream framework, with one stream learning from the time series of joint position and the other from the time series of relative joint displacement. We further develop a mid-layer fusion module to combine the discovered patterns in these two streams for diagnosis, which results in a complementary representation of the data for better prediction performance. We validate our system with a benchmark dataset of 3D skeleton motion that involves 45 patients with musculoskeletal and neurological disorders, and achieve a prediction accuracy of 95.56%, outperforming state-of-the-art methods.

Bimodal EEG-fNIRS in Neuroergonomics. Current Evidence and Prospects for Future Research

Neuroergonomics focuses on the brain signatures and associated mental states underlying behavior to design human-machine interfaces enhancing performance in the cognitive and physical domains. Brain imaging techniques such as functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) have been considered key methods for achieving this goal. Recent research stresses the value of combining EEG and fNIRS in improving these interface systems' mental state decoding abilities, but little is known about whether these improvements generalize over different paradigms and methodologies, nor about the potentialities for using these systems in the real world. We review 33 studies comparing mental state decoding accuracy between bimodal EEG-fNIRS and unimodal EEG and fNIRS in several subdomains of neuroergonomics. In light of these studies, we also consider the challenges of exploiting wearable versions of these systems in real-world contexts. Overall the studies reviewed suggest that bimodal EEG-fNIRS outperforms unimodal EEG or fNIRS despite major differences in their conceptual and methodological aspects. Much work however remains to be done to reach practical applications of bimodal EEG-fNIRS in naturalistic conditions. We consider these points to identify aspects of bimodal EEG-fNIRS research in which progress is expected or desired.

Concurrent fNIRS and EEG for Brain Function Investigation: A Systematic, Methodology-Focused Review

Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) stand as state-of-the-art techniques for non-invasive functional neuroimaging. On a unimodal basis, EEG has poor spatial resolution while presenting high temporal resolution. In contrast, fNIRS offers better spatial resolution, though it is constrained by its poor temporal resolution. One important merit shared by the EEG and fNIRS is that both modalities have favorable portability and could be integrated into a compatible experimental setup, providing a compelling ground for the development of a multimodal fNIRS-EEG integration analysis approach. Despite a growing number of studies using concurrent fNIRS-EEG designs reported in recent years, the methodological reference of past studies remains unclear. To fill this knowledge gap, this review critically summarizes the status of analysis methods currently used in concurrent fNIRS-EEG studies, providing an up-to-date overview and guideline for future projects to conduct concurrent fNIRS-EEG studies. A literature search was conducted using PubMed and Web of Science through 31 August 2021. After screening and qualification assessment, 92 studies involving concurrent fNIRS-EEG data recordings and analyses were included in the final methodological review. Specifically, three methodological categories of concurrent fNIRS-EEG data analyses, including EEG-informed fNIRS analyses, fNIRS-informed EEG analyses, and parallel fNIRS-EEG analyses, were identified and explained with detailed description. Finally, we highlighted current challenges and potential directions in concurrent fNIRS-EEG data analyses in future research.

fNIRS-based brain state transition features to signify functional degeneration after Parkinson's disease

OBJECTIVE

Parkinson's disease (PD) is a common neurodegenerative brain disorder, and early diagnosis is of vital importance for treatment. Existing methods are mainly focused on behavior examination, while the functional neurodegeneration after PD has not been well explored. This paper aims to investigate the brain functional variation of PD patients in comparison with healthy controls.

APPROACH

In this work, we propose brain hemodynamic states and state transition features to signify functional degeneration after PD. Firstly, a functional near-infrared spectroscopy (fNIRS)-based experimental paradigm was designed to capture brain activation during dual-task walking from PD patients and healthy controls. Then, three brain states, named expansion, contraction, and intermediate states, were defined with respect to the oxyhemoglobin and deoxyhemoglobin responses. After that, two features were designed from a constructed transition factor and concurrent variations of oxy- and deoxy-hemoglobin over time, to quantify the transitions of brain states. Further, a support vector machine classifier was trained with the proposed features to distinguish PD patients and healthy controls.

RESULTS

Experimental results showed that our method with the proposed brain state transition features achieved classification accuracy of 0:8200 and F score of 0:9091, and outperformed existing fNIRS-based methods. Compared with healthy controls, PD patients had significantly smaller transition acceleration and transition angle.

SIGNIFICANCE

The proposed brain state transition features well signify functional degeneration of PD patients and may serve as promising functional biomarkers for PD diagnosis.

A Multimodal Data Driven Rehabilitation Strategy Auxiliary Feedback Method: A Case Study

In Industry 4.0, medical data present a trend of multisource development. However, in complex information networks, an information gap often exists in data exchange between doctors and patients. In the case of diseases with complex manifestations, doctors often perform qualitative analysis, which is macroscopic and fuzzy, to present treatment recommendations for patients. Improving the reliability of data acquisition and maximizing the potential of data, require attention. To solve these problems, a multimodal data-driven rehabilitation strategy auxiliary feedback method is proposed. In this study, depth sensor and functional near-infrared spectroscopy (fNIRS) were used to obtain ethology and brain function data, and skeleton tracking analysis and ethology discrete statistics were performed to assist the diagnostic feedback of rehabilitation strategies. This study takes rhythm rehabilitation training of autistic children as a case, and results show that the multimodal data-driven rehabilitation strategy auxiliary feedback method can provide effective feedback for individuals or groups. The proposed auxiliary decision method increases the dimension of data analysis and improves the reliability of analysis. Through discrete statistical results, the potential of data are maximized, thereby assisting the proposed rehabilitation strategy diagnostic feedback.

Characterization of Bimanual Cyclical Tasks from Single-trial EEG-fNIRS Measurements

Robot-assisted bimanual training is promising to improve motor function and cortical reorganization for hemiparetic stroke patients. Closing the rehabilitation training loop with neurofeedback can help refine training protocols in time for better engagements and outcomes. However, due to the low signal-to- noise ratio (SNR) and non-stationary properties of neural signals, reliable characterization of bimanual training-induced neural activities from single-trial measurement is challenging. In this study, ten human participants were recruited conducting robot-assisted bimanual cyclical tasks (in-phase, 90° out-of-phase, and anti-phase) when concurrent electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) were recorded. A unified EEG-fNIRS bimodal signal processing framework was proposed to characterize neural activities induced by three types of bimanual cyclical tasks. In this framework, novel artifact removal methods were used to improve the SNR and the task-related component analysis (TRCA) was introduced to increase the reproducibility of EEG-fNIRS bimodal features. The optimized features were transformed into low-dimensional indicators to reliably characterize bimanual training-induced neural activation. The SVM classification results of three bimanual cyclical tasks revealed a good discrimination ability of EEG-fNIRS bimodal indicators (90.1%), which was higher than that using EEG (74.8%) or fNIRS (82.2%) alone, supporting the proposed method as a feasible technique to characterize neural activities during robot-assisted bimanual training.

Development of an Integrated EEG/fNIRS Brain Function Monitoring System

In this study, a fully integrated electroencephalogram/functional near-infrared spectroscopy (EEG/fNIRS) brain monitoring system was designed to fulfill the demand for a miniaturized, light-weight, low-power-consumption, and low-cost brain monitoring system as a potential tool with which to screen for brain diseases. The system is based on the ADS1298IPAG Analog Front-End (AFE) and can simultaneously acquire two-channel EEG signals with a sampling rate of 250 SPS and six-channel fNIRS signals with a sampling rate of 8 SPS. AFE is controlled by Teensy 3.2 and powered by a lithium polymer battery connected to two protection circuits and regulators. The acquired EEG and fNIRS signals are monitored and stored using a Graphical User Interface (GUI). The system was evaluated by implementing several tests to verify its ability to simultaneously acquire EEG and fNIRS signals. The implemented system can acquire EEG and fNIRS signals with a CMRR of -115 dB, power consumption of 0.75 mW/ch, system weight of 70.5 g, probe weight of 3.1 g, and a total cost of USD 130. The results proved that this system can be qualified as a low-cost, light-weight, low-power-consumption, and fully integrated EEG/fNIRS brain monitoring system.

Wearable, Integrated EEG-fNIRS Technologies: A Review

There has been considerable interest in applying electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) simultaneously for multimodal assessment of brain function. EEG–fNIRS can provide a comprehensive picture of brain electrical and hemodynamic function and has been applied across various fields of brain science. The development of wearable, mechanically and electrically integrated EEG–fNIRS technology is a critical next step in the evolution of this field. A suitable system design could significantly increase the data/image quality, the wearability, patient/subject comfort, and capability for long-term monitoring. Here, we present a concise, yet comprehensive, review of the progress that has been made toward achieving a wearable, integrated EEG–fNIRS system. Significant marks of progress include the development of both discrete component-based and microchip-based EEG–fNIRS technologies; modular systems; miniaturized, lightweight form factors; wireless capabilities; and shared analogue-to-digital converter (ADC) architecture between fNIRS and EEG data acquisitions. In describing the attributes, advantages, and disadvantages of current technologies, this review aims to provide a roadmap toward the next generation of wearable, integrated EEG–fNIRS systems.

Tracking Brain Development From Neonates to the Elderly by Hemoglobin Phase Measurement Using Functional Near-Infrared Spectroscopy

The biological and neurological processes during the lifespan are dynamic with significant alterations associated with different stages of life. The phase and coupling of oxy-hemoglobin (Δ[HbO]) and deoxy-hemoglobin concentration changes (Δ[Hb]) measured by functional near-infrared spectroscopy (fNIRS) are shown to characterize the neurovascular and metabolic development of infants. However, the changes in phase and coupling across the human lifespan remain mostly unknown. Here, fNIRS measurements of Δ[HbO] and Δ[Hb] conducted at two sites on different age populations (from newborns to elderly) were combined. Firstly, we assessed the influence of random noise on the calculation of the phase difference and phase-locking index (PLI) in fNIRS measurement. The results showed that the phase difference is close to π as the noise intensity approaches -8 dB, and the coupling strength (i.e., PLI) presents a u-shape curve as the noise increase. Secondly, phase difference and PLI in the frequency range 0.01-0.10 Hz were calculated after denoising. It showed that the phase difference increases from newborns to 3-4-month-olds babies. This phase difference persists throughout adulthood until finally being disrupted in the old age. The children's PLI is the highest, followed by that of adults. These two groups' PLI are significantly higher than those of infants and the elderly (p < 0.001). Lastly, a hemodynamic model was used to explain the observations and found close associations with cerebral autoregulation and speed of blood flow. These results demonstrate that the phase-related parameters measured by fNIRS can be used to study the brain and assess brain health throughout the lifespan.

Virtual Reality-Based Framework to Simulate Control Algorithms for Robotic Assistance and Rehabilitation Tasks through a Standing Wheelchair

The implementation of control algorithms oriented to robotic assistance and rehabilitation tasks for people with motor disabilities has been of increasing interest in recent years. However, practical implementation cannot be carried out unless one has the real robotic system availability. To overcome this drawback, this article presents the development of an interactive virtual reality (VR)-based framework that allows one to simulate the execution of rehabilitation tasks and robotic assistance through a robotic standing wheelchair. The virtual environment developed considers the kinematic and dynamic model of the standing human–wheelchair system with a displaced center of mass, since it can be displaced for different reasons, e.g.,: bad posture, limb amputations, obesity, etc. The standing wheelchair autonomous control scheme has been implemented through the Full Simulation (FS) and Hardware in the Loop (HIL) techniques. Finally, the performance of the virtual control schemes has been shown by means of several experiments based on robotic assistance and rehabilitation for people with motor disabilities.

Cerebral Hemodynamic Responses to the Difficulty Level of Ambulatory Tasks in Patients With Parkinson's Disease: A Systematic Review and Meta-Analysis

Background. Ambulatory tasks are the important components of balance training which effectively improve postural stability and functional activities in persons with Parkinson's disease (PD). The difficulty level of an ambulatory task is usually set in the form of attention, direction, speed, or amplitude requirement. Objectives. This study aimed to explore the neural mechanisms of cerebral hemodynamic responses to the difficulty level of ambulatory tasks in persons with PD. Methods. We included ten studies that examined cerebral hemodynamic responses during ambulatory tasks at different difficulty levels in persons with PD. The change in hemodynamic responses was synthesized and meta-analyzed. Results. Patients during "ON" medication had higher relative change in oxygenated hemoglobin (ΔHBO₂) in the prefrontal cortex in response to difficulty levels of ambulatory tasks, which is comparable to that in healthy elderly individuals. However, patients during "OFF" medication did not show cortical activation in response to difficulty levels. During the lower-difficulty tasks, patients during "ON" medication demonstrated higher ΔHBO₂ than healthy elderly participants and patients during "OFF" medication. Factors found to significantly contribute to the heterogeneity across studies included subjects' type and cognitive status, task duration, setting, and filter used for functional near-infrared spectroscopy (fNIRS) data pre-processing. Conclusions. The findings suggest that ambulatory task at a higher difficulty level could be necessary to train the cortical capacity of PD persons, which should be conducted during "ON" medication; meanwhile, the contributing factors to the heterogeneity of studies would be useful as a reference when designing comparable fNIRS studies.

Wearable sensors in the diagnosis and study of Parkinson's disease symptoms: a systematic review

Nowadays, there are several diseases which affect different systems of the body, producing changes in the correct functioning of the organism and the people lifestyles. One of them is Parkinson's disease (PD), which is defined as a neurodegenerative disorder provoked by the destruction of dopaminergic neurons in the brain, resulting in a set of motor and non-motor symptoms. As this disease affects principally to ancient people, several researchers have studied different treatments and therapies for stopping neurodegeneration and diminishing symptoms, to improve the quality patients' lives. The most common therapies created for PD are based on pharmacological treatment for controlling the degeneration advance and the physical ones which do not reveal the progress of patients. For this reason, this review paper opens the possibility for using wearable motion capture systems as an option for the control and study of PD. Therefore, it aims to (1) study the different wearable systems used for capture the movements of PD patients and (2) determine which of them bring better results for monitoring and assess PD people. For the analysis, it uses papers based on experiments that prove the functioning of several motion systems in different aspects as monitoring, treatment and diagnose of the disease. As a result, it works with 30 papers which describe the factors mentioned before. Additionally, the paper uses journals and literature review about the pathology, its characteristics and the function of wearable sensors for the correct understanding of the topic.

Gait Recognition Using Optical Motion Capture: A Decision Fusion Based Method

Human identification based on motion capture data has received signification attentions for its wide applications in authentication and surveillance systems. The optical motion capture system (OMCS) can dynamically capture the high-precision three-dimensional locations of optical trackers that are implemented on a human body, but its potential in applications on gait recognition has not been studied in existing works. On the other hand, a typical OMCS can only support one player one time, which limits its capability and efficiency. In this paper, our goals are investigating the performance of OMCS-based gait recognition performance, and realizing gait recognition in OMCS such that it can support multiple players at the same time. We develop a gait recognition method based on decision fusion, and it includes the following four steps: feature extraction, unreliable feature calibration, classification of single motion frame, and decision fusion of multiple motion frame. We use kernel extreme learning machine (KELM) for single motion classification, and in particular we propose a reliability weighted sum (RWS) decision fusion method to combine the fuzzy decisions of the motion frames. We demonstrate the performance of the proposed method by using walking gait data collected from 76 participants, and results show that KELM significantly outperforms support vector machine (SVM) and random forest in the single motion frame classification task, and demonstrate that the proposed RWS decision fusion rule can achieve better fusion accuracy compared with conventional fusion rules. Our results also show that, with 10 motion trackers that are implemented on lower body locations, the proposed method can achieve 100% validation accuracy with less than 50 gait motion frames.

Disruptions of cortico-kinematic interactions in Parkinson's disease

The cortical role of the motor symptoms reflected by kinematic characteristics in Parkinson's disease (PD) is poorly understood. In this study, we aim to explore how PD affects cortico-kinematic interactions. Electroencephalographic (EEG) and kinematic data were recorded from seven healthy participants and eight participants diagnosed with PD during a set of self-paced finger tapping tasks. Event-related desynchronization (ERD) was compared between groups in the α (8-14 Hz), low-ß (14-20 Hz), and high-ß (20-35 Hz) frequency bands to investigate between-group differences in the cortical activities associated with movement. Average kinematic peak amplitudes and latencies were extracted alongside Sample Entropy (SaEn), a measure of signal complexity, as variables for comparison between groups. These variables were further correlated with average EEG power in each frequency band to establish within-group interactions between cortical motor functions and kinematic motor output. High ß-band power correlated with mean kinematic peak latency and signal complexity in the healthy group, while no correlation was found in the PD group. Also, the healthy group demonstrated stronger ERD in the broad ß-band than the PD participants. Our results suggest that cortical ß-band power in healthy populations is graded to finger tapping latency and complexity of movement, but this relationship is impaired in PD. These insights could help further enhance our understanding of the role of cortical ß-band oscillations in healthy movement and the possible disruption of that relationship in PD. These outcomes can provide further directions for treatment and therapeutic applications and potentially establish cortical biomarkers of Parkinson's disease.

Analysis of Human Gait Using Hybrid EEG-fNIRS-Based BCI System: A Review

Human gait is a complex activity that requires high coordination between the central nervous system, the limb, and the musculoskeletal system. More research is needed to understand the latter coordination's complexity in designing better and more effective rehabilitation strategies for gait disorders. Electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) are among the most used technologies for monitoring brain activities due to portability, non-invasiveness, and relatively low cost compared to others. Fusing EEG and fNIRS is a well-known and established methodology proven to enhance brain-computer interface (BCI) performance in terms of classification accuracy, number of control commands, and response time. Although there has been significant research exploring hybrid BCI (hBCI) involving both EEG and fNIRS for different types of tasks and human activities, human gait remains still underinvestigated. In this article, we aim to shed light on the recent development in the analysis of human gait using a hybrid EEG-fNIRS-based BCI system. The current review has followed guidelines of preferred reporting items for systematic reviews and meta-Analyses (PRISMA) during the data collection and selection phase. In this review, we put a particular focus on the commonly used signal processing and machine learning algorithms, as well as survey the potential applications of gait analysis. We distill some of the critical findings of this survey as follows. First, hardware specifications and experimental paradigms should be carefully considered because of their direct impact on the quality of gait assessment. Second, since both modalities, EEG and fNIRS, are sensitive to motion artifacts, instrumental, and physiological noises, there is a quest for more robust and sophisticated signal processing algorithms. Third, hybrid temporal and spatial features, obtained by virtue of fusing EEG and fNIRS and associated with cortical activation, can help better identify the correlation between brain activation and gait. In conclusion, hBCI (EEG + fNIRS) system is not yet much explored for the lower limb due to its complexity compared to the higher limb. Existing BCI systems for gait monitoring tend to only focus on one modality. We foresee a vast potential in adopting hBCI in gait analysis. Imminent technical breakthroughs are expected using hybrid EEG-fNIRS-based BCI for gait to control assistive devices and Monitor neuro-plasticity in neuro-rehabilitation. However, although those hybrid systems perform well in a controlled experimental environment when it comes to adopting them as a certified medical device in real-life clinical applications, there is still a long way to go.