Neural Activity and Decoding of Action Observation Using Combined EEG and fNIRS Measurement

Action Observation and Motor Imagery as a Treatment in Patients with Parkinson's Disease

Action observation (AO) and motor imagery (MI) has emerged as promising tool for physiotherapy intervention in Parkinson's disease (PD). This narrative review summarizes why, how, and when applying AO and MI training in individual with PD. We report the neural underpinning of AO and MI and their effects on motor learning. We examine the characteristics and the current evidence regarding the effectiveness of physiotherapy interventions and we provide suggestions about their implementation with technologies. Neurophysiological data suggest a substantial correct activation of brain networks underlying AO and MI in people with PD, although the occurrence of compensatory mechanisms has been documented. Regarding the efficacy of training, in general evidence indicates that both these techniques improve mobility and functional activities in PD. However, these findings should be interpreted with caution due to variety of the study designs, training characteristics, and the modalities in which AO and MI were applied. Finally, results on long-term effects are still uncertain. Several elements should be considered to optimize the use of AO and MI in clinical setting, such as the selection of the task, the imagery or the video perspectives, the modalities of training. However, a comprehensive individual assessment, including motor and cognitive abilities, is essential to select which between AO and MI suite the best to each PD patients. Much unrealized potential exists for the use AO and MI training to provide personalized intervention aimed at fostering motor learning in both the clinic and home setting.

Bimodal EEG-fNIRS in Neuroergonomics. Current Evidence and Prospects for Future Research

Neuroergonomics focuses on the brain signatures and associated mental states underlying behavior to design human-machine interfaces enhancing performance in the cognitive and physical domains. Brain imaging techniques such as functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) have been considered key methods for achieving this goal. Recent research stresses the value of combining EEG and fNIRS in improving these interface systems' mental state decoding abilities, but little is known about whether these improvements generalize over different paradigms and methodologies, nor about the potentialities for using these systems in the real world. We review 33 studies comparing mental state decoding accuracy between bimodal EEG-fNIRS and unimodal EEG and fNIRS in several subdomains of neuroergonomics. In light of these studies, we also consider the challenges of exploiting wearable versions of these systems in real-world contexts. Overall the studies reviewed suggest that bimodal EEG-fNIRS outperforms unimodal EEG or fNIRS despite major differences in their conceptual and methodological aspects. Much work however remains to be done to reach practical applications of bimodal EEG-fNIRS in naturalistic conditions. We consider these points to identify aspects of bimodal EEG-fNIRS research in which progress is expected or desired.

Intelligent Classification Technique of Hand Motor Imagery Using EEG Beta Rebound Follow-Up Pattern

To apply EEG-based brain-machine interfaces during rehabilitation, separating various tasks during motor imagery (MI) and assimilating MI into motor execution (ME) are needed. Previous studies were focusing on classifying different MI tasks based on complex algorithms. In this paper, we implement intelligent, straightforward, comprehensible, time-efficient, and channel-reduced methods to classify ME versus MI and left- versus right-hand MI. EEG of 30 healthy participants undertaking motional tasks is recorded to investigate two classification tasks. For the first task, we first propose a “follow-up” pattern based on the beta rebound. This method achieves an average classification accuracy of 59.77% ± 11.95% and can be up to 89.47% for finger-crossing. Aside from time-domain information, we map EEG signals to feature space using extraction methods including statistics, wavelet coefficients, average power, sample entropy, and common spatial patterns. To evaluate their practicability, we adopt a support vector machine as an intelligent classifier model and sparse logistic regression as a feature selection technique and achieve 79.51% accuracy. Similar approaches are taken for the second classification reaching 75.22% accuracy. The classifiers we propose show high accuracy and intelligence. The achieved results make our approach highly suitable to be applied to the rehabilitation of paralyzed limbs.

Study Protocol of tDCS Based Pain Modulation in Head and Neck Cancer Patients Under Chemoradiation Therapy Condition: An fNIRS-EEG Study

Background

Multiple therapeutic strategies have been adopted to reduce pain, odynophagia, and oral mucositis in head and neck cancer patients. Among them, transcranial direct current stimulation (tDCS) represents a unique analgesic modality. However, the details of tDCS mechanisms in pain treatment are still unclear.

Aims

(1) to study the analgesic effects of a protocol that encompassed supervised-remote and in-clinic tDCS sessions applied in head and neck patients undergoing chemoradiation therapy; (2) to explore the underlining brain mechanisms of such modulation process, using a novel protocol that combined functional near-infrared spectroscopy (fNIRS), and electroencephalograph (EEG), two distinct neuroimaging methods that bring information regarding changes in the hemodynamic as well as in the electrical activity of the brain, respectively.

Methods

This proof-of-concept study was performed on two subjects. The study protocol included a 7-week-long tDCS stimulation procedure, a pre-tDCS baseline session, and two post-tDCS follow-up sessions. Two types of tDCS devices were used. One was used in the clinical setting and the other remotely. Brain imaging was obtained in weeks 1, 2, 5, 7, 8, and after 1 month.

Results

The protocol implemented was safe and reliable. Preliminary results of the fNIRS analysis in weeks 2 and 7 showed a decrease in functional connections between the bilateral prefrontal cortex (PFC) and the primary sensory cortex (S1) (p < 0.05, FDR corrected). Changes in EEG power spectra were found in the PFC when comparing the seventh with the first week of tDCS.

Conclusion

The protocol combining remote and in-clinic administered tDCS and integrated fNIRS and EEG to evaluate the brain activity is feasible. The preliminary results suggest that the mechanisms of tDCS in reducing the pain of head and neck cancer patients may be related to its effects on the connections between the S1 and the PFC.

Viewing neurovascular coupling through the lens of combined EEG-fNIRS: A systematic review of current methods

Neurovascular coupling is a key physiological mechanism that occurs in the healthy human brain, and understanding this process has implications for understanding the aging and neuropsychiatric populations. Combined electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) has emerged as a promising, noninvasive tool for probing neurovascular interactions in humans. However, the utility of this approach critically depends on the methodological quality used for multimodal integration. Despite a growing number of combined EEG-fNIRS applications reported in recent years, the methodological rigor of past studies remains unclear, limiting the accurate interpretation of reported findings and hindering the translational application of this multimodal approach. To fill this knowledge gap, we critically evaluated various methodological aspects of previous combined EEG-fNIRS studies performed in healthy individuals. A literature search was conducted using PubMed and PsycINFO on June 28, 2021. Studies involving concurrent EEG and fNIRS measurements in awake and healthy individuals were selected. After screening and eligibility assessment, 96 studies were included in the methodological evaluation. Specifically, we critically reviewed various aspects of participant sampling, experimental design, signal acquisition, data preprocessing, outcome selection, data analysis, and results presentation reported in these studies. Altogether, we identified several notable strengths and limitations of the existing EEG-fNIRS literature. In light of these limitations and the features of combined EEG-fNIRS, recommendations are made to improve and standardize research practices to facilitate the use of combined EEG-fNIRS when studying healthy neurovascular coupling processes and alterations in neurovascular coupling among various populations.

Multi-class classification of action intention understanding brain signals based on thresholding graph metric

BACKGROUND: Classification of action intention understanding is extremely important for human computer interaction. Many studies on the action intention understanding classification mainly focus on binary classification, while the classification accuracy is often unsatisfactory, not to mention multi-class classification. METHOD: To complete the multi-class classification task of action intention understanding brain signals effectively, we propose a novel feature extraction procedure based on thresholding graph metric. RESULTS: Both the alpha frequency band and full-band obtained considerable classification accuracies. Compared with other methods, the novel method has better classification accuracy. CONCLUSIONS: Brain activity of action intention understanding is closely related to the alpha band. The new feature extraction procedure is an effective method for the multi-class classification of action intention understanding brain signals.

Characterization of Bimanual Cyclical Tasks from Single-trial EEG-fNIRS Measurements

Robot-assisted bimanual training is promising to improve motor function and cortical reorganization for hemiparetic stroke patients. Closing the rehabilitation training loop with neurofeedback can help refine training protocols in time for better engagements and outcomes. However, due to the low signal-to- noise ratio (SNR) and non-stationary properties of neural signals, reliable characterization of bimanual training-induced neural activities from single-trial measurement is challenging. In this study, ten human participants were recruited conducting robot-assisted bimanual cyclical tasks (in-phase, 90° out-of-phase, and anti-phase) when concurrent electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) were recorded. A unified EEG-fNIRS bimodal signal processing framework was proposed to characterize neural activities induced by three types of bimanual cyclical tasks. In this framework, novel artifact removal methods were used to improve the SNR and the task-related component analysis (TRCA) was introduced to increase the reproducibility of EEG-fNIRS bimodal features. The optimized features were transformed into low-dimensional indicators to reliably characterize bimanual training-induced neural activation. The SVM classification results of three bimanual cyclical tasks revealed a good discrimination ability of EEG-fNIRS bimodal indicators (90.1%), which was higher than that using EEG (74.8%) or fNIRS (82.2%) alone, supporting the proposed method as a feasible technique to characterize neural activities during robot-assisted bimanual training.

Advanced Applications of Industrial Robotics: New Trends and Possibilities

This review is dedicated to the advanced applications of robotic technologies in the industrial field. Robotic solutions in areas with non-intensive applications are presented, and their implementations are analysed. We also provide an overview of survey publications and technical reports, classified by application criteria, and the development of the structure of existing solutions, and identify recent research gaps. The analysis results reveal the background to the existing obstacles and problems. These issues relate to the areas of psychology, human nature, special artificial intelligence (AI) implementation, and the robot-oriented object design paradigm. Analysis of robot applications shows that the existing emerging applications in robotics face technical and psychological obstacles. The results of this review revealed four directions of required advancement in robotics: development of intelligent companions; improved implementation of AI-based solutions; robot-oriented design of objects; and psychological solutions for robot–human collaboration.

Combining Action Observation Treatment with a Brain-Computer Interface System: Perspectives on Neurorehabilitation

Action observation treatment (AOT) exploits a neurophysiological mechanism, matching an observed action on the neural substrates where that action is motorically represented. This mechanism is also known as mirror mechanism. In a typical AOT session, one can distinguish an observation phase and an execution phase. During the observation phase, the patient observes a daily action and soon after, during the execution phase, he/she is asked to perform the observed action at the best of his/her ability. Indeed, the execution phase may sometimes be difficult for those patients where motor impairment is severe. Although, in the current practice, the physiotherapist does not intervene on the quality of the execution phase, here, we propose a stimulation system based on neurophysiological parameters. This perspective article focuses on the possibility to combine AOT with a brain–computer interface system (BCI) that stimulates upper limb muscles, thus facilitating the execution of actions during a rehabilitation session. Combining a rehabilitation tool that is well-grounded in neurophysiology with a stimulation system, such as the one proposed, may improve the efficacy of AOT in the treatment of severe neurological patients, including stroke patients, Parkinson’s disease patients, and children with cerebral palsy.

A bimodal deep learning architecture for EEG-fNIRS decoding of overt and imagined speech

OBJECTIVE

Brain-computer interfaces (BCI) studies are increasingly leveraging different attributes of multiple signal modalities simultaneously. Bimodal data acquisition protocols combining the temporal resolution of electroencephalography (EEG) with the spatial resolution of functional near-infrared spectroscopy (fNIRS) require novel approaches to decoding.

METHODS

We present an EEG-fNIRS Hybrid BCI that employs a new bimodal deep neural network architecture consisting of two convolutional sub-networks (subnets) to decode overt and imagined speech. Features from each subnet are fused before further feature extraction and classification. Nineteen participants performed overt and imagined speech in a novel cue-based paradigm enabling investigation of stimulus and linguistic effects on decoding.

RESULTS

Using the hybrid approach, classification accuracies (46.31% and 34.29% for overt and imagined speech, respectively (chance: 25%)) indicated a significant improvement on EEG used independently for imagined speech (p=0.020) while tending towards significance for overt speech (p=0.098). In comparison with fNIRS, significant improvements for both speech-types were achieved with bimodal decoding (p<0.001). There was a mean difference of ~12.02% between overt and imagined speech with accuracies as high as 87.18% and 53%. Deeper subnets enhanced performance while stimulus effected overt and imagined speech in significantly different ways.

CONCLUSION

The bimodal approach was a significant improvement on unimodal results for several tasks. Results indicate the potential of multi-modal deep learning for enhancing neural signal decoding.

SIGNIFICANCE

This novel architecture can be used to enhance speech decoding from bimodal neural signals.

EEG Electrode Selection for a Two-Class Motor Imagery Task in a BCI Using fNIRS Prior Data

This study investigated the possibility of using functional near infrared spectroscopy (fNIRS) during right- and left-hand motor imagery tasks to select an optimum set of electroencephalography (EEG) electrodes for a brain computer interface. fNIRS has better spatial resolution allowing areas of brain activity to more readily be identified. The ReliefF algorithm was used to identify the most reliable fNIRS channels. Then, EEG electrodes adjacent to those channels were selected for classification. This study used three different classifiers of linear and quadratic discriminant analyses, and support vector machine to examine the proposed method.Clinical Relevance- Reducing the number of sensors in a BCI makes the system more usable for patients with severe disabilities.

Developmental Differences in Cortical Activation During Action Observation, Action Execution and Interpersonal Synchrony: An fNIRS Study

Interpersonal synchrony (IPS) is an important everyday behavior influencing social cognitive development; however, few studies have investigated the developmental differences and underlying neural mechanisms of IPS. functional near-infrared spectroscopy (fNIRS) is a novel neuroimaging tool that allows the study of cortical activation in the presence of natural movements. Using fNIRS, we compared cortical activation patterns between children and adults during action observation, execution, and IPS. Seventeen school-age children and 15 adults completed a reach to cleanup task while we obtained cortical activation data from bilateral inferior frontal gyrus (IFG), superior temporal sulcus (STS), and inferior parietal lobes (IPL). Children showed lower spatial and temporal accuracy during IPS compared to adults (i.e., spatial synchrony scores (Mean ± SE) in children: 2.67 ± 0.08 and adults: 2.85 ± 0.06; temporal synchrony scores (Mean ± SE) in children: 2.74 ± 0.06 and adults: 2.88 ± 0.05). For both groups, the STS regions were more activated during action observation, while the IFG and STS were more activated during action execution and IPS. The IPS condition involved more right-sided activation compared to action execution suggesting that IPS is a higher-order process involving more bilateral cortical activation. In addition, adults showed more left lateralization compared to the children during movement conditions (execution and IPS); which indicated greater inhibition of ipsilateral cortices in the adults compared to children. These findings provide a neuroimaging framework to study imitation and IPS impairments in special populations such as children with Autism Spectrum Disorder.