1
|
Das Chakladar D, Roy PP. Cognitive workload estimation using physiological measures: a review. Cogn Neurodyn 2024; 18:1445-1465. [PMID: 39104683 PMCID: PMC11297869 DOI: 10.1007/s11571-023-10051-3] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/16/2023] [Revised: 11/14/2023] [Accepted: 11/28/2023] [Indexed: 08/07/2024] Open
Abstract
Estimating cognitive workload levels is an emerging research topic in the cognitive neuroscience domain, as participants' performance is highly influenced by cognitive overload or underload results. Different physiological measures such as Electroencephalography (EEG), Functional Magnetic Resonance Imaging, Functional near-infrared spectroscopy, respiratory activity, and eye activity are efficiently used to estimate workload levels with the help of machine learning or deep learning techniques. Some reviews focus only on EEG-based workload estimation using machine learning classifiers or multimodal fusion of different physiological measures for workload estimation. However, a detailed analysis of all physiological measures for estimating cognitive workload levels still needs to be discovered. Thus, this survey highlights the in-depth analysis of all the physiological measures for assessing cognitive workload. This survey emphasizes the basics of cognitive workload, open-access datasets, the experimental paradigm of cognitive tasks, and different measures for estimating workload levels. Lastly, we emphasize the significant findings from this review and identify the open challenges. In addition, we also specify future scopes for researchers to overcome those challenges.
Collapse
Affiliation(s)
- Debashis Das Chakladar
- Department of Computer Science and Engineering, Indian Institute of Technology Roorkee, Roorkee, Uttarakhand India
| | - Partha Pratim Roy
- Department of Computer Science and Engineering, Indian Institute of Technology Roorkee, Roorkee, Uttarakhand India
| |
Collapse
|
2
|
Blanco-Diaz CF, Guerrero-Mendez CD, de Andrade RM, Badue C, De Souza AF, Delisle-Rodriguez D, Bastos-Filho T. Decoding lower-limb kinematic parameters during pedaling tasks using deep learning approaches and EEG. Med Biol Eng Comput 2024:10.1007/s11517-024-03147-3. [PMID: 39028484 DOI: 10.1007/s11517-024-03147-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/18/2023] [Accepted: 05/29/2024] [Indexed: 07/20/2024]
Abstract
Stroke is a neurological condition that usually results in the loss of voluntary control of body movements, making it difficult for individuals to perform activities of daily living (ADLs). Brain-computer interfaces (BCIs) integrated into robotic systems, such as motorized mini exercise bikes (MMEBs), have been demonstrated to be suitable for restoring gait-related functions. However, kinematic estimation of continuous motion in BCI systems based on electroencephalography (EEG) remains a challenge for the scientific community. This study proposes a comparative analysis to evaluate two artificial neural network (ANN)-based decoders to estimate three lower-limb kinematic parameters: x- and y-axis position of the ankle and knee joint angle during pedaling tasks. Long short-term memory (LSTM) was used as a recurrent neural network (RNN), which reached Pearson correlation coefficient (PCC) scores close to 0.58 by reconstructing kinematic parameters from the EEG features on the delta band using a time window of 250 ms. These estimates were evaluated through kinematic variance analysis, where our proposed algorithm showed promising results for identifying pedaling and rest periods, which could increase the usability of classification tasks. Additionally, negative linear correlations were found between pedaling speed and decoder performance, thereby indicating that kinematic parameters between slower speeds may be easier to estimate. The results allow concluding that the use of deep learning (DL)-based methods is feasible for the estimation of lower-limb kinematic parameters during pedaling tasks using EEG signals. This study opens new possibilities for implementing controllers most robust for MMEBs and BCIs based on continuous decoding, which may allow for maximizing the degrees of freedom and personalized rehabilitation.
Collapse
Affiliation(s)
| | | | | | - Claudine Badue
- Department of Informatics, Federal University of Espirito Santo, Vitoria, Brazil
| | | | - Denis Delisle-Rodriguez
- Edmond and Lily Safra International Institute of Neurosciences, Santos Dumont Institute, Macaiba, RN, Brazil
| | - Teodiano Bastos-Filho
- Postgraduate Program in Electrical Engineering, Federal University of Espirito Santo, Vitoria, Brazil
| |
Collapse
|
3
|
Lee SH, Paik SH, Kang SY, Phillips Z, Kim JB, Kim BJ, Kim BM. Convolutional neural networks can detect orthostatic hypotension in Parkinson's disease using resting-state functional near-infrared spectroscopy data. JOURNAL OF BIOPHOTONICS 2024:e202400138. [PMID: 38952169 DOI: 10.1002/jbio.202400138] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/02/2024] [Revised: 05/10/2024] [Accepted: 06/10/2024] [Indexed: 07/03/2024]
Abstract
Neurological disorders such as Parkinson's disease (PD) often adversely affect the vascular system, leading to alterations in blood flow patterns. Functional near-infrared spectroscopy (fNIRS) is used to monitor hemodynamic changes via signal measurement. This study investigated the potential of using resting-state fNIRS data through a convolutional neural network (CNN) to evaluate PD with orthostatic hypotension. The CNN demonstrated significant efficacy in analyzing fNIRS data, and it outperformed the other machine learning methods. The results indicate that judicious input data selection can enhance accuracy by over 85%, while including the correlation matrix as an input further improves the accuracy to more than 90%. This study underscores the promising role of CNN-based fNIRS data analysis in the diagnosis and management of the PD. This approach enhances diagnostic accuracy, particularly in resting-state conditions, and can reduce the discomfort and risks associated with current diagnostic methods, such as the head-up tilt test.
Collapse
Affiliation(s)
- Seung Hyun Lee
- Global Health Technology Research Center, Korea University, Seoul, Republic of Korea
| | | | - Shin-Young Kang
- Department of Biomedical Engineering, Korea University, Seoul, Republic of Korea
| | - Zephaniah Phillips
- Global Health Technology Research Center, Korea University, Seoul, Republic of Korea
| | - Jung Bin Kim
- Department of Neurology, Korea University Anam Hospital, Korea University College of Medicine, Seoul, Republic of Korea
| | - Byung-Jo Kim
- Department of Neurology, Korea University Anam Hospital, Korea University College of Medicine, Seoul, Republic of Korea
| | - Beop-Min Kim
- Department of Biomedical Engineering, Korea University, Seoul, Republic of Korea
| |
Collapse
|
4
|
Liao L, Lu J, Wang L, Zhang Y, Gao D, Wang M. CT-Net: an interpretable CNN-Transformer fusion network for fNIRS classification. Med Biol Eng Comput 2024:10.1007/s11517-024-03138-4. [PMID: 38816665 DOI: 10.1007/s11517-024-03138-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/27/2023] [Accepted: 05/23/2024] [Indexed: 06/01/2024]
Abstract
Functional near-infrared spectroscopy (fNIRS), an optical neuroimaging technique, has been widely used in the field of brain activity recognition and brain-computer interface. Existing works have proposed deep learning-based algorithms for the fNIRS classification problem. In this paper, a novel approach based on convolutional neural network and Transformer, named CT-Net, is established to guide the deep modeling for the classification of mental arithmetic (MA) tasks. We explore the effect of data representations, and design a temporal-level combination of two raw chromophore signals to improve the data utilization and enrich the feature learning of the model. We evaluate our model on two open-access datasets and achieve the classification accuracy of 98.05% and 77.61%, respectively. Moreover, we explain our model by the gradient-weighted class activation mapping, which presents a high consistent between the contributing value of features learned by the model and the mapping of brain activity in the MA task. The results suggest the feasibility and interpretability of CT-Net for decoding MA tasks.
Collapse
Affiliation(s)
- Lingxiang Liao
- School of Computer Science, Chengdu University of Information Technology, Chengdu, 610225, China
| | - Jingqing Lu
- School of Life Sciences and Technology, University of Electronic Science and Technology of China, Chengdu, 611731, China
| | - Lutao Wang
- School of Computer Science, Chengdu University of Information Technology, Chengdu, 610225, China
| | - Yongqing Zhang
- School of Computer Science, Chengdu University of Information Technology, Chengdu, 610225, China
| | - Dongrui Gao
- School of Computer Science, Chengdu University of Information Technology, Chengdu, 610225, China
| | - Manqing Wang
- School of Computer Science, Chengdu University of Information Technology, Chengdu, 610225, China.
- School of Life Sciences and Technology, University of Electronic Science and Technology of China, Chengdu, 611731, China.
| |
Collapse
|
5
|
Akila V, Christaline JA, Edward AS. Novel Feature Generation for Classification of Motor Activity from Functional Near-Infrared Spectroscopy Signals Using Machine Learning. Diagnostics (Basel) 2024; 14:1008. [PMID: 38786306 PMCID: PMC11119315 DOI: 10.3390/diagnostics14101008] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2024] [Revised: 05/03/2024] [Accepted: 05/07/2024] [Indexed: 05/25/2024] Open
Abstract
Recent research in the field of cognitive motor action decoding focuses on data acquired from Functional Near-Infrared Spectroscopy (fNIRS) and its analysis. This research aims to classify two different motor activities, namely, mental drawing (MD) and spatial navigation (SN), using fNIRS data from non-motor baseline data and other motor activities. Accurate activity detection in non-stationary signals like fNIRS is challenging and requires complex feature descriptors. As a novel framework, a new feature generation by fusion of wavelet feature, Hilbert, symlet, and Hjorth parameters is proposed for improving the accuracy of the classification. This new fused feature has statistical descriptor elements, time-localization in the frequency domain, edge feature, texture features, and phase information to detect and locate the activity accurately. Three types of independent component analysis, including FastICA, Picard, and Infomax were implemented for preprocessing which removes noises and motion artifacts. Two independent binary classifiers are designed to handle the complexity of classification in which one is responsible for mental drawing (MD) detection and the other one is spatial navigation (SN). Four different types of algorithms including nearest neighbors (KNN), Linear Discriminant Analysis (LDA), light gradient-boosting machine (LGBM), and Extreme Gradient Boosting (XGBOOST) were implemented. It has been identified that the LGBM classifier gives high accuracies-98% for mental drawing and 97% for spatial navigation. Comparison with existing research proves that the proposed method gives the highest classification accuracies. Statistical validation of the proposed new feature generation by the Kruskal-Wallis H-test and Mann-Whitney U non-parametric test proves the reliability of the proposed mechanism.
Collapse
Affiliation(s)
- V. Akila
- Department of ECE, SRM Institute of Science and Technology, Vadapalani, Chennai 600026, India; (J.A.C.); (A.S.E.)
| | | | | |
Collapse
|
6
|
Pang R, Sang H, Yi L, Gao C, Xu H, Wei Y, Zhang L, Sun J. Working memory load recognition with deep learning time series classification. BIOMEDICAL OPTICS EXPRESS 2024; 15:2780-2797. [PMID: 38855665 PMCID: PMC11161351 DOI: 10.1364/boe.516063] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 12/14/2023] [Revised: 01/19/2024] [Accepted: 01/31/2024] [Indexed: 06/11/2024]
Abstract
Working memory load (WML) is one of the widely applied signals in the areas of human-machine interaction. The precise evaluation of the WML is crucial for this kind of application. This study aims to propose a deep learning (DL) time series classification (TSC) model for inter-subject WML decoding. We used fNIRS to record the hemodynamic signals of 27 participants during visual working memory tasks. Traditional machine learning and deep time series classification algorithms were respectively used for intra-subject and inter-subject WML decoding from the collected blood oxygen signals. The intra-subject classification accuracy of LDA and SVM were 94.6% and 79.1%. Our proposed TAResnet-BiLSTM model had the highest inter-subject WML decoding accuracy, reaching 92.4%. This study provides a new idea and method for the brain-computer interface application of fNIRS in real-time WML detection.
Collapse
Affiliation(s)
- Richong Pang
- Barco Technology Limited, Zhuhai 519031, China
- Joint Laboratory of Brain-Verse Digital Convergence, Guangdong Institute of Intelligence Science and Technology, Zhuhai 519031, China
| | - Haojun Sang
- Chinese Institute for Brain Research, Beijing 102206, China
| | - Li Yi
- School of Mechatronic Engineering and Automation, Foshan University, Foshan 528000, China
| | - Chenyang Gao
- Institute of Biomedical Engineering, Chinese Academy of Medical Sciences and Peking Union Medical College, Tianjin 300110, China
| | - Hongkai Xu
- Barco Technology Limited, Zhuhai 519031, China
- Joint Laboratory of Brain-Verse Digital Convergence, Guangdong Institute of Intelligence Science and Technology, Zhuhai 519031, China
| | - Yanzhao Wei
- Barco Technology Limited, Zhuhai 519031, China
- Joint Laboratory of Brain-Verse Digital Convergence, Guangdong Institute of Intelligence Science and Technology, Zhuhai 519031, China
| | - Lei Zhang
- Chinese Institute for Brain Research, Beijing 102206, China
| | - Jinyan Sun
- School of Medicine, Foshan University, Foshan 528000, China
| |
Collapse
|
7
|
Khanam F, Ahmad M, Hossain ABMA. Investigation of the neural correlation with task performance and its effect on cognitive load level classification. PLoS One 2023; 18:e0291576. [PMID: 38127869 PMCID: PMC10735190 DOI: 10.1371/journal.pone.0291576] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2023] [Accepted: 08/31/2023] [Indexed: 12/23/2023] Open
Abstract
Electroencephalogram (EEG)-based cognitive load assessment is now an important assignment in psychological research. This type of research work is conducted by providing some mental task to the participants and their responses are counted through their EEG signal. In general assumption, it is considered that during different tasks, the cognitive workload is increased. This paper has investigated this specific idea and showed that the conventional hypothesis is not correct always. This paper showed that cognitive load can be varied according to the performance of the participants. In this paper, EEG data of 36 participants are taken against their resting and task (mental arithmetic) conditions. The features of the signal were extracted using the empirical mode decomposition (EMD) method and classified using the support vector machine (SVM) model. Based on the classification accuracy, some hypotheses are built upon the impact of subjects' performance on cognitive load. Based on some statistical consideration and graphical justification, it has been shown how the hypotheses are valid. This result will help to construct the machine learning-based model in predicting the cognitive load assessment more appropriately in a subject-independent approach.
Collapse
Affiliation(s)
- Farzana Khanam
- Department of Biomedical Engineering, Khulna University of Engineering & Technology (KUET), Khulna, Bangladesh
- Department of Biomedical Engineering, Jashore University of Science and Technology (JUST), Jashore, Bangladesh
| | - Mohiuddin Ahmad
- Department of Electrical and Electronic Engineering, Khulna University of Engineering & Technology (KUET), Khulna, Bangladesh
| | - A. B. M. Aowlad Hossain
- Department of Electronics and Communication Engineering, Khulna University of Engineering & Technology (KUET), Khulna, Bangladesh
| |
Collapse
|
8
|
Barnova K, Mikolasova M, Kahankova RV, Jaros R, Kawala-Sterniuk A, Snasel V, Mirjalili S, Pelc M, Martinek R. Implementation of artificial intelligence and machine learning-based methods in brain-computer interaction. Comput Biol Med 2023; 163:107135. [PMID: 37329623 DOI: 10.1016/j.compbiomed.2023.107135] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/20/2023] [Revised: 05/13/2023] [Accepted: 06/04/2023] [Indexed: 06/19/2023]
Abstract
Brain-computer interfaces are used for direct two-way communication between the human brain and the computer. Brain signals contain valuable information about the mental state and brain activity of the examined subject. However, due to their non-stationarity and susceptibility to various types of interference, their processing, analysis and interpretation are challenging. For these reasons, the research in the field of brain-computer interfaces is focused on the implementation of artificial intelligence, especially in five main areas: calibration, noise suppression, communication, mental condition estimation, and motor imagery. The use of algorithms based on artificial intelligence and machine learning has proven to be very promising in these application domains, especially due to their ability to predict and learn from previous experience. Therefore, their implementation within medical technologies can contribute to more accurate information about the mental state of subjects, alleviate the consequences of serious diseases or improve the quality of life of disabled patients.
Collapse
Affiliation(s)
- Katerina Barnova
- Department of Cybernetics and Biomedical Engineering, Faculty of Electrical Engineering and Computer Science, VSB-Technical University of Ostrava, Czechia.
| | - Martina Mikolasova
- Department of Cybernetics and Biomedical Engineering, Faculty of Electrical Engineering and Computer Science, VSB-Technical University of Ostrava, Czechia.
| | - Radana Vilimkova Kahankova
- Department of Cybernetics and Biomedical Engineering, Faculty of Electrical Engineering and Computer Science, VSB-Technical University of Ostrava, Czechia
| | - Rene Jaros
- Department of Cybernetics and Biomedical Engineering, Faculty of Electrical Engineering and Computer Science, VSB-Technical University of Ostrava, Czechia.
| | - Aleksandra Kawala-Sterniuk
- Faculty of Electrical Engineering, Automatic Control and Informatics, Opole University of Technology, Poland.
| | - Vaclav Snasel
- Department of Computer Science, Faculty of Electrical Engineering and Computer Science, VSB-Technical University of Ostrava, Czechia.
| | - Seyedali Mirjalili
- Centre for Artificial Intelligence Research and Optimisation, Torrens University Australia, Australia.
| | - Mariusz Pelc
- Faculty of Electrical Engineering, Automatic Control and Informatics, Opole University of Technology, Poland; School of Computing and Mathematical Sciences, University of Greenwich, London, UK.
| | - Radek Martinek
- Department of Cybernetics and Biomedical Engineering, Faculty of Electrical Engineering and Computer Science, VSB-Technical University of Ostrava, Czechia; Faculty of Electrical Engineering, Automatic Control and Informatics, Opole University of Technology, Poland.
| |
Collapse
|
9
|
Gado S, Lingelbach K, Wirzberger M, Vukelić M. Decoding Mental Effort in a Quasi-Realistic Scenario: A Feasibility Study on Multimodal Data Fusion and Classification. SENSORS (BASEL, SWITZERLAND) 2023; 23:6546. [PMID: 37514840 PMCID: PMC10383122 DOI: 10.3390/s23146546] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/20/2023] [Revised: 07/17/2023] [Accepted: 07/17/2023] [Indexed: 07/30/2023]
Abstract
Humans' performance varies due to the mental resources that are available to successfully pursue a task. To monitor users' current cognitive resources in naturalistic scenarios, it is essential to not only measure demands induced by the task itself but also consider situational and environmental influences. We conducted a multimodal study with 18 participants (nine female, M = 25.9 with SD = 3.8 years). In this study, we recorded respiratory, ocular, cardiac, and brain activity using functional near-infrared spectroscopy (fNIRS) while participants performed an adapted version of the warship commander task with concurrent emotional speech distraction. We tested the feasibility of decoding the experienced mental effort with a multimodal machine learning architecture. The architecture comprised feature engineering, model optimisation, and model selection to combine multimodal measurements in a cross-subject classification. Our approach reduces possible overfitting and reliably distinguishes two different levels of mental effort. These findings contribute to the prediction of different states of mental effort and pave the way toward generalised state monitoring across individuals in realistic applications.
Collapse
Affiliation(s)
- Sabrina Gado
- Experimental Clinical Psychology, Department of Psychology, Julius-Maximilians-University of Würzburg, 97070 Würzburg, Germany
| | - Katharina Lingelbach
- Applied Neurocognitive Systems, Fraunhofer Institute for Industrial Engineering IAO, 70569 Stuttgart, Germany
- Applied Neurocognitive Psychology Lab, Department of Psychology, Carl von Ossietzky University, 26129 Oldenburg, Germany
| | - Maria Wirzberger
- Department of Teaching and Learning with Intelligent Systems, University of Stuttgart, 70174 Stuttgart, Germany
- LEAD Graduate School & Research Network, University of Tübingen, 72072 Tübingen, Germany
| | - Mathias Vukelić
- Applied Neurocognitive Systems, Fraunhofer Institute for Industrial Engineering IAO, 70569 Stuttgart, Germany
| |
Collapse
|
10
|
Zhang Y, Liu D, Li T, Zhang P, Li Z, Gao F. CGAN-rIRN: a data-augmented deep learning approach to accurate classification of mental tasks for a fNIRS-based brain-computer interface. BIOMEDICAL OPTICS EXPRESS 2023; 14:2934-2954. [PMID: 37342712 PMCID: PMC10278643 DOI: 10.1364/boe.489179] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/03/2023] [Revised: 05/11/2023] [Accepted: 05/15/2023] [Indexed: 06/23/2023]
Abstract
Functional near-infrared spectroscopy (fNIRS) is increasingly used to investigate different mental tasks for brain-computer interface (BCI) control due to its excellent environmental and motion robustness. Feature extraction and classification strategy for fNIRS signal are essential to enhance the classification accuracy of voluntarily controlled BCI systems. The limitation of traditional machine learning classifiers (MLCs) lies in manual feature engineering, which is considered as one of the drawbacks that reduce accuracy. Since the fNIRS signal is a typical multivariate time series with multi-dimensionality and complexity, it makes the deep learning classifier (DLC) ideal for classifying neural activation patterns. However, the inherent bottleneck of DLCs is the requirement of substantial-scale, high-quality labeled training data and expensive computational resources to train deep networks. The existing DLCs for classifying mental tasks do not fully consider the temporal and spatial properties of fNIRS signals. Therefore, a specifically-designed DLC is desired to classify multi-tasks with high accuracy in fNIRS-BCI. To this end, we herein propose a novel data-augmented DLC to accurately classify mental tasks, which employs a convolution-based conditional generative adversarial network (CGAN) for data augmentation and a revised Inception-ResNet (rIRN) based DLC. The CGAN is utilized to generate class-specific synthetic fNIRS signals to augment the training dataset. The network architecture of rIRN is elaborately designed in accordance with the characteristics of the fNIRS signal, with serial multiple spatial and temporal feature extraction modules (FEMs), where each FEM performs deep and multi-scale feature extraction and fusion. The results of the paradigm experiments show that the proposed CGAN-rIRN approach improves the single-trial accuracy for mental arithmetic and mental singing tasks in both the data augmentation and classifier, as compared to the traditional MLCs and the commonly used DLCs. The proposed fully data-driven hybrid deep learning approach paves a promising way to improve the classification performance of volitional control fNIRS-BCI.
Collapse
Affiliation(s)
- Yao Zhang
- College of Precision Instrument and Optoelectronics Engineering, Tianjin University, Tianjin 300070, China
| | - Dongyuan Liu
- College of Precision Instrument and Optoelectronics Engineering, Tianjin University, Tianjin 300070, China
| | - Tieni Li
- College of Precision Instrument and Optoelectronics Engineering, Tianjin University, Tianjin 300070, China
| | - Pengrui Zhang
- College of Precision Instrument and Optoelectronics Engineering, Tianjin University, Tianjin 300070, China
| | - Zhiyong Li
- College of Precision Instrument and Optoelectronics Engineering, Tianjin University, Tianjin 300070, China
| | - Feng Gao
- College of Precision Instrument and Optoelectronics Engineering, Tianjin University, Tianjin 300070, China
- Tianjin Key Laboratory of Biomedical Detecting Techniques and Instruments, Tianjin 300070, China
| |
Collapse
|
11
|
Masengo G, Zhang X, Dong R, Alhassan AB, Hamza K, Mudaheranwa E. Lower limb exoskeleton robot and its cooperative control: A review, trends, and challenges for future research. Front Neurorobot 2023; 16:913748. [PMID: 36714152 PMCID: PMC9875327 DOI: 10.3389/fnbot.2022.913748] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/06/2022] [Accepted: 12/19/2022] [Indexed: 01/12/2023] Open
Abstract
Effective control of an exoskeleton robot (ER) using a human-robot interface is crucial for assessing the robot's movements and the force they produce to generate efficient control signals. Interestingly, certain surveys were done to show off cutting-edge exoskeleton robots. The review papers that were previously published have not thoroughly examined the control strategy, which is a crucial component of automating exoskeleton systems. As a result, this review focuses on examining the most recent developments and problems associated with exoskeleton control systems, particularly during the last few years (2017-2022). In addition, the trends and challenges of cooperative control, particularly multi-information fusion, are discussed.
Collapse
Affiliation(s)
- Gilbert Masengo
- School of Mechanical Engineering, Xi'an Jiaotong University, Xi'an, China,Shaanxi Key Laboratory of Intelligent Robot, Xi'an Jiaotong University, Xi'an, China,Department of Mechanical Engineering, Rwanda Polytechnic/Integrated Polytechnic Regional College (IPRC) Karongi, Kigali, Rwanda,*Correspondence: Gilbert Masengo ✉
| | - Xiaodong Zhang
- School of Mechanical Engineering, Xi'an Jiaotong University, Xi'an, China,Shaanxi Key Laboratory of Intelligent Robot, Xi'an Jiaotong University, Xi'an, China
| | - Runlin Dong
- School of Mechanical Engineering, Xi'an Jiaotong University, Xi'an, China,Shaanxi Key Laboratory of Intelligent Robot, Xi'an Jiaotong University, Xi'an, China
| | - Ahmad B. Alhassan
- School of Mechanical Engineering, Xi'an Jiaotong University, Xi'an, China,Shaanxi Key Laboratory of Intelligent Robot, Xi'an Jiaotong University, Xi'an, China
| | - Khaled Hamza
- School of Mechanical Engineering, Xi'an Jiaotong University, Xi'an, China,Shaanxi Key Laboratory of Intelligent Robot, Xi'an Jiaotong University, Xi'an, China
| | - Emmanuel Mudaheranwa
- Department of Mechanical Engineering, Rwanda Polytechnic/Integrated Polytechnic Regional College (IPRC) Karongi, Kigali, Rwanda,Department of Engineering, Cardiff University, Cardiff, United Kingdom
| |
Collapse
|
12
|
Liang P, Li Z, Li J, Wei J, Li J, Zhang S, Xu S, Liu Z, Wang J. Impacts of complex electromagnetic radiation and low-frequency noise exposure conditions on the cognitive function of operators. Front Public Health 2023; 11:1138118. [PMID: 37033075 PMCID: PMC10076881 DOI: 10.3389/fpubh.2023.1138118] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2023] [Accepted: 03/07/2023] [Indexed: 04/11/2023] Open
Abstract
Background Both electromagnetic radiation (EMR) and low-frequency noise (LFN) are widespread and influential environmental factors, and operators are inevitably exposed to both EMR and LFN within a complex exposure environment. The potential adverse effects of such exposure on human health must be considered seriously. This study aimed to investigate the effects of EMR and LFN on cognitive function as well as their interaction effect, which remain unclear. Methods Sixty young male college students were randomly grouped and experiments were conducted with a 2 × 2 factorial design in a shielded chamber. Mental workload (MWL) levels of the study subjects were measured and assessed using the NASA-task load index (TLX) subjective scale, an n-back task paradigm, and the functional near-infrared spectroscopy (fNIRS) imaging technique. Results For the 3-back task, the NASA-TLX subjective scale revealed a statistically significant main effect of LFN intensity, which enhanced the subjects' MWL level (F = 8.716, p < 0.01). Behavioral performance revealed that EMR intensity (430.1357 MHz, 10.75 W/m2) and LFN intensity (0-200 Hz, 72.9 dB) had a synergistic interaction effect, and the correct response time was statistically significantly prolonged by the combined exposure (F = 4.343, p < 0.05). The fNIRS imaging technique revealed a synergistic interaction effect between operational EMR intensity and operational LFN intensity, with statistically significant effects on the activation levels in the left and right dorsolateral prefrontal cortex (DLPFC). The mean β values of DLPFC were significantly increased (L-DLPFC F = 5.391, p < 0.05, R-DLPFC F = 4.222, p < 0.05), and the relative concentrations of oxyhemoglobin in the DLPFC were also significantly increased (L-DLPFC F = 4.925, p < 0.05, R-DLPFC F = 9.715, p < 0.01). Conclusion We found a statistically significant interaction effect between EMR (430.1357 MHz, 10.75 W/m2) and LFN (0-200 Hz, 72.9 dB) when simultaneously exposing subjects to both for 30 min. We conclude that exposure to this complex environment can cause a statistically significant increase in the MWL level of operators, and even alterations in their cognitive function.
Collapse
Affiliation(s)
- Peng Liang
- Department of Rehabilitative Physioltherapy, The Second Affiliated Hospital of Air Force Medical University, Xi’an, China
- Hospital of No. 95007 Unit of PLA, Guangzhou, China
| | - Zenglei Li
- Department of Rehabilitative Physioltherapy, The Second Affiliated Hospital of Air Force Medical University, Xi’an, China
| | - Jiangjing Li
- Department of Anesthesiology, The Second Affiliated Hospital of Air Force Medical University, Xi’an, China
| | - Jing Wei
- Ministry of Education Key Lab of Hazard Assessment and Control in Special Operational Environment, Department of Radiation Medical Protection, School of Military Preventive Medicine, Fourth Military Medical University, Xi’an, China
| | - Jing Li
- Ministry of Education Key Lab of Hazard Assessment and Control in Special Operational Environment, Department of Radiation Medical Protection, School of Military Preventive Medicine, Fourth Military Medical University, Xi’an, China
| | - Shenghao Zhang
- Department of Neurosurgery, The 940th Hospital of PLA Joint Logistics Support Force, Lanzhou, China
| | - Shenglong Xu
- Department of Neurosurgery, The 940th Hospital of PLA Joint Logistics Support Force, Lanzhou, China
| | - Zhaohui Liu
- Department of Orthopaedics, The Second Affiliated Hospital of Air Force Medical University, Xi’an, China
- *Correspondence: Zhaohui Liu,
| | - Jin Wang
- Ministry of Education Key Lab of Hazard Assessment and Control in Special Operational Environment, Department of Radiation Medical Protection, School of Military Preventive Medicine, Fourth Military Medical University, Xi’an, China
- Jin Wang,
| |
Collapse
|
13
|
Rafidison MA, Ramafiarisona HM, Randriamitantsoa PA, Rafanantenana SHJ, Toky FMR, Rakotondrazaka LP, Rakotomihamina AH. Image Classification Based on Light Convolutional Neural Network Using Pulse Couple Neural Network. COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE 2023; 2023:7371907. [PMID: 36959839 PMCID: PMC10030221 DOI: 10.1155/2023/7371907] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/24/2022] [Revised: 02/25/2023] [Accepted: 02/27/2023] [Indexed: 03/16/2023]
Abstract
Recently, most image classification studies solicit the intervention of convolutional neural networks because these DL-based classification methods generally outperform other methodologies with higher accuracy. However, this type of deep learning networks require many parameters and have a complex structure with multiple convolutional and pooling layers depending on the objective. These layers compute a large volume of data and it may impact the processing time and the performance. Therefore, this paper proposes a new method of image classification based on the light convolutional neural network. It consists of replacing the feature extraction layers of standard convolutional neural network with a single pulse coupled neural network by introducing the notion of foveation. This module provides the feature map of input image and the data compression using Discrete Wavelet Transform which is an optional step depending on the information quantity of this signature. The fully connected neural network, which has six hidden layers, classifies the image. With this technique, the computation time is reduced, and the network architecture is identical and simple independent of the type of dataset. The number of parameter is less than that in current research. The proposed method was validated with different dataset such as Caltech-101, Caltech-256, CIFAR-10, CIFAR-100, and ImageNet, and the accuracy reaches 92%, 90%, 99%, 94%, and 91%, respectively, which are better than the previous related works.
Collapse
Affiliation(s)
- Maminiaina Alphonse Rafidison
- Telecommunication-Automatic-Signal-Image-Research, Laboratory/Doctoral School in Science and Technology of Engineering and Innovation/University of Antananarivo, Antananarivo 101, Madagascar
| | - Hajasoa Malalatiana Ramafiarisona
- Telecommunication-Automatic-Signal-Image-Research, Laboratory/Doctoral School in Science and Technology of Engineering and Innovation/University of Antananarivo, Antananarivo 101, Madagascar
| | - Paul Auguste Randriamitantsoa
- Telecommunication-Automatic-Signal-Image-Research, Laboratory/Doctoral School in Science and Technology of Engineering and Innovation/University of Antananarivo, Antananarivo 101, Madagascar
| | - Sabine Harisoa Jacques Rafanantenana
- Telecommunication-Automatic-Signal-Image-Research, Laboratory/Doctoral School in Science and Technology of Engineering and Innovation/University of Antananarivo, Antananarivo 101, Madagascar
| | - Faniriharisoa Maxime Rajaonarison Toky
- Telecommunication-Automatic-Signal-Image-Research, Laboratory/Doctoral School in Science and Technology of Engineering and Innovation/University of Antananarivo, Antananarivo 101, Madagascar
| | - Lovasoa Patrick Rakotondrazaka
- Telecommunication-Automatic-Signal-Image-Research, Laboratory/Doctoral School in Science and Technology of Engineering and Innovation/University of Antananarivo, Antananarivo 101, Madagascar
| | - Andry Harivony Rakotomihamina
- Telecommunication-Automatic-Signal-Image-Research, Laboratory/Doctoral School in Science and Technology of Engineering and Innovation/University of Antananarivo, Antananarivo 101, Madagascar
| |
Collapse
|
14
|
Zhang Y, Liu D, Zhang P, Li T, Li Z, Gao F. Combining robust level extraction and unsupervised adaptive classification for high-accuracy fNIRS-BCI: An evidence on single-trial differentiation between mentally arithmetic- and singing-tasks. Front Neurosci 2022; 16:938518. [PMID: 36300170 PMCID: PMC9589108 DOI: 10.3389/fnins.2022.938518] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2022] [Accepted: 09/12/2022] [Indexed: 11/18/2022] Open
Abstract
Functional near-infrared spectroscopy (fNIRS) is a safe and non-invasive optical imaging technique that is being increasingly used in brain-computer interfaces (BCIs) to recognize mental tasks. Unlike electroencephalography (EEG) which directly measures neural activation, fNIRS signals reflect neurovascular-coupling inducing hemodynamic response that can be slow in time and varying in the pattern. The established classifiers extend the EEG-ones by mostly employing the feature based supervised models such as the support vector machine (SVM) and linear discriminant analysis (LDA), and fail to timely characterize the level-sensitive hemodynamic pattern. A dedicated classifier is desired for intentional activity recognition of fNIRS-BCI, including the adaptive acquisition of response relevant features and accurate discrimination of implied ideas. To this end, we herein propose a specifically-designed joint adaptive classification method that combines a Kalman filtering (KF) for robust level extraction and an adaptive Gaussian mixture model (a-GMM) for enhanced pattern recognition. The simulative investigations and paradigm experiments have shown that the proposed KF/a-GMM classification method can effectively track the random variations of task-evoked brain activation patterns, and improve the accuracy of single-trial classification task of mental arithmetic vs. mental singing, as compared to the conventional methods, e.g., those that employ combinations of the band-pass filtering (BPF) based feature extractors (mean, slope, and variance, etc.) and the classical recognizers (GMM, SVM, and LDA). The proposed approach paves a promising way for developing the real-time fNIRS-BCI technique.
Collapse
Affiliation(s)
- Yao Zhang
- College of Precision Instrument and Optoelectronics Engineering, Tianjin University, Tianjin, China
| | - Dongyuan Liu
- College of Precision Instrument and Optoelectronics Engineering, Tianjin University, Tianjin, China
| | - Pengrui Zhang
- College of Precision Instrument and Optoelectronics Engineering, Tianjin University, Tianjin, China
| | - Tieni Li
- College of Precision Instrument and Optoelectronics Engineering, Tianjin University, Tianjin, China
| | - Zhiyong Li
- College of Precision Instrument and Optoelectronics Engineering, Tianjin University, Tianjin, China
| | - Feng Gao
- College of Precision Instrument and Optoelectronics Engineering, Tianjin University, Tianjin, China
- Tianjin Key Laboratory of Biomedical Detecting Techniques and Instruments, Tianjin University, Tianjin, China
- *Correspondence: Feng Gao
| |
Collapse
|
15
|
Eastmond C, Subedi A, De S, Intes X. Deep learning in fNIRS: a review. NEUROPHOTONICS 2022; 9:041411. [PMID: 35874933 PMCID: PMC9301871 DOI: 10.1117/1.nph.9.4.041411] [Citation(s) in RCA: 26] [Impact Index Per Article: 13.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/26/2022] [Accepted: 06/22/2022] [Indexed: 05/28/2023]
Abstract
Significance: Optical neuroimaging has become a well-established clinical and research tool to monitor cortical activations in the human brain. It is notable that outcomes of functional near-infrared spectroscopy (fNIRS) studies depend heavily on the data processing pipeline and classification model employed. Recently, deep learning (DL) methodologies have demonstrated fast and accurate performances in data processing and classification tasks across many biomedical fields. Aim: We aim to review the emerging DL applications in fNIRS studies. Approach: We first introduce some of the commonly used DL techniques. Then, the review summarizes current DL work in some of the most active areas of this field, including brain-computer interface, neuro-impairment diagnosis, and neuroscience discovery. Results: Of the 63 papers considered in this review, 32 report a comparative study of DL techniques to traditional machine learning techniques where 26 have been shown outperforming the latter in terms of the classification accuracy. In addition, eight studies also utilize DL to reduce the amount of preprocessing typically done with fNIRS data or increase the amount of data via data augmentation. Conclusions: The application of DL techniques to fNIRS studies has shown to mitigate many of the hurdles present in fNIRS studies such as lengthy data preprocessing or small sample sizes while achieving comparable or improved classification accuracy.
Collapse
Affiliation(s)
- Condell Eastmond
- Center for Modeling, Simulation and Imaging for Medicine, Rensselaer Polytechnic, Department of Biomedical Engineering, Troy, New York, United States
| | - Aseem Subedi
- Center for Modeling, Simulation and Imaging for Medicine, Rensselaer Polytechnic, Department of Biomedical Engineering, Troy, New York, United States
| | - Suvranu De
- Center for Modeling, Simulation and Imaging for Medicine, Rensselaer Polytechnic, Department of Biomedical Engineering, Troy, New York, United States
| | - Xavier Intes
- Center for Modeling, Simulation and Imaging for Medicine, Rensselaer Polytechnic, Department of Biomedical Engineering, Troy, New York, United States
| |
Collapse
|
16
|
Jangwan NS, Ashraf GM, Ram V, Singh V, Alghamdi BS, Abuzenadah AM, Singh MF. Brain augmentation and neuroscience technologies: current applications, challenges, ethics and future prospects. Front Syst Neurosci 2022; 16:1000495. [PMID: 36211589 PMCID: PMC9538357 DOI: 10.3389/fnsys.2022.1000495] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/22/2022] [Accepted: 08/31/2022] [Indexed: 12/02/2022] Open
Abstract
Ever since the dawn of antiquity, people have strived to improve their cognitive abilities. From the advent of the wheel to the development of artificial intelligence, technology has had a profound leverage on civilization. Cognitive enhancement or augmentation of brain functions has become a trending topic both in academic and public debates in improving physical and mental abilities. The last years have seen a plethora of suggestions for boosting cognitive functions and biochemical, physical, and behavioral strategies are being explored in the field of cognitive enhancement. Despite expansion of behavioral and biochemical approaches, various physical strategies are known to boost mental abilities in diseased and healthy individuals. Clinical applications of neuroscience technologies offer alternatives to pharmaceutical approaches and devices for diseases that have been fatal, so far. Importantly, the distinctive aspect of these technologies, which shapes their existing and anticipated participation in brain augmentations, is used to compare and contrast them. As a preview of the next two decades of progress in brain augmentation, this article presents a plausible estimation of the many neuroscience technologies, their virtues, demerits, and applications. The review also focuses on the ethical implications and challenges linked to modern neuroscientific technology. There are times when it looks as if ethics discussions are more concerned with the hypothetical than with the factual. We conclude by providing recommendations for potential future studies and development areas, taking into account future advancements in neuroscience innovation for brain enhancement, analyzing historical patterns, considering neuroethics and looking at other related forecasts.
Collapse
Affiliation(s)
- Nitish Singh Jangwan
- Department of Pharmacology, School of Pharmaceutical Sciences and Technology, Sardar Bhagwan Singh University, Balawala, India
| | - Ghulam Md Ashraf
- Pre-Clinical Research Unit, King Fahd Medical Research Center, King Abdulaziz University, Jeddah, Saudi Arabia
- Department of Medical Laboratory Sciences, Faculty of Applied Medical Sciences, King Abdulaziz University, Jeddah, Saudi Arabia
| | - Veerma Ram
- Department of Pharmacology, School of Pharmaceutical Sciences and Technology, Sardar Bhagwan Singh University, Balawala, India
| | - Vinod Singh
- Prabha Harji Lal College of Pharmacy and Paraclinical Sciences, University of Jammu, Jammu, India
| | - Badrah S. Alghamdi
- Pre-Clinical Research Unit, King Fahd Medical Research Center, King Abdulaziz University, Jeddah, Saudi Arabia
- Department of Physiology, Neuroscience Unit, Faculty of Medicine, King Abdulaziz University, Jeddah, Saudi Arabia
| | - Adel Mohammad Abuzenadah
- Faculty of Applied Medical Sciences, King Abdulaziz University, Jeddah, Saudi Arabia
- King Fahd Medical Research Center, King Abdulaziz University, Jeddah, Saudi Arabia
| | - Mamta F. Singh
- Department of Pharmacology, School of Pharmaceutical Sciences and Technology, Sardar Bhagwan Singh University, Balawala, India
| |
Collapse
|
17
|
Mughal NE, Khan MJ, Khalil K, Javed K, Sajid H, Naseer N, Ghafoor U, Hong KS. EEG-fNIRS-based hybrid image construction and classification using CNN-LSTM. Front Neurorobot 2022; 16:873239. [PMID: 36119719 PMCID: PMC9472125 DOI: 10.3389/fnbot.2022.873239] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/10/2022] [Accepted: 07/20/2022] [Indexed: 11/13/2022] Open
Abstract
The constantly evolving human–machine interaction and advancement in sociotechnical systems have made it essential to analyze vital human factors such as mental workload, vigilance, fatigue, and stress by monitoring brain states for optimum performance and human safety. Similarly, brain signals have become paramount for rehabilitation and assistive purposes in fields such as brain–computer interface (BCI) and closed-loop neuromodulation for neurological disorders and motor disabilities. The complexity, non-stationary nature, and low signal-to-noise ratio of brain signals pose significant challenges for researchers to design robust and reliable BCI systems to accurately detect meaningful changes in brain states outside the laboratory environment. Different neuroimaging modalities are used in hybrid settings to enhance accuracy, increase control commands, and decrease the time required for brain activity detection. Functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) measure the hemodynamic and electrical activity of the brain with a good spatial and temporal resolution, respectively. However, in hybrid settings, where both modalities enhance the output performance of BCI, their data compatibility due to the huge discrepancy between their sampling rate and the number of channels remains a challenge for real-time BCI applications. Traditional methods, such as downsampling and channel selection, result in important information loss while making both modalities compatible. In this study, we present a novel recurrence plot (RP)-based time-distributed convolutional neural network and long short-term memory (CNN-LSTM) algorithm for the integrated classification of fNIRS EEG for hybrid BCI applications. The acquired brain signals are first projected into a non-linear dimension with RPs and fed into the CNN to extract essential features without performing any downsampling. Then, LSTM is used to learn the chronological features and time-dependence relation to detect brain activity. The average accuracies achieved with the proposed model were 78.44% for fNIRS, 86.24% for EEG, and 88.41% for hybrid EEG-fNIRS BCI. Moreover, the maximum accuracies achieved were 85.9, 88.1, and 92.4%, respectively. The results confirm the viability of the RP-based deep-learning algorithm for successful BCI systems.
Collapse
Affiliation(s)
- Nabeeha Ehsan Mughal
- School of Mechanical and Manufacturing Engineering, National University of Sciences and Technology (NUST), Islamabad, Pakistan
| | - Muhammad Jawad Khan
- School of Mechanical and Manufacturing Engineering, National University of Sciences and Technology (NUST), Islamabad, Pakistan
- National Center of Artificial Intelligence (NCAI) – NUST, Islamabad, Pakistan
| | - Khurram Khalil
- School of Mechanical and Manufacturing Engineering, National University of Sciences and Technology (NUST), Islamabad, Pakistan
| | - Kashif Javed
- School of Mechanical and Manufacturing Engineering, National University of Sciences and Technology (NUST), Islamabad, Pakistan
| | - Hasan Sajid
- School of Mechanical and Manufacturing Engineering, National University of Sciences and Technology (NUST), Islamabad, Pakistan
- National Center of Artificial Intelligence (NCAI) – NUST, Islamabad, Pakistan
| | - Noman Naseer
- Department of Mechatronics and Biomedical Engineering, Air University, Islamabad, Pakistan
| | - Usman Ghafoor
- School of Mechanical Engineering, Pusan National University, Busan, South Korea
| | - Keum-Shik Hong
- School of Mechanical Engineering, Pusan National University, Busan, South Korea
- *Correspondence: Keum-Shik Hong
| |
Collapse
|
18
|
Wang Z, Zhang J, Xia Y, Chen P, Wang B. A General and Scalable Vision Framework for Functional Near-Infrared Spectroscopy Classification. IEEE Trans Neural Syst Rehabil Eng 2022; 30:1982-1991. [PMID: 35830404 DOI: 10.1109/tnsre.2022.3190431] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
Functional near-infrared spectroscopy (fNIRS), a non-invasive optical technique, is widely used to monitor brain activities for disease diagnosis and brain-computer interfaces (BCIs). Deep learning-based fNIRS classification faces three major barriers: limited datasets, confusing evaluation criteria, and domain barriers. We apply more appropriate evaluation methods to three open-access datasets to solve the first two barriers. For domain barriers, we propose a general and scalable vision fNIRS framework that converts multi-channel fNIRS signals into multi-channel virtual images using the Gramian angular difference field (GADF). We use the framework to train state-of-the-art visual models from computer vision (CV) within a few minutes, and the classification performance is competitive with the latest fNIRS models. In cross-validation experiments, visual models achieve the highest average classification results of 78.68% and 73.92% on mental arithmetic and word generation tasks, respectively. Although visual models are slightly lower than the fNIRS models on unilateral finger- and foot-tapping tasks, the F1-score and kappa coefficient indicate that these differences are insignificant in subject-independent experiments. Furthermore, we study fNIRS signal representations and the classification performance of sequence-to-image methods. We hope to introduce rich achievements from the CV domain to improve fNIRS classification research.
Collapse
|
19
|
Kant P, Laskar SH, Hazarika J. Transfer learning-based EEG analysis of visual attention and working memory on motor cortex for BCI. Neural Comput Appl 2022. [DOI: 10.1007/s00521-022-07580-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
|
20
|
Raufi B, Longo L. An Evaluation of the EEG Alpha-to-Theta and Theta-to-Alpha Band Ratios as Indexes of Mental Workload. Front Neuroinform 2022; 16:861967. [PMID: 35651718 PMCID: PMC9149374 DOI: 10.3389/fninf.2022.861967] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/25/2022] [Accepted: 04/25/2022] [Indexed: 12/25/2022] Open
Abstract
Many research works indicate that EEG bands, specifically the alpha and theta bands, have been potentially helpful cognitive load indicators. However, minimal research exists to validate this claim. This study aims to assess and analyze the impact of the alpha-to-theta and the theta-to-alpha band ratios on supporting the creation of models capable of discriminating self-reported perceptions of mental workload. A dataset of raw EEG data was utilized in which 48 subjects performed a resting activity and an induced task demanding exercise in the form of a multitasking SIMKAP test. Band ratios were devised from frontal and parietal electrode clusters. Building and model testing was done with high-level independent features from the frequency and temporal domains extracted from the computed ratios over time. Target features for model training were extracted from the subjective ratings collected after resting and task demand activities. Models were built by employing Logistic Regression, Support Vector Machines and Decision Trees and were evaluated with performance measures including accuracy, recall, precision and f1-score. The results indicate high classification accuracy of those models trained with the high-level features extracted from the alpha-to-theta ratios and theta-to-alpha ratios. Preliminary results also show that models trained with logistic regression and support vector machines can accurately classify self-reported perceptions of mental workload. This research contributes to the body of knowledge by demonstrating the richness of the information in the temporal, spectral and statistical domains extracted from the alpha-to-theta and theta-to-alpha EEG band ratios for the discrimination of self-reported perceptions of mental workload.
Collapse
|
21
|
Training Monitoring in Sports: It Is Time to Embrace Cognitive Demand. Sports (Basel) 2022; 10:sports10040056. [PMID: 35447866 PMCID: PMC9028378 DOI: 10.3390/sports10040056] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/01/2022] [Revised: 03/27/2022] [Accepted: 04/06/2022] [Indexed: 11/16/2022] Open
Abstract
Appropriate training burden monitoring is still a challenge for the support staff, athletes, and coaches. Extensive research has been done in recent years that proposes several external and internal indicators. Among all measurements, the importance of cognitive factors has been indicated but has never been really considered in the training monitoring process. While there is strong evidence supporting the use of cognitive demand indicators in cognitive neuroscience, their importance in training monitoring for multiple sports settings must be better emphasized. The aims of this scoping review are to (1) provide an overview of the cognitive demand concept beside the physical demand in training; (2) highlight the current methods for assessing cognitive demand in an applied setting to sports in part through a neuroergonomics approach; (3) show how cognitive demand metrics can be exploited and applied to our better understanding of fatigue, sport injury, overtraining and individual performance capabilities. This review highlights also the potential new ways of brain imaging approaches for monitoring in situ. While assessment of cognitive demand is still in its infancy in sport, it may represent a very fruitful approach if applied with rigorous protocols and deep knowledge of both the neurobehavioral and cognitive aspects. It is time now to consider the cognitive demand to avoid underestimating the total training burden and its management.
Collapse
|
22
|
Hamid H, Naseer N, Nazeer H, Khan MJ, Khan RA, Shahbaz Khan U. Analyzing Classification Performance of fNIRS-BCI for Gait Rehabilitation Using Deep Neural Networks. SENSORS (BASEL, SWITZERLAND) 2022; 22:1932. [PMID: 35271077 PMCID: PMC8914987 DOI: 10.3390/s22051932] [Citation(s) in RCA: 11] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/28/2022] [Revised: 02/21/2022] [Accepted: 02/25/2022] [Indexed: 05/11/2023]
Abstract
This research presents a brain-computer interface (BCI) framework for brain signal classification using deep learning (DL) and machine learning (ML) approaches on functional near-infrared spectroscopy (fNIRS) signals. fNIRS signals of motor execution for walking and rest tasks are acquired from the primary motor cortex in the brain's left hemisphere for nine subjects. DL algorithms, including convolutional neural networks (CNNs), long short-term memory (LSTM), and bidirectional LSTM (Bi-LSTM) are used to achieve average classification accuracies of 88.50%, 84.24%, and 85.13%, respectively. For comparison purposes, three conventional ML algorithms, support vector machine (SVM), k-nearest neighbor (k-NN), and linear discriminant analysis (LDA) are also used for classification, resulting in average classification accuracies of 73.91%, 74.24%, and 65.85%, respectively. This study successfully demonstrates that the enhanced performance of fNIRS-BCI can be achieved in terms of classification accuracy using DL approaches compared to conventional ML approaches. Furthermore, the control commands generated by these classifiers can be used to initiate and stop the gait cycle of the lower limb exoskeleton for gait rehabilitation.
Collapse
Affiliation(s)
- Huma Hamid
- Department of Mechatronics and Biomedical Engineering, Air University, Islamabad 44000, Pakistan; (H.H.); (H.N.)
| | - Noman Naseer
- Department of Mechatronics and Biomedical Engineering, Air University, Islamabad 44000, Pakistan; (H.H.); (H.N.)
| | - Hammad Nazeer
- Department of Mechatronics and Biomedical Engineering, Air University, Islamabad 44000, Pakistan; (H.H.); (H.N.)
| | - Muhammad Jawad Khan
- School of Mechanical and Manufacturing Engineering, National University of Science and Technology, Islamabad 44000, Pakistan;
| | - Rayyan Azam Khan
- Department of Mechanical Engineering, University of Saskatchewan, Saskatoon, SK S7N 5A9, Canada;
| | - Umar Shahbaz Khan
- Department of Mechatronics Engineering, National University of Sciences and Technology, Islamabad 44000, Pakistan;
- National Centre of Robotics and Automation (NCRA), Rawalpindi 46000, Pakistan
| |
Collapse
|
23
|
Khalil K, Asgher U, Ayaz Y. Novel fNIRS study on homogeneous symmetric feature-based transfer learning for brain-computer interface. Sci Rep 2022; 12:3198. [PMID: 35210460 PMCID: PMC8873341 DOI: 10.1038/s41598-022-06805-4] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/18/2021] [Accepted: 01/04/2022] [Indexed: 01/23/2023] Open
Abstract
The brain-computer interface (BCI) provides an alternate means of communication between the brain and external devices by recognizing the brain activities and translating them into external commands. The functional Near-Infrared Spectroscopy (fNIRS) is becoming popular as a non-invasive modality for brain activity detection. The recent trends show that deep learning has significantly enhanced the performance of the BCI systems. But the inherent bottleneck for deep learning (in the domain of BCI) is the requirement of the vast amount of training data, lengthy recalibrating time, and expensive computational resources for training deep networks. Building a high-quality, large-scale annotated dataset for deep learning-based BCI systems is exceptionally tedious, complex, and expensive. This study investigates the novel application of transfer learning for fNIRS-based BCI to solve three objective functions (concerns), i.e., the problem of insufficient training data, reduced training time, and increased accuracy. We applied symmetric homogeneous feature-based transfer learning on convolutional neural network (CNN) designed explicitly for fNIRS data collected from twenty-six (26) participants performing the n-back task. The results suggested that the proposed method achieves the maximum saturated accuracy sooner and outperformed the traditional CNN model on averaged accuracy by 25.58% in the exact duration of training time, reducing the training time, recalibrating time, and computational resources.
Collapse
Affiliation(s)
- Khurram Khalil
- National Center of Artificial Intelligence (NCAI), School of Mechanical and Manufacturing Engineering (SMME), National University of Sciences and Technology (NUST), Islamabad, 44000, Pakistan
| | - Umer Asgher
- National Center of Artificial Intelligence (NCAI), School of Mechanical and Manufacturing Engineering (SMME), National University of Sciences and Technology (NUST), Islamabad, 44000, Pakistan.,Department of Mechatronics Engineering, College of Electrical and Mechanical Engineering, National University of Sciences and Technology (NUST), Islamabad, 44000, Pakistan
| | - Yasar Ayaz
- National Center of Artificial Intelligence (NCAI), School of Mechanical and Manufacturing Engineering (SMME), National University of Sciences and Technology (NUST), Islamabad, 44000, Pakistan.
| |
Collapse
|
24
|
Chu H, Cao Y, Jiang J, Yang J, Huang M, Li Q, Jiang C, Jiao X. Optimized electroencephalogram and functional near-infrared spectroscopy-based mental workload detection method for practical applications. Biomed Eng Online 2022; 21:9. [PMID: 35109879 PMCID: PMC8812267 DOI: 10.1186/s12938-022-00980-1] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/03/2021] [Accepted: 01/21/2022] [Indexed: 11/14/2022] Open
Abstract
Background Mental workload is a critical consideration in complex man–machine systems design. Among various mental workload detection techniques, multimodal detection techniques integrating electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) signals have attracted considerable attention. However, existing EEG–fNIRS-based mental workload detection methods have certain defects, such as complex signal acquisition channels and low detection accuracy, which restrict their practical application. Methods The signal acquisition configuration was optimized by analyzing the feature importance in mental workload recognition model and a more accurate and convenient EEG–fNIRS-based mental workload detection method was constructed. A classical Multi-Task Attribute Battery (MATB) task was conducted with 20 participating volunteers. Subjective scale data, 64-channel EEG data, and two-channel fNIRS data were collected. Results A higher number of EEG channels correspond to higher detection accuracy. However, there is no obvious improvement in accuracy once the number of EEG channels reaches 26, with a four-level mental workload detection accuracy of 76.25 ± 5.21%. Partial results of physiological analysis verify the results of previous studies, such as that the θ power of EEG and concentration of O2Hb in the prefrontal region increase while the concentration of HHb decreases with task difficulty. It was further observed, for the first time, that the energy of each band of EEG signals was significantly different in the occipital lobe region, and the power of \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$\beta_{1}$$\end{document}β1 and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$\beta_{2}$$\end{document}β2 bands in the occipital region increased significantly with task difficulty. The changing range and the mean amplitude of O2Hb in high-difficulty tasks were significantly higher compared with those in low-difficulty tasks. Conclusions The channel configuration of EEG–fNIRS-based mental workload detection was optimized to 26 EEG channels and two frontal fNIRS channels. A four-level mental workload detection accuracy of 76.25 ± 5.21% was obtained, which is higher than previously reported results. The proposed configuration can promote the application of mental workload detection technology in military, driving, and other complex human–computer interaction systems.
Collapse
Affiliation(s)
- Hongzuo Chu
- National Key Laboratory of Human Factors Engineering, China Astronaut Research and Training Center, Beijing, China.,Space Engineering University, Beijing, China
| | - Yong Cao
- National Key Laboratory of Human Factors Engineering, China Astronaut Research and Training Center, Beijing, China
| | - Jin Jiang
- National Key Laboratory of Human Factors Engineering, China Astronaut Research and Training Center, Beijing, China
| | - Jiehong Yang
- National Key Laboratory of Human Factors Engineering, China Astronaut Research and Training Center, Beijing, China.,Space Engineering University, Beijing, China
| | - Mengyin Huang
- National Key Laboratory of Human Factors Engineering, China Astronaut Research and Training Center, Beijing, China.,Space Engineering University, Beijing, China
| | - Qijie Li
- National Key Laboratory of Human Factors Engineering, China Astronaut Research and Training Center, Beijing, China
| | - Changhua Jiang
- National Key Laboratory of Human Factors Engineering, China Astronaut Research and Training Center, Beijing, China.
| | - Xuejun Jiao
- National Key Laboratory of Human Factors Engineering, China Astronaut Research and Training Center, Beijing, China. .,Space Engineering University, Beijing, China.
| |
Collapse
|
25
|
Wang Z, Zhang J, Zhang X, Chen P, Wang B. Transformer Model for Functional Near-Infrared Spectroscopy Classification. IEEE J Biomed Health Inform 2022; 26:2559-2569. [PMID: 34986110 DOI: 10.1109/jbhi.2022.3140531] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
Functional near-infrared spectroscopy (fNIRS) is a promising neuroimaging technology. The fNIRS classification problem has always been the focus of the brain-computer interface (BCI). Inspired by the success of Transformer based on self-attention mechanism in the fields of natural language processing and computer vision, we propose an fNIRS classification network based on Transformer, named fNIRS-T. We explore the spatial-level and channel-level representation of fNIRS signals to improve data utilization and network representation capacity. Besides, a preprocessing module, which consists of one-dimensional average pooling and layer normalization, is designed to replace filtering and baseline correction of data preprocessing. It makes fNIRS-T an end-to-end network, called fNIRS-PreT. Compared with traditional machine learning classifiers, convolutional neural network (CNN), and long short-term memory (LSTM), the proposed models obtain the best accuracy on three open-access datasets. Specifically, in the most extensive ternary classification task (30 subjects) that includes three types of overt movements, fNIRS-T, CNN, and LSTM obtain 75.49%, 72.89%, and 61.94% on test sets, respectively. Compared to traditional classifiers, fNIRS-T is at least 27.41% higher than statistical features and 6.79% higher than well-designed features. In the individual subject experiment of the ternary classification task, fNIRS-T achieves an average subject accuracy of 78.22% and surpasses CNN and LSTM by a large margin of +4.75% and +11.33%. fNIRS-PreT using raw data also achieves competitive performance to fNIRS-T. Therefore, the proposed models improve the performance of fNIRS-based BCI significantly.
Collapse
|
26
|
Li H, Gong A, Zhao L, Wang F, Qian Q, Zhou J, Fu Y. Identification of gait imagery based on fNIRS and class-dependent sparse representation. Biomed Signal Process Control 2021. [DOI: 10.1016/j.bspc.2021.102597] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
|
27
|
|
28
|
Arif S, Khan MJ, Naseer N, Hong KS, Sajid H, Ayaz Y. Vector Phase Analysis Approach for Sleep Stage Classification: A Functional Near-Infrared Spectroscopy-Based Passive Brain-Computer Interface. Front Hum Neurosci 2021; 15:658444. [PMID: 33994983 PMCID: PMC8121150 DOI: 10.3389/fnhum.2021.658444] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/25/2021] [Accepted: 03/09/2021] [Indexed: 11/13/2022] Open
Abstract
A passive brain-computer interface (BCI) based upon functional near-infrared spectroscopy (fNIRS) brain signals is used for earlier detection of human drowsiness during driving tasks. This BCI modality acquired hemodynamic signals of 13 healthy subjects from the right dorsolateral prefrontal cortex (DPFC) of the brain. Drowsiness activity is recorded using a continuous-wave fNIRS system and eight channels over the right DPFC. During the experiment, sleep-deprived subjects drove a vehicle in a driving simulator while their cerebral oxygen regulation (CORE) state was continuously measured. Vector phase analysis (VPA) was used as a classifier to detect drowsiness state along with sleep stage-based threshold criteria. Extensive training and testing with various feature sets and classifiers are done to justify the adaptation of threshold criteria for any subject without requiring recalibration. Three statistical features (mean oxyhemoglobin, signal peak, and the sum of peaks) along with six VPA features (trajectory slopes of VPA indices) were used. The average accuracies for the five classifiers are 90.9% for discriminant analysis, 92.5% for support vector machines, 92.3% for nearest neighbors, 92.4% for both decision trees, and ensembles over all subjects' data. Trajectory slopes of CORE vector magnitude and angle: m(|R|) and m(∠R) are the best-performing features, along with ensemble classifier with the highest accuracy of 95.3% and minimum computation time of 40 ms. The statistical significance of the results is validated with a p-value of less than 0.05. The proposed passive BCI scheme demonstrates a promising technique for online drowsiness detection using VPA along with sleep stage classification.
Collapse
Affiliation(s)
- Saad Arif
- School of Mechanical and Manufacturing Engineering, National University of Sciences and Technology, Islamabad, Pakistan
| | - Muhammad Jawad Khan
- School of Mechanical and Manufacturing Engineering, National University of Sciences and Technology, Islamabad, Pakistan.,National Center of Artificial Intelligence (NCAI), Islamabad, Pakistan
| | - Noman Naseer
- Department of Mechatronics Engineering, Air University, Islamabad, Pakistan
| | - Keum-Shik Hong
- School of Mechanical Engineering, Pusan National University, Busan, South Korea
| | - Hasan Sajid
- School of Mechanical and Manufacturing Engineering, National University of Sciences and Technology, Islamabad, Pakistan.,National Center of Artificial Intelligence (NCAI), Islamabad, Pakistan
| | - Yasar Ayaz
- School of Mechanical and Manufacturing Engineering, National University of Sciences and Technology, Islamabad, Pakistan.,National Center of Artificial Intelligence (NCAI), Islamabad, Pakistan
| |
Collapse
|
29
|
Asgher U, Khan MJ, Asif Nizami MH, Khalil K, Ahmad R, Ayaz Y, Naseer N. Motor Training Using Mental Workload (MWL) With an Assistive Soft Exoskeleton System: A Functional Near-Infrared Spectroscopy (fNIRS) Study for Brain-Machine Interface (BMI). Front Neurorobot 2021; 15:605751. [PMID: 33815084 PMCID: PMC8012849 DOI: 10.3389/fnbot.2021.605751] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/13/2020] [Accepted: 02/05/2021] [Indexed: 11/24/2022] Open
Abstract
Mental workload is a neuroergonomic human factor, which is widely used in planning a system's safety and areas like brain-machine interface (BMI), neurofeedback, and assistive technologies. Robotic prosthetics methodologies are employed for assisting hemiplegic patients in performing routine activities. Assistive technologies' design and operation are required to have an easy interface with the brain with fewer protocols, in an attempt to optimize mobility and autonomy. The possible answer to these design questions may lie in neuroergonomics coupled with BMI systems. In this study, two human factors are addressed: designing a lightweight wearable robotic exoskeleton hand that is used to assist the potential stroke patients with an integrated portable brain interface using mental workload (MWL) signals acquired with portable functional near-infrared spectroscopy (fNIRS) system. The system may generate command signals for operating a wearable robotic exoskeleton hand using two-state MWL signals. The fNIRS system is used to record optical signals in the form of change in concentration of oxy and deoxygenated hemoglobin (HbO and HbR) from the pre-frontal cortex (PFC) region of the brain. Fifteen participants participated in this study and were given hand-grasping tasks. Two-state MWL signals acquired from the PFC region of the participant's brain are segregated using machine learning classifier-support vector machines (SVM) to utilize in operating a robotic exoskeleton hand. The maximum classification accuracy is 91.31%, using a combination of mean-slope features with an average information transfer rate (ITR) of 1.43. These results show the feasibility of a two-state MWL (fNIRS-based) robotic exoskeleton hand (BMI system) for hemiplegic patients assisting in the physical grasping tasks.
Collapse
Affiliation(s)
- Umer Asgher
- School of Mechanical and Manufacturing Engineering (SMME), National University of Sciences and Technology (NUST), Islamabad, Pakistan
| | - Muhammad Jawad Khan
- School of Mechanical and Manufacturing Engineering (SMME), National University of Sciences and Technology (NUST), Islamabad, Pakistan
| | - Muhammad Hamza Asif Nizami
- School of Mechanical and Manufacturing Engineering (SMME), National University of Sciences and Technology (NUST), Islamabad, Pakistan
- Florida State University College of Engineering, Florida A&M University, Tallahassee, FL, United States
| | - Khurram Khalil
- School of Mechanical and Manufacturing Engineering (SMME), National University of Sciences and Technology (NUST), Islamabad, Pakistan
| | - Riaz Ahmad
- School of Mechanical and Manufacturing Engineering (SMME), National University of Sciences and Technology (NUST), Islamabad, Pakistan
- Directorate of Quality Assurance and International Collaboration, National University of Sciences and Technology (NUST), Islamabad, Pakistan
| | - Yasar Ayaz
- School of Mechanical and Manufacturing Engineering (SMME), National University of Sciences and Technology (NUST), Islamabad, Pakistan
- National Center of Artificial Intelligence (NCAI), National University of Sciences and Technology, Islamabad, Pakistan
| | - Noman Naseer
- Department of Mechatronics and Biomedical Engineering, Air University, Islamabad, Pakistan
| |
Collapse
|
30
|
Kwon J, Im CH. Subject-Independent Functional Near-Infrared Spectroscopy-Based Brain-Computer Interfaces Based on Convolutional Neural Networks. Front Hum Neurosci 2021; 15:646915. [PMID: 33776674 PMCID: PMC7994252 DOI: 10.3389/fnhum.2021.646915] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/28/2020] [Accepted: 02/19/2021] [Indexed: 11/22/2022] Open
Abstract
Functional near-infrared spectroscopy (fNIRS) has attracted increasing attention in the field of brain–computer interfaces (BCIs) owing to their advantages such as non-invasiveness, user safety, affordability, and portability. However, fNIRS signals are highly subject-specific and have low test-retest reliability. Therefore, individual calibration sessions need to be employed before each use of fNIRS-based BCI to achieve a sufficiently high performance for practical BCI applications. In this study, we propose a novel deep convolutional neural network (CNN)-based approach for implementing a subject-independent fNIRS-based BCI. A total of 18 participants performed the fNIRS-based BCI experiments, where the main goal of the experiments was to distinguish a mental arithmetic task from an idle state task. Leave-one-subject-out cross-validation was employed to evaluate the average classification accuracy of the proposed subject-independent fNIRS-based BCI. As a result, the average classification accuracy of the proposed method was reported to be 71.20 ± 8.74%, which was higher than the threshold accuracy for effective BCI communication (70%) as well as that obtained using conventional shrinkage linear discriminant analysis (65.74 ± 7.68%). To achieve a classification accuracy comparable to that of the proposed subject-independent fNIRS-based BCI, 24 training trials (of approximately 12 min) were necessary for the traditional subject-dependent fNIRS-based BCI. It is expected that our CNN-based approach would reduce the necessity of long-term individual calibration sessions, thereby enhancing the practicality of fNIRS-based BCIs significantly.
Collapse
Affiliation(s)
- Jinuk Kwon
- Department of Biomedical Engineering, Hanyang University, Seoul, South Korea.,Department of Electronic Engineering, Hanyang University, Seoul, South Korea
| | - Chang-Hwan Im
- Department of Biomedical Engineering, Hanyang University, Seoul, South Korea.,Department of Electronic Engineering, Hanyang University, Seoul, South Korea
| |
Collapse
|
31
|
Manco L, Maffei N, Strolin S, Vichi S, Bottazzi L, Strigari L. Basic of machine learning and deep learning in imaging for medical physicists. Phys Med 2021; 83:194-205. [DOI: 10.1016/j.ejmp.2021.03.026] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/02/2020] [Revised: 03/07/2021] [Accepted: 03/16/2021] [Indexed: 02/08/2023] Open
|
32
|
Khan MU, Hasan MAH. Hybrid EEG-fNIRS BCI Fusion Using Multi-Resolution Singular Value Decomposition (MSVD). Front Hum Neurosci 2020; 14:599802. [PMID: 33363459 PMCID: PMC7753369 DOI: 10.3389/fnhum.2020.599802] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/28/2020] [Accepted: 11/12/2020] [Indexed: 12/16/2022] Open
Abstract
Brain-computer interface (BCI) multi-modal fusion has the potential to generate multiple commands in a highly reliable manner by alleviating the drawbacks associated with single modality. In the present work, a hybrid EEG-fNIRS BCI system—achieved through a fusion of concurrently recorded electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) signals—is used to overcome the limitations of uni-modality and to achieve higher tasks classification. Although the hybrid approach enhances the performance of the system, the improvements are still modest due to the lack of availability of computational approaches to fuse the two modalities. To overcome this, a novel approach is proposed using Multi-resolution singular value decomposition (MSVD) to achieve system- and feature-based fusion. The two approaches based up different features set are compared using the KNN and Tree classifiers. The results obtained through multiple datasets show that the proposed approach can effectively fuse both modalities with improvement in the classification accuracy.
Collapse
Affiliation(s)
- Muhammad Umer Khan
- Department of Mechatronics Engineering, Atilim University, Ankara, Turkey
| | - Mustafa A H Hasan
- Department of Mechatronics Engineering, Atilim University, Ankara, Turkey
| |
Collapse
|
33
|
Yang D, Nguyen TH, Chung WY. A Bipolar-Channel Hybrid Brain-Computer Interface System for Home Automation Control Utilizing Steady-State Visually Evoked Potential and Eye-Blink Signals. SENSORS 2020; 20:s20195474. [PMID: 32987871 PMCID: PMC7582823 DOI: 10.3390/s20195474] [Citation(s) in RCA: 14] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/27/2020] [Revised: 09/19/2020] [Accepted: 09/21/2020] [Indexed: 11/22/2022]
Abstract
The goal of this study was to develop and validate a hybrid brain-computer interface (BCI) system for home automation control. Over the past decade, BCIs represent a promising possibility in the field of medical (e.g., neuronal rehabilitation), educational, mind reading, and remote communication. However, BCI is still difficult to use in daily life because of the challenges of the unfriendly head device, lower classification accuracy, high cost, and complex operation. In this study, we propose a hybrid BCI system for home automation control with two brain signals acquiring electrodes and simple tasks, which only requires the subject to focus on the stimulus and eye blink. The stimulus is utilized to select commands by generating steady-state visually evoked potential (SSVEP). The single eye blinks (i.e., confirm the selection) and double eye blinks (i.e., deny and re-selection) are employed to calibrate the SSVEP command. Besides that, the short-time Fourier transform and convolution neural network algorithms are utilized for feature extraction and classification, respectively. The results show that the proposed system could provide 38 control commands with a 2 s time window and a good accuracy (i.e., 96.92%) using one bipolar electroencephalogram (EEG) channel. This work presents a novel BCI approach for the home automation application based on SSVEP and eye blink signals, which could be useful for the disabled. In addition, the provided strategy of this study—a friendly channel configuration (i.e., one bipolar EEG channel), high accuracy, multiple commands, and short response time—might also offer a reference for the other BCI controlled applications.
Collapse
|