1
|
Aviles M, Alvarez-Alvarado JM, Robles-Ocampo JB, Sevilla-Camacho PY, Rodríguez-Reséndiz J. Optimizing RNNs for EMG Signal Classification: A Novel Strategy Using Grey Wolf Optimization. Bioengineering (Basel) 2024; 11:77. [PMID: 38247954 PMCID: PMC10813014 DOI: 10.3390/bioengineering11010077] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2023] [Revised: 01/08/2024] [Accepted: 01/11/2024] [Indexed: 01/23/2024] Open
Abstract
Accurate classification of electromyographic (EMG) signals is vital in biomedical applications. This study evaluates different architectures of recurrent neural networks for the classification of EMG signals associated with five movements of the right upper extremity. A Butterworth filter was implemented for signal preprocessing, followed by segmentation into 250 ms windows, with an overlap of 190 ms. The resulting dataset was divided into training, validation, and testing subsets. The Grey Wolf Optimization algorithm was applied to the gated recurrent unit (GRU), long short-term memory (LSTM) architectures, and bidirectional recurrent neural networks. In parallel, a performance comparison with support vector machines (SVMs) was performed. The results obtained in the first experimental phase revealed that all the RNN networks evaluated reached a 100% accuracy, standing above the 93% achieved by the SVM. Regarding classification speed, LSTM ranked as the fastest architecture, recording a time of 0.12 ms, followed by GRU with 0.134 ms. Bidirectional recurrent neural networks showed a response time of 0.2 ms, while SVM had the longest time at 2.7 ms. In the second experimental phase, a slight decrease in the accuracy of the RNN models was observed, standing at 98.46% for LSTM, 96.38% for GRU, and 97.63% for the bidirectional network. The findings of this study highlight the effectiveness and speed of recurrent neural networks in the EMG signal classification task.
Collapse
Affiliation(s)
- Marcos Aviles
- Facultad de Ingeniería, Universidad Autónoma de Querétaro, Santiago de Querétaro 76010, Mexico;
| | | | - Jose-Billerman Robles-Ocampo
- Programa de Postgrado en Energías Renovables, Universidad Politécnica de Chiapas, Suchiapa 29150, Mexico; (J.-B.R.-O.); (P.Y.S.-C.)
- Departamento de Ingeniería Energética, Universidad Politécnica de Chiapas, Suchiapa 29150, Mexico
| | - Perla Yazmín Sevilla-Camacho
- Programa de Postgrado en Energías Renovables, Universidad Politécnica de Chiapas, Suchiapa 29150, Mexico; (J.-B.R.-O.); (P.Y.S.-C.)
- Departamento de Ingeniería Mecatrónica, Universidad Politécnica de Chiapas, Suchiapa 29150, Mexico
| | | |
Collapse
|
2
|
Zhang W, Wang Y, Zhang J, Pang G. EMG-FRNet: A feature reconstruction network for EMG irrelevant gesture recognition. Biosci Trends 2023:2023.01116. [PMID: 37394614 DOI: 10.5582/bst.2023.01116] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/04/2023]
Abstract
With the development of deep learning technology, gesture recognition based on surface electromyography (EMG) signals has shown broad application prospects in various human-computer interaction fields. Most current gesture recognition technologies can achieve high recognition accuracy on a wide range of gesture actions. However, in practical applications, gesture recognition based on surface EMG signals is susceptible to interference from irrelevant gesture movements, which affects the accuracy and security of the system. Therefore, it is crucial to design an irrelevant gesture recognition method. This paper introduces the GANomaly network from the field of image anomaly detection into surface EMG-based irrelevant gesture recognition. The network has a small feature reconstruction error for target samples and a large feature reconstruction error for irrelevant samples. By comparing the relationship between the feature reconstruction error and the predefined threshold, we can determine whether the input samples are from the target category or the irrelevant category. In order to improve the performance of EMG irrelevant gesture recognition, this paper proposes a feature reconstruction network named EMG-FRNet for EMG irrelevant gesture recognition. This network is based on GANomaly and incorporates structures such as channel cropping (CC), cross-layer encoding-decoding feature fusion (CLEDFF), and SE channel attention (SE). In this paper, Ninapro DB1, Ninapro DB5 and self-collected datasets were used to verify the performance of the proposed model. The Area Under the receiver operating characteristic Curve (AUC) values of EMG-FRNet on the above three datasets were 0.940, 0.926 and 0.962, respectively. Experimental results demonstrate that the proposed model achieves the highest accuracy among related research.
Collapse
Affiliation(s)
- Wenli Zhang
- Faculty of Imformation Technology, Beijing University of Technology, Beijing, China
| | - Yufei Wang
- Faculty of Imformation Technology, Beijing University of Technology, Beijing, China
| | - Jianyi Zhang
- College of Art and Design, Beijing University of Technology, Beijing, China
| | - Gongpeng Pang
- Faculty of Imformation Technology, Beijing University of Technology, Beijing, China
| |
Collapse
|
3
|
Zheng K, Liu S, Yang J, Al-Selwi M, Li J. sEMG-Based Continuous Hand Action Prediction by Using Key State Transition and Model Pruning. SENSORS (BASEL, SWITZERLAND) 2022; 22:s22249949. [PMID: 36560318 PMCID: PMC9787629 DOI: 10.3390/s22249949] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/20/2022] [Revised: 12/12/2022] [Accepted: 12/14/2022] [Indexed: 06/12/2023]
Abstract
Conventional classification of hand motions and continuous joint angle estimation based on sEMG have been widely studied in recent years. The classification task focuses on discrete motion recognition and shows poor real-time performance, while continuous joint angle estimation evaluates the real-time joint angles by the continuity of the limb. Few researchers have investigated continuous hand action prediction based on hand motion continuity. In our study, we propose the key state transition as a condition for continuous hand action prediction and simulate the prediction process using a sliding window with long-term memory. Firstly, the key state modeled by GMM-HMMs is set as the condition. Then, the sliding window is used to dynamically look for the key state transition. The prediction results are given while finding the key state transition. To extend continuous multigesture action prediction, we use model pruning to improve reusability. Eight subjects participated in the experiment, and the results show that the average accuracy of continuous two-hand actions is 97% with a 70 ms time delay, which is better than LSTM (94.15%, 308 ms) and GRU (93.83%, 300 ms). In supplementary experiments with continuous four-hand actions, over 85% prediction accuracy is achieved with an average time delay of 90 ms.
Collapse
Affiliation(s)
- Kaikui Zheng
- School of Advanced Manufacturing, Fuzhou University, Quanzhou 362200, China
| | - Shuai Liu
- School of Advanced Manufacturing, Fuzhou University, Quanzhou 362200, China
| | - Jinxing Yang
- Quanzhou Institute of Equipment Manufacturing, Chinese Academy of Sciences, Quanzhou 362216, China
| | - Metwalli Al-Selwi
- Quanzhou Institute of Equipment Manufacturing, Chinese Academy of Sciences, Quanzhou 362216, China
| | - Jun Li
- Quanzhou Institute of Equipment Manufacturing, Chinese Academy of Sciences, Quanzhou 362216, China
| |
Collapse
|
4
|
Karrenbach M, Preechayasomboon P, Sauer P, Boe D, Rombokas E. Deep learning and session-specific rapid recalibration for dynamic hand gesture recognition from EMG. Front Bioeng Biotechnol 2022; 10:1034672. [PMID: 36588953 PMCID: PMC9797837 DOI: 10.3389/fbioe.2022.1034672] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2022] [Accepted: 12/05/2022] [Indexed: 12/23/2022] Open
Abstract
We anticipate wide adoption of wrist and forearm electomyographic (EMG) interface devices worn daily by the same user. This presents unique challenges that are not yet well addressed in the EMG literature, such as adapting for session-specific differences while learning a longer-term model of the specific user. In this manuscript we present two contributions toward this goal. First, we present the MiSDIREKt (Multi-Session Dynamic Interaction Recordings of EMG and Kinematics) dataset acquired using a novel hardware design. A single participant performed four kinds of hand interaction tasks in virtual reality for 43 distinct sessions over 12 days, totaling 814 min. Second, we analyze this data using a non-linear encoder-decoder for dimensionality reduction in gesture classification. We find that an architecture which recalibrates with a small amount of single session data performs at an accuracy of 79.5% on that session, as opposed to architectures which learn solely from the single session (49.6%) or learn only from the training data (55.2%).
Collapse
Affiliation(s)
- Maxim Karrenbach
- Department of Electrical and Computer Engineering, University of Washington, Seattle, WA, United States
| | | | - Peter Sauer
- Department of Statistics and Data Science, Carnegie Mellon University, Pittsburgh, PA, United States
| | - David Boe
- Department of Mechanical Engineering, University of Washington, Seattle, WA, United States
| | - Eric Rombokas
- Department of Electrical and Computer Engineering, University of Washington, Seattle, WA, United States
- Department of Mechanical Engineering, University of Washington, Seattle, WA, United States
| |
Collapse
|
5
|
Wang T, Zhao Y, Wang Q. A Flexible Iontronic Capacitive Sensing Array for Hand Gesture Recognition Using Deep Convolutional Neural Networks. Soft Robot 2022. [DOI: 10.1089/soro.2021.0209] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/05/2022] Open
Affiliation(s)
- Tiantong Wang
- Department of Advanced Manufacturing and Robotics, College of Engineering, Peking University, Beijing, China
- Beijing Engineering Research Center of Intelligent Rehabilitation Engineering, Beijing, China
| | - Yunbiao Zhao
- Department of Advanced Manufacturing and Robotics, College of Engineering, Peking University, Beijing, China
- Beijing Engineering Research Center of Intelligent Rehabilitation Engineering, Beijing, China
| | - Qining Wang
- Department of Advanced Manufacturing and Robotics, College of Engineering, Peking University, Beijing, China
- Beijing Engineering Research Center of Intelligent Rehabilitation Engineering, Beijing, China
- Institute for Artificial Intelligence, Peking University, Beijing, China
- Beijing Institute for General Artificial Intelligence, Beijing, China
| |
Collapse
|
6
|
Jiang Y, Song L, Zhang J, Song Y, Yan M. Multi-Category Gesture Recognition Modeling Based on sEMG and IMU Signals. SENSORS (BASEL, SWITZERLAND) 2022; 22:5855. [PMID: 35957417 PMCID: PMC9371015 DOI: 10.3390/s22155855] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/26/2022] [Revised: 08/01/2022] [Accepted: 08/02/2022] [Indexed: 05/14/2023]
Abstract
Gesture recognition based on wearable devices is one of the vital components of human-computer interaction systems. Compared with skeleton-based recognition in computer vision, gesture recognition using wearable sensors has attracted wide attention for its robustness and convenience. Recently, many studies have proposed deep learning methods based on surface electromyography (sEMG) signals for gesture classification; however, most of the existing datasets are built for surface EMG signals, and there is a lack of datasets for multi-category gestures. Due to model limitations and inadequate classification data, the recognition accuracy of these methods cannot satisfy multi-gesture interaction scenarios. In this paper, a multi-category dataset containing 20 gestures is recorded with the help of a wearable device that can acquire surface electromyographic and inertial (IMU) signals. Various two-stream deep learning models are established and improved further. The basic convolutional neural network (CNN), recurrent neural network (RNN), and Transformer models are experimented on with our dataset as the classifier. The CNN and the RNN models' test accuracy is over 95%; however, the Transformer model has a lower test accuracy of 71.68%. After further improvements, the CNN model is introduced into the residual network and augmented to the CNN-Res model, achieving 98.24% accuracy; moreover, it has the shortest training and testing time. Then, after combining the RNN model and the CNN-Res model, the long short term memory (LSTM)-Res model and gate recurrent unit (GRU)-Res model achieve the highest classification accuracy of 99.67% and 99.49%, respectively. Finally, the fusion of the Transformer model and the CNN model enables the Transformer-CNN model to be constructed. Such improvement dramatically boosts the performance of the Transformer module, increasing the recognition accuracy from 71.86% to 98.96%.
Collapse
Affiliation(s)
- Yujian Jiang
- State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, China
- Key Laboratory of Acoustic Visual Technology and Intelligent Control System, Ministry of Culture and Tourism, Communication University of China, Beijing 100024, China
- Beijing Key Laboratory of Modern Entertainment Technology, Communication University of China, Beijing 100024, China
- School of Information and Communication Engineering, Communication University of China, Beijing 100024, China
| | - Lin Song
- State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, China
- Key Laboratory of Acoustic Visual Technology and Intelligent Control System, Ministry of Culture and Tourism, Communication University of China, Beijing 100024, China
- Beijing Key Laboratory of Modern Entertainment Technology, Communication University of China, Beijing 100024, China
- School of Information and Communication Engineering, Communication University of China, Beijing 100024, China
| | - Junming Zhang
- State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, China
- Key Laboratory of Acoustic Visual Technology and Intelligent Control System, Ministry of Culture and Tourism, Communication University of China, Beijing 100024, China
- Beijing Key Laboratory of Modern Entertainment Technology, Communication University of China, Beijing 100024, China
- School of Information and Communication Engineering, Communication University of China, Beijing 100024, China
| | - Yang Song
- State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, China
- Key Laboratory of Acoustic Visual Technology and Intelligent Control System, Ministry of Culture and Tourism, Communication University of China, Beijing 100024, China
- Beijing Key Laboratory of Modern Entertainment Technology, Communication University of China, Beijing 100024, China
- School of Information and Communication Engineering, Communication University of China, Beijing 100024, China
| | - Ming Yan
- State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, China
- Key Laboratory of Acoustic Visual Technology and Intelligent Control System, Ministry of Culture and Tourism, Communication University of China, Beijing 100024, China
- Beijing Key Laboratory of Modern Entertainment Technology, Communication University of China, Beijing 100024, China
- School of Information and Communication Engineering, Communication University of China, Beijing 100024, China
| |
Collapse
|
7
|
Fatayer A, Gao W, Fu Y. sEMG-based Gesture Recognition using Deep Learning from Noisy Labels. IEEE J Biomed Health Inform 2022; 26:4462-4473. [PMID: 35653452 DOI: 10.1109/jbhi.2022.3179630] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
Gesture recognition for myoelectric prosthesis control utilizing sparse multichannel surface Electromyography (sEMG) is a challenging task, and from a Muscle-Computer Interface (MCI) standpoint, the performance is still far from optimal. However, the design of a well-performed sEMG recognition system depends on the flexibility of the input-output function and the dataset's quality. To improve the performance of MCI, we proposed a novel gesture recognition framework that (i) Enrich the spectral information of the sparse sEMG signals by constructing a fused map image (denoted as sEMG-Map) that integrates a multiresolution decomposition (by means of orthogonal wavelets) through the raw signals then rely upon the Convolutional Neural Network (CNN) capacity to exploit the composite hierarchies in the constructed sEMGMap input. (ii) deals with the label noise by proposing a data-centric method (denoted as ALR-CNN) that synchronously refines the falsely labeled samples and optimizes the CNN model based on two basic assumptions. First, the deep model accuracy improves as the training progress. Second, a set of successive learnable max-activated outputs of a well-performed deep model is a reliable estimator for motion detection in the muscle activation pattern. Our proposed framework is evaluated on three large-scale public databases. The average classification accuracy is 95.50%, 95.85%, and 85.58% for NinaPro DB2, NinaPro DB7, and NinaPro DB3, respectively. The experimental results verify the effectuality of the proposed method and show high accuracy.
Collapse
|
8
|
Hoshino T, Kanoga S, Tsubaki M, Aoyama A. Comparing subject-to-subject transfer learning methods in surface electromyogram-based motion recognition with shallow and deep classifiers. Neurocomputing 2022. [DOI: 10.1016/j.neucom.2021.12.081] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
|
9
|
Fang Y, Yang J, Zhou D, Ju Z. Modelling EMG driven wrist movements using a bio-inspired neural network. Neurocomputing 2022. [DOI: 10.1016/j.neucom.2021.10.104] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/05/2023]
|
10
|
Ghislieri M, Cerone GL, Knaflitz M, Agostini V. Long short-term memory (LSTM) recurrent neural network for muscle activity detection. J Neuroeng Rehabil 2021; 18:153. [PMID: 34674720 PMCID: PMC8532313 DOI: 10.1186/s12984-021-00945-w] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2021] [Accepted: 10/13/2021] [Indexed: 11/18/2022] Open
Abstract
BACKGROUND The accurate temporal analysis of muscle activation is of great interest in many research areas, spanning from neurorobotic systems to the assessment of altered locomotion patterns in orthopedic and neurological patients and the monitoring of their motor rehabilitation. The performance of the existing muscle activity detectors is strongly affected by both the SNR of the surface electromyography (sEMG) signals and the set of features used to detect the activation intervals. This work aims at introducing and validating a powerful approach to detect muscle activation intervals from sEMG signals, based on long short-term memory (LSTM) recurrent neural networks. METHODS First, the applicability of the proposed LSTM-based muscle activity detector (LSTM-MAD) is studied through simulated sEMG signals, comparing the LSTM-MAD performance against other two widely used approaches, i.e., the standard approach based on Teager-Kaiser Energy Operator (TKEO) and the traditional approach, used in clinical gait analysis, based on a double-threshold statistical detector (Stat). Second, the effect of the Signal-to-Noise Ratio (SNR) on the performance of the LSTM-MAD is assessed considering simulated signals with nine different SNR values. Finally, the newly introduced approach is validated on real sEMG signals, acquired during both physiological and pathological gait. Electromyography recordings from a total of 20 subjects (8 healthy individuals, 6 orthopedic patients, and 6 neurological patients) were included in the analysis. RESULTS The proposed algorithm overcomes the main limitations of the other tested approaches and it works directly on sEMG signals, without the need for background-noise and SNR estimation (as in Stat). Results demonstrate that LSTM-MAD outperforms the other approaches, revealing higher values of F1-score (F1-score > 0.91) and Jaccard similarity index (Jaccard > 0.85), and lower values of onset/offset bias (average absolute bias < 6 ms), both on simulated and real sEMG signals. Moreover, the advantages of using the LSTM-MAD algorithm are particularly evident for signals featuring a low to medium SNR. CONCLUSIONS The presented approach LSTM-MAD revealed excellent performances against TKEO and Stat. The validation carried out both on simulated and real signals, considering normal as well as pathological motor function during locomotion, demonstrated that it can be considered a powerful tool in the accurate and effective recognition/distinction of muscle activity from background noise in sEMG signals.
Collapse
Affiliation(s)
- Marco Ghislieri
- Department of Electronics and Telecommunications, Politecnico Di Torino, 10129, Turin, Italy.
- PoliToBIOMed Lab, Politecnico di Torino, 10129, Turin, Italy.
| | - Giacinto Luigi Cerone
- PoliToBIOMed Lab, Politecnico di Torino, 10129, Turin, Italy
- Laboratory for Engineering of the Neuromuscular System (LISiN), Departement of Electronics and Telecommunications, Politecnico di Torino, 10129, Turin, Italy
| | - Marco Knaflitz
- Department of Electronics and Telecommunications, Politecnico Di Torino, 10129, Turin, Italy
- PoliToBIOMed Lab, Politecnico di Torino, 10129, Turin, Italy
| | - Valentina Agostini
- Department of Electronics and Telecommunications, Politecnico Di Torino, 10129, Turin, Italy
- PoliToBIOMed Lab, Politecnico di Torino, 10129, Turin, Italy
| |
Collapse
|
11
|
Su Z, Liu H, Qian J, Zhang Z, Zhang L. Hand Gesture Recognition Based on sEMG Signal and Convolutional Neural Network. INT J PATTERN RECOGN 2021. [DOI: 10.1142/s0218001421510125] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
Abstract
Recently, deep learning has become a promising technique for constructing gesture recognition classifiers from surface electromyography (sEMG) signals in human–computer interaction. In this paper, we propose a gesture recognition method with sEMG signals based on a deep multi-parallel convolutional neural network (CNN), which solves the problem that traditional machine learning methods may lose too much useful information during feature extraction. CNNs provide an efficient way to constrain the complexity of feedforward neural networks by weight sharing and restriction to local connections. Sophisticated feature extraction is to be avoided and hand gestures are to be classified directly. A multi-parallel and multi-convolution layer convolution structure is proposed to classify hand gestures. Experiment results show that in comparison with five traditional machine learning methods, the proposed method could achieve higher accuracy.
Collapse
Affiliation(s)
- Ziyi Su
- School of Mechatronic Engineering and Automation, Shanghai University, 99 Shangda Road, Baoshan District, Shanghai, P. R. China
| | - Handong Liu
- School of Mechatronic Engineering and Automation, Shanghai University, 99 Shangda Road, Baoshan District, Shanghai, P. R. China
| | - Jinwu Qian
- School of Mechatronic Engineering and Automation, Shanghai University, 99 Shangda Road, Baoshan District, Shanghai, P. R. China
| | - Zhen Zhang
- School of Mechatronic Engineering and Automation, Shanghai University, 99 Shangda Road, Baoshan District, Shanghai, P. R. China
| | - Lunwei Zhang
- School of Aerospace Engineering and Mechanics, Tongji University, 1239 Siping Road, Yangpu District, Shanghai, P. R. China
| |
Collapse
|
12
|
Li W, Shi P, Yu H. Gesture Recognition Using Surface Electromyography and Deep Learning for Prostheses Hand: State-of-the-Art, Challenges, and Future. Front Neurosci 2021; 15:621885. [PMID: 33981195 PMCID: PMC8107289 DOI: 10.3389/fnins.2021.621885] [Citation(s) in RCA: 21] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/27/2020] [Accepted: 02/23/2021] [Indexed: 01/09/2023] Open
Abstract
Amputation of the upper limb brings heavy burden to amputees, reduces their quality of life, and limits their performance in activities of daily life. The realization of natural control for prosthetic hands is crucial to improving the quality of life of amputees. Surface electromyography (sEMG) signal is one of the most widely used biological signals for the prediction of upper limb motor intention, which is an essential element of the control systems of prosthetic hands. The conversion of sEMG signals into effective control signals often requires a lot of computational power and complex process. Existing commercial prosthetic hands can only provide natural control for very few active degrees of freedom. Deep learning (DL) has performed surprisingly well in the development of intelligent systems in recent years. The significant improvement of hardware equipment and the continuous emergence of large data sets of sEMG have also boosted the DL research in sEMG signal processing. DL can effectively improve the accuracy of sEMG pattern recognition and reduce the influence of interference factors. This paper analyzes the applicability and efficiency of DL in sEMG-based gesture recognition and reviews the key techniques of DL-based sEMG pattern recognition for the prosthetic hand, including signal acquisition, signal preprocessing, feature extraction, classification of patterns, post-processing, and performance evaluation. Finally, the current challenges and future prospects in clinical application of these techniques are outlined and discussed.
Collapse
Affiliation(s)
- Wei Li
- Institute of Rehabilitation Engineering and Technology, University of Shanghai for Science and Technology, Shanghai, China
| | - Ping Shi
- Institute of Rehabilitation Engineering and Technology, University of Shanghai for Science and Technology, Shanghai, China
| | - Hongliu Yu
- Institute of Rehabilitation Engineering and Technology, University of Shanghai for Science and Technology, Shanghai, China
| |
Collapse
|
13
|
Yu Z, Zhao J, Wang Y, He L, Wang S. Surface EMG-Based Instantaneous Hand Gesture Recognition Using Convolutional Neural Network with the Transfer Learning Method. SENSORS (BASEL, SWITZERLAND) 2021; 21:2540. [PMID: 33916379 PMCID: PMC8038633 DOI: 10.3390/s21072540] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 01/21/2021] [Revised: 03/15/2021] [Accepted: 04/02/2021] [Indexed: 02/03/2023]
Abstract
In recent years, surface electromyography (sEMG)-based human-computer interaction has been developed to improve the quality of life for people. Gesture recognition based on the instantaneous values of sEMG has the advantages of accurate prediction and low latency. However, the low generalization ability of the hand gesture recognition method limits its application to new subjects and new hand gestures, and brings a heavy training burden. For this reason, based on a convolutional neural network, a transfer learning (TL) strategy for instantaneous gesture recognition is proposed to improve the generalization performance of the target network. CapgMyo and NinaPro DB1 are used to evaluate the validity of our proposed strategy. Compared with the non-transfer learning (non-TL) strategy, our proposed strategy improves the average accuracy of new subject and new gesture recognition by 18.7% and 8.74%, respectively, when up to three repeated gestures are employed. The TL strategy reduces the training time by a factor of three. Experiments verify the transferability of spatial features and the validity of the proposed strategy in improving the recognition accuracy of new subjects and new gestures, and reducing the training burden. The proposed TL strategy provides an effective way of improving the generalization ability of the gesture recognition system.
Collapse
Affiliation(s)
- Zhipeng Yu
- Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China; (Z.Y.); (Y.W.); (S.W.)
- University of Science and Technology of China, Hefei 230026, China;
| | - Jianghai Zhao
- Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China; (Z.Y.); (Y.W.); (S.W.)
| | - Yucheng Wang
- Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China; (Z.Y.); (Y.W.); (S.W.)
| | - Linglong He
- University of Science and Technology of China, Hefei 230026, China;
| | - Shaonan Wang
- Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China; (Z.Y.); (Y.W.); (S.W.)
- University of Science and Technology of China, Hefei 230026, China;
| |
Collapse
|
14
|
Fang Y, Zhang X, Zhou D, Liu H. Improve Inter-day Hand Gesture Recognition Via Convolutional Neural Network-based Feature Fusion. INT J HUM ROBOT 2021. [DOI: 10.1142/s0219843620500255] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
Abstract
The learning of inter-day representation of electromyographic (EMG) signals across multiple days remains a challenging topic and not fully accommodated yet. This study aims to improve the inter-day hand motion classification accuracy via convolutional neural network (CNN)-based data feature fusion. An EMG database (ISRMyo-I) was recorded from six subjects on 10 days via a low density electrode setting. This study investigated CNNs’ capability of feature learning, and found that the output of the first fully connected layer (CNNFeats) was a decent supplement feature set to the most prevalent Hudgins’ time domain features in combination with fourth-order autoregressive coefficients (TDAR). Through adding the automatically learned CNNFeats to the handcrafted TDAR feature set, both linear discriminant analysis (LDA) and support vector machine (SVM) classifiers received [Formula: see text]3% accuracy improvement. Similarly, taking TDAR as additional input to the CNN improved the accuracy by [Formula: see text]1% in the comparison with the basic CNN. Our results also demonstrated that the CNN approach outperformed conventional approaches when multiple subjects’ data were available for training, while traditional approaches were more adept at presenting motion patterns for single subject. A preliminary conclusion is drawn that substantial “common knowledge/features” can be learned by CNNs from the raw EMG signals across multiple days and multiple subjects, and thus it is believed that a pre-trained CNN model would contribute to higher accuracy as well as the reduction of learning burden.
Collapse
Affiliation(s)
- Yinfeng Fang
- College of Telecommunication, Hangzhou Dianzi University, 1158, No. 2 Avenue, Xiasha, Hangzhou 310018, P. R. China
| | - Xuguang Zhang
- College of Telecommunication, Hangzhou Dianzi University, 1158, No. 2 Avenue, Xiasha, Hangzhou 310018, P. R. China
| | - Dalin Zhou
- Intelligent Systems and Biomedical Robotics Group, School of Computing, University of Portsmouth, Portsmouth PO1 3HE, UK
| | - Honghai Liu
- Intelligent Systems and Biomedical Robotics Group, School of Computing, University of Portsmouth, Portsmouth PO1 3HE, UK
- School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai 200240, P. R. China
| |
Collapse
|
15
|
Tateno S, Liu H, Ou J. Development of Sign Language Motion Recognition System for Hearing-Impaired People Using Electromyography Signal. SENSORS (BASEL, SWITZERLAND) 2020; 20:E5807. [PMID: 33066452 PMCID: PMC7602266 DOI: 10.3390/s20205807] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/22/2020] [Revised: 10/10/2020] [Accepted: 10/12/2020] [Indexed: 06/11/2023]
Abstract
Sign languages are developed around the world for hearing-impaired people to communicate with others who understand them. Different grammar and alphabets limit the usage of sign languages between different sign language users. Furthermore, training is required for hearing-intact people to communicate with them. Therefore, in this paper, a real-time motion recognition system based on an electromyography signal is proposed for recognizing actual American Sign Language (ASL) hand motions for helping hearing-impaired people communicate with others and training normal people to understand the sign languages. A bilinear model is applied to deal with the on electromyography (EMG) data for decreasing the individual difference among different people. A long short-term memory neural network is used in this paper as the classifier. Twenty sign language motions in the ASL library are selected for recognition in order to increase the practicability of the system. The results indicate that this system can recognize these twenty motions with high accuracy among twenty participants. Therefore, this system has the potential to be widely applied to help hearing-impaired people for daily communication and normal people to understand the sign languages.
Collapse
Affiliation(s)
- Shigeyuki Tateno
- Graduate School of Information, Production and Systems, Waseda University, Kitakyushu 808-0135, Japan; (H.L.); (J.O.)
| | | | | |
Collapse
|
16
|
On the Use of Fuzzy and Permutation Entropy in Hand Gesture Characterization from EMG Signals: Parameters Selection and Comparison. APPLIED SCIENCES-BASEL 2020. [DOI: 10.3390/app10207144] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
The surface electromyography signal (sEMG) is widely used for gesture characterization; its reliability is strongly connected to the features extracted from sEMG recordings. This study aimed to investigate the use of two complexity measures, i.e., fuzzy entropy (FEn) and permutation entropy (PEn) for hand gesture characterization. Fourteen upper limb movements, sorted into three sets, were collected on ten subjects and the performances of FEn and PEn for gesture descriptions were analyzed for different computational parameters. FEn and PEn were able to properly cluster the expected numbers of gestures, but computational parameters were crucial for ensuring clusters’ separability and proper gesture characterization. FEn and PEn were also compared with other eighteen classical time and frequency domain features through the minimum redundancy maximum relevance algorithm and showed the best predictive importance scores in two gesture sets; they also had scores within the subset of the best five features in the remaining one. Further, the classification accuracies of four different feature sets presented remarkable increases when FEn and PEn are included as additional features. Outcomes support the use of FEn and PEn for hand gesture description when computational parameters are properly selected, and they could be useful in supporting the development of robotic arms and prostheses myoelectric control.
Collapse
|