The effect of arm position on classification of hand gestures with intramuscular EMG

Unravelling Influence Factors in Pattern Recognition Myoelectric Control Systems: The Impact of Limb Positions and Electrode Shifts

Pattern recognition (PR)-based myoelectric control systems can naturally provide multifunctional and intuitive control of upper limb prostheses and restore lost limb function, but understanding their robustness remains an open scientific question. This study investigates how limb positions and electrode shifts-two factors that have been suggested to cause classification deterioration-affect classifiers' performance by quantifying changes in the class distribution using each factor as a class and computing the repeatability and modified separability indices. Ten intact-limb participants took part in the study. Linear discriminant analysis (LDA) was used as the classifier. The results confirmed previous studies that limb positions and electrode shifts deteriorate classification performance (14-21% decrease) with no difference between factors (p > 0.05). When considering limb positions and electrode shifts as classes, we could classify them with an accuracy of 96.13 ± 1.44% and 65.40 ± 8.23% for single and all motions, respectively. Testing on five amputees corroborated the above findings. We have demonstrated that each factor introduces changes in the feature space that are statistically new class instances. Thus, the feature space contains two statistically classifiable clusters when the same motion is collected in two different limb positions or electrode shifts. Our results are a step forward in understanding PR schemes' challenges for myoelectric control of prostheses and further validation needs be conducted on more amputee-related datasets.

TraHGR: Transformer for Hand Gesture Recognition via Electromyography

Deep learning-based Hand Gesture Recognition (HGR) via surface Electromyogram (sEMG) signals have recently shown considerable potential for development of advanced myoelectric-controlled prosthesis. Although deep learning techniques can improve HGR accuracy compared to their classical counterparts, classifying hand movements based on sparse multichannel sEMG signals is still a challenging task. Furthermore, existing deep learning approaches, typically, include only one model as such can hardly extract representative features. In this paper, we aim to address this challenge by capitalizing on the recent advances in hybrid models and transformers. In other words, we propose a hybrid framework based on the transformer architecture, which is a relatively new and revolutionizing deep learning model. The proposed hybrid architecture, referred to as the Transformer for Hand Gesture Recognition (TraHGR), consists of two parallel paths followed by a linear layer that acts as a fusion center to integrate the advantage of each module. We evaluated the proposed architecture TraHGR based on the commonly used second Ninapro dataset, referred to as the DB2. The sEMG signals in the DB2 dataset are measured in real-life conditions from 40 healthy users, each performing 49 gestures. We have conducted an extensive set of experiments to test and validate the proposed TraHGR architecture, and compare its achievable accuracy with several recently proposed HGR classification algorithms over the same dataset. We have also compared the results of the proposed TraHGR architecture with each individual path and demonstrated the distinguishing power of the proposed hybrid architecture. The recognition accuracies of the proposed TraHGR architecture for the window of size 200ms and step size of 100ms are 86.00%, 88.72%, 81.27%, and 93.74%, which are 2.30%, 4.93%, 8.65%, and 4.20% higher than the state-of-the-art performance for DB2 (49 gestures), DB2-B (17 gestures), DB2-C (23 gestures), and DB2-D (9 gestures), respectively.

Two-stage classification of electromyogram signals from hand grasps in the transverse plane

This paper presents a two-stage classification to resolve the effect of arm position changes on electromyogram (EMG) classification for hand grasps in the transverse plane. The proposed method combines the EMG signals with the signals from an inertial measurement unit in both the position and motion classification stages. To improve accuracy, we incorporate EMG data from the upper arm and shoulder with the forearm EMG signals. When evaluated on the five alternative object grasps placed on the nine positions, the proposed technique yields an average total classification error of 0.9%, which is a substantial improvement over the single-stage classification (4.3%).

Use of regenerative peripheral nerve interfaces and intramuscular electrodes to improve prosthetic grasp selection: a case study

Objective.Advanced myoelectric hands enable users to select from multiple functional grasps. Current methods for controlling these hands are unintuitive and require frequent recalibration. This case study assessed the performance of tasks involving grasp selection, object interaction, and dynamic postural changes using intramuscular electrodes with regenerative peripheral nerve interfaces (RPNIs) and residual muscles.Approach.One female with unilateral transradial amputation participated in a series of experiments to compare the performance of grasp selection controllers with RPNIs and intramuscular control signals with controllers using surface electrodes. These experiments included a virtual grasp-matching task with and without a concurrent cognitive task and physical tasks with a prosthesis including standardized functional assessments and a functional assessment where the individual made a cup of coffee ('Coffee Task') that required grasp transitions.Main results.In the virtual environment, the participant was able to select between four functional grasps with higher accuracy using the RPNI controller (92.5%) compared to surface controllers (81.9%). With the concurrent cognitive task, performance of the virtual task was more consistent with RPNI controllers (reduced accuracy by 1.1%) compared to with surface controllers (4.8%). When RPNI signals were excluded from the controller with intramuscular electromyography (i.e. residual muscles only), grasp selection accuracy decreased by up to 24%. The participant completed the Coffee Task with 11.7% longer completion time with the surface controller than with the RPNI controller. She also completed the Coffee Task with 11 fewer transition errors out of a maximum of 25 total errors when using the RPNI controller compared to surface controller.Significance.The use of RPNI signals in concert with residual muscles and intramuscular electrodes can improve grasp selection accuracy in both virtual and physical environments. This approach yielded consistent performance without recalibration needs while reducing cognitive load associated with pattern recognition for myoelectric control (clinical trial registration number NCT03260400).

A Co-driven Functional Electrical Stimulation Control Strategy by Dynamic Surface Electromyography and Joint Angle

Functional electrical stimulation (FES) is widely used in neurorehabilitation to improve patients’ motion ability. It has been verified to promote neural remodeling and relearning, during which FES has to produce an accurate movement to obtain a good efficacy. Therefore, many studies have focused on the relationship between FES parameters and the generated movements. However, most of the relationships have been established in static contractions, which leads to an unsatisfactory result when applied to dynamic conditions. Therefore, this study proposed a FES control strategy based on the surface electromyography (sEMG) and kinematic information during dynamic contractions. The pulse width (PW) of FES was determined by a direct transfer function (DTF) with sEMG features and joint angles as the input. The DTF was established by combing the polynomial transfer functions of sEMG and joint torque and the polynomial transfer functions of joint torque and FES. Moreover, the PW of two FES channels was set based on the muscle synergy ratio obtained through sEMG. A total of six healthy right-handed subjects were recruited in this experiment to verify the validity of the strategy. The PW of FES applied to the left arm was evaluated based on the sEMG of the right extensor carpi radialis (ECR) and the right wrist angle. The coefficient of determination (R²) and the normalized root mean square error (NRMSE) of FES-included and voluntary wrist angles and torques were used to verify the performance of the strategy. The result showed that this study achieved a high accuracy (R² = 0.965 and NRMSE = 0.047) of joint angle and a good accuracy (R² = 0.701 and NRMSE = 0.241) of joint torque reproduction during dynamic movements. Moreover, the DTF in real-time FES system also had a nice performance of joint angle fitting (R² = 0.940 and NRMSE = 0.071) and joint torque fitting (R² = 0.607 and NRMSE = 0.303). It is concluded that the proposed strategy is able to generate proper FES parameters based on sEMG and kinematic information for dynamic movement reproduction and can be used in a real-time FES system combined with bilateral movements for better rehabilitation.

Evaluation of feature projection techniques in object grasp classification using electromyogram signals from different limb positions

A myoelectric prosthesis is manipulated using electromyogram (EMG) signals from the existing muscles for performing the activities of daily living. A feature vector that is formed by concatenating data from many EMG channels may result in a high dimensional space, which may cause prolonged computation time, redundancy, and irrelevant information. We evaluated feature projection techniques, namely principal component analysis (PCA), linear discriminant analysis (LDA), t-Distributed Stochastic Neighbor Embedding (t-SNE), and spectral regression extreme learning machine (SRELM), applied to object grasp classification. These represent feature projections that are combinations of either linear or nonlinear, and supervised or unsupervised types. All pairs of the four types of feature projection with seven types of classifiers were evaluated, with data from six EMG channels and an IMU sensors for nine upper limb positions in the transverse plane. The results showed that SRELM outperformed LDA with supervised feature projections, and t-SNE was superior to PCA with unsupervised feature projections. The classification errors from SRELM and t-SNE paired with the seven classifiers were from 1.50% to 2.65% and from 1.27% to 17.15%, respectively. A one-way ANOVA test revealed no statistically significant difference by classifier type when using the SRELM projection, which is a nonlinear supervised feature projection (p = 0.334). On the other hand, we have to carefully select an appropriate classifier for use with t-SNE, which is a nonlinear unsupervised feature projection. We achieved the lowest classification error 1.27% using t-SNE paired with a k-nearest neighbors classifier. For SRELM, the lowest 1.50% classification error was obtained when paired with a neural network classifier.

Review on electromyography based intention for upper limb control using pattern recognition for human-machine interaction

Upper limb myoelectric prosthetic control is an essential topic in the field of rehabilitation. The technique controls prostheses using surface electromyogram (sEMG) and intramuscular EMG (iEMG) signals. EMG signals are extensively used in controlling prosthetic upper and lower limbs, virtual reality entertainment, and human-machine interface (HMI). EMG signals are vital parameters for machine learning and deep learning algorithms and help to give an insight into the human brain’s function and mechanisms. Pattern recognition techniques pertaining to support vector machine (SVM), k-nearest neighbor (KNN) and Bayesian classifiers have been utilized to classify EMG signals. This paper presents a review on current EMG signal techniques, including electrode array utilization, signal acquisition, signal preprocessing and post-processing, feature selection and extraction, data dimensionality reduction, classification, and ultimate application to the community. The paper also discusses using alternatives to EMG signals, such as force sensors, to measure muscle activity with reliable results. Future implications for EMG classification include employing deep learning techniques such as artificial neural networks (ANN) and recurrent neural networks (RNN) for achieving robust results.

Towards Robust, Adaptive and Reliable Upper-limb Motion Estimation Using Machine Learning and Deep Learning--A Survey in Myoelectric Control

To develop multi-functional human-machine interfaces that can help disabled people reconstruct lost functions of upper-limbs, machine learning (ML) and deep learning (DL) techniques have been widely implemented to decode human movement intentions from surface electromyography (sEMG) signals. However, due to the high complexity of upper-limb movements and the inherent non-stable characteristics of sEMG, the usability of ML/DL based control schemes is still greatly limited in practical scenarios. To this end, tremendous efforts have been made to improve model robustness, adaptation, and reliability. In this article, we provide a systematic review on recent achievements, mainly from three categories: multi-modal sensing fusion to gain additional information of the user, transfer learning (TL) methods to eliminate domain shift impacts on estimation models, and post-processing approaches to obtain more reliable outcomes. Special attention is given to fusion strategies, deep TL frameworks, and confidence estimation. \textcolor{red}{Research challenges and emerging opportunities, with respect to hardware development, public resources, and decoding strategies, are also analysed to provide perspectives for future developments.

Activities of Daily Living-based Rehabilitation System for Arm and Hand Motor Function Retraining after Stroke

Most stroke survivors have difficulties completing activities of daily living (ADLs) independently. However, few rehabilitation systems have focused on ADLs-related training for gross and fine motor function together. We propose an ADLs-based serious game rehabilitation system for the training of motor function and coordination of both arm and hand movement where the user performs corresponding ADLs movements to interact with the target in the serious game. A multi-sensor fusion model based on electromyographic (EMG), force myographic (FMG), and inertial sensing was developed to estimate users' natural upper limb movement. Eight healthy subjects and three stroke patients were recruited in an experiment to validate the system's effectiveness. The performance of different sensor and classifier configurations on hand gesture classification against the arm position variations were analyzed, and qualitative patient questionnaires were conducted. Results showed that elbow extension/flexion has a more significant negative influence on EMG-based, FMG-based, and EMG+FMG-based hand gesture recognition than shoulder abduction/adduction does. In addition, there was no significant difference in the negative influence of shoulder abduction/adduction and shoulder flexion/extension on hand gesture recognition. However, there was a significant interaction between sensor configurations and algorithm configurations in both offline and real-time recognition accuracy. The EMG+FMG-combined multi-position classifier model had the best performance against arm position change. In addition, all the stroke patients reported their ADLs-related ability could be restored by using the system. These results demonstrate that the multi-sensor fusion model could estimate hand gestures and gross movement accurately, and the proposed training system has the potential to improve patients' ability to perform ADLs.

Inter-classifier comparison for upper extremity EMG signal at different hand postures and arm positions using pattern recognition

The utilization of surface EMG and intramuscular EMG signals has been observed to create significant improvement in pattern recognition approaches and myoelectric control. However, there is less data of different arm positions and hand postures available. Hand postures and arm positions tend to affect the combination of surface and intramuscular EMG signal acquisition in terms of classifier accuracy. Hence, this study aimed to find a robust classifier for two scenarios: (1) at fixed arm position (FAP) where classifiers classify different hand postures and (2) at fixed hand posture (FHP) where classifiers classify different arm positions. A total of 20 healthy male participants (30.62 ± 3.87 years old) were recruited for this study. They were asked to perform five motion classes including hand grasp, hand open, rest, hand extension, and hand flexion at four different arm positions at 0°, 45°, 90°, and 135°. SVM, KNN, and LDA classifier were deployed. Statistical analysis in the form of pairwise comparisons was carried out using SPSS. It is concluded that there is no significant difference among the three classifiers. SVM gave highest accuracy of 75.35% and 58.32% at FAP and FHP respectively for each motion classification. KNN yielded the highest accuracies of 69.11% and 79.04% when data was pooled and was classified at different arm positions and at different hand postures respectively. The results exhibited that there is no significant effect of changing arm position and hand posture on the classifier accuracy.

EMG based classification for pick and place task

The hand amputee is deprived of number of activities of daily living. To help the hand amputee, it is important to learn the pattern of muscles activity. There are several elements of tasks, which involve forearm along with the wrist and hand. The one very important task is pick and place activity performed by the hand. A pick and place action is a compilation of different finger motions for the grasping of objects at different force levels. This action may be better understood by learning the electromyography signals of forearm muscles. Electromyography is the technique to acquire electrical muscle activity that is used for the pattern recognition technique of assistive devices. Regarding this, the different classification characterizations of EMG signals involved in the pick and place action, subjected to variable grip span and weights were considered in this study. A low-level force measuring gripper, capable to bear the changes in weights and object spans was designed and developed to simulate the task. The grip span varied from 6 cm to 9 cm and the maximum weight used in this study was 750 gms. The pattern recognition classification methodology was performed for the differentiation of phases of the pick and place activity, grip force, and the angular deviation of metacarpal phalangeal (MCP) joint. The classifiers used in this study were decision tree (DT), support vector machines (SVM) and k-nearest neighbor (k-NN) based on the feature sets of the EMG signals. After analyses, it was found that k-NN performed best to classify different phases of the activity and relative deviation of MCP joint with an average classification accuracy of 82% and 91% respectively. However; the SVM performed best in classification of force with a particular feature set. The findings of the study would be helpful in designing the assistive devices for hand amputee.

Effective recognition of human lower limb jump locomotion phases based on multi-sensor information fusion and machine learning

Jump locomotion is the basic movement of human. However, no thorough research on the recognition of jump sub-phases has been carried so far. This paper aims to use multi-sensor information fusion and machine learning to recognize the human jump phase, which is crucial to the development of exoskeleton that assists jumping. The method of information fusion for sensors including sEMG, IMU, and footswitch sensor is studied. The footswitch signals are filtered by median filter. A processing method of synthesizing Euler angles into phase angle is proposed, which is beneficial to data integration. The jump locomotion is creatively segmented into five phases. The onset and offset of active segment are detected by sample entropy of sEMG and standard deviation of acceleration signal. The features are extracted from analysis windows using multi-sensor information fusion, and the dimension of feature matrix is selected. By comparing the performances of state-of-the-art machine learning classifiers, feature subsets of sEMG, IMU, and footswitch signals are selected from time domain features in a series of analysis window parameters. The average recognition accuracy of sEMG and IMU is 91.76% and 97.68%, respectively. When using the combination of sEMG, IMU, and footswitch signals, the average accuracy is 98.70%, which outperforms the combination of sEMG and IMU (97.97%, p < 0.01). Graphical Abstract The sub-phases of human locomotion are recognized based on multi-sensor information fusion and machine learning method. The feature data of the sub-phases is visualized in 3-dimensional space. The predicted states and the true states in a complete jump are compared along the time axis.

Evaluation of windowing techniques for intramuscular EMG-based diagnostic, rehabilitative and assistive devices

Objective.Intramuscular electromyography (iEMG) signals, invasively recorded, directly from the muscles are used to diagnose various neuromuscular disorders/diseases and to control rehabilitative and assistive robotic devices. iEMG signals are potentially being used in neurology, kinesiology, rehabilitation and ergonomics, to detect/diagnose various diseases/disorders. Electromyography-based classification and analysis systems are being designed and tested for the classification of various neuromuscular disorders and to control rehabilitative and assistive robotic devices. Many studies have explored parameters such as the pre-processing, feature extraction and selection of classifiers that can affect the performance and efficacy of iEMG-based classification systems. The pre-processing stage includes the removal of any unwanted noise from the original signal and windowing of the signal.Approach.This study investigated and presented the optimum windowing configurations for robust control and better performance results of an iEMG-based analysis system based on the stationarity rate (SR) and classification accuracy. Both disjoint and overlap, windowing techniques with varying window and overlap sizes have been investigated using a machine learning-based classification algorithm called linear discriminant analysis.Main results.The optimum window size ranges are from 200-300 ms for the disjoint and 225-300 ms for the overlap windowing technique, respectively. The inferred results show that for the overlap windowing technique the optimum range of overlap size is from 10%-30% of the length of window size. The mean classification accuracy (MCA) and mean stationarity rate (MSR) were found to be lower in the disjoint windowing technique compared to overlap windowing at all investigated overlap sizes. Statistical analysis (two-way analysis of variance test) showed that the MSR and MCA of the overlap windowing technique was significantly different at overlap sizes of 10%-30% (p-values < 0.05).Significance.The presented results can be used to achieve the best possible classification results and SR for any iEMG-based real-time diagnosis, detection and control system, which can enhance the performance of the system significantly.

A Transferable Adaptive Domain Adversarial Neural Network for Virtual Reality Augmented EMG-Based Gesture Recognition

Within the field of electromyography-based (EMG) gesture recognition, disparities exist between the offline accuracy reported in the literature and the real-time usability of a classifier. This gap mainly stems from two factors: 1) The absence of a controller, making the data collected dissimilar to actual control. 2) The difficulty of including the four main dynamic factors (gesture intensity, limb position, electrode shift, and transient changes in the signal), as including their permutations drastically increases the amount of data to be recorded. Contrarily, online datasets are limited to the exact EMG-based controller used to record them, necessitating the recording of a new dataset for each control method or variant to be tested. Consequently, this paper proposes a new type of dataset to serve as an intermediate between offline and online datasets, by recording the data using a real-time experimental protocol. The protocol, performed in virtual reality, includes the four main dynamic factors and uses an EMG-independent controller to guide movements. This EMG-independent feedback ensures that the user is in-the-loop during recording, while enabling the resulting dynamic dataset to be used as an EMG-based benchmark. The dataset is comprised of 20 able-bodied participants completing three to four sessions over a period of 14 to 21 days. The ability of the dynamic dataset to serve as a benchmark is leveraged to evaluate the impact of different recalibration techniques for long-term (across-day) gesture recognition, including a novel algorithm, named TADANN. TADANN consistently and significantly ( [Formula: see text]) outperforms using fine-tuning as the recalibration technique.

Decoding Attempted Hand Movements in Stroke Patients Using Surface Electromyography

Brain- and muscle-triggered exoskeletons have been proposed as a means for motor training after a stroke. With the possibility of performing different movement types with an exoskeleton, it is possible to introduce task variability in training. It is difficult to decode different movement types simultaneously from brain activity, but it may be possible from residual muscle activity that many patients have or quickly regain. This study investigates whether nine different motion classes of the hand and forearm could be decoded from forearm EMG in 15 stroke patients. This study also evaluates the test-retest reliability of a classical, but simple, classifier (linear discriminant analysis) and advanced, but more computationally intensive, classifiers (autoencoders and convolutional neural networks). Moreover, the association between the level of motor impairment and classification accuracy was tested. Three channels of surface EMG were recorded during the following motion classes: Hand Close, Hand Open, Wrist Extension, Wrist Flexion, Supination, Pronation, Lateral Grasp, Pinch Grasp, and Rest. Six repetitions of each motion class were performed on two different days. Hudgins time-domain features were extracted and classified using linear discriminant analysis and autoencoders, and raw EMG was classified with convolutional neural networks. On average, 79 ± 12% and 80 ± 12% (autoencoders) of the movements were correctly classified for days 1 and 2, respectively, with an intraclass correlation coefficient of 0.88. No association was found between the level of motor impairment and classification accuracy (Spearman correlation: 0.24). It was shown that nine motion classes could be decoded from residual EMG, with autoencoders being the best classification approach, and that the results were reliable across days; this may have implications for the development of EMG-controlled exoskeletons for training in the patient’s home.

Selection of Features and Classifiers for EMG-EEG-Based Upper Limb Assistive Devices-A Review

Bio-signals are distinctive factors in the design of human-machine interface, essentially useful for prosthesis, orthosis, and exoskeletons. Despite the progress in the analysis of pattern recognition based devices; the acceptance of these devices is still questionable. One reason is the lack of information to identify the possible combinations of features and classifiers. Besides; there is also a need for optimal selection of various sensors for sensations such as touch, force, texture, along with EMGs/EEGs. This article reviews the two bio-signal techniques, named as electromyography and electroencephalography. The details of the features and the classifiers used in the data processing for upper limb assist devices are summarised here. Various features and their sets are surveyed and different classifiers for feature sets are discussed on the basis of the classification rate. The review was carried out on the basis of the last 10-12 years of published research in this area. This article also outlines the influence of modality of EMGs and EEGs with other sensors on classifications. Also, other bio-signals used in upper limb devices and future aspects are considered.