Bilinear modeling of EMG signals to extract user-independent features for multiuser myoelectric interface

Wearable high-density EMG sleeve for complex hand gesture classification and continuous joint angle estimation

High-density electromyography (HD-EMG) can provide a natural interface to enhance human-computer interaction (HCI). This study aims to demonstrate the capability of a novel HD-EMG forearm sleeve equipped with up to 150 electrodes to capture high-resolution muscle activity, decode complex hand gestures, and estimate continuous hand position via joint angle predictions. Ten able-bodied participants performed 37 hand movements and grasps while EMG was recorded using the HD-EMG sleeve. Simultaneously, an 18-sensor motion capture glove calculated 23 joint angles from the hand and fingers across all movements for training regression models. For classifying across the 37 gestures, our decoding algorithm was able to differentiate between sequential movements with 97.3 ± 0.3 % accuracy calculated on a 100 ms bin-by-bin basis. In a separate mixed dataset consisting of 19 movements randomly interspersed, decoding performance achieved an average bin-wise accuracy of 92.8 ± 0.8 % . When evaluating decoders for use in real-time scenarios, we found that decoders can reliably decode both movements and movement transitions, achieving an average accuracy of 93.3 ± 0.9 % on the sequential set and 88.5 ± 0.9 % on the mixed set. Furthermore, we estimated continuous joint angles from the EMG sleeve data, achieving a R 2 of 0.884 ± 0.003 in the sequential set and 0.750 ± 0.008 in the mixed set. Median absolute error (MAE) was kept below 10° across all joints, with a grand average MAE of 1.8 ± 0 . 04 ∘ and 3.4 ± 0 . 07 ∘ for the sequential and mixed datasets, respectively. We also assessed two algorithm modifications to address specific challenges for EMG-driven HCI applications. To minimize decoder latency, we used a method that accounts for reaction time by dynamically shifting cue labels in time. To reduce training requirements, we show that pretraining models with historical data provided an increase in decoding performance compared with models that were not pretrained when reducing the in-session training data to only one attempt of each movement. The HD-EMG sleeve, combined with sophisticated machine learning algorithms, can be a powerful tool for hand gesture recognition and joint angle estimation. This technology holds significant promise for applications in HCI, such as prosthetics, assistive technology, rehabilitation, and human-robot collaboration.

Cross-User Electromyography Pattern Recognition Based on a Novel Spatial-Temporal Graph Convolutional Network

With the goal of promoting the development of myoelectric control technology, this paper focuses on exploring graph neural network (GNN) based robust electromyography (EMG) pattern recognition solutions. Given that high-density surface EMG (HD-sEMG) signal contains rich temporal and spatial information, the multi-view spatial-temporal graph convolutional network (MSTGCN)is adopted as the basic classifier, and a feature extraction convolutional neural network (CNN) module is designed and integrated into MSTGCN to generate a new model called CNN-MSTGCN. The EMG pattern recognition experiments are conducted on HD-sEMG data of 17 gestures from 11 subjects. The ablation experiments show that each functional module of the proposed CNN-MSTGCN network has played a more or less positive role in improving the performance of EMG pattern recognition. The user-independent recognition experiments and the transfer learning-based cross-user recognition experiments verify the advantages of the proposed CNN-MSTGCN network in improving recognition rate and reducing user training burden. In the user-independent recognition experiments, CNN-MSTGCN achieves the recognition rate of 68%, which is significantly better than those obtained by residual network-50 (ResNet50, 47.5%, p < 0.001) and long-short-term-memory (LSTM, 57.1%, p=0.045). In the transfer learning-based cross-user recognition experiments, TL-CMSTGCN achieves an impressive recognition rate of 92.3%, which is significantly superior to both TL-ResNet50 (84.6%, p = 0.003) and TL-LSTM (85.3%, p = 0.008). The research results of this paper indicate that GNN has certain advantages in overcoming the impact of individual differences, and can be used to provide possible solutions for achieving robust EMG pattern recognition technology.

Cybersecurity in neural interfaces: Survey and future trends

With the joint advancement in areas such as pervasive neural data sensing, neural computing, neuromodulation and artificial intelligence, neural interface has become a promising technology facilitating both the closed-loop neurorehabilitation for neurologically impaired patients and the intelligent man-machine interactions for general application purposes. However, although neural interface has been widely studied, few previous studies focused on the cybersecurity issues in related applications. In this survey, we systematically investigated possible cybersecurity risks in neural interfaces, together with potential solutions to these problems. Importantly, our survey considers interfacing techniques on both central nervous systems (i.e., brain-computer interfaces) and peripheral nervous systems (i.e., general neural interfaces), covering diverse neural modalities such as electroencephalography, electromyography and more. Moreover, our survey is organized on three different levels: (1) the data level, which mainly focuses on the privacy leakage issue via attacking and analyzing neural database of users; (2) the permission level, which mainly focuses on the prospects and risks to directly use real time neural signals as biometrics for continuous and unobtrusive user identity verification; and (3) the model level, which mainly focuses on adversarial attacks and defenses on both the forward neural decoding models (e.g. via machine learning) and the backward feedback implementation models (e.g. via neuromodulation and stimulation). This is the first study to systematically investigate cybersecurity risks and possible solutions in neural interfaces which covers both central and peripheral nervous systems, and considers multiple different levels to provide a complete picture of this issue.

A Review of Myoelectric Control for Prosthetic Hand Manipulation

Myoelectric control for prosthetic hands is an important topic in the field of rehabilitation. Intuitive and intelligent myoelectric control can help amputees to regain upper limb function. However, current research efforts are primarily focused on developing rich myoelectric classifiers and biomimetic control methods, limiting prosthetic hand manipulation to simple grasping and releasing tasks, while rarely exploring complex daily tasks. In this article, we conduct a systematic review of recent achievements in two areas, namely, intention recognition research and control strategy research. Specifically, we focus on advanced methods for motion intention types, discrete motion classification, continuous motion estimation, unidirectional control, feedback control, and shared control. In addition, based on the above review, we analyze the challenges and opportunities for research directions of functionality-augmented prosthetic hands and user burden reduction, which can help overcome the limitations of current myoelectric control research and provide development prospects for future research.

Research on Lower Limb Step Speed Recognition Method Based on Electromyography

Wearable exoskeletons play an important role in people's lives, such as helping stroke and amputation patients to carry out rehabilitation training and so on. How to make the exoskeleton accurately judge the human action intention is the basic requirement to ensure that it can complete the corresponding task. Traditional exoskeleton control signals include pressure values, joint angles and acceleration values, which can only reflect the current motion information of the human lower limbs and cannot be used to predict motion. The electromyography (EMG) signal always occurs before a certain movement; it can be used to predict the target's gait speed and movement as the input signal. In this study, the generalization ability of a BP neural network and the timing property of a hidden Markov chain are used to properly fuse the two, and are finally used in the research of this paper. Experiments show that, using the same training samples, the recognition accuracy of the three-layer BP neural network is only 91%, while the recognition accuracy of the fusion discriminant model proposed in this paper can reach 95.1%. The results show that the fusion of BP neural network and hidden Markov chain has a strong solving ability for the task of wearable exoskeleton recognition of target step speed.

Multi-source domain generalization and adaptation toward cross-subject myoelectric pattern recognition

Objective.Myoelectric pattern recognition (MPR) has shown satisfactory performance under ideal laboratory conditions. Nevertheless, the individual variances lead to dramatic performance degradation in cross-user MPR applications. It is crucial to enable the myoelectric interface to adapt to multiple users' surface electromyography (sEMG) distributions in practical.Approach.Domain adaptation (DA) is a promising approach to tackle cross-user challenges due to its ability to diminish the divergence between individual users' EMG distributions and escalate model generalization performance. However, existing DA methods in sEMG control are based on single-source domain adaptation (SDA). SDA solely mixes multiple training users' data as a combined source domain and attempts to align with a novel user. This simple data mixing manner ignores the sEMG distribution variations between disparate training users, leading to an insufficient variance elimination and lower performance. To this end, this paper proposes a multi-source synchronize domain adaptation framework with both DA and domain generalization (DG) capability. This multi-source framework aligns each source user and the new user in individual feature spaces, which better transfers the knowledge of existing users to the new user. Moreover, we retain the source-combined data to preserve the effectiveness of SDA. The property was further confirmed by evaluating the performance of the proposed method on data from nine subjects performing six tasks.Main results.Experiment results prove that the proposed multi-source framework achieved both positive DG and DA performance in a cross-user classification manner.Significance.This work demonstrates the usability and feasibility of the proposed multi-source framework in cross-user myoelectric control.

Time–frequency feature transform suite for deep learning-based gesture recognition using sEMG signals

Recently, deep learning methods have achieved considerable performance in gesture recognition using surface electromyography signals. However, improving the recognition accuracy in multi-subject gesture recognition remains a challenging problem. In this study, we aimed to improve recognition performance by adding subject-specific prior knowledge to provide guidance for multi-subject gesture recognition. We proposed a time–frequency feature transform suite (TFFT) that takes the maps generated by continuous wavelet transform (CWT) as input. The TFFT can be connected to a neural network to obtain an end-to-end architecture. Thus, we integrated the suite into traditional neural networks, such as convolutional neural networks and long short-term memory, to adjust the intermediate features. The results of comparative experiments showed that the deep learning models with the TFFT suite based on CWT improved the recognition performance of the original architectures without the TFFT suite in gesture recognition tasks. Our proposed TFFT suite has promising applications in multi-subject gesture recognition and prosthetic control.

Multi-user motion recognition using sEMG via discriminative canonical correlation analysis and adaptive dimensionality reduction

The inability of new users to adapt quickly to the surface electromyography (sEMG) interface has greatly hindered the development of sEMG in the field of rehabilitation. This is due mainly to the large differences in sEMG signals produced by muscles when different people perform the same motion. To address this issue, a multi-user sEMG framework is proposed, using discriminative canonical correlation analysis and adaptive dimensionality reduction (ADR). The interface projects the feature sets for training users and new users into a low-dimensional uniform style space, overcoming the problem of individual differences in sEMG. The ADR method removes the redundant information in sEMG features and improves the accuracy of system motion recognition. The presented framework was validated on eight subjects with intact limbs, with an average recognition accuracy of 92.23% in 12 categories of upper-limb movements. In rehabilitation laboratory experiments, the average recognition rate reached 90.52%. The experimental results suggest that the framework offers a good solution to enable new rehabilitation users to adapt quickly to the sEMG interface.

Natural grasping movement recognition and force estimation using electromyography

Electromyography (EMG) generated by human hand movements is usually used to decode different action types with high accuracy. However, the classifications of the gestures rarely consider the impact of force, and the estimation of the grasp force when performing natural grasping movements is so far overlooked. Decoding natural grasping movements and estimating the force generated by the associated movements can help patients to improve the accuracy of prosthesis control. This study mainly focused on two aspects: the classification of four natural grasping movements and the force estimation of these actions. For this purpose, we designed an experimental platform where subjects could perform four common natural grasping movements in daily life, including pinch, palmar, twist, and plug grasp, to complete target profiles. On the one hand, the results showed that, for natural grasping movements with different levels of force (three levels at 20, 50, and 80%), the average accuracy could reach from 91.43 to 97.33% under five classification schemes. On the other hand, the feasibility of force estimation for natural grasping movements was demonstrated. Furthermore, in the process of force estimation, we confirmed that the regression performance about plug grasp was the best, and the average R2 could reach 0.9082. Besides, we found that the regression results were affected by the speed of force application. These findings contribute to the natural control of myoelectric prosthesis and the EMG-based rehabilitation training system, improving the user’s experience and acceptance.

Domain Adaptation with Self-Guided Adaptive Sampling Strategy: Feature Alignment for Cross-User Myoelectric Pattern Recognition

Gestural interfaces based on surface electromyo-graphic (sEMG) signal have been widely explored. Nevertheless, due to the individual differences in the sEMG signals, it is very challenging for a myoelectric pattern recognition control system to adapt cross-user variability. Unsupervised domain adaptation (UDA) has achieved unprecedented success in improving the cross-domain robustness, and it is a promising approach to solve the cross-user challenge. Existing UDA methods largely ignore the instantaneous data distribution during model updating, thus deteriorating the feature representation given a large domain shift. To address this issue, a novel method is proposed based on a UDA model incorporated with a self-guided adaptive sampling (SGAS) strategy. This strategy is designed to utilize the domain distance in a kernel space as an indicator to screen out reliable instantaneous samples for updating the classifier. Thus, it enables improved alignment of feature representations of myoelectric patterns across users. To evaluate the performance of the proposed method, sEMG data were recorded from forearm muscles of nine subjects performing six finger and wrist gestures. Experiment results show that the UDA method with the SGAS strategy achieved a mean accuracy of 90.41% ± 14.44% in a cross-user classification manner, outperformed the state-of-the-art methods with statistical significance. This study demonstrates the effectiveness of the proposed UDA framework and offers a novel tool for implementing cross-user myoelectric pattern recognition towards a multi-user and user-independent control.

Review on electromyography based intention for upper limb control using pattern recognition for human-machine interaction

Upper limb myoelectric prosthetic control is an essential topic in the field of rehabilitation. The technique controls prostheses using surface electromyogram (sEMG) and intramuscular EMG (iEMG) signals. EMG signals are extensively used in controlling prosthetic upper and lower limbs, virtual reality entertainment, and human-machine interface (HMI). EMG signals are vital parameters for machine learning and deep learning algorithms and help to give an insight into the human brain’s function and mechanisms. Pattern recognition techniques pertaining to support vector machine (SVM), k-nearest neighbor (KNN) and Bayesian classifiers have been utilized to classify EMG signals. This paper presents a review on current EMG signal techniques, including electrode array utilization, signal acquisition, signal preprocessing and post-processing, feature selection and extraction, data dimensionality reduction, classification, and ultimate application to the community. The paper also discusses using alternatives to EMG signals, such as force sensors, to measure muscle activity with reliable results. Future implications for EMG classification include employing deep learning techniques such as artificial neural networks (ANN) and recurrent neural networks (RNN) for achieving robust results.

Towards Robust, Adaptive and Reliable Upper-limb Motion Estimation Using Machine Learning and Deep Learning--A Survey in Myoelectric Control

To develop multi-functional human-machine interfaces that can help disabled people reconstruct lost functions of upper-limbs, machine learning (ML) and deep learning (DL) techniques have been widely implemented to decode human movement intentions from surface electromyography (sEMG) signals. However, due to the high complexity of upper-limb movements and the inherent non-stable characteristics of sEMG, the usability of ML/DL based control schemes is still greatly limited in practical scenarios. To this end, tremendous efforts have been made to improve model robustness, adaptation, and reliability. In this article, we provide a systematic review on recent achievements, mainly from three categories: multi-modal sensing fusion to gain additional information of the user, transfer learning (TL) methods to eliminate domain shift impacts on estimation models, and post-processing approaches to obtain more reliable outcomes. Special attention is given to fusion strategies, deep TL frameworks, and confidence estimation. \textcolor{red}{Research challenges and emerging opportunities, with respect to hardware development, public resources, and decoding strategies, are also analysed to provide perspectives for future developments.

User-Independent Hand Gesture Recognition Classification Models Using Sensor Fusion

Recently, it has been proven that targeting motor impairments as early as possible while using wearable mechatronic devices for assisted therapy can improve rehabilitation outcomes. However, despite the advanced progress on control methods for wearable mechatronic devices, the need for a more natural interface that allows for better control remains. To address this issue, electromyography (EMG)-based gesture recognition systems have been studied as a potential solution for human–machine interface applications. Recent studies have focused on developing user-independent gesture recognition interfaces to reduce calibration times for new users. Unfortunately, given the stochastic nature of EMG signals, the performance of these interfaces is negatively impacted. To address this issue, this work presents a user-independent gesture classification method based on a sensor fusion technique that combines EMG data and inertial measurement unit (IMU) data. The Myo Armband was used to measure muscle activity and motion data from healthy subjects. Participants were asked to perform seven types of gestures in four different arm positions while using the Myo on their dominant limb. Data obtained from 22 participants were used to classify the gestures using three different classification methods. Overall, average classification accuracies in the range of 67.5–84.6% were obtained, with the Adaptive Least-Squares Support Vector Machine model obtaining accuracies as high as 92.9%. These results suggest that by using the proposed sensor fusion approach, it is possible to achieve a more natural interface that allows better control of wearable mechatronic devices during robot assisted therapies.

How adaptation, training, and customization contribute to benefits from exoskeleton assistance

Exoskeletons can enhance human mobility, but we still know little about why they are effective. For example, we do not know the relative importance of training, how much is required, or what type is most effective; how people adapt with the device; or the relative benefits of customizing assistance. We conducted experiments in which naïve users learned to walk with ankle exoskeletons under one of three training regimens characterized by different levels of variation in device behavior. Assistance was also customized for one group. After moderate-variation training, the benefits of customized assistance were large; metabolic rate was reduced by 39% compared with walking with the exoskeleton turned off. Training contributed about half of this benefit and customization about one-quarter; a generic controller reduced energy cost by 10% before training and 31% afterward. Training required much more exposure than typical of exoskeleton studies, about 109 minutes of assisted walking. Type of training also had a strong effect; the low-variation group required twice as long as the moderate-variation group to become expert, and the high-variation group never acquired this level of expertise. Curiously, all users adapted in a way that resulted in less mechanical power from the exoskeleton as they gained expertise. Customizing assistance required less time than training for all parameters except peak torque magnitude, which grew slowly over the study, suggesting a longer time scale adaptation in the person. These results underscore the importance of training to the benefits of exoskeleton assistance and suggest the topic deserves more attention.

Dynamic Hand Gesture Recognition in In-Vehicle Environment Based on FMCW Radar and Transformer

Hand gesture recognition technology plays an important role in human-computer interaction and in-vehicle entertainment. Under in-vehicle conditions, it is a great challenge to design gesture recognition systems due to variable driving conditions, complex backgrounds, and diversified gestures. In this paper, we propose a gesture recognition system based on frequency-modulated continuous-wave (FMCW) radar and transformer for an in-vehicle environment. Firstly, the original range-Doppler maps (RDMs), range-azimuth maps (RAMs), and range-elevation maps (REMs) of the time sequence of each gesture are obtained by radar signal processing. Then we preprocess the obtained data frames by region of interest (ROI) extraction, vibration removal algorithm, background removal algorithm, and standardization. We propose a transformer-based radar gesture recognition network named RGTNet. It fully extracts and fuses the spatial-temporal information of radar feature maps to complete the classification of various gestures. The experimental results show that our method can better complete the eight gesture classification tasks in the in-vehicle environment. The recognition accuracy is 97.56%.

Performance enhancement of facial electromyogram-based facial-expression recognition for social virtual reality applications using linear discriminant analysis adaptation

Recent studies have indicated that facial electromyogram (fEMG)-based facial-expression recognition (FER) systems are promising alternatives to the conventional camera-based FER systems for virtual reality (VR) environments because they are economical, do not depend on the ambient lighting, and can be readily incorporated into existing VR headsets. In our previous study, we applied a Riemannian manifold-based feature extraction approach to fEMG signals recorded around the eyes and demonstrated that 11 facial expressions could be classified with a high accuracy of 85.01%, with only a single training session. However, the performance of the conventional fEMG-based FER system was not high enough to be applied in practical scenarios. In this study, we developed a new method for improving the FER performance by employing linear discriminant analysis (LDA) adaptation with labeled datasets of other users. Our results indicated that the mean classification accuracy could be increased to 89.40% by using the LDA adaptation method (p < .001, Wilcoxon signed-rank test). Additionally, we demonstrated the potential of a user-independent FER system that could classify 11 facial expressions with a classification accuracy of 82.02% without any training sessions. To the best of our knowledge, this was the first study in which the LDA adaptation approach was employed in a cross-subject manner. It is expected that the proposed LDA adaptation approach would be used as an important method to increase the usability of fEMG-based FER systems for social VR applications.

Score, Rank, and Decision-Level Fusion Strategies of Multicode Electromyogram-based Verification and Identification Biometrics

Recent advances in biometric research have established surface electromyogram (sEMG) as a potential spoof-free solution to address some key limitations in current biometric traits. The nature of sEMG signals provide a unique dual-mode security: sEMGs have individual-specific characteristics (biometrics), and users can customize and change gestures just like passcodes. Such security also facilitates the use of code sequences (multicode) to further enhance the security. In this study, three levels of fusion, score, rank, and decision were investigated for two biometric applications, verification and identification. This study involved 24 subjects performing 16 hand/finger gestures, and code sequences with varying codelengths were generated. The performance of the verification and identification system was analyzed for varying codelength (M: 16) and rank (K: 14) to determine the best fusion scheme and desirable parameter values for a multicode sEMG biometric system. The results showed that the decision-level fusion scheme using a weighted majority voting resulted in an average equal error rate of 0.6% for the verification system when M=4. For the identification system, the score-level fusion scheme with score normalization based on fitting a Weibull distribution resulted in a minimum false rejection rate of 0.01% and false acceptance rate of 4.7% using a combination of K=2 and M=4. The results also suggested that the parameters M and K could be adjusted based on the number of users in the database to facilitate optimal performance. In summary, a multicode sEMG biometric system was developed to provide improved dual-mode security based on the personalized codes and biometric traits of individuals, with the combination of enhanced security and flexibility.

A Dual-Modal Approach Using Electromyography and Sonomyography Improves Prediction of Dynamic Ankle Movement: A Case Study

For decades, surface electromyography (sEMG) has been a popular non-invasive bio-sensing technology for predicting human joint motion. However, cross-talk, interference from adjacent muscles, and its inability to measure deeply located muscles limit its performance in predicting joint motion. Recently, ultrasound (US) imaging has been proposed as an alternative non-invasive technology to predict joint movement due to its high signal-to-noise ratio, direct visualization of targeted tissue, and ability to access deep-seated muscles. This paper proposes a dual-modal approach that combines US imaging and sEMG for predicting volitional dynamic ankle dorsiflexion movement. Three feature sets: 1) a uni-modal set with four sEMG features, 2) a uni-modal set with four US imaging features, and 3) a dual-modal set with four dominant sEMG and US imaging features, together with measured ankle dorsiflexion angles, were used to train multiple machine learning regression models. The experimental results from a seated posture and five walking trials at different speeds, ranging from 0.50 m/s to 1.50 m/s, showed that the dual-modal set significantly reduced the prediction root mean square errors (RMSEs). Compared to the uni-modal sEMG feature set, the dual-modal set reduced RMSEs by up to 47.84% for the seated posture and up to 77.72% for the walking trials. Similarly, when compared to the US imaging feature set, the dual-modal set reduced RMSEs by up to 53.95% for the seated posture and up to 58.39% for the walking trials. The findings show that potentially the dual-modal sensing approach can be used as a superior sensing modality to predict human intent of a continuous motion and implemented for volitional control of clinical rehabilitative and assistive devices.

Fusion Learning for sEMG Recognition of Multiple Upper-Limb Rehabilitation Movements

Surface electromyogram (sEMG) signals have been used in human motion intention recognition, which has significant application prospects in the fields of rehabilitation medicine and cognitive science. However, some valuable dynamic information on upper-limb motions is lost in the process of feature extraction for sEMG signals, and there exists the fact that only a small variety of rehabilitation movements can be distinguished, and the classification accuracy is easily affected. To solve these dilemmas, first, a multiscale time-frequency information fusion representation method (MTFIFR) is proposed to obtain the time-frequency features of multichannel sEMG signals. Then, this paper designs the multiple feature fusion network (MFFN), which aims at strengthening the ability of feature extraction. Finally, a deep belief network (DBN) was introduced as the classification model of the MFFN to boost the generalization performance for more types of upper-limb movements. In the experiments, 12 kinds of upper-limb rehabilitation actions were recognized utilizing four sEMG sensors. The maximum identification accuracy was 86.10% and the average classification accuracy of the proposed MFFN was 73.49%, indicating that the time-frequency representation approach combined with the MFFN is superior to the traditional machine learning and convolutional neural network.

Inter-Subject Domain Adaptation for CNN-Based Wrist Kinematics Estimation Using sEMG

Recently, convolutional neural network (CNN) has been widely investigated to decode human intentions using surface Electromyography (sEMG) signals. However, a pre-trained CNN model usually suffers from severe degradation when testing on a new individual, and this is mainly due to domain shift where characteristics of training and testing sEMG data differ substantially. To enhance inter-subject performances of CNN in the wrist kinematics estimation, we propose a novel regression scheme for supervised domain adaptation (SDA), based on which domain shift effects can be effectively reduced. Specifically, a two-stream CNN with shared weights is established to exploit source and target sEMG data simultaneously, such that domain-invariant features can be extracted. To tune CNN weights, both regression losses and a domain discrepancy loss are employed, where the former enable supervised learning and the latter minimizes distribution divergences between two domains. In this study, eight healthy subjects were recruited to perform wrist flexion-extension movements. Experiment results illustrated that the proposed regression SDA outperformed fine-tuning, a state-of-the-art transfer learning method, in both single-single and multiple-single scenarios of kinematics estimation. Unlike fine-tuning which suffers from catastrophic forgetting, regression SDA can maintain much better performances in original domains, which boosts the model reusability among multiple subjects.

Deep Cross-User Models Reduce the Training Burden in Myoelectric Control

The effort, focus, and time to collect data and train EMG pattern recognition systems is one of the largest barriers to their widespread adoption in commercial applications. In addition to multiple repetitions of motions, including exemplars of confounding factors during the training protocol has been shown to be critical for robust machine learning models. This added training burden is prohibitive for most regular use cases, so cross-user models have been proposed that could leverage inter-repetition variability supplied by other users. Existing cross-user models have not yet achieved performance levels sufficient for commercialization and require users to closely adhere to a training protocol that is impractical without expert guidance. In this work, we extend a previously reported adaptive domain adversarial neural network (ADANN) to a cross-subject framework that requires very little training data from the end-user. We compare its performance to single-repetition within-user training and the previous state-of-the-art cross-subject technique, canonical correlation analysis (CCA). ADANN significantly outperformed CCA for both intact-limb (86.8–96.2%) and amputee (64.1–84.2%) populations. Moreover, the ADANN adaptation computation time was substantially lower than the time otherwise devoted to conducting a full within-subject training protocol. This study shows that cross-user models, enabled by deep-learned adaptations, may be a viable option for improved generalized pattern recognition-based myoelectric control.

Force-Invariant Improved Feature Extraction Method for Upper-Limb Prostheses of Transradial Amputees

A force-invariant feature extraction method derives identical information for all force levels. However, the physiology of muscles makes it hard to extract this unique information. In this context, we propose an improved force-invariant feature extraction method based on nonlinear transformation of the power spectral moments, changes in amplitude, and the signal amplitude along with spatial correlation coefficients between channels. Nonlinear transformation balances the forces and increases the margin among the gestures. Additionally, the correlation coefficient between channels evaluates the amount of spatial correlation; however, it does not evaluate the strength of the electromyogram signal. To evaluate the robustness of the proposed method, we use the electromyogram dataset containing nine transradial amputees. In this study, the performance is evaluated using three classifiers with six existing feature extraction methods. The proposed feature extraction method yields a higher pattern recognition performance, and significant improvements in accuracy, sensitivity, specificity, precision, and F1 score are found. In addition, the proposed method requires comparatively less computational time and memory, which makes it more robust than other well-known feature extraction methods.

Cancelable HD-sEMG-Based Biometrics for Cross-Application Discrepant Personal Identification

With the soaring development of body sensor network (BSN)-based health informatics, information security in such medical devices has attracted increasing attention in recent years. Employing the biosignals acquired directly by the BSN as biometrics for personal identification is an effective approach. Noncancelability and cross-application invariance are two natural flaws of most traditional biometric modalities. Once the biometric template is exposed, it is compromised forever. Even worse, because the same biometrics may be employed as tokens for different accounts in multiple applications, the exposed template can be used to compromise other accounts. In this work, we propose a cancelable and cross-application discrepant biometric approach based on high-density surface electromyogram (HD-sEMG) for personal identification. We enrolled two accounts for each user. HD-sEMG signals from the right dorsal hand under isometric contractions of different finger muscles were employed as biometric tokens. Since isometric contraction, in contrast to dynamic contraction, requires no actual movement, the users' choice to login to different accounts is greatly protected against impostors. We realized a promising identification accuracy of 85.8% for 44 identities (22 subjects × 2 accounts) with training and testing data acquired 9 days apart. The high identification accuracy of different accounts for the same user demonstrates the promising cancelability and cross-application discrepancy of the proposed HD-sEMG-based biometrics. To the best of our knowledge, this is the first study to employ HD-sEMG in personal identification applications, with signal variation across days considered.

Generalized Finger Motion Classification Model Based on Motor Unit Voting

Surface electromyogram-based finger motion classification has shown its potential for prosthetic control. However, most current finger motion classification models are subject-specific, requiring calibration when applied to new subjects. Generalized subject-nonspecific models are essential for real-world applications. In this study, the authors developed a subject-nonspecific model based on motor unit (MU) voting. A high-density surface electromyogram was first decomposed into individual MUs. The features extracted from each MU were then fed into a random forest classifier to obtain the finger label (primary prediction). The final prediction was selected by voting for all primary predictions provided by the decomposed MUs. Experiments conducted on 14 subjects demonstrated that our method significantly outperformed traditional methods in the context of subject-nonspecific finger motion classification models.

Development of Sign Language Motion Recognition System for Hearing-Impaired People Using Electromyography Signal

Sign languages are developed around the world for hearing-impaired people to communicate with others who understand them. Different grammar and alphabets limit the usage of sign languages between different sign language users. Furthermore, training is required for hearing-intact people to communicate with them. Therefore, in this paper, a real-time motion recognition system based on an electromyography signal is proposed for recognizing actual American Sign Language (ASL) hand motions for helping hearing-impaired people communicate with others and training normal people to understand the sign languages. A bilinear model is applied to deal with the on electromyography (EMG) data for decreasing the individual difference among different people. A long short-term memory neural network is used in this paper as the classifier. Twenty sign language motions in the ASL library are selected for recognition in order to increase the practicability of the system. The results indicate that this system can recognize these twenty motions with high accuracy among twenty participants. Therefore, this system has the potential to be widely applied to help hearing-impaired people for daily communication and normal people to understand the sign languages.

Current Trends and Confounding Factors in Myoelectric Control: Limb Position and Contraction Intensity

This manuscript presents a hybrid study of a comprehensive review and a systematic(research) analysis. Myoelectric control is the cornerstone ofmany assistive technologies used in clinicalpractice, such as prosthetics and orthoses, and human-computer interaction, such as virtual reality control.Although the classification accuracy of such devices exceeds 90% in a controlled laboratory setting,myoelectric devices still face challenges in robustness to variability of daily living conditions.The intrinsic physiological mechanisms limiting practical implementations of myoelectric deviceswere explored: the limb position effect and the contraction intensity effect. The degradationof electromyography (EMG) pattern recognition in the presence of these factors was demonstratedon six datasets, where classification performance was 13% and 20% lower than the controlledsetting for the limb position and contraction intensity effect, respectively. The experimental designsof limb position and contraction intensity literature were surveyed. Current state-of-the-art trainingstrategies and robust algorithms for both effects were compiled and presented. Recommendationsfor future limb position effect studies include: the collection protocol providing exemplars of at least 6positions (four limb positions and three forearm orientations), three-dimensional space experimentaldesigns, transfer learning approaches, and multi-modal sensor configurations. Recommendationsfor future contraction intensity effect studies include: the collection of dynamic contractions, nonlinearcomplexity features, and proportional control.

Biometric From Surface Electromyogram (sEMG): Feasibility of User Verification and Identification Based on Gesture Recognition

Electrical biosignals are favored as biometric traits due to their hidden nature and allowing for liveness detection. This study explored the feasibility of surface electromyogram (sEMG), the electrical manifestation of muscle activities, as a biometric trait. The accurate gesture recognition from sEMG provided a unique advantage over two traditional electrical biosignal traits, electrocardiogram (ECG), and electroencephalogram (EEG), enabling users to customize their own gesture codes. The performance of 16 static wrist and hand gestures was systematically investigated in two identity management modes: verification and identification. The results showed that for a single fixed gesture, using only 0.8-second data, the averaged equal error rate (EER) for verification was 3.5%, and the averaged rank-1 for identification was 90.3%, both comparable to the reported performance of ECG and EEG. The function of customizing gesture code could further improve the verification performance to 1.1% EER. This work demonstrated the potential and effectiveness of sEMG as a biometric trait in user verification and identification, beneficial for the design of future biometric systems.

Training-Free Bayesian Self-Adaptive Classification for sEMG Pattern Recognition Including Motion Transition

A direct, ready-to-use surface electromyogram (sEMG) pattern classification algorithm that does not require prerequisite training, regardless of the user, is proposed herein. In addition to data collection, conventional supervised learning approaches for sEMG require labeling and segmenting the data and additional time for the learning algorithm. Consequently, these approaches cannot cope well with sEMG patterns during motion transitions of various movement speeds. The proposed unsupervised and self-adaptive method employs an iterative self-adaptive procedure realized by the probabilistic methods of diffusion, updating, and registration to cluster the activation patterns simultaneously in real time, and classify the current sEMG as new clustered patterns. Experiments demonstrated that even for the same motion, the proposed method could autonomously detect changes in muscular activation patterns varying with the speed of motion. Furthermore, some patterns of both steady- and transient-state motions could be distinguished. In addition, it was verified that the classified sEMG pattern could be correlated consistently with the actual motion, thereby realizing a high level of motion classification.

A Subject-Transfer Framework Based on Single-Trial EMG Analysis Using Convolutional Neural Networks

In recent years, electromyography (EMG)-based practical myoelectric interfaces have been developed to improve the quality of daily life for people with physical disabilities. With these interfaces, it is very important to decode a user's movement intention, to properly control the external devices. However, improving the performance of these interfaces is difficult due to the high variations in the EMG signal patterns caused by intra-user variability. Therefore, this paper proposes a novel subject-transfer framework for decoding hand movements, which is robust in terms of intra-user variability. In the proposed framework, supportive convolutional neural network (CNN) classifiers, which are pre-trained using the EMG data of several subjects, are selected and fine-tuned for the target subject via single-trial analysis. Then, the target subject's hand movements are classified by voting the outputs of the supportive CNN classifiers. The feasibility of the proposed framework is validated with NinaPro databases 2 and 3, which comprise 49 hand movements of 40 healthy and 11 amputee subjects, respectively. The experimental results indicate that, when compared to the self-decoding framework, which uses only the target subject's data, the proposed framework can successfully decode hand movements with improved performance in both healthy and amputee subjects. From the experimental results, the proposed subject-transfer framework can be seen to represent a useful tool for EMG-based practical myoelectric interfaces controlling external devices.

Design of User-Independent Hand Gesture Recognition Using Multilayer Perceptron Networks and Sensor Fusion Techniques

According to the World Health Organization, stroke is the third leading cause of disability. A common consequence of stroke is hemiparesis, which leads to the impairment of one side of the body and affects the performance of activities of daily living. It has been proven that targeting the motor impairments as early as possible while using wearable mechatronic devices as a robot assisted therapy, and letting the patient be in control of the robotic system, can improve the rehabilitation outcomes. However, despite the increased progress on control methods for wearable mechatronic devices, a need for a more natural interface that allows for better control remains. In this work, a user-independent gesture classification method based on a sensor fusion technique using surface electromyography (EMG) and an inertial measurement unit (IMU) is presented. The Myo Armband was used to extract EMG and IMU data from healthy subjects. Participants were asked to perform 10 types of gestures in 4 different arm positions while using the Myo on their dominant limb. Data obtained from 14 participants were used to classify the gestures using a Multilayer Perceptron Network. Finally, the classification algorithm was tested on 5 novel users, obtaining an average accuracy of 78.94%. These results demonstrate that by using the proposed approach, it is possible to achieve a more natural human machine interface that allows better control of wearable mechatronic devices during robot assisted therapies.

Classification and regression of spatio-temporal signals using NeuCube and its realization on SpiNNaker neuromorphic hardware

OBJECTIVE

The objective of this work is to use the capability of spiking neural networks to capture the spatio-temporal information encoded in time-series signals and decode them without the use of hand-crafted features and vector-based learning and the realization of the spiking model on low-power neuromorphic hardware.

APPROACH

The NeuCube spiking model was used to classify different grasp movements directly from raw surface electromyography signals (sEMG), the estimations of the applied finger forces as well as the classification of two motor imagery movements from raw electroencephalography (EEG). In a parallel investigation, the designed spiking decoder was implemented on SpiNNaker neuromorphic hardware, which allows low-energy real-time processing.

MAIN RESULTS

Experimental results reveal a better classification accuracy using the NeuCube model compared to traditional machine learning methods. For sEMG classification, we reached a training accuracy of 85% and a test accuracy of 84.8%, as well as less than 19% of relative root mean square error (rRMSE) when estimating finger forces from six subjects. For the EEG classification, a mean accuracy of 75% was obtained when tested on raw EEG data from nine subjects from the existing 2b dataset from 'BCI competition IV'.

SIGNIFICANCE

This work provides a proof of concept for a successful implementation of the NeuCube spiking model on the SpiNNaker neuromorphic platform for raw sEMG and EEG decoding, which could chart a route ahead for a new generation of portable closed-loop and low-power neuroprostheses.

Causes of Performance Degradation in Non-invasive Electromyographic Pattern Recognition in Upper Limb Prostheses

Surface Electromyography (EMG)-based pattern recognition methods have been investigated over the past years as a means of controlling upper limb prostheses. Despite the very good reported performance of myoelectric controlled prosthetic hands in lab conditions, real-time performance in everyday life conditions is not as robust and reliable, explaining the limited clinical use of pattern recognition control. The main reason behind the instability of myoelectric pattern recognition control is that EMG signals are non-stationary in real-life environments and present a lot of variability over time and across subjects, hence affecting the system's performance. This can be the result of one or many combined changes, such as muscle fatigue, electrode displacement, difference in arm posture, user adaptation on the device over time and inter-subject singularity. In this paper an extensive literature review is performed to present the causes of the drift of EMG signals, ways of detecting them and possible techniques to counteract for their effects in the application of upper limb prostheses. The suggested techniques are organized in a table that can be used to recognize possible problems in the clinical application of EMG-based pattern recognition methods for upper limb prosthesis applications and state-of-the-art methods to deal with such problems.

Navigating features: a topologically informed chart of electromyographic features space

The success of biological signal pattern recognition depends crucially on the selection of relevant features. Across signal and imaging modalities, a large number of features have been proposed, leading to feature redundancy and the need for optimal feature set identification. A further complication is that, due to the inherent biological variability, even the same classification problem on different datasets can display variations in the respective optimal sets, casting doubts on the generalizability of relevant features. Here, we approach this problem by leveraging topological tools to create charts of features spaces. These charts highlight feature sub-groups that encode similar information (and their respective similarities) allowing for a principled and interpretable choice of features for classification and analysis. Using multiple electromyographic (EMG) datasets as a case study, we use this feature chart to identify functional groups among 58 state-of-the-art EMG features, and to show that they generalize across three different forearm EMG datasets obtained from able-bodied subjects during hand and finger contractions. We find that these groups describe meaningful non-redundant information, succinctly recapitulating information about different regions of feature space. We then recommend representative features from each group based on maximum class separability, robustness and minimum complexity.

Deciphering the functional role of spatial and temporal muscle synergies in whole-body movements

Voluntary movement is hypothesized to rely on a limited number of muscle synergies, the recruitment of which translates task goals into effective muscle activity. In this study, we investigated how to analytically characterize the functional role of different types of muscle synergies in task performance. To this end, we recorded a comprehensive dataset of muscle activity during a variety of whole-body pointing movements. We decomposed the electromyographic (EMG) signals using a space-by-time modularity model which encompasses the main types of synergies. We then used a task decoding and information theoretic analysis to probe the role of each synergy by mapping it to specific task features. We found that the temporal and spatial aspects of the movements were encoded by different temporal and spatial muscle synergies, respectively, consistent with the intuition that there should a correspondence between major attributes of movement and major features of synergies. This approach led to the development of a novel computational method for comparing muscle synergies from different participants according to their functional role. This functional similarity analysis yielded a small set of temporal and spatial synergies that describes the main features of whole-body reaching movements.

Latent Factors Limiting the Performance of sEMG-Interfaces

Recent advances in recording and real-time analysis of surface electromyographic signals (sEMG) have fostered the use of sEMG human-machine interfaces for controlling personal computers, prostheses of upper limbs, and exoskeletons among others. Despite a relatively high mean performance, sEMG-interfaces still exhibit strong variance in the fidelity of gesture recognition among different users. Here, we systematically study the latent factors determining the performance of sEMG-interfaces in synthetic tests and in an arcade game. We show that the degree of muscle cooperation and the amount of the body fatty tissue are the decisive factors in synthetic tests. Our data suggest that these factors can only be adjusted by long-term training, which promotes fine-tuning of low-level neural circuits driving the muscles. Short-term training has no effect on synthetic tests, but significantly increases the game scoring. This implies that it works at a higher decision-making level, not relevant for synthetic gestures. We propose a procedure that enables quantification of the gestures' fidelity in a dynamic gaming environment. For each individual subject, the approach allows identifying "problematic" gestures that decrease gaming performance. This information can be used for optimizing the training strategy and for adapting the signal processing algorithms to individual users, which could be a way for a qualitative leap in the development of future sEMG-interfaces.

A comparison of open-loop and closed-loop adaptive calibration for pattern recognition based myoelectric control

This study presented a closed-loop adaptive calibration (CLAC) scheme where subjects could get instantaneous feedback of their movements and alter their motions immediately to update the model parameters to enhance its ability. The real-time performance was compared between the conventional open-loop calibration (OLC) and the presented CLAC based on three metrics (motion-selection time, motion-completion time and motion-completion rate). The CLAC performed slightly better than the OLC, but the difference was not significant. This was the first study designed to investigate the effects of CLAC for pattern recognition-based myoelectric control (discrete movement). The CLAC could be potentially applied in the multiuser interface to make the adaptation of the common model to a novel user efficiently and flexibly.

Real-time muscle state estimation from EMG signals during isometric contractions using Kalman filters

State-space control of myoelectric devices and real-time visualization of muscle forces in virtual rehabilitation require measuring or estimating muscle dynamic states: neuromuscular activation, tendon force and muscle length. This paper investigates whether regular (KF) and extended Kalman filters (eKF), derived directly from Hill-type muscle mechanics equations, can be used as real-time muscle state estimators for isometric contractions using raw electromyography signals (EMG) as the only available measurement. The estimators' amplitude error, computational cost, filtering lags and smoothness are compared with usual EMG-driven analysis, performed offline, by integrating the nonlinear Hill-type muscle model differential equations (offline simulations-OS). EMG activity of the three triceps surae components (soleus, gastrocnemius medialis and gastrocnemius lateralis), in three torque levels, was collected for ten subjects. The actualization interval (AI) between two updates of the KF and eKF was also varied. The results show that computational costs are significantly reduced (70x for KF and 17[Formula: see text] for eKF). The filtering lags presented sharp linear relationships with the AI (0-300 ms), depending on the state and activation level. Under maximum excitation, amplitude errors varied in the range 10-24% for activation, 5-8% for tendon force and 1.4-1.8% for muscle length, reducing linearly with the excitation level. Smoothness, measured by the ratio between the average standard variations of KF/eKF and OS estimations, was greatly reduced for activation but converged exponentially to 1 for the other states by increasing AI. Compared to regular KF, extended KF does not seem to improve estimation accuracy significantly. Depending on the particular application requirements, the most appropriate KF actualization interval can be selected.

Improved prosthetic hand control with concurrent use of myoelectric and inertial measurements

Background

Myoelectric pattern recognition systems can decode movement intention to drive upper-limb prostheses. Despite recent advances in academic research, the commercial adoption of such systems remains low. This limitation is mainly due to the lack of classification robustness and a simultaneous requirement for a large number of electromyogram (EMG) electrodes. We propose to address these two issues by using a multi-modal approach which combines surface electromyography (sEMG) with inertial measurements (IMs) and an appropriate training data collection paradigm. We demonstrate that this can significantly improve classification performance as compared to conventional techniques exclusively based on sEMG signals.

Methods

We collected and analyzed a large dataset comprising recordings with 20 able-bodied and two amputee participants executing 40 movements. Additionally, we conducted a novel real-time prosthetic hand control experiment with 11 able-bodied subjects and an amputee by using a state-of-the-art commercial prosthetic hand. A systematic performance comparison was carried out to investigate the potential benefit of incorporating IMs in prosthetic hand control.

Results

The inclusion of IM data improved performance significantly, by increasing classification accuracy (CA) in the offline analysis and improving completion rates (CRs) in the real-time experiment. Our findings were consistent across able-bodied and amputee subjects. Integrating the sEMG electrodes and IM sensors within a single sensor package enabled us to achieve high-level performance by using on average 4-6 sensors.

Conclusions

The results from our experiments suggest that IMs can form an excellent complimentary source signal for upper-limb myoelectric prostheses. We trust that multi-modal control solutions have the potential of improving the usability of upper-extremity prostheses in real-life applications.

Electronic supplementary material

The online version of this article (doi:10.1186/s12984-017-0284-4) contains supplementary material, which is available to authorized users.