Continuous estimation of upper limb joint angle from sEMG signals based on SCA-LSTM deep learning approach

A Novel TCN-LSTM Hybrid Model for sEMG-Based Continuous Estimation of Wrist Joint Angles

Surface electromyography (sEMG) offers a novel method in human-machine interactions (HMIs) since it is a distinct physiological electrical signal that conceals human movement intention and muscle information. Unfortunately, the nonlinear and non-smooth features of sEMG signals often make joint angle estimation difficult. This paper proposes a joint angle prediction model for the continuous estimation of wrist motion angle changes based on sEMG signals. The proposed model combines a temporal convolutional network (TCN) with a long short-term memory (LSTM) network, where the TCN can sense local information and mine the deeper information of the sEMG signals, while LSTM, with its excellent temporal memory capability, can make up for the lack of the ability of the TCN to capture the long-term dependence of the sEMG signals, resulting in a better prediction. We validated the proposed method in the publicly available Ninapro DB1 dataset by selecting the first eight subjects and picking three types of wrist-dependent movements: wrist flexion (WF), wrist ulnar deviation (WUD), and wrist extension and closed hand (WECH). Finally, the proposed TCN-LSTM model was compared with the TCN and LSTM models. The proposed TCN-LSTM outperformed the TCN and LSTM models in terms of the root mean square error (RMSE) and average coefficient of determination (R2). The TCN-LSTM model achieved an average RMSE of 0.064, representing a 41% reduction compared to the TCN model and a 52% reduction compared to the LSTM model. The TCN-LSTM also achieved an average R2 of 0.93, indicating an 11% improvement over the TCN model and an 18% improvement over the LSTM model.

Feature evaluation for myoelectric pattern recognition of multiple nearby reaching targets

Intention detection of the reaching movement is considerable for myoelectric human and machine collaboration applications. A comprehensive set of handcrafted features was mined from windows of electromyogram (EMG) of the upper-limb muscles while reaching nine nearby targets like activities of daily living. The feature selection-based scoring method, neighborhood component analysis (NCA), selected the relevant feature subset. Finally, the target was recognized by the support vector machine (SVM) model. The classification performance was generalized by a nested cross-validation structure that selected the optimal feature subset in the inner loop. According to the low spatial resolution of the target location on display and following the slight discrimination of signals between targets, the best classification accuracy of 77.11 % was achieved for concatenating the features of two segments with a length of 2 and 0.25 s. Due to the lack of subtle variation in EMG, while reaching different targets, a wide range of features was applied to consider additional aspects of the knowledge contained in EMG signals. Furthermore, since NCA selected features that provided more discriminant power, it became achievable to employ various combinations of features and even concatenated features extracted from different movement parts to improve classification performance.

Continuous Motion Intention Prediction Using sEMG for Upper-Limb Rehabilitation: A Systematic Review of Model-Based and Model-Free Approaches

Upper limb functional impairments persisting after stroke significantly affect patients' quality of life. Precise adjustment of robotic assistance levels based on patients' motion intentions using sEMG signals is crucial for active rehabilitation. This paper systematically reviews studies on continuous prediction of upper limb single joints and multi-joint combinations motion intention using Model-Based (MB) and Model-Free (MF) approaches over the past decade, based on 186 relevant studies screened from six major electronic databases. The findings indicate ongoing challenges in terms of subject composition, algorithm robustness and generalization, and algorithm feasibility for practical applications. Moreover, it suggests integrating the strengths of both MB and MF approaches to improve existing algorithms. Therefore, future research should further explore personalized MB-MF combination methods incorporating deep learning, attention mechanisms, muscle synergy features, motor unit features, and closed-loop feedback to achieve precise, real-time, and long-duration prediction of multi-joint complex movements, while further refining the transfer learning strategy for rapid algorithm deployment across days and subjects. Overall, this review summarizes the current research status, significant findings, and challenges, aiming to inspire future research on predicting upper limb motion intentions based on sEMG.

A Multimodal Multilevel Converged Attention Network for Hand Gesture Recognition With Hybrid sEMG and A-Mode Ultrasound Sensing

Gesture recognition based on surface electromyography (sEMG) has been widely used in the field of human-machine interaction (HMI). However, sEMG has limitations, such as low signal-to-noise ratio and insensitivity to fine finger movements, so we consider adding A-mode ultrasound (AUS) to enhance the recognition impact. To explore the influence of multisource sensing data on gesture recognition and better integrate the features of different modules. We proposed a multimodal multilevel converged attention network (MMCANet) model for multisource signals composed of sEMG and AUS. The proposed model extracts the hidden features of the AUS signal with a convolutional neural network (CNN). Meanwhile, a CNN-LSTM (long-short memory network) hybrid structure extracts some spatial-temporal features from the sEMG signal. Then, two types of CNN features from AUS and sEMG are spliced and transmitted to a transformer encoder to fuse the information and interact with sEMG features to produce hybrid features. Finally, the classification results are output employing fully connected layers. Attention mechanisms are used to adjust the weights of feature channels. We compared MMCANet's feature extraction and classification performance with that of manually extracted sEMG-AUS features using four traditional machine-learning (ML) algorithms. The recognition accuracy increased by at least 5.15%. In addition, we tried deep learning (DL) methods with CNN on single modals. The experimental results showed that the proposed model improved 14.31% and 3.80% over the CNN method with single sEMG and AUS, respectively. Compared with some state-of-the-art fusion techniques, our method also achieved better results.

sEMG-Based End-to-End Continues Prediction of Human Knee Joint Angles Using the Tightly Coupled Convolutional Transformer Model

Wearable exoskeleton robots can promote the rehabilitation of patients with physical dysfunction. And improving human-computer interaction performance is a significant challenge for exoskeleton robots. The traditional feature extraction process based on surface Electromyography(sEMG) is complex and requires manual intervention, making real-time performance difficult to guarantee. In this study, we propose an end-to-end method to predict human knee joint angles based on sEMG signals using a tightly coupled convolutional transformer (TCCT) model. We first collected sEMG signals from 5 healthy subjects. Then, the envelope was extracted from the noise-removed sEMG signal and used as the input to the model. Finally, we developed the TCCT model to predict the knee joint angle after 100 ms. For the prediction performance, we used the Root Mean Square Error(RMSE), Pearson Correlation Coefficient(CC), and Adjustment R2 as metrics to evaluate the error between the actual knee angle and the predicted knee angle. The results show that the model can predict the human knee angle quickly and accurately. The mean RMSE, Adjustment R2, and (CC) values of the model are 3.79°, 0.96, and 0.98, respectively, which are better than traditional deep learning models such as Informer (4.14, 0.95, 0.98), CNN (5.56, 0.89, 0.96) and CNN-BiLSTM (3.97, 0.95, 0.98). In addition, the prediction time of our proposed model is only 11.67 ± 0.67 ms, which is less than 100 ms. Therefore, the real-time and accuracy of the model can meet the continuous prediction of human knee joint angle in practice.

Continuous online prediction of lower limb joints angles based on sEMG signals by deep learning approach

Continuous online prediction of human joints angles is a key point to improve the performance of man-machine cooperative control. In this study, a framework of online prediction method of joints angles by long short-term memory (LSTM) neural network only based on surface electromyography (sEMG) signals was proposed. The sEMG signals from eight muscles of five subjects' right leg and three joints angles and plantar pressure signals of subjects were collected simultaneously. Different inputs (only sEMG (unimodal), sEMG combined with plantar pressure (multimodal)) after online feature extraction and standardization were used for training the angle online prediction model by LSTM. The results indicate that there is no significant difference between the two kinds of inputs for LSTM model and the proposed method can make up for the shortage of using a single type of sensor. The range of mean values of root square mean error, mean absolute error and Pearson correlation coefficient of the three joints angles achieved by the proposed model only with the input of sEMG under four kinds of predicted time (50, 100, 150, and 200 ms) are [1.63°,3.20°],[1.27°, 2.36°] and [0.9747, 0.9935]. Three popular machine learning algorithms with different inputs were compared to the proposed model only based on sEMG. Experiment results demonstrate that the proposed method has the best prediction performance and there are highly significant differences between it and other methods. The difference of prediction results under different gait phases by the proposed method was also analyzed. The results indicate that the prediction effect of support phases is generally better than that of swing phases. Above experimental results show that the proposed method can realize accurate online joint angle prediction and has better performance to promote man-machine cooperation.

Continuous Motion Estimation of Knee Joint Based on a Parameter Self-Updating Mechanism Model

Estimation of continuous motion of human joints using surface electromyography (sEMG) signals has a critical part to play in intelligent rehabilitation. Traditional methods always use sEMG signals as inputs to build regression or biomechanical models to estimate continuous joint motion variables. However, it is challenging to accurately estimate continuous joint motion in new subjects due to the non-stationarity and individual differences in sEMG signals, which greatly limits the generalisability of the method. In this paper, a continuous motion estimation model for the human knee joint with a parameter self-updating mechanism based on the fusion of particle swarm optimization (PSO) and deep belief network (DBN) is proposed. According to the original sEMG signals of different subjects, the method adaptively optimized the parameters of the DBN model and completed the optimal reconstruction of signal feature structure in high-dimensional space to achieve the optimal estimation of continuous joint motion. Extensive experiments were conducted on knee joint motions. The results suggested that the average root mean square errors (RMSEs) of the proposed method were 9.42° and 7.36°, respectively, which was better than the results obtained by common neural networks. This finding lays a foundation for the human-robot interaction (HRI) of the exoskeleton robots based on the sEMG signals.

A Review of Myoelectric Control for Prosthetic Hand Manipulation

Myoelectric control for prosthetic hands is an important topic in the field of rehabilitation. Intuitive and intelligent myoelectric control can help amputees to regain upper limb function. However, current research efforts are primarily focused on developing rich myoelectric classifiers and biomimetic control methods, limiting prosthetic hand manipulation to simple grasping and releasing tasks, while rarely exploring complex daily tasks. In this article, we conduct a systematic review of recent achievements in two areas, namely, intention recognition research and control strategy research. Specifically, we focus on advanced methods for motion intention types, discrete motion classification, continuous motion estimation, unidirectional control, feedback control, and shared control. In addition, based on the above review, we analyze the challenges and opportunities for research directions of functionality-augmented prosthetic hands and user burden reduction, which can help overcome the limitations of current myoelectric control research and provide development prospects for future research.

Enhanced Deep Transfer Learning Model based on Spatial-Temporal driven Scalograms for Precise Decoding of Motor Intent in Stroke Survivors

Intelligent rehabilitation robotics (RR) have been proposed in recent years to aid post-stroke survivors recover their lost limb functions. However, a large proportion of these robotic systems operate in a passive mode that restricts users to predefined trajectories that rarely align with their intended limb movements, precluding full functional recovery. To address this issue, an efficient Transfer Learning based Convolutional Neural Network (TL-CNN) model is proposed to decode post-stroke patients' motion intentions toward realizing dexterously active robotic training during rehabilitation. For the first time, we use Spatial-Temporal Descriptor based Continuous Wavelet Transform (STD-CWT) as input to TL-CNN to optimally decode limb movement intent patterns. We evaluated the STD-CWT method on three distinct wavelets including the Morse, Amor, and Bump, and compared their decoding outcomes with those of the commonly adopted CWT technique under similar experimental conditions. We then validated the method using electromyogram signals of five stroke survivors who performed twenty-one distinct motor tasks. The results showed that the proposed technique recorded a significantly higher (p<0.05) decoding accuracy and faster convergence compared to the common method. Our method equally recorded obvious class separability for individual motor tasks across subjects. The findings suggest that the STD-CWT Scalograms have the potential for robust decoding of motor intention and could facilitate intuitive and active motor training in stroke RR.Clinical Relevance- The study demonstrated the potential of Spatial Temporal based Scalograms in aiding precise and robust decoding of multi-class motor tasks, upon which dexterously active rehabilitation robotic training for full motor function restoration could be realized.

A kinematic, imaging and electromyography dataset for human muscular manipulability index prediction

Human Muscular Manipulability is a metric that measures the comfort of an specific pose and it can be used for a variety of applications related to healthcare. For this reason, we introduce KIMHu: a Kinematic, Imaging and electroMyography dataset for Human muscular manipulability index prediction. The dataset is comprised of images, depth maps, skeleton tracking data, electromyography recordings and 3 different Human Muscular Manipulability indexes of 20 participants performing different physical exercises with their arm. The methodology followed to acquire and process the data is also presented for future replication. A specific analysis framework for Human Muscular Manipulability is proposed in order to provide benchmarking tools based on this dataset.

A Feature-Encoded Physics-Informed Parameter Identification Neural Network for Musculoskeletal Systems

Identification of muscle-tendon force generation properties and muscle activities from physiological measurements, e.g., motion data and raw surface electromyography (sEMG), offers opportunities to construct a subject-specific musculoskeletal (MSK) digital twin system for health condition assessment and motion prediction. While machine learning approaches with capabilities in extracting complex features and patterns from a large amount of data have been applied to motion prediction given sEMG signals, the learned data-driven mapping is black-box and may not satisfy the underlying physics and has reduced generality. In this work, we propose a feature-encoded physics-informed parameter identification neural network (FEPI-PINN) for simultaneous prediction of motion and parameter identification of human MSK systems. In this approach, features of high-dimensional noisy sEMG signals are projected onto a low-dimensional noise-filtered embedding space for the enhancement of forwarding dynamics prediction. This FEPI-PINN model can be trained to relate sEMG signals to joint motion and simultaneously identify key MSK parameters. The numerical examples demonstrate that the proposed framework can effectively identify subject-specific muscle parameters and the trained physics-informed forward-dynamics surrogate yields accurate motion predictions of elbow flexion-extension motion that are in good agreement with the measured joint motion data.

Comparison of machine learning and deep learning-based methods for locomotion mode recognition using a single inertial measurement unit

Locomotion mode recognition provides the prosthesis control with the information on when to switch between different walking modes, whereas the gait phase detection indicates where we are in the gait cycle. But powered prostheses often implement a different control strategy for each locomotion mode to improve the functionality of the prosthesis. Existing studies employed several classical machine learning methods for locomotion mode recognition. However, these methods were less effective for data with complex decision boundaries and resulted in misclassifications of motion recognition. Deep learning-based methods potentially resolve these limitations as it is a special type of machine learning method with more sophistication. Therefore, this study evaluated three deep learning-based models for locomotion mode recognition, namely recurrent neural network (RNN), long short-term memory (LSTM) neural network, and convolutional neural network (CNN), and compared the recognition performance of deep learning models to the machine learning model with random forest classifier (RFC). The models are trained from data of one inertial measurement unit (IMU) placed on the lower shanks of four able-bodied subjects to perform four walking modes, including level ground walking (LW), standing (ST), and stair ascent/stair descent (SA/SD). The results indicated that CNN and LSTM models outperformed other models, and these models were promising for applying locomotion mode recognition in real-time for robotic prostheses.

A deep learning method to predict ankle joint moment during walking at different speeds with ultrasound imaging: A framework for assistive devices control

Robotic assistive or rehabilitative devices are promising aids for people with neurological disorders as they help regain normative functions for both upper and lower limbs. However, it remains challenging to accurately estimate human intent or residual efforts non-invasively when using these robotic devices. In this article, we propose a deep learning approach that uses a brightness mode, that is, B-mode, of ultrasound (US) imaging from skeletal muscles to predict the ankle joint net plantarflexion moment while walking. The designed structure of customized deep convolutional neural networks (CNNs) guarantees the convergence and robustness of the deep learning approach. We investigated the influence of the US imaging's region of interest (ROI) on the net plantarflexion moment prediction performance. We also compared the CNN-based moment prediction performance utilizing B-mode US and sEMG spectrum imaging with the same ROI size. Experimental results from eight young participants walking on a treadmill at multiple speeds verified an improved accuracy by using the proposed US imaging + deep learning approach for net joint moment prediction. With the same CNN structure, compared to the prediction performance by using sEMG spectrum imaging, US imaging significantly reduced the normalized prediction root mean square error by 37.55% ( < .001) and increased the prediction coefficient of determination by 20.13% ( < .001). The findings show that the US imaging + deep learning approach personalizes the assessment of human joint voluntary effort, which can be incorporated with assistive or rehabilitative devices to improve clinical performance based on the assist-as-needed control strategy.

Appropriate Feature Set and Window Parameters Selection for Efficient Motion Intent Characterization towards Intelligently Smart EMG-PR System

The constantly rising number of limb stroke survivors and amputees has motivated the development of intelligent prosthetic/rehabilitation devices for their arm function restoration. The device often integrates a pattern recognition (PR) algorithm that decodes amputees’ limb movement intent from electromyogram (EMG) signals, characterized by neural information and symmetric distribution. However, the control performance of the prostheses mostly rely on the interrelations among multiple dynamic factors of feature set, windowing parameters, and signal conditioning that have rarely been jointly investigated to date. This study systematically investigated the interaction effects of these dynamic factors on the performance of EMG-PR system towards constructing optimal parameters for accurately robust movement intent decoding in the context of prosthetic control. In this regard, the interaction effects of various features across window lengths (50 ms~300 ms), increments (50 ms~125 ms), robustness to external interferences and sensor channels (2 ch~6 ch), were examined using EMG signals obtained from twelve subjects through a symmetrical movement elicitation protocol. Compared to single features, multiple features consistently achieved minimum decoding error below 10% across optimal windowing parameters of 250 ms/100 ms. Also, the multiple features showed high robustness to additive noise with obvious trade-offs between accuracy and computation time. Consequently, our findings may provide proper insight for appropriate parameter selection in the context of robust PR-based control strategy for intelligent rehabilitation device.