Sonomyography for Control of Upper-Limb Prostheses: Current State and Future Directions

Problem Statement

Despite the recent advancements in technology, many individuals with upper-limb loss struggle to achieve stable control over multiple degrees of freedom in a prosthesis. There is an ongoing need to develop noninvasive prosthesis control modalities that could improve functional patient outcomes.

Proposed Solution

Ultrasound-based sensing of muscle deformation, known as sonomyography, is an emerging sensing modality for upper-limb prosthesis control with the potential to significantly improve functionality. Sonomyography enables spatiotemporal characterization of both superficial and deep muscle activity, making it possible to distinguish the contributions of individual muscles during functional movements and derive a large set of independent prosthesis control signals. Using sonomyography to control a prosthesis has shown great promise in the research literature but has not yet been fully adapted for clinical use. This article describes the implementation of sonomyography for upper-limb prosthesis control, ongoing technological development, considerations for deploying this technology in clinical settings, and recommendations for future study.

Clinical Relevance

Sonomyography may soon become a clinically viable modality for upper-limb prosthesis control that could offer prosthetists an additional solution when selecting optimal treatment plans for their patients.

Bio-robotics research for non-invasive myoelectric neural interfaces for upper-limb prosthetic control: a 10-year perspective review

A decade ago, a group of researchers from academia and industry identified a dichotomy between the industrial and academic state-of-the-art in upper-limb prosthesis control, a widely used bio-robotics application. They proposed that four key technical challenges, if addressed, could bridge this gap and translate academic research into clinically and commercially viable products. These challenges are unintuitive control schemes, lack of sensory feedback, poor robustness and single sensor modality. Here, we provide a perspective review on the research effort that occurred in the last decade, aiming at addressing these challenges. In addition, we discuss three research areas essential to the recent development in upper-limb prosthetic control research but were not envisioned in the review 10 years ago: deep learning methods, surface electromyogram decomposition and open-source databases. To conclude the review, we provide an outlook into the near future of the research and development in upper-limb prosthetic control and beyond.

Two-stage classification of electromyogram signals from hand grasps in the transverse plane

This paper presents a two-stage classification to resolve the effect of arm position changes on electromyogram (EMG) classification for hand grasps in the transverse plane. The proposed method combines the EMG signals with the signals from an inertial measurement unit in both the position and motion classification stages. To improve accuracy, we incorporate EMG data from the upper arm and shoulder with the forearm EMG signals. When evaluated on the five alternative object grasps placed on the nine positions, the proposed technique yields an average total classification error of 0.9%, which is a substantial improvement over the single-stage classification (4.3%).

Evaluation of feature projection techniques in object grasp classification using electromyogram signals from different limb positions

A myoelectric prosthesis is manipulated using electromyogram (EMG) signals from the existing muscles for performing the activities of daily living. A feature vector that is formed by concatenating data from many EMG channels may result in a high dimensional space, which may cause prolonged computation time, redundancy, and irrelevant information. We evaluated feature projection techniques, namely principal component analysis (PCA), linear discriminant analysis (LDA), t-Distributed Stochastic Neighbor Embedding (t-SNE), and spectral regression extreme learning machine (SRELM), applied to object grasp classification. These represent feature projections that are combinations of either linear or nonlinear, and supervised or unsupervised types. All pairs of the four types of feature projection with seven types of classifiers were evaluated, with data from six EMG channels and an IMU sensors for nine upper limb positions in the transverse plane. The results showed that SRELM outperformed LDA with supervised feature projections, and t-SNE was superior to PCA with unsupervised feature projections. The classification errors from SRELM and t-SNE paired with the seven classifiers were from 1.50% to 2.65% and from 1.27% to 17.15%, respectively. A one-way ANOVA test revealed no statistically significant difference by classifier type when using the SRELM projection, which is a nonlinear supervised feature projection (p = 0.334). On the other hand, we have to carefully select an appropriate classifier for use with t-SNE, which is a nonlinear unsupervised feature projection. We achieved the lowest classification error 1.27% using t-SNE paired with a k-nearest neighbors classifier. For SRELM, the lowest 1.50% classification error was obtained when paired with a neural network classifier.

First Demonstration of Functional Task Performance Using a Sonomyographic Prosthesis: A Case Study

Ultrasound-based sensing of muscle deformation, known as sonomyography, has shown promise for accurately classifying the intended hand grasps of individuals with upper limb loss in offline settings. Building upon this previous work, we present the first demonstration of real-time prosthetic hand control using sonomyography to perform functional tasks. An individual with congenital bilateral limb absence was fitted with sockets containing a low-profile ultrasound transducer placed over forearm muscle tissue in the residual limbs. A classifier was trained using linear discriminant analysis to recognize ultrasound images of muscle contractions for three discrete hand configurations (rest, tripod grasp, index finger point) under a variety of arm positions designed to cover the reachable workspace. A prosthetic hand mounted to the socket was then controlled using this classifier. Using this real-time sonomyographic control, the participant was able to complete three functional tasks that required selecting different hand grasps in order to grasp and move one-inch wooden blocks over a broad range of arm positions. Additionally, these tests were successfully repeated without retraining the classifier across 3 hours of prosthesis use and following simulated donning and doffing of the socket. This study supports the feasibility of using sonomyography to control upper limb prostheses in real-world applications.

Stacking classifier to improve the classification of shoulder motion in transhumeral amputees

In recent years surface electromyography signals-based machine learning models are rapidly establishing. The efficacy of prosthetic arm growth for transhumeral amputees is aided by efficient classifiers. The paper aims to propose a stacking classifier-based classification system for sEMG shoulder movements. It presents the possibility of various shoulder motions classification of transhumeral amputees. To improve the system performance, adaptive threshold method and wavelet transformation have been applied for features extraction. Six different classifiers Support Vector Machines (SVM), Tree, Random Forest (RF), K-Nearest Neighbour (KNN), AdaBoost and Naïve Bayes (NB) are designed to extract the sEMG data classification accuracy. With cross-validation, the accuracy of RF, Tree and Ada Boost is 97%, 92% and 92% respectively. Stacking classifiers provides an accuracy as 99.4% after combining the best predicted multiple classifiers.

Activities of Daily Living-based Rehabilitation System for Arm and Hand Motor Function Retraining after Stroke

Most stroke survivors have difficulties completing activities of daily living (ADLs) independently. However, few rehabilitation systems have focused on ADLs-related training for gross and fine motor function together. We propose an ADLs-based serious game rehabilitation system for the training of motor function and coordination of both arm and hand movement where the user performs corresponding ADLs movements to interact with the target in the serious game. A multi-sensor fusion model based on electromyographic (EMG), force myographic (FMG), and inertial sensing was developed to estimate users' natural upper limb movement. Eight healthy subjects and three stroke patients were recruited in an experiment to validate the system's effectiveness. The performance of different sensor and classifier configurations on hand gesture classification against the arm position variations were analyzed, and qualitative patient questionnaires were conducted. Results showed that elbow extension/flexion has a more significant negative influence on EMG-based, FMG-based, and EMG+FMG-based hand gesture recognition than shoulder abduction/adduction does. In addition, there was no significant difference in the negative influence of shoulder abduction/adduction and shoulder flexion/extension on hand gesture recognition. However, there was a significant interaction between sensor configurations and algorithm configurations in both offline and real-time recognition accuracy. The EMG+FMG-combined multi-position classifier model had the best performance against arm position change. In addition, all the stroke patients reported their ADLs-related ability could be restored by using the system. These results demonstrate that the multi-sensor fusion model could estimate hand gestures and gross movement accurately, and the proposed training system has the potential to improve patients' ability to perform ADLs.

Research on the Improved CNN Deep Learning Method for Motion Intention Recognition of Dynamic Lower Limb Prosthesis

Objective

In order to study the motion recognition intention of lower limb prosthesis based on the CNN deep learning algorithm.

Methods

A convolutional neural network (CNN) model was established to reconstruct the motion pattern. Before the movement mode of the affected side was converted, the sensor was bound to the healthy side. The classifier was employed to extract and classify the features, so as to realize the accurate description of the movement intention of the disabled.

Results

The method proposed in this research can achieve 98.2% recognition rate of the movement intention of patients with lower limb amputation under different terrains, and the recognition rate can reach 97% after the pattern converted between the five modes was added.

Conclusion

The deep learning algorithm that automatically recognized and extracted features can effectively improve the control performance on the intelligent lower limb prosthesis and realize the natural and seamless conversion of the intelligent prosthesis in a variety of motion modes.

Position Identification for Robust Myoelectric Control Against Electrode Shift

The vulnerability to the electrode shift was one of the key barriers to the wide application of pattern recognition-based (PR-based) myoelectric control systems outside the controlled laboratory conditions. To overcome this challenge, a novel framework named position identification (PI) was proposed. In the PI framework, an anchor gesture performed by the user was first analyzed to identify the current electrode position from a pool of potential electrode shift positions. Next, the classifier calibrated by the data of the identified position would be selected for following myoelectric control tasks. The results of the amputee and able-bodied participants both demonstrated that the differential filter combined with majority voting improved the PI accuracy. With only one second contraction of the chosen anchor gesture (hand close), the subsequent PR-based myoelectric control performance was fully restored from eight different electrode shift scenarios, with 1 cm in either or both perpendicular and parallel directions. The classification accuracies with PI framework were not significant before and after the shift ( 0.001). The advantage of restoring performance fully in just one second made it a practical solution to improve the robustness of PR-based myoelectric control systems in a wide range of real-world applications.

sEMG dataset of routine activities

In this paper, we present the data set of surface electromyography (sEMG) and an Inertial Measurement Unit (IMU) against human muscle activity during routine activities. The Myo Thalamic Armband is used to acquire the signals from muscles below the elbow. The dataset comprises of raw sEMG, accelerometer, gyro and derived orientation signals for four different activities. The four activities, which are selected for this dataset acquisition, are resting, typing, push up exercise and lifting a heavy object. Therefore, there are five associated files against each activity. The IMU data can be fused with the sEMG data for better classification of activities especially to separate aggressive and normal activities. The data is valuable for researchers working on assistive computer aided support systems for subjects with disabilities due to physical or mental disorder.

Understanding Limb Position and External Load Effects on Real-Time Pattern Recognition Control in Amputees

Limb position is a factor that negatively affects myoelectric pattern recognition classification accuracy. However, prior studies evaluating impact on real-time control for upper-limb amputees have done so without a physical prosthesis on the residual limb. It remains unclear how limb position affects real-time pattern recognition control in amputees when their residual limb is supporting various weights. We used a virtual reality target achievement control test to evaluate the effects of limb position and external load on real-time pattern recognition control in fourteen intact limb subjects and six major upper limb amputee subjects. We also investigated how these effects changed based on different control system training methods. In a static training method, subjects kept their unloaded arm by their side with the elbow bent whereas in the dynamic training method, subjects moved their arm throughout a workspace while supporting a load. When static training was used, limb position significantly affected real-time control in all subjects. However, amputee subjects were still able to adequately complete tasks in all conditions, even in untrained limb positions. Moreover, increasing external loads decreased controller performance, albeit to a lesser extent in amputee subjects. The effects of limb position did not change as load increased, and vice versa. In intact limb subjects, dynamic training significantly reduced the limb position effect but did not completely remove them. In contrast, in amputee subjects, dynamic training eliminated the limb position effect in three out of four outcome measures. However, it did not reduce the effects of load for either subject population. These findings suggest that results obtained from intact limb subjects may not generalize to amputee subjects and that advanced training methods can substantially improve controller robustness to different limb positions regardless of limb loading.

Efficient correction of armband rotation for myoelectric-based gesture control interface

OBJECTIVE

The appearance of commercial myoelectric armbands has greatly increased the portability and convenience of myoelectric controlled interfaces (MCIs). However, one limitation of the current state-of-the-art myoelectric control algorithms is that they have poor robustness against armband displacements, especially rotation, leading to great algorithmic performance degradation. The traditional remedy, retraining the interface, requires the data collection of all gestures and is impractical in many applications. The recently proposed position verification (PV) framework focused on quickly identifying and correcting the electrode positions after the displacement, showing the potential to restore the performance of MCI in a faster way. However, its online effectiveness is still yet to be validated.

APPROACH

This work proposed a novel algorithm of identifying the rotation direction to improve the efficiency of the PV framework and demonstrated the real-time capability of the PV framework using a commercially available armband.

MAIN RESULTS

The results showed that with PV, a 1.5-cm rotation could be corrected with an average of 3.1 ± 1.5 interactive adjustments, equivalent to around 15.5 ± 7.5 s, which was greatly reduced compared to retraining. There was no significant difference in the real-time control performance between before the armband displacement and after the PV correction.

SIGNIFICANCE

To the best of our knowledge, this study was the first maintaining pattern recognition-based myoelectric control performance in the presence of electrode shifts without recollecting the entire training data. It suggested the feasibility of the PV framework used in the myoelectric armband and MCI for practical applications.

The Merits of Dynamic Data Acquisition for Realistic Myocontrol

Natural myocontrol is the intuitive control of a prosthetic limb via the user's voluntary muscular activations. This type of control is usually implemented by means of pattern recognition, which uses a set of training data to create a model that can decipher these muscular activations. A consequence of this approach is that the reliability of a myocontrol system depends on how representative this training data is for all types of signal variability that may be encountered when the amputee puts the prosthesis into real use. Myoelectric signals are indeed known to vary according to the position and orientation of the limb, among other factors, which is why it has become common practice to take this variability into account by acquiring training data in multiple body postures. To shed further light on this problem, we compare two ways of collecting data: while the subjects hold their limb statically in several positions one at a time, which is the traditional way, or while they dynamically move their limb at a constant pace through those same positions. Since our interest is to investigate any differences when controlling an actual prosthetic device, we defined an evaluation protocol that consisted of a series of complex, bimanual daily-living tasks. Fourteen intact participants performed these tasks while wearing prosthetic hands mounted on splints, which were controlled via either a statically or dynamically built myocontrol model. In both cases all subjects managed to complete all tasks and participants without previous experience in myoelectric control manifested a significant learning effect; moreover, there was no significant difference in the task completion times achieved with either model. When evaluated in a simulated scenario with traditional offline performance evaluation, on the other hand, the dynamically-trained system showed significantly better accuracy. Regardless of the setting, the dynamic data acquisition was faster, less tiresome, and better accepted by the users. We conclude that dynamic data acquisition is advantageous and confirm the limited relevance of offline analyses for online myocontrol performance.

Current Trends and Confounding Factors in Myoelectric Control: Limb Position and Contraction Intensity

This manuscript presents a hybrid study of a comprehensive review and a systematic(research) analysis. Myoelectric control is the cornerstone ofmany assistive technologies used in clinicalpractice, such as prosthetics and orthoses, and human-computer interaction, such as virtual reality control.Although the classification accuracy of such devices exceeds 90% in a controlled laboratory setting,myoelectric devices still face challenges in robustness to variability of daily living conditions.The intrinsic physiological mechanisms limiting practical implementations of myoelectric deviceswere explored: the limb position effect and the contraction intensity effect. The degradationof electromyography (EMG) pattern recognition in the presence of these factors was demonstratedon six datasets, where classification performance was 13% and 20% lower than the controlledsetting for the limb position and contraction intensity effect, respectively. The experimental designsof limb position and contraction intensity literature were surveyed. Current state-of-the-art trainingstrategies and robust algorithms for both effects were compiled and presented. Recommendationsfor future limb position effect studies include: the collection protocol providing exemplars of at least 6positions (four limb positions and three forearm orientations), three-dimensional space experimentaldesigns, transfer learning approaches, and multi-modal sensor configurations. Recommendationsfor future contraction intensity effect studies include: the collection of dynamic contractions, nonlinearcomplexity features, and proportional control.

Selection of Features and Classifiers for EMG-EEG-Based Upper Limb Assistive Devices-A Review

Bio-signals are distinctive factors in the design of human-machine interface, essentially useful for prosthesis, orthosis, and exoskeletons. Despite the progress in the analysis of pattern recognition based devices; the acceptance of these devices is still questionable. One reason is the lack of information to identify the possible combinations of features and classifiers. Besides; there is also a need for optimal selection of various sensors for sensations such as touch, force, texture, along with EMGs/EEGs. This article reviews the two bio-signal techniques, named as electromyography and electroencephalography. The details of the features and the classifiers used in the data processing for upper limb assist devices are summarised here. Various features and their sets are surveyed and different classifiers for feature sets are discussed on the basis of the classification rate. The review was carried out on the basis of the last 10-12 years of published research in this area. This article also outlines the influence of modality of EMGs and EEGs with other sensors on classifications. Also, other bio-signals used in upper limb devices and future aspects are considered.

Real-Time EMG Based Pattern Recognition Control for Hand Prostheses: A Review on Existing Methods, Challenges and Future Implementation

Upper limb amputation is a condition that significantly restricts the amputees from performing their daily activities. The myoelectric prosthesis, using signals from residual stump muscles, is aimed at restoring the function of such lost limbs seamlessly. Unfortunately, the acquisition and use of such myosignals are cumbersome and complicated. Furthermore, once acquired, it usually requires heavy computational power to turn it into a user control signal. Its transition to a practical prosthesis solution is still being challenged by various factors particularly those related to the fact that each amputee has different mobility, muscle contraction forces, limb positional variations and electrode placements. Thus, a solution that can adapt or otherwise tailor itself to each individual is required for maximum utility across amputees. Modified machine learning schemes for pattern recognition have the potential to significantly reduce the factors (movement of users and contraction of the muscle) affecting the traditional electromyography (EMG)-pattern recognition methods. Although recent developments of intelligent pattern recognition techniques could discriminate multiple degrees of freedom with high-level accuracy, their efficiency level was less accessible and revealed in real-world (amputee) applications. This review paper examined the suitability of upper limb prosthesis (ULP) inventions in the healthcare sector from their technical control perspective. More focus was given to the review of real-world applications and the use of pattern recognition control on amputees. We first reviewed the overall structure of pattern recognition schemes for myo-control prosthetic systems and then discussed their real-time use on amputee upper limbs. Finally, we concluded the paper with a discussion of the existing challenges and future research recommendations.

Dynamic time warping for reducing the effect of force variation on myoelectric control of hand prostheses

Research in pattern recognition (PR) for myoelectric control of the upper limb prostheses has been extensive. However, there has been limited attention to the factors that influence the clinical translation of this technology. A relevant factor of influence in clinical performance of EMG PR-based control of prostheses is the variation in muscle activation level, which modifies the EMG patterns even when the amputee attempts the same movement. To decrease the effect of muscle activation level variations on EMG PR, this work proposes to use dynamic time warping (DTW) and is validated on two databases. The first database, which has data from ten intact-limbed subjects, was used to test the baseline performance of DTW, resulting in an average classification accuracy of more than 90%. The second database comprised data from nine upper limb amputees recorded at three levels of force for six hand grips. The results showed that DTW trained at a single force level achieved an average classification accuracy of 60 ± 9%, 70 ± 8%, and 60 ± 7% at the low, medium and high force levels respectively across all amputee subjects. The proposed scheme with DTW achieved a significant 10% improvement in classification accuracy when trained at a low force level when compared to the traditional time-dependent power spectrum descriptors (TD-PSD) method.

Towards Wearable A-Mode Ultrasound Sensing for Real-Time Finger Motion Recognition

It is evident that surface electromyography (sEMG) based human-machine interfaces (HMI) have inherent difficulty in predicting dexterous musculoskeletal movements such as finger motions. This paper is an attempt to investigate a plausible alternative to sEMG, ultrasound-driven HMI, for dexterous motion recognition due to its characteristic of detecting morphological changes of deep muscles and tendons. A multi-channel A-mode ultrasound lightweight device is adopted to evaluate the performance of finger motion recognition; an experiment is designed for both widely acceptable offline and online algorithms with eight able-bodied subjects employed. The experiment result presents that the offline recognition accuracy is up to 98.83% ± 0.79%. The real-time motion completion rate is 95.4% ± 8.7% and online motion selection time is 0.243 ± 0.127 s. The outcomes confirm the feasibility of A-mode ultrasound based wearable HMI and its prosperous applications in prosthetic devices, virtual reality, and remote manipulation.

Natural Myocontrol in a Realistic Setting: a Comparison Between Static and Dynamic Data Acquisition

Natural myocontrol employs pattern recognition to allow users to control a robotic limb intuitively using their own voluntary muscular activations. The reliability of myocontrol strongly depends on the signals initially collected from the users, which must appropriately capture the variability encountered later on during operation. Since myoelectric signals can vary based on the position and orientation of the limb, it has become best practice to gather data in multiple body postures. We hereby concentrate on this acquisition protocol and investigate the relative merits of collecting data either statically or dynamically. In the static case, data for a desired hand configuration is collected while the users keep their hand still in certain positions, whereas in the dynamic case, data is collected while users move their limbs, passing through the required positions with a roughly constant velocity.Fourteen able-bodied subjects were asked to naturally control two dexterous hand prostheses mounted on splints, performing a set of complex, realistic bimanual activities of daily living. We could not find any significant difference between the protocols in terms of the total execution times, although the dynamic data acquisition was faster and less tiring. This would indicate that dynamic data acquisition should be preferred over the static one.

Improving the functionality, robustness, and adaptability of myoelectric control for dexterous motion restoration

The development of advanced and effective human-machine interfaces, especially for amputees to control their prostheses, is very high priority and a very active area of research. An intuitive control method should retain an adequate level of functionality for dexterous operation, provide robustness against confounding factors, and supply adaptability for diverse long-term usage, all of which are current problems being tackled by researchers. This paper reviews the state-of-the-art, as well as, the limitations of current myoelectric signal control (MSC) methods. To address the research topic on functionality, we review different approaches to prosthetic hand control (DOF configuration, discrete or simultaneous, etc.), and how well the control is performed (accuracy, response, intuitiveness, etc.). To address the research on robustness, we review the confounding factors (limb positions, electrode shift, force variance, and inadvertent activity) that affect the stability of the control performance. Lastly, to address adaptability, we review the strategies that can automatically adjust the classifier for different individuals and for long-term usage. This review provides a thorough overview of the current MSC methods and helps highlight the current areas of research focus and resulting clinic usability for the MSC methods for upper-limb prostheses.

Causes of Performance Degradation in Non-invasive Electromyographic Pattern Recognition in Upper Limb Prostheses

Surface Electromyography (EMG)-based pattern recognition methods have been investigated over the past years as a means of controlling upper limb prostheses. Despite the very good reported performance of myoelectric controlled prosthetic hands in lab conditions, real-time performance in everyday life conditions is not as robust and reliable, explaining the limited clinical use of pattern recognition control. The main reason behind the instability of myoelectric pattern recognition control is that EMG signals are non-stationary in real-life environments and present a lot of variability over time and across subjects, hence affecting the system's performance. This can be the result of one or many combined changes, such as muscle fatigue, electrode displacement, difference in arm posture, user adaptation on the device over time and inter-subject singularity. In this paper an extensive literature review is performed to present the causes of the drift of EMG signals, ways of detecting them and possible techniques to counteract for their effects in the application of upper limb prostheses. The suggested techniques are organized in a table that can be used to recognize possible problems in the clinical application of EMG-based pattern recognition methods for upper limb prosthesis applications and state-of-the-art methods to deal with such problems.

A Novel Unsupervised Adaptive Learning Method for Long-Term Electromyography (EMG) Pattern Recognition

Performance degradation will be caused by a variety of interfering factors for pattern recognition-based myoelectric control methods in the long term. This paper proposes an adaptive learning method with low computational cost to mitigate the effect in unsupervised adaptive learning scenarios. We presents a particle adaptive classifier (PAC), by constructing a particle adaptive learning strategy and universal incremental least square support vector classifier (LS-SVC). We compared PAC performance with incremental support vector classifier (ISVC) and non-adapting SVC (NSVC) in a long-term pattern recognition task in both unsupervised and supervised adaptive learning scenarios. Retraining time cost and recognition accuracy were compared by validating the classification performance on both simulated and realistic long-term EMG data. The classification results of realistic long-term EMG data showed that the PAC significantly decreased the performance degradation in unsupervised adaptive learning scenarios compared with NSVC (9.03% ± 2.23%, p < 0.05) and ISVC (13.38% ± 2.62%, p = 0.001), and reduced the retraining time cost compared with ISVC (2 ms per updating cycle vs. 50 ms per updating cycle).