The role of muscle synergies in myoelectric control: trends and challenges for simultaneous multifunction control

Arm muscle synergies enhance hand posture prediction in combination with forearm muscle synergies

Objective.We analyze and interpret arm and forearm muscle activity in relation with the kinematics of hand pre-shaping during reaching and grasping from the perspective of human synergistic motor control.Approach.Ten subjects performed six tasks involving reaching, grasping and object manipulation. We recorded electromyographic (EMG) signals from arm and forearm muscles with a mix of bipolar electrodes and high-density grids of electrodes. Motion capture was concurrently recorded to estimate hand kinematics. Muscle synergies were extracted separately for arm and forearm muscles, and postural synergies were extracted from hand joint angles. We assessed whether activation coefficients of postural synergies positively correlate with and can be regressed from activation coefficients of muscle synergies. Each type of synergies was clustered across subjects.Main results.We found consistency of the identified synergies across subjects, and we functionally evaluated synergy clusters computed across subjects to identify synergies representative of all subjects. We found a positive correlation between pairs of activation coefficients of muscle and postural synergies with important functional implications. We demonstrated a significant positive contribution in the combination between arm and forearm muscle synergies in estimating hand postural synergies with respect to estimation based on muscle synergies of only one body segment, either arm or forearm (p< 0.01). We found that dimensionality reduction of multi-muscle EMG root mean square (RMS) signals did not significantly affect hand posture estimation, as demonstrated by comparable results with regression of hand angles from EMG RMS signals.Significance.We demonstrated that hand posture prediction improves by combining activity of arm and forearm muscles and we evaluate, for the first time, correlation and regression between activation coefficients of arm muscle and hand postural synergies. Our findings can be beneficial for myoelectric control of hand prosthesis and upper-limb exoskeletons, and for biomarker evaluation during neurorehabilitation.

Nonlinear spatio-temporal filter to reduce crosstalk in bipolar electromyogram

Objective.The wide detection volume of surface electromyogram (EMG) makes it prone to crosstalk, i.e. the signal from other muscles than the target one. Removing this perturbation from bipolar recordings is an important open problem for many applications.Approach.An innovative nonlinear spatio-temporal filter is developed to estimate the EMG generated by the target muscle by processing noisy signals from two bipolar channels, placed over the target and the crosstalk muscle, respectively. The filter is trained on some calibration data and then can be applied on new signals. Tests are provided in simulations (considering different thicknesses of the subcutaneous tissue, inter-electrode distances, locations of the EMG channels, force levels) and experiments (from pronator teres and flexor carpi radialis of 8 healthy subjects).Main results.The proposed filter allows to reduce the effect of crosstalk in all investigated conditions, with a statistically significant reduction of its root mean squared of about 20%, both in simulated and experimental data. Its performances are also superior to those of a blind source separation method applied to the same data.Significance.The proposed filter is simple to be applied and feasible in applications in which single bipolar channels are placed over the muscles of interest. It can be useful in many fields, such as in gait analysis, tests of myoelectric fatigue, rehabilitation with EMG biofeedback, clinical studies, prosthesis control.

Classification of EMG signals with CNN features and voting ensemble classifier

Electromyography (EMG) signals are primarily used to control prosthetic hands. Classifying hand gestures efficiently with EMG signals presents numerous challenges. In addition to overcoming these challenges, a successful combination of feature extraction and classification approaches will improve classification accuracy. In the current work, convolutional neural network (CNN) features are used to reduce the redundancy problems associated with time and frequency domain features to improve classification accuracy. The features from the EMG signal are extracted using a CNN and are fed to the 'k' nearest neighbor (KNN) classifier with a different number of neighbors ( 1 N N , 3 N N , 5 N N , and   7 N N ) . It results in an ensemble of classifiers that are combined using a hard voting-based classifier. Based on the benchmark Ninapro DB4 database and CapgMyo database, the proposed framework obtained 91.3 % classification accuracy on CapgMyo and 89.5 % on Ninapro DB4.

Progressive unsupervised control of myoelectric upper limbs

Objective.Unsupervised myocontrol methods aim to create control models for myoelectric prostheses while avoiding the complications of acquiring reliable, regular, and sufficient labeled training data. A limitation of current unsupervised methods is that they fix the number of controlled prosthetic functions a priori, thus requiring an initial assessment of the user's motor skills and neglecting the development of novel motor skills over time.Approach.We developed a progressive unsupervised myocontrol (PUM) paradigm in which the user and the control model coadaptively identify distinct muscle synergies, which are then used to control arbitrarily associated myocontrol functions, each corresponding to a hand or wrist movement. The interaction starts with learning a single function and the user may request additional functions after mastering the available ones, which aligns the evolution of their motor skills with an increment in system complexity. We conducted a multi-session user study to evaluate PUM and compare it against a state-of-the-art non-progressive unsupervised alternative. Two participants with congenital upper-limb differences tested PUM, while ten non-disabled control participants tested either PUM or the non-progressive baseline. All participants engaged in myoelectric control of a virtual hand and wrist.Main results.PUM enabled autonomous learning of three myocontrol functions for participants with limb differences, and of all four available functions for non-disabled subjects, using both existing or newly identified muscle synergies. Participants with limb differences achieved similar success rates to non-disabled ones on myocontrol tests, but faced greater difficulties in internalizing new motor skills and exhibited slightly inferior movement quality. The performance was comparable with either PUM or the non-progressive baseline for the group of non-disabled participants.Significance.The PUM paradigm enables users to autonomously learn to operate the myocontrol system, adapts to the users' varied preexisting motor skills, and supports the further development of those skills throughout practice.

Restoring natural upper limb movement through a wrist prosthetic module for partial hand amputees

BACKGROUND

Most partial hand amputees experience limited wrist movement. The limited rotational wrist movement deteriorates natural upper limb system related to hand use and the usability of the prosthetic hand, which may cause secondary damage to the musculoskeletal system due to overuse of the upper limb affected by repetitive compensatory movement patterns. Nevertheless, partial hand prosthetics, in common, have only been proposed without rotational wrist movement because patients have various hand shapes, and a prosthetic hand should be attached to a narrow space.

METHODS

We hypothesized that partial hand amputees, when using a prosthetic hand with a wrist rotation module, would achieve natural upper limb movement muscle synergy and motion analysis comparable to a control group. To validate the proposed prototype design with the wrist rotation module and verify our hypothesis, we compared a control group with partial hand amputees wearing hand prostheses, both with and without the wrist rotation module prototype. The study contained muscle synergy analysis through non-negative matrix factorization (NMF) using surface electromyography (sEMG) and motion analyses employing a motion capture system during the reach-to-grasp task. Additionally, we assessed the usability of the prototype design for partial hand amputees using the Jebsen-Taylor hand function test (JHFT).

RESULTS

The results showed that the number of muscle synergies identified through NMF remained consistent at 3 for both the control group and amputees using a hand prosthesis with a wrist rotation module. In the motion analysis, a statistically significant difference was observed between the control group and the prosthetic hand without the wrist rotation module, indicating the presence of compensatory movements when utilizing a prosthetic hand lacking this module. Furthermore, among the amputees, the JHFT demonstrated a greater improvement in total score when using the prosthetic hand equipped with a wrist rotation module compared to the prosthetic hand without this module.

CONCLUSION

In conclusion, integrating a wrist rotation module in prosthetic hand designs for partial hand amputees restores natural upper limb movement patterns, reduces compensatory movements, and prevent the secondary musculoskeletal. This highlights the importance of this module in enhancing overall functionality and quality of life.

Exploring Muscle Synergies for Performance Enhancement and Learning in Myoelectric Control Maps

This work proposes two myoelectric control maps based on a DoF-wise synergy algorithm, inspired by human motor control studies. One map, called intuitive, matches control outputs with body movement directions. The second one, named non-intuitive, takes advantage of different synergies contribution to each DoF, without specific correlation to body movement directions. The effectiveness and learning process for the two maps is evaluated through performance metrics in ten able-bodied individuals. The analysis was conducted using a 2-DoFs center-reach-out task and a survey. Results showed equivalent performance and perception for both mappings. However, learning is only visible in subjects that performed better in non-intuitive mapping, that required some familiarization to then exploit its features. Most of the myoelectric control designs use intuitive mappings. Nevertheless, non-intuitive mapping could provide more design flexibility, which can be especially interesting for patients with motor disabilities.

Developing new intermuscular coordination patterns through an electromyographic signal-guided training in the upper extremity

BACKGROUND

Muscle synergies, computationally identified intermuscular coordination patterns, have been utilized to characterize neuromuscular control and learning in humans. However, it is unclear whether it is possible to alter the existing muscle synergies or develop new ones in an intended way through a relatively short-term motor exercise in adulthood. This study aimed to test the feasibility of expanding the repertoire of intermuscular coordination patterns through an isometric, electromyographic (EMG) signal-guided exercise in the upper extremity (UE) of neurologically intact individuals.

METHODS

10 participants were trained for six weeks to induce independent control of activating a pair of elbow flexor muscles that tended to be naturally co-activated in force generation. An untrained isometric force generation task was performed to assess the effect of the training on the intermuscular coordination of the trained UE. We applied a non-negative matrix factorization on the EMG signals recorded from 12 major UE muscles during the assessment to identify the muscle synergies. In addition, the performance of training tasks and the characteristics of individual muscles' activity in both time and frequency domains were quantified as the training outcomes.

RESULTS

Typically, in two weeks of the training, participants could use newly developed muscle synergies when requested to perform new, untrained motor tasks by activating their UE muscles in the trained way. Meanwhile, their habitually expressed muscle synergies, the synergistic muscle activation groups that were used before the training, were conserved throughout the entire training period. The number of muscle synergies activated for the task performance remained the same. As the new muscle synergies were developed, the neuromotor control of the trained muscles reflected in the metrics, such as the ratio between the targeted muscles, number of matched targets, and task completion time, was improved.

CONCLUSION

These findings suggest that our protocol can increase the repertoire of readily available muscle synergies and improve motor control by developing the activation of new muscle coordination patterns in healthy adults within a relatively short period. Furthermore, the study shows the potential of the isometric EMG-guided protocol as a neurorehabilitation tool for aiming motor deficits induced by abnormal intermuscular coordination after neurological disorders.

TRIAL REGISTRATION

This study was registered at the Clinical Research Information Service (CRiS) of the Korea National Institute of Health (KCT0005803) on 1/22/2021.

Gait Event Prediction Using Surface Electromyography in Parkinsonian Patients

Gait disturbances are common manifestations of Parkinson's disease (PD), with unmet therapeutic needs. Inertial measurement units (IMUs) are capable of monitoring gait, but they lack neurophysiological information that may be crucial for studying gait disturbances in these patients. Here, we present a machine learning approach to approximate IMU angular velocity profiles and subsequently gait events using electromyographic (EMG) channels during overground walking in patients with PD. We recorded six parkinsonian patients while they walked for at least three minutes. Patient-agnostic regression models were trained on temporally embedded EMG time series of different combinations of up to five leg muscles bilaterally (i.e., tibialis anterior, soleus, gastrocnemius medialis, gastrocnemius lateralis, and vastus lateralis). Gait events could be detected with high temporal precision (median displacement of <50 ms), low numbers of missed events (<2%), and next to no false-positive event detections (<0.1%). Swing and stance phases could thus be determined with high fidelity (median F1-score of ~0.9). Interestingly, the best performance was obtained using as few as two EMG probes placed on the left and right vastus lateralis. Our results demonstrate the practical utility of the proposed EMG-based system for gait event prediction, which allows the simultaneous acquisition of an electromyographic signal to be performed. This gait analysis approach has the potential to make additional measurement devices such as IMUs and force plates less essential, thereby reducing financial and preparation overheads and discomfort factors in gait studies.

A Proposal of Bioinspired Soft Active Hand Prosthesis

Soft robotics have broken the rigid wall of interaction between humans and robots due to their own definition and manufacturing principles, allowing robotic systems to adapt to humans and enhance or restore their capabilities. In this research we propose a dexterous bioinspired soft active hand prosthesis based in the skeletal architecture of the human hand. The design includes the imitation of the musculoskeletal components and morphology of the human hand, allowing the prosthesis to emulate the biomechanical properties of the hand, which results in better grips and a natural design. CAD models for each of the bones were developed and 3D printing was used to manufacture the skeletal structure of the prosthesis, also soft materials were used for the musculoskeletal components. A myoelectric control system was developed using a recurrent neural network (RNN) to classify the hand gestures using electromyography signals; the RNN model achieved an accuracy of 87% during real time testing. Objects with different size, texture and shape were tested to validate the grasping performance of the prosthesis, showing good adaptability, soft grasping and mechanical compliance to object of the daily life.

A Co-driven Functional Electrical Stimulation Control Strategy by Dynamic Surface Electromyography and Joint Angle

Functional electrical stimulation (FES) is widely used in neurorehabilitation to improve patients’ motion ability. It has been verified to promote neural remodeling and relearning, during which FES has to produce an accurate movement to obtain a good efficacy. Therefore, many studies have focused on the relationship between FES parameters and the generated movements. However, most of the relationships have been established in static contractions, which leads to an unsatisfactory result when applied to dynamic conditions. Therefore, this study proposed a FES control strategy based on the surface electromyography (sEMG) and kinematic information during dynamic contractions. The pulse width (PW) of FES was determined by a direct transfer function (DTF) with sEMG features and joint angles as the input. The DTF was established by combing the polynomial transfer functions of sEMG and joint torque and the polynomial transfer functions of joint torque and FES. Moreover, the PW of two FES channels was set based on the muscle synergy ratio obtained through sEMG. A total of six healthy right-handed subjects were recruited in this experiment to verify the validity of the strategy. The PW of FES applied to the left arm was evaluated based on the sEMG of the right extensor carpi radialis (ECR) and the right wrist angle. The coefficient of determination (R²) and the normalized root mean square error (NRMSE) of FES-included and voluntary wrist angles and torques were used to verify the performance of the strategy. The result showed that this study achieved a high accuracy (R² = 0.965 and NRMSE = 0.047) of joint angle and a good accuracy (R² = 0.701 and NRMSE = 0.241) of joint torque reproduction during dynamic movements. Moreover, the DTF in real-time FES system also had a nice performance of joint angle fitting (R² = 0.940 and NRMSE = 0.071) and joint torque fitting (R² = 0.607 and NRMSE = 0.303). It is concluded that the proposed strategy is able to generate proper FES parameters based on sEMG and kinematic information for dynamic movement reproduction and can be used in a real-time FES system combined with bilateral movements for better rehabilitation.

Unsupervised Myocontrol of a Virtual Hand Based on a Coadaptive Abstract Motor Mapping

Applications of simultaneous and proportional control for upper-limb prostheses typically rely on supervised machine learning to map muscle activations to prosthesis movements. This scheme often poses problems for individuals with limb differences, as they may not be able to reliably reproduce the training activations required to construct a natural motor mapping. We propose an unsupervised myocontrol paradigm that eliminates the need for labeled data by mapping the most salient muscle synergies in arbitrary order to a number of predefined prosthesis actions. The paradigm is coadaptive, in the sense that while the user learns to control the system via interaction, the system continually refines the identification of the user's muscular synergies. Our evaluation consisted of eight subjects without limb-loss performing target achievement control tasks of four actions of the hand and wrist. The subjects achieved comparable performance using the proposed unsupervised myocontrol paradigm and a supervised benchmark method, despite reporting increased mental load with the former.

Learning Non-Euclidean Representations With SPD Manifold for Myoelectric Pattern Recognition

How to learn informative representations from Electromyography (EMG) signals is of vital importance for myoelectric control systems. Traditionally, hand-crafted features are extracted from individual EMG channels and combined together for pattern recognition. The spatial topological information between different channels can also be informative, which is seldom considered. This paper presents a radically novel approach to extract spatial structural information within diverse EMG channels based on the symmetric positive definite (SPD) manifold. The object is to learn non-Euclidean representations inside EMG signals for myoelectric pattern recognition. The performance is compared with two classical feature sets using accuracy and F1-score. The algorithm is tested on eleven gestures collected from ten subjects, and the best accuracy reaches 84.85%±5.15% with an improvement of 4.04%~20.25%, which outperforms the contrast method, and reaches a significant improvement with the Wilcoxon signed-rank test. Eleven gestures from three public databases involving Ninapro DB2, DB4, and DB5 are also evaluated, and better performance is observed. Furthermore, the computational cost is less than the contrast method, making it more suitable for low-cost systems. It shows the effectiveness of the presented approach and contributes a new way for myoelectric pattern recognition.

Real-Time Control of a Multi-Degree-of-Freedom Mirror Myoelectric Interface During Functional Task Training

Motor learning mediated by motor training has in the past been explored for rehabilitation. Myoelectric interfaces together with exoskeletons allow patients to receive real-time feedback about their muscle activity. However, the number of degrees of freedom that can be simultaneously controlled is limited, which hinders the training of functional tasks and the effectiveness of the rehabilitation therapy. The objective of this study was to develop a myoelectric interface that would allow multi-degree-of-freedom control of an exoskeleton involving arm, wrist and hand joints, with an eye toward rehabilitation. We tested the effectiveness of a myoelectric decoder trained with data from one upper limb and mirrored to control a multi-degree-of-freedom exoskeleton with the opposite upper limb (i.e., mirror myoelectric interface) in 10 healthy participants. We demonstrated successful simultaneous control of multiple upper-limb joints by all participants. We showed evidence that subjects learned the mirror myoelectric model within the span of a five-session experiment, as reflected by a significant decrease in the time to execute trials and in the number of failed trials. These results are the necessary precursor to evaluating if a decoder trained with EMG from the healthy limb could foster learning of natural EMG patterns and lead to motor rehabilitation in stroke patients.

Co-adaptive control of bionic limbs via unsupervised adaptation of muscle synergies

OBJECTIVE

In this work, we present a myoelectric interface that extracts natural motor synergies from multi-muscle signals and adapts in real-time with new user inputs. With this unsupervised adaptive myocontrol (UAM) system, optimal synergies for control are continuously co-adapted with changes in user motor control, or as a function of perturbed conditions via online non-negative matrix factorization guided by physiologically informed sparseness constraints in lieu of explicit data labelling.

METHODS

UAM was tested in a set of virtual target reaching tasks completed by able-bodied and amputee subjects. Tests were conducted under normative and electrode perturbed conditions to gauge control robustness with comparisons to non-adaptive and supervised adaptive myocontrol schemes. Furthermore, UAM was used to interface an amputee with a multi-functional powered hand prosthesis during standardized Clothespin Relocation Tests, also conducted in normative and perturbed conditions.

RESULTS

In virtual tests, UAM effectively mitigated performance degradation caused by electrode displacement, affording greater resilience over an existing supervised adaptive system for amputee subjects. Induced electrode shifts also had negligible effect on the real world control performance of UAM with consistent completion times (23.91±1.33 s) achieved across Clothespin Relocation Tests in the normative and electrode perturbed conditions.

CONCLUSION

UAM affords comparable robustness improvements to existing supervised adaptive myocontrol interfaces whilst providing additional practical advantages for clinical deployment.

SIGNIFICANCE

The proposed system uniquely incorporates neuromuscular control principles with unsupervised online learning methods and presents a working example of a freely co-adaptive bionic interface.

Rejecting Impulse Artifacts from Surface EMG Signals using Real-time Cumulative Histogram Filtering

This paper presents a cumulative histogram filtering (CHF) algorithm to filter impulsive artifacts within surface electromyograhy (sEMG) signal for time-domain signal feature extraction. The proposed CHF algorithm filters sEMG signals by extracting a continuous subset of amplitude-sorted values within a real-time window of measured samples using information about the probabilistic distribution of sEMG amplitude. For real-time deployment of the proposed CHF algorithm on an embedded computing platform, we also present an efficient, iterative implementation of the proposed algorithm. The proposed CHF algorithm was evaluated on synthetic impulse artifacts superimposed upon undisturbed sEMG recorded from a subject with transtibial amputation. Results suggest that the CHF algorithm effectively suppresses the simulated impulse artifacts while preserving a minimum signal-to-noise ratio of 95% and an average Pearson correlation of 0.99 compared to the undisturbed sEMG recordings.

Upper-Limb Electromyogram Classification of Reaching-to-Grasping Tasks Based on Convolutional Neural Networks for Control of a Prosthetic Hand

In recent years, myoelectric interfaces using surface electromyogram (EMG) signals have been developed for assisting people with physical disabilities. Especially, in the myoelectric interfaces for robotic hands or arms, decoding the user’s upper-limb movement intentions is cardinal to properly control the prosthesis. However, because previous experiments were implemented with only healthy subjects, the possibility of classifying reaching-to-grasping based on the EMG signals from the residual limb without the below-elbow muscles was not investigated yet. Therefore, we aimed to investigate the possibility of classifying reaching-to-grasping tasks using the EMG from the upper arm and upper body without considering wrist muscles for prosthetic users. In our study, seven healthy subjects, one trans-radial amputee, and one wrist amputee were participated and performed 10 repeatable 12 reaching-to-grasping tasks based on the Southampton Hand Assessment Procedure (SHAP) with 12 different weighted (light and heavy) objects. The acquired EMG was processed using the principal component analysis (PCA) and convolutional neural network (CNN) to decode the tasks. The PCA–CNN method showed that the average accuracies of the healthy subjects were 69.4 ± 11.4%, using only the EMG signals by the upper arm and upper body. The result with the PCA–CNN method showed 8% significantly higher accuracies than the result with the widely used time domain and auto-regressive-support vector machine (TDAR–SVM) method as 61.6 ± 13.7%. However, in the cases of the amputees, the PCA–CNN showed slightly lower performance. In addition, in the aspects of assistant daily living, because grip force is also important when grasping an object after reaching, the possibility of classifying the two light and heavy objects in each reaching-to-grasping task was also investigated. Consequently, the PCA–CNN method showed higher accuracy at 70.1 ± 9.8%. Based on our results, the PCA–CNN method can help to improve the performance of classifying reaching-to-grasping tasks without wrist EMG signals. Our findings and decoding method can be implemented to further develop a practical human–machine interface using EMG signals.

An Improvement of Robot Stiffness-Adaptive Skill Primitive Generalization Using the Surface Electromyography in Human-Robot Collaboration

Learning from Demonstration in robotics has proved its efficiency in robot skill learning. The generalization goals of most skill expression models in real scenarios are specified by humans or associated with other perceptual data. Our proposed framework using the Probabilistic Movement Primitives (ProMPs) modeling to resolve the shortcomings of the previous research works; the coupling between stiffness and motion is inherently established in a single model. Such a framework can request a small amount of incomplete observation data to infer the entire skill primitive. It can be used as an intuitive generalization command sending tool to achieve collaboration between humans and robots with human-like stiffness modulation strategies on either side. Experiments (human–robot hand-over, object matching, pick-and-place) were conducted to prove the effectiveness of the work. Myo armband and Leap motion camera are used as surface electromyography (sEMG) signal and motion capture sensors respective in the experiments. Also, the experiments show that the proposed framework strengthened the ability to distinguish actions with similar movements under observation noise by introducing the sEMG signal into the ProMP model. The usage of the mixture model brings possibilities in achieving automation of multiple collaborative tasks.

Rethinking Autonomous Surgery: Focusing on Enhancement over Autonomy

CONTEXT

As robot-assisted surgery is increasingly used in surgical care, the engineering research effort towards surgical automation has also increased significantly. Automation promises to enhance surgical outcomes, offload mundane or repetitive tasks, and improve workflow. However, we must ask an important question: should autonomous surgery be our long-term goal?

OBJECTIVE

To provide an overview of the engineering requirements for automating control systems, summarize technical challenges in automated robotic surgery, and review sensing and modeling techniques to capture real-time human behaviors for integration into the robotic control loop for enhanced shared or collaborative control.

EVIDENCE ACQUISITION

We performed a nonsystematic search of the English language literature up to March 25, 2021. We included original studies related to automation in robot-assisted laparoscopic surgery and human-centered sensing and modeling.

EVIDENCE SYNTHESIS

We identified four comprehensive review papers that present techniques for automating portions of surgical tasks. Sixteen studies relate to human-centered sensing technologies and 23 to computer vision and/or advanced artificial intelligence or machine learning methods for skill assessment. Twenty-two studies evaluate or review the role of haptic or adaptive guidance during some learning task, with only a few applied to robotic surgery. Finally, only three studies discuss the role of some form of training in patient outcomes and none evaluated the effects of full or semi-autonomy on patient outcomes.

CONCLUSIONS

Rather than focusing on autonomy, which eliminates the surgeon from the loop, research centered on more fully understanding the surgeon's behaviors, goals, and limitations could facilitate a superior class of collaborative surgical robots that could be more effective and intelligent than automation alone.

PATIENT SUMMARY

We reviewed the literature for studies on automation in surgical robotics and on modeling of human behavior in human-machine interaction. The main application is to enhance the ability of surgical robotic systems to collaborate more effectively and intelligently with human surgeon operators.

Brain-machine and muscle-machine bio-sensing methods for gesture intent acquisition in upper-limb prosthesis control: a review

This paper presents a review of a number of bio-sensing methods for gesture intent signal acquisition in control tasks for upper-limb prosthesis. The paper specifically provides a breakdown of the control task in myoelectric prosthesis, and in addition, highlights and describes the importance of the acquisition of a high-quality bio-signal. The paper also describes commonly used invasive and non-invasive brain and muscle machine interfaces such as electroencephalography, electrocorticography, electroneurography, surface electromyography, sonomyography, mechanomyography, near infra-red, force sensitive resistance/pressure, and magnetoencephalography. Each modality is reviewed based on its operating principle and limitations in gesture recognition, followed by respective advantages and disadvantages. Also described within this paper, are multimodal sensing approaches, which involve data fusion of information from various sensing modalities for an enhanced neuromuscular bio-sensing source. Using a semi-systematic review methodology, we are able to derive a novel tabular approach towards contrasting the various strengths and weaknesses of the reviewed bio-sensing methods towards gesture recognition in a prosthesis interface. This would allow for a streamlined method of down selection of an appropriate bio-sensor given specific prosthesis design criteria and requirements. The paper concludes by highlighting a number of research areas that require more work for strides to be made towards improving and enhancing the connection between man and machine as it concerns upper-limb prosthesis. Such areas include classifier augmentation for gesture recognition, filtering techniques for sensor disturbance rejection, feeling of tactile sensations with an artificial limb.

Dimensionality analysis of forearm muscle activation for myoelectric control in transradial amputees

Establishing a natural communication interface between the user and the terminal device is one of the central challenges of hand neuroprosthetics research. Surface electromyography (EMG) is the most common source of neural signals for interpreting a user’s intent in these interfaces. However, how the capacity of EMG generation is affected by various clinical parameters remains largely unknown. In this study, we examined the EMG activity of forearm muscles recorded from 11 transradially amputated subjects who performed a wide range of movements. EMG recordings from 40 able-bodied subjects were also analyzed to provide comparative benchmarks. By using non-negative matrix factorization, we extracted the synergistic EMG patterns for each subject to estimate the dimensionality of muscle control, under the framework of motor synergies. We found that amputees exhibited less than four synergies (with substantial variability related to the length of remaining limb and age), whereas able-bodied subjects commonly demonstrate five or more synergies. The results of this study provide novel insight into the muscle synergy framework and the design of natural myoelectric control interfaces.

A Muscle Synergy Framework for Cross-Limb Reconstruction of Hand Muscle Activity Distal to a Virtual Wrist-Level Disarticulation

Currently, myoelectric prostheses lack dexterity and ease of control, in part because of inadequate schemes to extract relevant muscle features that can approximate muscle activation patterns that enable individuated dexterous finger motion. This project seeks to apply a novel algorithm pipeline that extracts muscle activation patterns from one limb, as well as from forearm muscles of the opposite limb, to predict muscle activation data of opposite limb intrinsic hand muscles, with the long-range goal of informing dexterous prosthetic control.

Automatic Myoelectric Control Site Detection Using Candid Covariance-Free Incremental Principal Component Analysis

The unknown composition of residual muscles surrounding the stump of an amputee makes optimal electrode placement challenging. This often causes the experimental set-up and calibration of upper-limb prostheses to be time consuming. In this work, we propose the use of existing dimensionality reduction techniques, typically used for muscle synergy analysis, to provide meaningful real-time functional information of the residual muscles during the calibration period. Two variations of principal component analysis (PCA) were applied to electromyography (EMG) data collected during a myoelectric task. Candid covariance-free incremental PCA (CCIPCA) detected task-specific muscle synergies with high accuracy using minimal amounts of data. Our findings offer a real-time solution towards optimizing calibration periods.

Task-Oriented Muscle Synergy Extraction Using An Autoencoder-Based Neural Model

The growing interest in wearable robots opens the challenge for developing intuitive and natural control strategies. Among several human–machine interaction approaches, myoelectric control consists of decoding the motor intention from muscular activity (or EMG signals) with the aim of driving prosthetic or assistive robotic devices accordingly, thus establishing an intimate human–machine connection. In this scenario, bio-inspired approaches, e.g., synergy-based controllers, are revealed to be the most robust. However, synergy-based myo-controllers already proposed in the literature consider muscle patterns that are computed considering only the total variance reconstruction rate of the EMG signals, without taking into account the performance of the controller in the task (or application) space. In this work, extending a previous study, the authors presented an autoencoder-based neural model able to extract muscles synergies for motion intention detection while optimizing the task performance in terms of force/moment reconstruction. The proposed neural topology has been validated with EMG signals acquired from the main upper limb muscles during planar isometric reaching tasks performed in a virtual environment while wearing an exoskeleton. The presented model has been compared with the non-negative matrix factorization algorithm (i.e., the most used approach in the literature) in terms of muscle synergy extraction quality, and with three techniques already presented in the literature in terms of goodness of shoulder and elbow predicted moments. The results of the experimental comparisons have showed that the proposed model outperforms the state-of-art synergy-based joint moment estimators at the expense of the quality of the EMG signals reconstruction. These findings demonstrate that a trade-off, between the capability of the extracted muscle synergies to better describe the EMG signals variability and the task performance in terms of force reconstruction, can be achieved. The results of this study might open new horizons on synergies extraction methodologies, optimized synergy-based myo-controllers and, perhaps, reveals useful hints about their origin.

A Review of Robot-Assisted Lower-Limb Stroke Therapy: Unexplored Paths and Future Directions in Gait Rehabilitation

Stroke affects one out of every six people on Earth. Approximately 90% of stroke survivors have some functional disability with mobility being a major impairment, which not only affects important daily activities but also increases the likelihood of falling. Originally intended to supplement traditional post-stroke gait rehabilitation, robotic systems have gained remarkable attention in recent years as a tool to decrease the strain on physical therapists while increasing the precision and repeatability of the therapy. While some of the current methods for robot-assisted rehabilitation have had many positive and promising outcomes, there is moderate evidence of improvement in walking and motor recovery using robotic devices compared to traditional practice. In order to better understand how and where robot-assisted rehabilitation has been effective, it is imperative to identify the main schools of thought that have prevailed. This review intends to observe those perspectives through three different lenses: the goal and type of interaction, the physical implementation, and the sensorimotor pathways targeted by robotic devices. The ways that researchers approach the problem of restoring gait function are grouped together in an intuitive way. Seeing robot-assisted rehabilitation in this unique light can naturally provoke the development of new directions to potentially fill the current research gaps and eventually discover more effective ways to provide therapy. In particular, the idea of utilizing the human inter-limb coordination mechanisms is brought up as an especially promising area for rehabilitation and is extensively discussed.

Modulation of muscle synergies for multiple forearm movements under variant force and arm position constraints

OBJECTIVE

To promote clinical applications of muscle-synergy-based neurorehabilitation techniques, this study aims to clarify any potential modulations of both the muscular compositions and temporal activations of forearm muscle synergies for multiple movements under variant force levels and arm positions.

APPROACH

Two groups of healthy subjects participated in this study. Electromyography (EMG) signals were collected when they performed four hand and wrist movements under variant constraints-three different force levels for one group and five arm positions for the other. Muscle synergies were extracted from the EMGs, and their robustness across variant force levels and arm positions was separately assessed by evaluating their across-condition structure similarity, cross-validation, and cluster analysis. The synergies' activation coefficients across the variant constraints were also compared, and the coefficients were used to discriminate the different force levels and the arm positions, respectively.

MAIN RESULTS

Overall, the muscle synergies were relatively fixed across variant constraints, but they were more robust to variant forces than to changing arm positions. The activations of muscle synergies depended largely on the level of contraction force and could discriminate the force levels very well, but the coefficients corresponding to different arm positions discriminated the positions with lower accuracy. Similar results were found for all types of forearm movement analyzed.

SIGNIFICANCE

With our experiment and subject-specific analysis, only slight modulation of the muscular compositions of forearm muscle synergies was found under variant force and arm position constraints. Our results may shed valuable insights to future design of both muscle-synergy-based assistive robots and motor-function assessments.

Human-in-the-Loop Assessment of an Ultralight, Low-Cost Body Posture Tracking Device

In rehabilitation, assistive and space robotics, the capability to track the body posture of a user in real time is highly desirable. In more specific cases, such as teleoperated extra-vehicular activity, prosthetics and home service robotics, the ideal posture-tracking device must also be wearable, light and low-power, while still enforcing the best possible accuracy. Additionally, the device must be targeted at effective human-machine interaction. In this paper, we present and test such a device based upon commercial inertial measurement units: it weighs 575 grams in total, lasts up to 10.5 hours of continual operation, can be donned and doffed in under a minute and costs less than 290 EUR. We assess the attainable performance in terms of error in an online trajectory-tracking task in Virtual Reality using the device through an experiment involving 10 subjects, showing that an average user can attain a precision of 0.66 cm during a static precision task and 6.33 cm while tracking a moving trajectory, when tested in the full peri-personal space of a user.

An Approach to Extract Nonlinear Muscle Synergies from sEMG through Multi-Model Learning

How does the Central Nervous System (CNS) controls a group of muscles is an important question in the field of motor control. A common conception is developed over the years that the CNS make use of predefined activation patterns, known as muscle synergies during task execution. These muscle synergies are extracted by applying any of the factorization algorithms such as Non-Negative Matrix Factorization (NNMF), Independent Component Analysis (ICA) or Principle Component Analysis (PCA) on a concatenated surface EMG data set recorded from the target muscles. However, the step to concatenate sEMG signals before they are given as input to these linear algorithm is crucial as the synergistic structure changes significantly based on the number and choice of muscles considered during concatenation step. To address this problem, we propose a new approach of extracting muscle synergies by treating sEMG signals from each muscle as an individual modality and then learning the synergistic structure among them if it exists using multi-view learning. In this study, we propose to use Manifold Relevance Determination (MRD) to find nonlinear synergies from sEMG by assuming the sEMG of a muscle as an individual modality. Results have shown that synergistic patterns extracted using our approach are consistent upon addition of sEMG signals from new muscles.

Extraction of Nonlinear Synergies for Proportional and Simultaneous Estimation of Finger Kinematics

OBJECTIVE

Proportional and simultaneous est-imation of finger kinematics from surface EMG based on the assumption that there exists a correlation between muscle activations and finger kinematics in low dimensional space.

METHODS

We employ Manifold Relevance Determination (MRD), a multi-view learning model with a nonparametric Bayesian approach, to extract the nonlinear muscle and kinematics synergies and the relationship between them by studying muscle activations (input-space) together with the finger kinematics (output-space).

RESULTS

This study finds that there exist muscle synergies which are associated with kinematic synergies. The acquired nonlinear synergies and the association between them has further been utilized for the estimation of finger kinematics from muscle activation inputs, and the proposed approach has outperformed other commonly used linear and nonlinear regression approaches with an average correlation coefficient of 0.91±0.03.

CONCLUSION

There exists an association between muscle and kinematic synergies which can be used for the proportional and simultaneous estimation of finger kinematics from the muscle activation inputs.

SIGNIFICANCE

The findings of this study not only presents a viable approach for accurate and intuitive myoelectric control but also provides a new perspective on the muscle synergies in the motor control community.

Selection of Features and Classifiers for EMG-EEG-Based Upper Limb Assistive Devices-A Review

Bio-signals are distinctive factors in the design of human-machine interface, essentially useful for prosthesis, orthosis, and exoskeletons. Despite the progress in the analysis of pattern recognition based devices; the acceptance of these devices is still questionable. One reason is the lack of information to identify the possible combinations of features and classifiers. Besides; there is also a need for optimal selection of various sensors for sensations such as touch, force, texture, along with EMGs/EEGs. This article reviews the two bio-signal techniques, named as electromyography and electroencephalography. The details of the features and the classifiers used in the data processing for upper limb assist devices are summarised here. Various features and their sets are surveyed and different classifiers for feature sets are discussed on the basis of the classification rate. The review was carried out on the basis of the last 10-12 years of published research in this area. This article also outlines the influence of modality of EMGs and EEGs with other sensors on classifications. Also, other bio-signals used in upper limb devices and future aspects are considered.

A Subject-Transfer Framework Based on Single-Trial EMG Analysis Using Convolutional Neural Networks

In recent years, electromyography (EMG)-based practical myoelectric interfaces have been developed to improve the quality of daily life for people with physical disabilities. With these interfaces, it is very important to decode a user's movement intention, to properly control the external devices. However, improving the performance of these interfaces is difficult due to the high variations in the EMG signal patterns caused by intra-user variability. Therefore, this paper proposes a novel subject-transfer framework for decoding hand movements, which is robust in terms of intra-user variability. In the proposed framework, supportive convolutional neural network (CNN) classifiers, which are pre-trained using the EMG data of several subjects, are selected and fine-tuned for the target subject via single-trial analysis. Then, the target subject's hand movements are classified by voting the outputs of the supportive CNN classifiers. The feasibility of the proposed framework is validated with NinaPro databases 2 and 3, which comprise 49 hand movements of 40 healthy and 11 amputee subjects, respectively. The experimental results indicate that, when compared to the self-decoding framework, which uses only the target subject's data, the proposed framework can successfully decode hand movements with improved performance in both healthy and amputee subjects. From the experimental results, the proposed subject-transfer framework can be seen to represent a useful tool for EMG-based practical myoelectric interfaces controlling external devices.

Human motor decoding from neural signals: a review

Many people suffer from movement disability due to amputation or neurological diseases. Fortunately, with modern neurotechnology now it is possible to intercept motor control signals at various points along the neural transduction pathway and use that to drive external devices for communication or control. Here we will review the latest developments in human motor decoding. We reviewed the various strategies to decode motor intention from human and their respective advantages and challenges. Neural control signals can be intercepted at various points in the neural signal transduction pathway, including the brain (electroencephalography, electrocorticography, intracortical recordings), the nerves (peripheral nerve recordings) and the muscles (electromyography). We systematically discussed the sites of signal acquisition, available neural features, signal processing techniques and decoding algorithms in each of these potential interception points. Examples of applications and the current state-of-the-art performance were also reviewed. Although great strides have been made in human motor decoding, we are still far away from achieving naturalistic and dexterous control like our native limbs. Concerted efforts from material scientists, electrical engineers, and healthcare professionals are needed to further advance the field and make the technology widely available in clinical use.

Automated Instability Detection for Interactive Myocontrol of Prosthetic Hands

Myocontrol is control of a prosthetic device using data obtained from (residual) muscle activity. In most myocontrol prosthetic systems, such biological data also denote the subject's intent: reliably interpreting what the user wants to do, exactly and only when she wants, is paramount to avoid instability, which can potentially lead to accidents, humiliation and trauma. Indeed, instability manifests itself as a failure of the myocontrol in interpreting the subject's intent, and the automated detection of such failures can be a specific step to improve myocontrol of prostheses—e.g., enabling the possibility of self-adaptation of the system via on-demand model updates for incremental learning, i.e., the interactive myocontrol paradigm. In this work we engaged six expert myocontrol users (five able-bodied subjects and one trans-radial amputee) in a simple, clear grasp-carry-release task, in which the subject's intent was reasonably determined by the task itself. We then manually ascertained when the intent would not coincide with the behavior of the prosthetic device, i.e., we labeled the failures of the myocontrol system. Lastly, we trained and tested a classifier to automatically detect such failures. Our results show that a standard classifier is able to detect myocontrol failures with a mean balanced error rate of 18.86% over all subjects. If confirmed in the large, this approach could pave the way to self-detection and correction of myocontrol errors, a tighter man-machine co-adaptation, and in the end the improvement of the reliability of myocontrol.

EMG-Based Real-Time Linear-Nonlinear Cascade Regression Decoding of Shoulder, Elbow, and Wrist Movements in Able-Bodied Persons and Stroke Survivors

OBJECTIVE

This study aimed to decode shoulder, elbow and wrist dynamic movements continuously and simultaneously based on multi-channel surface electromyography signals, useful for electromyography controlled exoskeleton robots for upper-limb rehabilitation.

METHODS

Ten able-bodied subjects and ten stroke subjects were instructed to voluntarily move the shoulder, elbow and wrist joints back and forth in a horizontal plane with an exoskeleton robot. The shoulder, elbow and wrist movements and surface electromyography signals from six muscles crossing the joints were recorded. A set of three parallel linear-nonlinear cascade decoders was developed to continuously estimate the selected shoulder, elbow and wrist movements based on a generalized linear model using the anterior deltoid, posterior deltoid, biceps brachii, long head triceps brachii, flexor carpi radialis, and extensor carpi radialis muscle electromyography signals as the model inputs.

RESULTS

The decoder performed well for both healthy and stroke populations. As movement smoothness decreased, decoding performance decreased for the stroke population.

CONCLUSION

The proposed method is capable of simultaneously and continuously estimating multi-joint movements of the human arm in real-time by characterizing the nonlinear mappings between muscle activity and kinematic signals based on linear regression.

SIGNIFICANCE

This may prove useful in developing myoelectric controlled exoskeletons for motor rehabilitation of neurological disorders.

Characterizing Intent Changes in Exoskeleton-Assisted Walking Through Onboard Sensors

Robotic exoskeletons are a promising technology for rehabilitation and locomotion following musculoskeletal injury, but their adoption outside the physical therapy clinic has been limited by relatively primitive methods for identifying and incorporating the user's gait intentions. Various intent detection approaches have been demonstrated using electromyography and electroencephalography signals. These technologies sense the human directly but introduce complications for donning/doffing the device and in measurement consistency. By contrast, sensors onboard the exoskeleton avoid these complications but sense the human indirectly via the human-robot interface. This pilot study examines if onboard sensors alone may enable identification of user intent. Joint positions and commanded motor currents are compared prior to and after changes in the user's intended gait speed. Preliminary experimental results confirm that these measures are significantly different following intent changes for both able-bodied and non-able-bodied users. The findings suggest that intent detection is possible with onboard sensors alone, but the intent signals depend on exoskeleton control settings, user ability, and temporal considerations.

Synergy-Based Myocontrol of a Multiple Degree-of-Freedom Humanoid Robot for Functional Tasks

In the context of sensor-based human-robot interaction, a particularly promising solution is represented by myoelectric control schemes based on synergy-derived signals. We developed and tested on healthy subjects a synergy-based control to achieve simultaneous, continuous actuation of three degrees of freedom of a humanoid robot, while performing functional reach-to-grasp movements. The control scheme exploits subject-specific muscle synergies extracted from eleven upper limb muscles through an easy semi-supervised calibration phase, and computes online activation coefficients to actuate the robot joints. The humanoid robot was able to well reproduce the subjects' motion, which consisted in free multi-degree-of-freedom reach-to-grasp movements at self-paced speeds. Furthermore, the synergy-based online control significantly outperformed a traditional muscle-pair approach (that uses a pair of antagonist muscles for each joint), in terms of decreased error, increased correlation, and peak correlation between the subjects' and the robot's joint angles. The delay introduced by the two algorithms was comparable. This work is a proof-of-concept for an intuitive and robust myocontrol interface, without the need for any training and practice. It has several potential applications, especially for functional assistive engaging devices in children with social and motor impairments.

Upper Limb Prosthesis Control for High-Level Amputees via Myoelectric Recognition of Leg Gestures

Recognition of motion intent via surface electromyography (EMG) has become increasingly practical for prosthesis control, but lacking residual muscle sites remains a major obstacle to its use by high-level amputees. Currently, there are few approaches to upper limb prosthesis control for individuals with amputations proximal to the elbow, all of which suffer from one or more of three primary problems: invasiveness, the need for intensive training, and lacking functionality. Using surface EMG sensors placed on the lower leg and a natural mapping between degrees of freedom of the leg and the arm, we tested a noninvasive control approach by which high-level amputees could control prosthetic elbow, wrist, and hand movements with minimal training. In this paper, we used able-bodied subjects to facilitate a direct comparison between control using intact arm and leg muscles. First, we found that foot gestures could be classified offline using time domain features and linear discriminant analysis with accuracy comparable to an equivalent system for recognizing arm movements. Second, we used the target achievement control test to evaluate real-time control performance in three and four degrees of freedom. After approximately 20 min of training, subjects tended to perform the task as well with the leg as with intact arm muscles, and performance overall was comparable to other control methods.

How are Muscle Synergies Affected by Electromyography Pre-Processing?

Muscle synergies have been used for decades to explain a variety of motor behaviors, both in humans and animals and, more recently, to steer rehabilitation strategies. However, many sources of variability such as factorization algorithms, criteria for dimensionality reduction and data pre-processing constitute a major obstacle to the successful comparison of the results obtained by different research groups. Starting from the canonical EMG processing we determined how variations in filter cut-off frequencies and normalization methods, commonly found in literature, affect synergy weights and inter-subject similarity (ISS) using experimental data related to a 15-muscles upper-limb reaching task. Synergy weights were not significantly altered by either normalization (maximum voluntary contraction - MVC - or maximum amplitude of the signal - SELF) or band-pass filter ([20-500 Hz] or [50-500] Hz). Normalization did, however, alter the amount of variance explained by a set of synergies, which is a criterion often used for model order selection. Comparing different low-pass (LP) filters (0.5 Hz, 4 Hz, 10 Hz, 20 Hz cut-offs) we showed that increasing the low pass filter cut-off had the effect of decreasing the variance accounted for by a set number of synergies and affected individual muscle contributions. Extreme smoothing (i.e., LP cut-off 0.5 Hz) enhanced the contrast between active and inactive muscles but had an unpredictable effect on the ISS. The results presented here constitute a further step towards a thoughtful EMG pre-processing for the extraction of muscle synergies.

Fixed muscle synergies and their potential to improve the intuitive control of myoelectric assistive technology for upper extremities

BACKGROUND

Users of myoelectric controlled assistive technology (AT) for upper extremities experience difficulties in controlling this technology in daily life, partly because the control is non-intuitive. Making the control of myoelectric AT intuitive may resolve the experienced difficulties. The present paper was inspired by the suggestion that intuitive control may be achieved if the control of myoelectric AT is based on neuromotor control principles. A significant approach within neurocomputational motor control suggests that myosignals are produced via a limited number of fixed muscle synergies. To effectively employ this approach in myoelectric AT, it is required that a limited number of muscle synergies is systematically exploited, also when muscles are used differently as required in controlling myoelectric AT. Therefore, the present study examined the systematic exploitation of muscle synergies when muscles were used differently to complete point-to-point movements with and without a rod.

METHODS

Healthy participants made multidirectional point-to-point movements with different end-effectors, i.e. with the index finger and with rods of different lengths. Myosignals were collected from 22 muscles in the arm, trunk, and back, and subsequently partitioned into muscle synergies per end-effector and for a pooled dataset including all end-effectors. The exploitation of these muscle synergies was assessed by evaluating the similarity of structure and explanatory ability of myosignals of per end-effector muscle synergies and the contribution of pooled muscle synergies across end-effectors.

RESULTS

Per end-effector, 3-5 muscle synergies could explain 73.8-81.1% of myosignal variation, whereas 6-8 muscle synergies from the pooled dataset also captured this amount of myosignal variation. Subsequent analyses showed that gradually different muscle synergies-extracted from separate end-effectors-were exploited across end-effectors. In line with this result, the order of contribution of muscle synergies extracted from the pooled dataset gradually reversed across end-effectors.

CONCLUSION

A limited number of muscle synergies was systematically exploited in the examined set of movements, indicating a potential for the fixed muscle synergy approach to improve the intuitive control of myoelectric AT. Given the gradual change in muscle synergy exploitation across end-effectors, future research should examine whether this potential can be extended to a larger range of movements and tasks.

Improving the functionality, robustness, and adaptability of myoelectric control for dexterous motion restoration

The development of advanced and effective human-machine interfaces, especially for amputees to control their prostheses, is very high priority and a very active area of research. An intuitive control method should retain an adequate level of functionality for dexterous operation, provide robustness against confounding factors, and supply adaptability for diverse long-term usage, all of which are current problems being tackled by researchers. This paper reviews the state-of-the-art, as well as, the limitations of current myoelectric signal control (MSC) methods. To address the research topic on functionality, we review different approaches to prosthetic hand control (DOF configuration, discrete or simultaneous, etc.), and how well the control is performed (accuracy, response, intuitiveness, etc.). To address the research on robustness, we review the confounding factors (limb positions, electrode shift, force variance, and inadvertent activity) that affect the stability of the control performance. Lastly, to address adaptability, we review the strategies that can automatically adjust the classifier for different individuals and for long-term usage. This review provides a thorough overview of the current MSC methods and helps highlight the current areas of research focus and resulting clinic usability for the MSC methods for upper-limb prostheses.