Effect of clinical parameters on the control of myoelectric robotic prosthetic hands

EMG gesture signal analysis towards diagnosis of upper limb using dual-pathway convolutional neural network

This research introduces a novel dual-pathway convolutional neural network (DP-CNN) architecture tailored for robust performance in Log-Mel spectrogram image analysis derived from raw multichannel electromyography signals. The primary objective is to assess the effectiveness of the proposed DP-CNN architecture across three datasets (NinaPro DB1, DB2, and DB3), encompassing both able-bodied and amputee subjects. Performance metrics, including accuracy, precision, recall, and F1-score, are employed for comprehensive evaluation. The DP-CNN demonstrates notable mean accuracies of 94.93 ± 1.71% and 94.00 ± 3.65% on NinaPro DB1 and DB2 for healthy subjects, respectively. Additionally, it achieves a robust mean classification accuracy of 85.36 ± 0.82% on amputee subjects in DB3, affirming its efficacy. Comparative analysis with previous methodologies on the same datasets reveals substantial improvements of 28.33%, 26.92%, and 39.09% over the baseline for DB1, DB2, and DB3, respectively. The DP-CNN's superior performance extends to comparisons with transfer learning models for image classification, reaffirming its efficacy. Across diverse datasets involving both able-bodied and amputee subjects, the DP-CNN exhibits enhanced capabilities, holding promise for advancing myoelectric control.

Motion intention recognition of the affected hand based on the sEMG and improved DenseNet network

The key to sEMG (surface electromyography)-based control of robotic hands is the utilization of sEMG signals from the affected hand of amputees to infer their motion intentions. With the advancements in deep learning, researchers have successfully developed viable solutions for CNN (Convolutional Neural Network)-based gesture recognition. However, most studies have primarily concentrated on utilizing sEMG data from the hands of healthy subjects, often relying on high-dimensional feature vectors obtained from a substantial number of electrodes. This approach has yielded high-performing sEMG recognition systems but has failed to consider the considerable inconvenience that the abundance of electrodes poses to the daily lives and work of patients. In this paper, we focused on transradial amputees and used sEMG data from the Ninapro DB3 database as our dataset. Firstly, we introduce a STFT (Short-Time Fourier Transform)-based time-frequency feature fusion map for sEMG. This map includes both time-frequency features and the time-frequency localization of sEMG signals. Secondly, we propose an Improved DenseNet (Dense Convolutional Network) model for recognizing motion intentions in the affected hand of amputees based on their sEMG signals. Finally, addressing the issue of optimizing the number of electrodes carried by amputees, we introduce the PCMIRR (Pearson Correlation and Motion Intention Recognition Rate) algorithm. This algorithm optimizes the number of channels by considering the Pearson correlation between the sEMG channels of amputees and the recognition rate of motion intentions in the affected hand based on single-channel sEMG data. The experimental results reveal that the recognition accuracy, recall, and F1 score achieved by the Improved DenseNet model were 93.82%, 93.61%, and 93.65%, respectively. When the number of electrodes was optimized to 8, the recognition accuracy reached 94.50%. In summary, this paper ultimately attained precise recognition of motion intentions in amputees' affected hands while utilizing the minimum number of sEMG channels. This method offers a novel approach to sEMG-based control of bionic robotic hands.

Deep transfer learning compared to subject-specific models for sEMG decoders

Objective. Accurate decoding of surface electromyography (sEMG) is pivotal for muscle-to-machine-interfaces and their application e.g. rehabilitation therapy. sEMG signals have high inter-subject variability, due to various factors, including skin thickness, body fat percentage, and electrode placement. Deep learning algorithms require long training time and tend to overfit if only few samples are available. In this study, we aim to investigate methods to calibrate deep learning models to a new user when only a limited amount of training data is available.Approach. Two methods are commonly used in the literature, subject-specific modeling and transfer learning. In this study, we investigate the effectiveness of transfer learning using weight initialization for recalibration of two different pretrained deep learning models on new subjects data and compare their performance to subject-specific models. We evaluate two models on three publicly available databases (non invasive adaptive prosthetics database 2-4) and compare the performance of both calibration schemes in terms of accuracy, required training data, and calibration time.Main results. On average over all settings, our transfer learning approach improves 5%-points on the pretrained models without fine-tuning, and 12%-points on the subject-specific models, while being trained for 22% fewer epochs on average. Our results indicate that transfer learning enables faster learning on fewer training samples than user-specific models.Significance. To the best of our knowledge, this is the first comparison of subject-specific modeling and transfer learning. These approaches are ubiquitously used in the field of sEMG decoding. But the lack of comparative studies until now made it difficult for scientists to assess appropriate calibration schemes. Our results guide engineers evaluating similar use cases.

Age-dependent Upper Limb Myoelectric Control Capability in Typically Developing Children

Research in EMG-based control of prostheses has mainly utilized adult subjects who have fully developed neuromuscular control. Little is known about children's ability to generate consistent EMG signals necessary to control artificial limbs with multiple degrees of freedom. As a first step to address this gap, experiments were designed to validate and benchmark two experimental protocols that quantify the ability to coordinate forearm muscle contractions in typically developing children. Able-bodied, healthy adults and children participated in our experiments that aimed to measure an individual's ability to use myoelectric control interfaces. In the first experiment, participants performed 8 repetitions of 16 different hand/wrist movements. Using offline classification analysis based on Support Vector Machine, we quantified their ability to consistently produce distinguishable muscle contraction patterns. We demonstrated that children had a smaller number of highly independent movements (can be classified with >90% accuracy) than adults did. The second experiment measured participants' ability to control the position of a cursor on a 1-DoF virtual slide using proportional EMG control with three different visuomotor gain levels. We found that children had higher failure rates and slower average target acquisitions than adults did, primarily due to longer correction times that did not improve over repetitive practice. We also found that the performance in both experiments was age-dependent in children. The results of this study provide novel insights into the technical and empirical basis to better understand neuromuscular development in children with upper-limb loss.

Dimensionality analysis of forearm muscle activation for myoelectric control in transradial amputees

Establishing a natural communication interface between the user and the terminal device is one of the central challenges of hand neuroprosthetics research. Surface electromyography (EMG) is the most common source of neural signals for interpreting a user’s intent in these interfaces. However, how the capacity of EMG generation is affected by various clinical parameters remains largely unknown. In this study, we examined the EMG activity of forearm muscles recorded from 11 transradially amputated subjects who performed a wide range of movements. EMG recordings from 40 able-bodied subjects were also analyzed to provide comparative benchmarks. By using non-negative matrix factorization, we extracted the synergistic EMG patterns for each subject to estimate the dimensionality of muscle control, under the framework of motor synergies. We found that amputees exhibited less than four synergies (with substantial variability related to the length of remaining limb and age), whereas able-bodied subjects commonly demonstrate five or more synergies. The results of this study provide novel insight into the muscle synergy framework and the design of natural myoelectric control interfaces.

Recurrent Neural Network for Contaminant Type Detector in Surface Electromyography Signals

A surface Electromyography (sEMG) contaminant type detector has been developed by using a Recurrent Neural Network (RNN) with Long Short-Term (LSMT) units in its hidden layer. This setup may reduce the contamination detection processing time since there is no need for feature extraction so that the classification occurs directly from the sEMG signal. The publicly available NINAPro (Non-Invasive Adaptive Prosthetics) database sEMG signals was used to train and test the network. Signals were contaminated with White Gaussian Noise, Movement Artifact, ECG and Power Line Interference. Two out of the 40 healthy subjects' data were considered to train the network and the other 38 to test it. Twelve models were trained under a -20dB contamination, one for each channel. ANOVA results showed that the training channel could affect the classification accuracy if SNR = -20dB and 0dB. An overall accuracy of 97.72% has been achieved by one of the models.

Should Hands Be Restricted When Measuring Able-Bodied Participants to Evaluate Machine Learning Controlled Prosthetic Hands?

OBJECTIVE

When evaluating methods for machine-learning controlled prosthetic hands, able-bodied participants are often recruited, for practical reasons, instead of participants with upper limb absence (ULA). However, able-bodied participants have been shown to often perform myoelectric control tasks better than participants with ULA. It has been suggested that this performance difference can be reduced by restricting the wrist and hand movements of able-bodied participants. However, the effect of such restrictions on the consistency and separability of the electromyogram's (EMG) features remains unknown. The present work investigates whether the EMG separability and consistency between unaffected and affected arms differ and whether they change after restricting the unaffected limb in persons with ULA.

METHODS

Both arms of participants with unilateral ULA were compared in two conditions: with the unaffected hand and wrist restricted or not. Furthermore, it was tested if the effect of arm and restriction is influenced by arm posture (arm down, arm in front, or arm up).

RESULTS

Fourteen participants (two women, age = 53.4±4.05) with acquired transradial limb loss were recruited. We found that the unaffected limb generated more separated EMG than the affected limb. Furthermore, restricting the unaffected hand and wrist lowered the separability of the EMG when the arm was held down.

CONCLUSION

Limb restriction is a viable method to make the EMG of able-bodied participants more similar to that of participants with ULA.

SIGNIFICANCE

Future research that evaluates methods for machine learning controlled hands in able-bodied participants should restrict the participants' hand and wrist.

Gaze, behavioral, and clinical data for phantom limbs after hand amputation from 15 amputees and 29 controls

Despite recent advances in prosthetics, many upper limb amputees still use prostheses with some reluctance. They often do not feel able to incorporate the artificial hand into their bodily self. Furthermore, prosthesis fitting is not usually tailored to accommodate the characteristics of an individual’s phantom limb sensations. These are experienced by almost all persons with an acquired amputation and comprise the motor and postural properties of the lost limb. This article presents and validates a multimodal dataset including an extensive qualitative and quantitative assessment of phantom limb sensations in 15 transradial amputees, surface electromyography and accelerometry data of the forearm, and measurements of gaze behavior during exercises requiring pointing or repositioning of the forearm and the phantom hand. The data also include acquisitions from 29 able-bodied participants, matched for gender and age. Special emphasis was given to tracking the visuo-motor coupling between eye-hand/eye-phantom during these exercises.

Measurement(s)	phantom limb sensation • muscle electrophysiology trait • first person video • body movement coordination trait • head movement trait • eye-hand coordination • completion time
Technology Type(s)	questionnaire • electromyography • eye tracking device • Accelerometer • accelerometer and gyroscope • data transformation • Foot Pedal Device
Factor Type(s)	age • sex • handedness • amputation side • amputation cause • years since amputation • residual limb length • prosthesis • painful limb pain • non-painful phantom limb sensations • body representation in dreams • referred sensations • obstacle shunning and obstacle tolerance • impact of the amputation on quality of life • motor imagery
Sample Characteristic - Organism	Homo sapiens

Machine-accessible metadata file describing the reported data: 10.6084/m9.figshare.11815263

Gaze, visual, myoelectric, and inertial data of grasps for intelligent prosthetics

A hand amputation is a highly disabling event, having severe physical and psychological repercussions on a person's life. Despite extensive efforts devoted to restoring the missing functionality via dexterous myoelectric hand prostheses, natural and robust control usable in everyday life is still challenging. Novel techniques have been proposed to overcome the current limitations, among them the fusion of surface electromyography with other sources of contextual information. We present a dataset to investigate the inclusion of eye tracking and first person video to provide more stable intent recognition for prosthetic control. This multimodal dataset contains surface electromyography and accelerometry of the forearm, and gaze, first person video, and inertial measurements of the head recorded from 15 transradial amputees and 30 able-bodied subjects performing grasping tasks. Besides the intended application for upper-limb prosthetics, we also foresee uses for this dataset to study eye-hand coordination in the context of psychophysics, neuroscience, and assistive robotics.

PaWFE: Fast Signal Feature Extraction Using Parallel Time Windows

Motivation: Hand amputations can dramatically affect the quality of life of a person. Researchers are developing surface electromyography and machine learning solutions to control dexterous and robotic prosthetic hands, however long computational times can slow down this process.

Objective: This paper aims at creating a fast signal feature extraction algorithm that can extract widely used features and allow researchers to easily add new ones.

Methods: PaWFE (Parallel Window Feature Extractor) extracts the signal features from several time windows in parallel. The MATLAB code is publicly available and supports several time domain and frequency features. The code was tested and benchmarked using 1,2,4,8,16,32, and 48 threads on a server with four Xeon E7- 4820 and 128 GB RAM using the first 5 datasets of the Ninapro database, that are recorded with different acquisition setups.

Results: The parallel time window analysis approach allows to reduce the computational time up to 20 times when using 32 cores, showing a very good scalability. Signal features can be extracted in few seconds from an entire data acquisition and in <100 ms from a single time window, easily reducing of up to over 15 times the feature extraction procedure in comparison to traditional approaches. The code allows users to easily add new signal feature extraction scripts, that can be added to the code and on the Ninapro website upon request.

Significance: The code allows researchers in machine learning and biosignals data analysis to easily and quickly test modern machine learning approaches on big datasets and it can be used as a resource for real time data analysis too.

Myoelectric Signal Classification of Targeted Muscles Using Dictionary Learning

Surface electromyography (sEMG) signals comprise electrophysiological information related to muscle activity. As this signal is easy to record, it is utilized to control several myoelectric prostheses devices. Several studies have been conducted to process sEMG signals more efficiently. However, research on optimal algorithms and electrode placements for the processing of sEMG signals is still inconclusive. In addition, very few studies have focused on minimizing the number of electrodes. In this study, we investigated the most effective method for myoelectric signal classification with a small number of electrodes. A total of 23 subjects participated in the study, and the sEMG data of 14 different hand movements of the subjects were acquired from targeted muscles and untargeted muscles. Furthermore, the study compared the classification accuracy of the sEMG data using discriminative feature-oriented dictionary learning (DFDL) and other conventional classifiers. DFDL demonstrated the highest classification accuracy among the classifiers, and its higher quality performance became more apparent as the number of channels decreased. The targeted method was superior to the untargeted method, particularly when classifying sEMG signals with DFDL. Therefore, it was concluded that the combination of the targeted method and the DFDL algorithm could classify myoelectric signals more effectively with a minimal number of channels.

A quantitative taxonomy of human hand grasps

Background

A proper modeling of human grasping and of hand movements is fundamental for robotics, prosthetics, physiology and rehabilitation. The taxonomies of hand grasps that have been proposed in scientific literature so far are based on qualitative analyses of the movements and thus they are usually not quantitatively justified.

Methods

This paper presents to the best of our knowledge the first quantitative taxonomy of hand grasps based on biomedical data measurements. The taxonomy is based on electromyography and kinematic data recorded from 40 healthy subjects performing 20 unique hand grasps. For each subject, a set of hierarchical trees are computed for several signal features. Afterwards, the trees are combined, first into modality-specific (i.e. muscular and kinematic) taxonomies of hand grasps and then into a general quantitative taxonomy of hand movements. The modality-specific taxonomies provide similar results despite describing different parameters of hand movements, one being muscular and the other kinematic.

Results

The general taxonomy merges the kinematic and muscular description into a comprehensive hierarchical structure. The obtained results clarify what has been proposed in the literature so far and they partially confirm the qualitative parameters used to create previous taxonomies of hand grasps. According to the results, hand movements can be divided into five movement categories defined based on the overall grasp shape, finger positioning and muscular activation. Part of the results appears qualitatively in accordance with previous results describing kinematic hand grasping synergies.

Conclusions

The taxonomy of hand grasps proposed in this paper clarifies with quantitative measurements what has been proposed in the field on a qualitative basis, thus having a potential impact on several scientific fields.

Phantom-Mobility-Based Prosthesis Control in Transhumeral Amputees Without Surgical Reinnervation: A Preliminary Study

Transhumeral amputees face substantial difficulties in efficiently controlling their prosthetic limb, leading to a high rate of rejection of these devices. Actual myoelectric control approaches make their use slow, sequential and unnatural, especially for these patients with a high level of amputation who need a prosthesis with numerous active degrees of freedom (powered elbow, wrist, and hand). While surgical muscle-reinnervation is becoming a generic solution for amputees to increase their control capabilities over a prosthesis, research is still being conducted on the possibility of using the surface myoelectric patterns specifically associated to voluntary Phantom Limb Mobilization (PLM), appearing naturally in most upper-limb amputees without requiring specific surgery. The objective of this study was to evaluate the possibility for transhumeral amputees to use a PLM-based control approach to perform more realistic functional grasping tasks. Two transhumeral amputated participants were asked to repetitively grasp one out of three different objects with an unworn eight-active-DoF prosthetic arm and release it in a dedicated drawer. The prosthesis control was based on phantom limb mobilization and myoelectric pattern recognition techniques, using only two repetitions of each PLM to train the classification architecture. The results show that the task could be successfully achieved with rather optimal strategies and joint trajectories, even if the completion time was increased in comparison with the performances obtained by a control group using a simple GUI control, and the control strategies required numerous corrections. While numerous limitations related to robustness of pattern recognition techniques and to the perturbations generated by actual wearing of the prosthesis remain to be solved, these preliminary results encourage further exploration and deeper understanding of the phenomenon of natural residual myoelectric activity related to PLM, since it could possibly be a viable option in some transhumeral amputees to extend their control abilities of functional upper limb prosthetics with multiple active joints without undergoing muscular reinnervation surgery.

Characteristics of phantom upper limb mobility encourage phantom-mobility-based prosthesis control

There is an increasing need to extend the control possibilities of upper limb amputees over their prosthetics, especially given the development of devices with numerous active joints. One way of feeding pattern recognition myoelectric control is to rely on the myoelectric activities of the residual limb associated with phantom limb movements (PLM). This study aimed to describe the types, characteristics, potential influencing factors and trainability of upper limb PLM. Seventy-six below- and above-elbow amputees with major amputation underwent a semi-directed interview about their phantom limb. Amputation level, elapsed time since amputation, chronic pain and use of prostheses of upper limb PLM were extracted from the interviews. Thirteen different PLM were found involving the hand, wrist and elbow. Seventy-six percent of the patients were able to produce at least one type of PLM; most of them could execute several. Amputation level, elapsed time since amputation, chronic pain and use of myoelectric prostheses were not found to influence PLM. Five above-elbow amputees participated in a PLM training program and consequently increased both endurance and speed of their PLM. These results clearly encourage further research on PLM-associated muscle activation patterns for future PLM-based modes of prostheses control.

Muscle Synergy Analysis of a Hand-Grasp Dataset: A Limited Subset of Motor Modules May Underlie a Large Variety of Grasps

Background: Kinematic and muscle patterns underlying hand grasps have been widely investigated in the literature. However, the identification of a reduced set of motor modules, generalizing across subjects and grasps, may be valuable for increasing the knowledge of hand motor control, and provide methods to be exploited in prosthesis control and hand rehabilitation.

Methods: Motor muscle synergies were extracted from a publicly available database including 28 subjects, executing 20 hand grasps selected for daily-life activities. The spatial synergies and temporal components were analyzed with a clustering algorithm to characterize the patterns underlying hand-grasps.

Results: Motor synergies were successfully extracted on all 28 subjects. Clustering orders ranging from 2 to 50 were tested. A subset of ten clusters, each one represented by a spatial motor module, approximates the original dataset with a mean maximum error of 5% on reconstructed modules; however, each spatial synergy might be employed with different timing and recruited at different grasp stages. Two temporal activation patterns are often recognized, corresponding to the grasp/hold phase, and to the pre-shaping and release phase.

Conclusions: This paper presents one of the biggest analysis of muscle synergies of hand grasps currently available. The results of 28 subjects performing 20 different grasps suggest that a limited number of time dependent motor modules (shared among subjects), correctly elicited by a control activation signal, may underlie the execution of a large variety of hand grasps. However, spatial synergies are not strongly related to specific motor functions but may be recruited at different stages, depending on subject and grasp. This result can lead to applications in rehabilitation and assistive robotics.

Classification of Transient Myoelectric Signals for the Control of Multi-Grasp Hand Prostheses

Understanding the neurophysiological signals underlying voluntary motor control and decoding them for controlling limb prostheses is one of the major challenges in applied neuroscience and rehabilitation engineering. While pattern recognition of continuous myoelectric (EMG) signals is arguably the most investigated approach for hand prosthesis control, its underlying assumption is poorly supported, i.e., that repeated muscular contractions produce consistent patterns of steady-state EMGs. In fact, it still remains to be shown that pattern recognition-based controllers allow natural control over multiple grasps in hand prosthesis outside well-controlled laboratory settings. Here, we propose an approach that relies on decoding the intended grasp from forearm EMG recordings associated with the onset of muscle contraction as opposed to the steady-state signals. Eight unimpaired individuals and two hand amputees performed four grasping movements with a variety of arm postures while EMG recordings subsequently processed to mimic signals picked up by conventional myoelectric sensors were obtained from their forearms and residual limbs, respectively. Off-line data analyses demonstrated the feasibility of the approach also with respect to the limb position effect. The sampling frequency and length of the classified EMG window that off-line resulted in optimal performance were applied to a controller of a research prosthesis worn by one hand amputee and proved functional in real-time when operated under realistic working conditions.

Comparison of six electromyography acquisition setups on hand movement classification tasks

Hand prostheses controlled by surface electromyography are promising due to the non-invasive approach and the control capabilities offered by machine learning. Nevertheless, dexterous prostheses are still scarcely spread due to control difficulties, low robustness and often prohibitive costs. Several sEMG acquisition setups are now available, ranging in terms of costs between a few hundred and several thousand dollars. The objective of this paper is the relative comparison of six acquisition setups on an identical hand movement classification task, in order to help the researchers to choose the proper acquisition setup for their requirements. The acquisition setups are based on four different sEMG electrodes (including Otto Bock, Delsys Trigno, Cometa Wave + Dormo ECG and two Thalmic Myo armbands) and they were used to record more than 50 hand movements from intact subjects with a standardized acquisition protocol. The relative performance of the six sEMG acquisition setups is compared on 41 identical hand movements with a standardized feature extraction and data analysis pipeline aimed at performing hand movement classification. Comparable classification results are obtained with three acquisition setups including the Delsys Trigno, the Cometa Wave and the affordable setup composed of two Myo armbands. The results suggest that practical sEMG tests can be performed even when costs are relevant (e.g. in small laboratories, developing countries or use by children). All the presented datasets can be used for offline tests and their quality can easily be compared as the data sets are publicly available.

Deep Learning with Convolutional Neural Networks Applied to Electromyography Data: A Resource for the Classification of Movements for Prosthetic Hands

Natural control methods based on surface electromyography (sEMG) and pattern recognition are promising for hand prosthetics. However, the control robustness offered by scientific research is still not sufficient for many real life applications, and commercial prostheses are capable of offering natural control for only a few movements. In recent years deep learning revolutionized several fields of machine learning, including computer vision and speech recognition. Our objective is to test its methods for natural control of robotic hands via sEMG using a large number of intact subjects and amputees. We tested convolutional networks for the classification of an average of 50 hand movements in 67 intact subjects and 11 transradial amputees. The simple architecture of the neural network allowed to make several tests in order to evaluate the effect of pre-processing, layer architecture, data augmentation and optimization. The classification results are compared with a set of classical classification methods applied on the same datasets. The classification accuracy obtained with convolutional neural networks using the proposed architecture is higher than the average results obtained with the classical classification methods, but lower than the results obtained with the best reference methods in our tests. The results show that convolutional neural networks with a very simple architecture can produce accurate results comparable to the average classical classification methods. They show that several factors (including pre-processing, the architecture of the net and the optimization parameters) can be fundamental for the analysis of sEMG data. Larger networks can achieve higher accuracy on computer vision and object recognition tasks. This fact suggests that it may be interesting to evaluate if larger networks can increase sEMG classification accuracy too.