Muscle Synergy Analysis of a Hand-Grasp Dataset: A Limited Subset of Motor Modules May Underlie a Large Variety of Grasps

Clustering user preferences for personalized teleoperation control schemes via trajectory similarity analysis

Successful operation of a teleoperated robot depends on a well-designed control scheme to translate human motion into robot motion; however, a single control scheme may not be suitable for all users. On the other hand, individual personalization of control schemes may be infeasible for designers to produce. In this paper, we present a method by which users may be classified into groups with mutually compatible control scheme preferences. Users are asked to demonstrate freehand motions to control a simulated robot in a virtual reality environment. Hand pose data is captured and compared with other users using SLAM trajectory similarity analysis techniques. The resulting pairwise trajectory error metrics are used to cluster participants based on their control motions, without foreknowledge of the number or types of control scheme preferences that may exist. The clusters identified for two different robots shows that a small number of clusters form stably for each case, each with its own control scheme paradigm. Survey data from participants validates that the clusters identified through this method correspond to the participants' control scheme rationales, and also identify nuances in participant control scheme descriptions that may not be obvious to designers relying only on participant explanations of their preferences.

Concurrent and continuous estimation of multi-finger forces by synergy mapping and reconstruction: a pilot study

Objective.The absence of intuitive control in present myoelectric interfaces makes it a challenge for users to communicate with assistive devices efficiently in real-world conditions. This study aims to tackle this difficulty by incorporating neurophysiological entities, namely muscle and force synergies, onto multi-finger force estimation to allow intuitive myoelectric control.Approach. Eleven healthy subjects performed six isometric grasping tasks at three muscle contraction levels. The exerted fingertip forces were collected concurrently with the surface electromyographic (sEMG) signals from six extrinsic and intrinsic muscles of hand. Muscle synergies were then extracted from recorded sEMG signals, while force synergies were identified from measured force data. Afterwards, a linear regressor was trained to associate the two types of synergies. This would allow us to predict multi-finger forces simply by multiplying the activation signals derived from muscle synergies with the weighting matrix of initially identified force synergies. To mitigate the false activation of unintended fingers, the force predictions were finally corrected by a finger state recognition procedure.Main results. We found that five muscle synergies and four force synergies are able to make a tradeoff between the computation load and the prediction accuracy for the proposed model; When trained and tested on all six grasping tasks, our method (SYN-II) achieved better performance (R2= 0.80 ± 0.04, NRMSE = 0.19 ± 0.01) than conventional sEMG amplitude-based method; Interestingly, SYN-II performed better than all other methods when tested on two unknown tasks outside the four training tasks (R2= 0.74 ± 0.03, NRMSE = 0.22 ± 0.02), which indicated better generalization ability.Significance. This study shows the first attempt to link between muscle and force synergies to allow concurrent and continuous estimation of multi-finger forces from sEMG. The proposed approach may lay the foundation for high-performance myoelectric interfaces that allow users to control robotic hands in a more natural and intuitive manner.

Exploring the EMG transient: the muscular activation sequences used as novel time-domain features for hand gestures classification

Introduction

Muscular activation sequences have been shown to be suitable time-domain features for classification of motion gestures. However, their clinical application in myoelectric prosthesis control was never investigated so far. The aim of the paper is to evaluate the robustness of these features extracted from the EMG signal in transient state, on the forearm, for classifying common hand tasks.

Methods

The signal associated to four hand gestures and the rest condition were acquired from ten healthy people and two persons with trans-radial amputation. A feature extraction algorithm allowed for encoding the EMG signals into muscular activation sequences, which were used to train four commonly used classifiers, namely Linear Discriminant Analysis (LDA), Support Vector Machine (SVM), Non-linear Logistic Regression (NLR) and Artificial Neural Network (ANN). The offline performances were assessed with the entire sample of recruited people. The online performances were assessed with the amputee subjects. Moreover, a comparison of the proposed method with approaches based on the signal envelope in the transient state and in the steady state was conducted.

Results

The highest performance were obtained with the NLR classifier. Using the sequences, the offline classification accuracy was higher than 93% for healthy and amputee subjects and always higher than the approach with the signal envelope in transient state. As regards the comparison with the steady state, the performances obtained with the proposed method are slightly lower (<4%), but the classification occurred at least 200 ms earlier. In the online application, the motion completion rate reached up to 85% of the total classification attempts, with a motion selection time that never exceeded 218 ms.

Discussion

Muscular activation sequences are suitable alternatives to the time-domain features commonly used in classification problems belonging to the sole EMG transient state and could be potentially exploited in control strategies of myoelectric prosthesis hands.

Spatial and Temporal Muscle Synergies Provide a Dual Characterization of Low-dimensional and Intermittent Control of Upper-limb Movements

Muscle synergy analysis investigates the neurophysiological mechanisms that the central nervous system employs to coordinate muscles. Several models have been developed to decompose electromyographic (EMG) signals into spatial and temporal synergies. However, using multiple approaches can complicate the interpretation of results. Spatial synergies represent invariant muscle weights modulated with variant temporal coefficients; temporal synergies are invariant temporal profiles that coordinate variant muscle weights. While non-negative matrix factorization allows to extract both spatial and temporal synergies, the comparison between the two approaches was rarely investigated targeting a large set of multi-joint upper-limb movements. Spatial and temporal synergies were extracted from two datasets with proximal (16 subjects, 10M, 6F) and distal upper-limb movements (30 subjects, 21M, 9F), focusing on their differences in reconstruction accuracy and inter-individual variability. We showed the existence of both spatial and temporal structure in the EMG data, comparing synergies with those from a surrogate dataset in which the phases were shuffled preserving the frequency content of the original data. The two models provide a compact characterization of motor coordination at the spatial or temporal level, respectively. However, a lower number of temporal synergies are needed to achieve the same reconstruction R²: spatial and temporal synergies may capture different hierarchical levels of motor control and are dual approaches to the characterization of low-dimensional coordination of the upper-limb. Last, a detailed characterization of the structure of the temporal synergies suggested that they can be related to intermittent control of the movement, allowing high flexibility and dexterity. These results improve neurophysiology understanding in several fields such as motor control, rehabilitation, and prosthetics.

The Number and Structure of Muscle Synergies Depend on the Number of Recorded Muscles: A Pilot Simulation Study with OpenSim

The muscle synergy approach is used to evaluate motor control and to quantitatively determine the number and structure of the modules underlying movement. In experimental studies regarding the upper limb, typically 8 to 16 EMG probes are used depending on the application, although the number of muscles involved in motor generation is higher. Therefore, the number of motor modules may be underestimated and the structure altered with the standard spatial synergy model based on the non-negative matrix factorization (NMF). In this study, we compared the number and structure of muscle synergies when considering 12 muscles (an "average" condition that represents previous studies) and 32 muscles of the upper limb, also including multiple muscle heads and deep muscles. First, we estimated the muscle activations with an upper-limb model in OpenSim using data from multi-directional reaching movements acquired in experimental sessions; then, spatial synergies were extracted from EMG activations from 12 muscles and from 32 muscles and their structures were compared. Finally, we compared muscle synergies obtained from OpenSim and from real experimental EMG signals to assess the reliability of the results. Interestingly, we found that on average, an additional synergy is needed to reconstruct the same R² level with 32 muscles with respect to 12 muscles; synergies have a very similar structure, although muscles with comparable physiological functions were added to the synergies extracted with 12 muscles. The additional synergies, instead, captured patterns that could not be identified with only 12 muscles. We concluded that current studies may slightly underestimate the number of controlled synergies, even though the main structure of synergies is not modified when adding more muscles. We also show that EMG activations estimated with OpenSim are in partial (but not complete) agreement with experimental recordings. These findings may have significative implications for motor control and clinical studies.

A Survey of Multifingered Robotic Manipulation: Biological Results, Structural Evolvements, and Learning Methods

Multifingered robotic hands (usually referred to as dexterous hands) are designed to achieve human-level or human-like manipulations for robots or as prostheses for the disabled. The research dates back 30 years ago, yet, there remain great challenges to effectively design and control them due to their high dimensionality of configuration, frequently switched interaction modes, and various task generalization requirements. This article aims to give a brief overview of multifingered robotic manipulation from three aspects: a) the biological results, b) the structural evolvements, and c) the learning methods, and discuss potential future directions. First, we investigate the structure and principle of hand-centered visual sensing, tactile sensing, and motor control and related behavioral results. Then, we review several typical multifingered dexterous hands from task scenarios, actuation mechanisms, and in-hand sensors points. Third, we report the recent progress of various learning-based multifingered manipulation methods, including but not limited to reinforcement learning, imitation learning, and other sub-class methods. The article concludes with open issues and our thoughts on future directions.

An Image-Based Benchmark Dataset and a Novel Object Detector for Water Surface Object Detection

Water surface object detection is one of the most significant tasks in autonomous driving and water surface vision applications. To date, existing public large-scale datasets collected from websites do not focus on specific scenarios. As a characteristic of these datasets, the quantity of the images and instances is also still at a low level. To accelerate the development of water surface autonomous driving, this paper proposes a large-scale, high-quality annotated benchmark dataset, named Water Surface Object Detection Dataset (WSODD), to benchmark different water surface object detection algorithms. The proposed dataset consists of 7,467 water surface images in different water environments, climate conditions, and shooting times. In addition, the dataset comprises a total of 14 common object categories and 21,911 instances. Simultaneously, more specific scenarios are focused on in WSODD. In order to find a straightforward architecture to provide good performance on WSODD, a new object detector, named CRB-Net, is proposed to serve as a baseline. In experiments, CRB-Net was compared with 16 state-of-the-art object detection methods and outperformed all of them in terms of detection precision. In this paper, we further discuss the effect of the dataset diversity (e.g., instance size, lighting conditions), training set size, and dataset details (e.g., method of categorization). Cross-dataset validation shows that WSODD significantly outperforms other relevant datasets and that the adaptability of CRB-Net is excellent.

U-Limb: A multi-modal, multi-center database on arm motion control in healthy and post-stroke conditions

BACKGROUND

Shedding light on the neuroscientific mechanisms of human upper limb motor control, in both healthy and disease conditions (e.g., after a stroke), can help to devise effective tools for a quantitative evaluation of the impaired conditions, and to properly inform the rehabilitative process. Furthermore, the design and control of mechatronic devices can also benefit from such neuroscientific outcomes, with important implications for assistive and rehabilitation robotics and advanced human-machine interaction. To reach these goals, we believe that an exhaustive data collection on human behavior is a mandatory step. For this reason, we release U-Limb, a large, multi-modal, multi-center data collection on human upper limb movements, with the aim of fostering trans-disciplinary cross-fertilization.

CONTRIBUTION

This collection of signals consists of data from 91 able-bodied and 65 post-stroke participants and is organized at 3 levels: (i) upper limb daily living activities, during which kinematic and physiological signals (electromyography, electro-encephalography, and electrocardiography) were recorded; (ii) force-kinematic behavior during precise manipulation tasks with a haptic device; and (iii) brain activity during hand control using functional magnetic resonance imaging.

A Novel Muscle Synergy Extraction Method Used for Motor Function Evaluation of Stroke Patients: A Pilot Study

In this paper, we present a novel muscle synergy extraction method based on multivariate curve resolution–alternating least squares (MCR-ALS) to overcome the limitation of the nonnegative matrix factorization (NMF) method for extracting non-sparse muscle synergy, and we study its potential application for evaluating motor function of stroke survivors. Nonnegative matrix factorization (NMF) is the most widely used method for muscle synergy extraction. However, NMF is susceptible to components’ sparseness and usually provides inferior reliability, which significantly limits the promotion of muscle synergy. In this study, MCR-ALS was employed to extract muscle synergy from electromyography (EMG) data. Its performance was compared with two other matrix factorization algorithms, NMF and self-modeling mixture analysis (SMMA). Simulated data sets were utilized to explore the influences of the sparseness and noise on the extracted synergies. As a result, the synergies estimated by MCR-ALS were the most similar to true synergies as compared with SMMA and NMF. MCR-ALS was used to analyze the muscle synergy characteristics of upper limb movements performed by healthy (n = 11) and stroke (n = 5) subjects. The repeatability and intra-subject consistency were used to evaluate the performance of MCR-ALS. As a result, MCR-ALS provided much higher repeatability and intra-subject consistency as compared with NMF, which were important for the reliability of the motor function evaluation. The stroke subjects had lower intra-subject consistency and seemingly had more synergies as compared with the healthy subjects. Thus, MCR-ALS is a promising muscle synergy analysis method for motor function evaluation of stroke patients.

A Systematic Review of EMG Applications for the Characterization of Forearm and Hand Muscle Activity during Activities of Daily Living: Results, Challenges, and Open Issues

The role of the hand is crucial for the performance of activities of daily living, thereby ensuring a full and autonomous life. Its motion is controlled by a complex musculoskeletal system of approximately 38 muscles. Therefore, measuring and interpreting the muscle activation signals that drive hand motion is of great importance in many scientific domains, such as neuroscience, rehabilitation, physiotherapy, robotics, prosthetics, and biomechanics. Electromyography (EMG) can be used to carry out the neuromuscular characterization, but it is cumbersome because of the complexity of the musculoskeletal system of the forearm and hand. This paper reviews the main studies in which EMG has been applied to characterize the muscle activity of the forearm and hand during activities of daily living, with special attention to muscle synergies, which are thought to be used by the nervous system to simplify the control of the numerous muscles by actuating them in task-relevant subgroups. The state of the art of the current results are presented, which may help to guide and foster progress in many scientific domains. Furthermore, the most important challenges and open issues are identified in order to achieve a better understanding of human hand behavior, improve rehabilitation protocols, more intuitive control of prostheses, and more realistic biomechanical models.

Variability of Muscle Synergies in Hand Grasps: Analysis of Intra- and Inter-Session Data

Background. Muscle synergy analysis is an approach to understand the neurophysiological mechanisms behind the hypothesized ability of the Central Nervous System (CNS) to reduce the dimensionality of muscle control. The muscle synergy approach is also used to evaluate motor recovery and the evolution of the patients’ motor performance both in single-session and longitudinal studies. Synergy-based assessments are subject to various sources of variability: natural trial-by-trial variability of performed movements, intrinsic characteristics of subjects that change over time (e.g., recovery, adaptation, exercise, etc.), as well as experimental factors such as different electrode positioning. These sources of variability need to be quantified in order to resolve challenges for the application of muscle synergies in clinical environments. The objective of this study is to analyze the stability and similarity of extracted muscle synergies under the effect of factors that may induce variability, including inter- and intra-session variability within subjects and inter-subject variability differentiation. The analysis was performed using the comprehensive, publicly available hand grasp NinaPro Database, featuring surface electromyography (EMG) measures from two EMG electrode bracelets. Methods. Intra-session, inter-session, and inter-subject synergy stability was analyzed using the following measures: variance accounted for (VAF) and number of synergies (NoS) as measures of reconstruction stability quality and cosine similarity for comparison of spatial composition of extracted synergies. Moreover, an approach based on virtual electrode repositioning was applied to shed light on the influence of electrode position on inter-session synergy similarity. Results. Inter-session synergy similarity was significantly lower with respect to intra-session similarity, both considering coefficient of variation of VAF (approximately 0.2–15% for inter vs. approximately 0.1% to 2.5% for intra, depending on NoS) and coefficient of variation of NoS (approximately 6.5–14.5% for inter vs. approximately 3–3.5% for intra, depending on VAF) as well as synergy similarity (approximately 74–77% for inter vs. approximately 88–94% for intra, depending on the selected VAF). Virtual electrode repositioning revealed that a slightly different electrode position can lower similarity of synergies from the same session and can increase similarity between sessions. Finally, the similarity of inter-subject synergies has no significant difference from the similarity of inter-session synergies (both on average approximately 84–90% depending on selected VAF). Conclusion. Synergy similarity was lower in inter-session conditions with respect to intra-session. This finding should be considered when interpreting results from multi-session assessments. Lastly, electrode positioning might play an important role in the lower similarity of synergies over different sessions.

Home-based rehabilitation using a soft robotic hand glove device leads to improvement in hand function in people with chronic spinal cord injury:a pilot study

BACKGROUND

Loss of hand function following high level spinal cord injury (SCI) is perceived as a high priority area for rehabilitation. Following discharge, it is often impractical for the specialist care centre to provide ongoing therapy for people living with chronic SCI at home, which can lead to further deterioration of hand function and a direct impact on an individual's capability to perform essential activities of daily living (ADL).

OBJECTIVE

This pilot study investigated the therapeutic effect of a self-administered home-based hand rehabilitation programme for people with cervical SCI using the soft extra muscle (SEM) Glove by Bioservo Technologies AB.

METHODS

Fifteen participants with chronic cervical motor incomplete (AIS C and D) SCI were recruited and provided with the glove device to use at home to complete a set task and perform their usual ADL for a minimum of 4 h a day for 12 weeks. Assessment was made at Week 0 (Initial), 6, 12 and 18 (6-week follow-up). The primary outcome measure was the Toronto Rehabilitation Institute hand function test (TRI-HFT), with secondary outcome measures including pinch dynamometry and the modified Ashworth scale.

RESULTS

The TRI-HFT demonstrated improvement in hand function at Week 6 of the therapy including improvement in object manipulation (58.3 ±3.2 to 66.9 ±1.8, p ≈ 0.01), and palmar grasp assessed as the length of the wooden bar that can be held using a pronated palmar grip (29.1 ±6.0 cm to 45.8 ±6.8 cm, p <0.01). A significant improvement in pinch strength, with reduced thumb muscle hypertonia was also detected. Improvements in function were present during the Week 12 assessment and also during the follow-up.

CONCLUSIONS

Self-administered rehabilitation using the SEM Glove is effective for improving and retaining gross and fine hand motor function for people living with chronic spinal cord injury at home. Retention of improved hand function suggests that an intensive activity-based rehabilitation programme in specific individuals is sufficient to improve long-term neuromuscular activity. Future studies should characterise the neuromuscular mechanism of action and the minimal rehabilitation programme necessary with the assistive device to improve ADL tasks following chronic cervical SCI.

TRIAL REGISTRATION NUMBER

Trial registration: ISRCTN, ISRCTN98677526, Registered 01/June/2017 - Retrospectively registered, http://www.isrctn.com/ISRCTN98677526.

Kinematic synergies of hand grasps: a comprehensive study on a large publicly available dataset

Background

Hand grasp patterns require complex coordination. The reduction of the kinematic dimensionality is a key process to study the patterns underlying hand usage and grasping. It allows to define metrics for motor assessment and rehabilitation, to develop assistive devices and prosthesis control methods. Several studies were presented in this field but most of them targeted a limited number of subjects, they focused on postures rather than entire grasping movements and they did not perform separate analysis for the tasks and subjects, which can limit the impact on rehabilitation and assistive applications. This paper provides a comprehensive mapping of synergies from hand grasps targeting activities of daily living. It clarifies several current limits of the field and fosters the development of applications in rehabilitation and assistive robotics.

Methods

In this work, hand kinematic data of 77 subjects, performing up to 20 hand grasps, were acquired with a data glove (a 22-sensor CyberGlove II data glove) and analyzed. Principal Component Analysis (PCA) and hierarchical cluster analysis were used to extract and group kinematic synergies that summarize the coordination patterns available for hand grasps.

Results

Twelve synergies were found to account for > 80% of the overall variation. The first three synergies accounted for more than 50% of the total amount of variance and consisted of: the flexion and adduction of the Metacarpophalangeal joint (MCP) of fingers 3 to 5 (synergy #1), palmar arching and flexion of the wrist (synergy #2) and opposition of the thumb (synergy #3). Further synergies refine movements and have higher variability among subjects.

Conclusion

Kinematic synergies are extracted from a large number of subjects (77) and grasps related to activities of daily living (20). The number of motor modules required to perform the motor tasks is higher than what previously described. Twelve synergies are responsible for most of the variation in hand grasping. The first three are used as primary synergies, while the remaining ones target finer movements (e.g. independence of thumb and index finger). The results generalize the description of hand kinematics, better clarifying several limits of the field and fostering the development of applications in rehabilitation and assistive robotics.

Kinect-Assisted Performance-Sensitive Upper Limb Exercise Platform for Post-stroke Survivors

One's ability to use upper limbs is critical for performing activities of daily living necessary for enjoying quality community life. However, after stroke, such abilities becomes adversely affected and it often deprives one of their capability to perform tasks that need coordinated movement in the upper limbs. To address issues with upper limb dysfunction, patients typically undergo rehabilitative exercises. Given the high patient to doctor ratio particularly in developing countries like India, conventional rehabilitation with patients undergoing exercises under one-on-one therapist's supervision often becomes a challenge. Thus, investigators are exploring technology such as computer-based platforms coupled with cameras that can alleviate the need for the continuous presence of a therapist and can offer a powerful complementary tool in the hands of the clinicians. Such marker-based imaging systems used for rehabilitation can offer real-time processing and high accuracy of data. However, these systems often require dedicated lab space and high set-up time. Often this is very expensive and suffers from portability issues. Investigators have been exploring marker-less imaging techniques e.g., Kinect integrated computer-based graphical user interfaces in stroke-rehabilitation such as tracking one's limb movement during rehabilitation. In our present study, we have developed a Kinect-assisted computer-based system that offered Human Computer Interaction (HCI) tasks of varying challenge levels. Execution of the tasks required one to use reaching and coordination skills of the upper limbs. Also, the system was Performance-sensitive i.e., adaptive to the individualized residual movement ability of one's upper limb quantified in terms of task performance score. We tested for the usability of our system by exposing 15 healthy participants to our system. Subsequently, seven post-stroke patients interacted with our system over a few sessions spread over 2 weeks. Also, we studied patient's mean tonic activity corresponding to the HCI tasks as a possible indicator of one's post-stroke functional recovery suggesting its potential of our system to serve as a rehabilitation platform. Our results indicate the potential of such systems toward the improvement of task performance capability of post-stroke patients with possibilities of upper limb movement rehabilitation.

Posture similarity index: a method to compare hand postures in synergy space

Background

The human hand can perform a range of manipulation tasks, from holding a pen to holding a hammer. The central nervous system (CNS) uses different strategies in different manipulation tasks based on task requirements. Attempts to compare postures of the hand have been made for use in robotics and animation industries. In this study, we developed an index called the posture similarity index to quantify the similarity between two human hand postures.

Methods

Twelve right-handed volunteers performed 70 postures, and lifted and held 30 objects (total of 100 different postures, each performed five times). A 16-sensor electromagnetic tracking system captured the kinematics of individual finger phalanges (segments). We modeled the hand as a 21-DoF system and computed the corresponding joint angles. We used principal component analysis to extract kinematic synergies from this 21-DoF data. We developed a posture similarity index (PSI), that represents the similarity between posture in the synergy (Principal component) space. First, we tested the performance of this index using a synthetic dataset. After confirming that it performs well with the synthetic dataset, we used it to analyze the experimental data. Further, we used PSI to identify postures that are “representative” in the sense that they have a greater overlap (in synergy space) with a large number of postures.

Results

Our results confirmed that PSI is a relatively accurate index of similarity in synergy space both with synthetic data and real experimental data. Also, more special postures than common postures were found among “representative” postures.

Conclusion

We developed an index for comparing posture similarity in synergy space and demonstrated its utility by using synthetic dataset and experimental dataset. Besides, we found that “special” postures are actually “special” in the sense that there are more of them in the “representative” postures as identified by our posture similarity index.