A Multi-Modal Under-Sensorized Wearable System for Optimal Kinematic and Muscular Tracking of Human Upper Limb Motion

Wearable sensing solutions have emerged as a promising paradigm for monitoring human musculoskeletal state in an unobtrusive way. To increase the deployability of these systems, considerations related to cost reduction and enhanced form factor and wearability tend to discourage the number of sensors in use. In our previous work, we provided a theoretical solution to the problem of jointly reconstructing the entire muscular-kinematic state of the upper limb, when only a limited amount of optimally retrieved sensory data are available. However, the effective implementation of these methods in a physical, under-sensorized wearable has never been attempted before. In this work, we propose to bridge this gap by presenting an under-sensorized system based on inertial measurement units (IMUs) and surface electromyography (sEMG) electrodes for the reconstruction of the upper limb musculoskeletal state, focusing on the minimization of the sensors' number. We found that, relying on two IMUs only and eight sEMG sensors, we can conjointly reconstruct all 17 degrees of freedom (five joints, twelve muscles) of the upper limb musculoskeletal state, yielding a median normalized RMS error of 8.5% on the non-measured joints and 2.5% on the non-measured muscles.

Simulative Evaluation of a Joint-Cartesian Hybrid Motion Mapping for Robot Hands Based on Spatial In-Hand Information

Two sub-problems are typically identified for the replication of human finger motions on artificial hands: the measurement of the motions on the human side and the mapping method of human hand movements (primary hand) on the robotic hand (target hand). In this study, we focus on the second sub-problem. During human to robot hand mapping, ensuring natural motions and predictability for the operator is a difficult task, since it requires the preservation of the Cartesian position of the fingertips and the finger shapes given by the joint values. Several approaches have been presented to deal with this problem, which is still unresolved in general. In this work, we exploit the spatial information available in-hand, in particular, related to the thumb-finger relative position, for combining joint and Cartesian mappings. In this way, it is possible to perform a large range of both volar grasps (where the preservation of finger shapes is more important) and precision grips (where the preservation of fingertip positions is more important) during primary-to-target hand mappings, even if kinematic dissimilarities are present. We therefore report on two specific realizations of this approach: a distance-based hybrid mapping, in which the transition between joint and Cartesian mapping is driven by the approaching of the fingers to the current thumb fingertip position, and a workspace-based hybrid mapping, in which the joint–Cartesian transition is defined on the areas of the workspace in which thumb and fingertips can get in contact. The general mapping approach is presented, and the two realizations are tested. In order to report the results of an evaluation of the proposed mappings for multiple robotic hand kinematic structures (both industrial grippers and anthropomorphic hands, with a variable number of fingers), a simulative evaluation was performed.

Analysis of length of finger segments with different hand postures to enhance glove design

It is important to understand how the hand and fingers elongate and contract with hand posture for optimally fitting and comfortable gloves. Nevertheless, the measurement and analysis of the finger segments for glove designs remain largely neglected. Here, the length and proportion of the finger segments when splayed and during gripping, and between the dorsal and palm sides of 30 participants are 3D scanned and analysed. The full digit lengths change by 7.6-11.9% with hand posture, but the finger segment changes are not proportional. The ratios of the fingertip to distal interphalangeal joint/full digit, and fingertip to the proximal interphalangeal joint/full digit, are important variables. The results are validated with 10 more subjects based on ratings of a ready-to-wear sports glove. Inaccurate proportioning of the finger regions causes shifting which results in displacement and discomfort. This research contributes to glove pattern engineering, with a focus on the finger segments.

Synergy-Based Sensor Reduction for Recording the Whole Hand Kinematics

Simultaneous measurement of the kinematics of all hand segments is cumbersome due to sensor placement constraints, occlusions, and environmental disturbances. The aim of this study is to reduce the number of sensors required by using kinematic synergies, which are considered the basic building blocks underlying hand motions. Synergies were identified from the public KIN-MUS UJI database (22 subjects, 26 representative daily activities). Ten synergies per subject were extracted as the principal components explaining at least 95% of the total variance of the angles recorded across all tasks. The 220 resulting synergies were clustered, and candidate angles for estimating the remaining angles were obtained from these groups. Different combinations of candidates were tested and the one providing the lowest error was selected, its goodness being evaluated against kinematic data from another dataset (KINE-ADL BE-UJI). Consequently, the original 16 joint angles were reduced to eight: carpometacarpal flexion and abduction of thumb, metacarpophalangeal and interphalangeal flexion of thumb, proximal interphalangeal flexion of index and ring fingers, metacarpophalangeal flexion of ring finger, and palmar arch. Average estimation errors across joints were below 10% of the range of motion of each joint angle for all the activities. Across activities, errors ranged between 3.1% and 16.8%.

A stochastic algorithm for automatic hand pose and motion estimation

In this paper, a novel, robust, and simple method for automatically estimating the hand pose is proposed and validated. The method uses a multi-camera optoelectronic system and a model-based stochastic algorithm. The approach is marker-based and relies on an Unscented Kalman Filter. A hand kinematic model is introduced for constraining relative marker's positions and improving the algorithm robustness with respect to outliers and possible occlusions. The algorithm outputs are 3D coordinate measures of markers and hand joint angle values. To validate the proposed algorithm, a comparison with ground truths for angular and 3D coordinate measures is carried out. The comparative analysis shows the advantages of using the model-based stochastic algorithm with respect to standard processing software of optoelectronic cameras in terms of implementation simplicity, time consumption, and user effort. The accuracy is remarkable, with a difference of maximum 0.035r a d and 4m m with respect to angular and 3D Cartesian coordinates ground truths, respectively.

Design, control, and testing of a thumb exoskeleton with series elastic actuation

We present an exoskeleton capable of assisting the human thumb through a large range of motion. Our novel thumb exoskeleton has the following unique features: (i) an underlying kinematic mechanism that is optimized to achieve a large range of motion, (ii) a design that actuates four degrees of freedom of the thumb, and (iii) a series elastic actuation based on a Bowden cable, allowing for bidirectional torque control of each thumb joint individually. We present a kinematic model of the coupled thumb exoskeleton system and use it to maximize the range of motion of the thumb. Finally, we carry out tests with the designed device on four subjects to evaluate its workspace and kinematic transparency using a motion capture system and torque control performance. Results show that the device allows for a large workspace with the thumb, is kinematically transparent to natural thumb motion to a high degree, and is capable of accurate torque control.

Can a Soft Robotic Probe Use Stiffness Control Like a Human Finger to Improve Efficacy of Haptic Perception?

When humans are asked to palpate a soft tissue to locate a hard nodule, they regulate the stiffness, speed, and force of the finger during examination. If we understand the relationship between these behavioral variables and haptic information gain (transfer entropy) during manual probing, we can improve the efficacy of soft robotic probes for soft tissue palpation, such as in tumor localization in minimally invasive surgery. Here, we recorded the muscle co-contraction activity of the finger using EMG sensors to address the question as to whether joint stiffness control during manual palpation plays an important role in the haptic information gain. To address this question, we used a soft robotic probe with a controllable stiffness joint and a force sensor mounted at the base to represent the function of the tendon in a biological finger. Then, we trained a Markov chain using muscle co-contraction patterns of human subjects, and used it to control the stiffness of the soft robotic probe in the same soft tissue palpation task. The soft robotic experiments showed that haptic information gain about the depth of the hard nodule can be maximized by varying the internal stiffness of the soft probe.

Recent Data Sets on Object Manipulation: A Survey

Data sets is crucial not only for model learning and evaluation but also to advance knowledge on human behavior, thus fostering mutual inspiration between neuroscience and robotics. However, choosing the right data set to use or creating a new data set is not an easy task, because of the variety of data that can be found in the related literature. The first step to tackle this issue is to collect and organize those that are available. In this work, we take a significant step forward by reviewing data sets that were published in the past 10 years and that are directly related to object manipulation and grasping. We report on modalities, activities, and annotations for each individual data set and we discuss our view on its use for object manipulation. We also compare the data sets and summarize them. Finally, we conclude the survey by providing suggestions and discussing the best practices for the creation of new data sets.

A Synergy-Based Optimally Designed Sensing Glove for Functional Grasp Recognition

Achieving accurate and reliable kinematic hand pose reconstructions represents a challenging task. The main reason for this is the complexity of hand biomechanics, where several degrees of freedom are distributed along a continuous deformable structure. Wearable sensing can represent a viable solution to tackle this issue, since it enables a more natural kinematic monitoring. However, the intrinsic accuracy (as well as the number of sensing elements) of wearable hand pose reconstruction (HPR) systems can be severely limited by ergonomics and cost considerations. In this paper, we combined the theoretical foundations of the optimal design of HPR devices based on hand synergy information, i.e., the inter-joint covariation patterns, with textile goniometers based on knitted piezoresistive fabrics (KPF) technology, to develop, for the first time, an optimally-designed under-sensed glove for measuring hand kinematics. We used only five sensors optimally placed on the hand and completed hand pose reconstruction (described according to a kinematic model with 19 degrees of freedom) leveraging upon synergistic information. The reconstructions we obtained from five different subjects were used to implement an unsupervised method for the recognition of eight functional grasps, showing a high degree of accuracy and robustness.

Wearable Textile Platform for Assessing Stroke Patient Treatment in Daily Life Conditions

Monitoring physical activities during post-stroke rehabilitation in daily life may help physicians to optimize and tailor the training program for patients. The European research project INTERACTION (FP7-ICT-2011-7-287351) evaluated motor capabilities in stroke patients during the recovery treatment period. We developed wearable sensing platform based on the sensor fusion among inertial, knitted piezoresistive sensors and textile EMG electrodes. The device was conceived in modular form and consists of a separate shirt, trousers, glove, and shoe. Thanks to the novel fusion approach it has been possible to develop a model for the shoulder taking into account the scapulo-thoracic joint of the scapular girdle, considerably improving the estimation of the hand position in reaching activities. In order to minimize the sensor set used to monitor gait, a single inertial sensor fused with a textile goniometer proved to reconstruct the orientation of all the body segments of the leg. Finally, the sensing glove, endowed with three textile goniometers and three force sensors showed good capabilities in the reconstruction of grasping activities and evaluating the interaction of the hand with the environment, according to the project specifications. This paper reports on the design and the technical evaluation of the performance of the sensing platform, tested on healthy subjects.

Hand synergies: Integration of robotics and neuroscience for understanding the control of biological and artificial hands

The term 'synergy' - from the Greek synergia - means 'working together'. The concept of multiple elements working together towards a common goal has been extensively used in neuroscience to develop theoretical frameworks, experimental approaches, and analytical techniques to understand neural control of movement, and for applications for neuro-rehabilitation. In the past decade, roboticists have successfully applied the framework of synergies to create novel design and control concepts for artificial hands, i.e., robotic hands and prostheses. At the same time, robotic research on the sensorimotor integration underlying the control and sensing of artificial hands has inspired new research approaches in neuroscience, and has provided useful instruments for novel experiments. The ambitious goal of integrating expertise and research approaches in robotics and neuroscience to study the properties and applications of the concept of synergies is generating a number of multidisciplinary cooperative projects, among which the recently finished 4-year European project "The Hand Embodied" (THE). This paper reviews the main insights provided by this framework. Specifically, we provide an overview of neuroscientific bases of hand synergies and introduce how robotics has leveraged the insights from neuroscience for innovative design in hardware and controllers for biomedical engineering applications, including myoelectric hand prostheses, devices for haptics research, and wearable sensing of human hand kinematics. The review also emphasizes how this multidisciplinary collaboration has generated new ways to conceptualize a synergy-based approach for robotics, and provides guidelines and principles for analyzing human behavior and synthesizing artificial robotic systems based on a theory of synergies.

Wearable soft sensing suit for human gait measurement

Wearable robots based on soft materials will augment mobility and performance of the host without restricting natural kinematics. Such wearable robots will need soft sensors to monitor the movement of the wearer and robot outside the lab. Until now wearable soft sensors have not demonstrated significant mechanical robustness nor been systematically characterized for human motion studies of walking and running. Here, we present the design and systematic characterization of a soft sensing suit for monitoring hip, knee, and ankle sagittal plane joint angles. We used hyper-elastic strain sensors based on microchannels of liquid metal embedded within elastomer, but refined their design with the use of discretized stiffness gradients to improve mechanical durability. We found that these robust sensors could stretch up to 396% of their original lengths, would restrict the wearer by less than 0.17% of any given joint’s torque, had gauge factor sensitivities of greater than 2.2, and exhibited less than 2% change in electromechanical specifications through 1500 cycles of loading–unloading. We also evaluated the accuracy and variability of the soft sensing suit by comparing it with joint angle data obtained through optical motion capture. The sensing suit had root mean square (RMS) errors of less than 5° for a walking speed of 0.89 m/s and reached a maximum RMS error of 15° for a running speed of 2.7 m/s. Despite the deviation of absolute measure, the relative repeatability of the sensing suit’s joint angle measurements were statistically equivalent to that of optical motion capture at all speeds. We anticipate that wearable soft sensing will also have applications beyond wearable robotics, such as in medical diagnostics and in human–computer interaction.

Synergy-based hand pose sensing: Reconstruction enhancement

Many applications in human–machine interfaces, information visualization, rehabilitation and entertainment require hand pose reconstruction systems that are both accurate and economic. Unfortunately, economically and ergonomically viable sensing gloves provide limited precision due to the imperfect and incomplete correspondence of sensing models with the anatomical degrees of freedom of the human hand, and because of measurement noise. This paper examines the problem of optimally estimating the posture of a human hand using non-ideal sensing gloves. The main idea is to maximize their performance by exploiting knowledge of how humans most frequently use their hands. To increase the accuracy of pose reconstruction without modifying the glove hardware — hence basically at no extra cost — we propose the collection, organization, and exploitation of information on the probabilistic distribution of human hand poses in common tasks. We discuss how a database of such a priori information can be built, represented in a hierarchy of correlation patterns or postural synergies, and fused with glove data in a consistent way, so as to provide good hand pose reconstruction in spite of insufficient and inaccurate sensing data. Simulations and experiments on a low-cost glove are reported which demonstrate the effectiveness of the proposed techniques.