A Multi-Modal Under-Sensorized Wearable System for Optimal Kinematic and Muscular Tracking of Human Upper Limb Motion

Wearable sensing solutions have emerged as a promising paradigm for monitoring human musculoskeletal state in an unobtrusive way. To increase the deployability of these systems, considerations related to cost reduction and enhanced form factor and wearability tend to discourage the number of sensors in use. In our previous work, we provided a theoretical solution to the problem of jointly reconstructing the entire muscular-kinematic state of the upper limb, when only a limited amount of optimally retrieved sensory data are available. However, the effective implementation of these methods in a physical, under-sensorized wearable has never been attempted before. In this work, we propose to bridge this gap by presenting an under-sensorized system based on inertial measurement units (IMUs) and surface electromyography (sEMG) electrodes for the reconstruction of the upper limb musculoskeletal state, focusing on the minimization of the sensors' number. We found that, relying on two IMUs only and eight sEMG sensors, we can conjointly reconstruct all 17 degrees of freedom (five joints, twelve muscles) of the upper limb musculoskeletal state, yielding a median normalized RMS error of 8.5% on the non-measured joints and 2.5% on the non-measured muscles.

A kinematic and EMG dataset of online adjustment of reach-to-grasp movements to visual perturbations

Control of reach-to-grasp movements for deft and robust interactions with objects requires rapid sensorimotor updating that enables online adjustments to changing external goals (e.g., perturbations or instability of objects we interact with). Rarely do we appreciate the remarkable coordination in reach-to-grasp, until control becomes impaired by neurological injuries such as stroke, neurodegenerative diseases, or even aging. Modeling online control of human reach-to-grasp movements is a challenging problem but fundamental to several domains, including behavioral and computational neuroscience, neurorehabilitation, neural prostheses, and robotics. Currently, there are no publicly available datasets that include online adjustment of reach-to-grasp movements to object perturbations. This work aims to advance modeling efforts of reach-to-grasp movements by making publicly available a large kinematic and EMG dataset of online adjustment of reach-to-grasp movements to instantaneous perturbations of object size and distance performed in immersive haptic-free virtual environment (hf-VE). The presented dataset is composed of a large number of perturbation types (10 for both object size and distance) applied at three different latencies after the start of the movement.

Planning Grasping Motions for Humanoid Robots

This paper addresses the problem of obtaining the required motions for a humanoid robot to perform grasp actions trying to mimic the coordinated hand–arm movements humans do. The first step is the data acquisition and analysis, which consists in capturing human movements while grasping several everyday objects (covering four possible grasp types), mapping them to the robot and computing the hand motion synergies for the pre-grasp and grasp phases (per grasp type). Then, the grasp and motion synthesis step is done, which consists in generating potential grasps for a given object using the four family types, and planning the motions using a bi-directional multi-goal sampling-based planner, which efficiently guides the motion planning following the synergies in a reduced search space, resulting in paths with human-like appearance. The approach has been tested in simulation, thoroughly compared with other state-of-the-art planning algorithms obtaining better results, and also implemented in a real robot.

Human hand kinematic data during feeding and cooking tasks

This work presents a database of human hand kinematics containing data collected during the performance of a wide variety of activities of daily living involving feeding and cooking. The data were recorded using CyberGlove instrumented gloves on both hands measuring 18 degrees of freedom on each. A total of 20 subjects participated in each part of the experiment, and the objects and their arrangement were the same across subjects, although they performed the tasks in a natural non-directed way. This dataset contains a total of 1160 continuous calibrated recordings taken at 100 Hz during the performance of the tasks, with filtered signal. Statistical descriptive analyses from these data are presented. This database can be useful for machine learning purposes and prostheses control, as well as for the characterization of healthy human hand kinematics.

DeepDynamicHand: A Deep Neural Architecture for Labeling Hand Manipulation Strategies in Video Sources Exploiting Temporal Information

Humans are capable of complex manipulation interactions with the environment, relying on the intrinsic adaptability and compliance of their hands. Recently, soft robotic manipulation has attempted to reproduce such an extraordinary behavior, through the design of deformable yet robust end-effectors. To this goal, the investigation of human behavior has become crucial to correctly inform technological developments of robotic hands that can successfully exploit environmental constraint as humans actually do. Among the different tools robotics can leverage on to achieve this objective, deep learning has emerged as a promising approach for the study and then the implementation of neuro-scientific observations on the artificial side. However, current approaches tend to neglect the dynamic nature of hand pose recognition problems, limiting the effectiveness of these techniques in identifying sequences of manipulation primitives underpinning action generation, e.g., during purposeful interaction with the environment. In this work, we propose a vision-based supervised Hand Pose Recognition method which, for the first time, takes into account temporal information to identify meaningful sequences of actions in grasping and manipulation tasks. More specifically, we apply Deep Neural Networks to automatically learn features from hand posture images that consist of frames extracted from grasping and manipulation task videos with objects and external environmental constraints. For training purposes, videos are divided into intervals, each associated to a specific action by a human supervisor. The proposed algorithm combines a Convolutional Neural Network to detect the hand within each video frame and a Recurrent Neural Network to predict the hand action in the current frame, while taking into consideration the history of actions performed in the previous frames. Experimental validation has been performed on two datasets of dynamic hand-centric strategies, where subjects regularly interact with objects and environment. Proposed architecture achieved a very good classification accuracy on both datasets, reaching performance up to 94%, and outperforming state of the art techniques. The outcomes of this study can be successfully applied to robotics, e.g., for planning and control of soft anthropomorphic manipulators.

Recent Data Sets on Object Manipulation: A Survey

Data sets is crucial not only for model learning and evaluation but also to advance knowledge on human behavior, thus fostering mutual inspiration between neuroscience and robotics. However, choosing the right data set to use or creating a new data set is not an easy task, because of the variety of data that can be found in the related literature. The first step to tackle this issue is to collect and organize those that are available. In this work, we take a significant step forward by reviewing data sets that were published in the past 10 years and that are directly related to object manipulation and grasping. We report on modalities, activities, and annotations for each individual data set and we discuss our view on its use for object manipulation. We also compare the data sets and summarize them. Finally, we conclude the survey by providing suggestions and discussing the best practices for the creation of new data sets.

A Synergy-Based Optimally Designed Sensing Glove for Functional Grasp Recognition

Achieving accurate and reliable kinematic hand pose reconstructions represents a challenging task. The main reason for this is the complexity of hand biomechanics, where several degrees of freedom are distributed along a continuous deformable structure. Wearable sensing can represent a viable solution to tackle this issue, since it enables a more natural kinematic monitoring. However, the intrinsic accuracy (as well as the number of sensing elements) of wearable hand pose reconstruction (HPR) systems can be severely limited by ergonomics and cost considerations. In this paper, we combined the theoretical foundations of the optimal design of HPR devices based on hand synergy information, i.e., the inter-joint covariation patterns, with textile goniometers based on knitted piezoresistive fabrics (KPF) technology, to develop, for the first time, an optimally-designed under-sensed glove for measuring hand kinematics. We used only five sensors optimally placed on the hand and completed hand pose reconstruction (described according to a kinematic model with 19 degrees of freedom) leveraging upon synergistic information. The reconstructions we obtained from five different subjects were used to implement an unsupervised method for the recognition of eight functional grasps, showing a high degree of accuracy and robustness.

Hand synergies: Integration of robotics and neuroscience for understanding the control of biological and artificial hands

The term 'synergy' - from the Greek synergia - means 'working together'. The concept of multiple elements working together towards a common goal has been extensively used in neuroscience to develop theoretical frameworks, experimental approaches, and analytical techniques to understand neural control of movement, and for applications for neuro-rehabilitation. In the past decade, roboticists have successfully applied the framework of synergies to create novel design and control concepts for artificial hands, i.e., robotic hands and prostheses. At the same time, robotic research on the sensorimotor integration underlying the control and sensing of artificial hands has inspired new research approaches in neuroscience, and has provided useful instruments for novel experiments. The ambitious goal of integrating expertise and research approaches in robotics and neuroscience to study the properties and applications of the concept of synergies is generating a number of multidisciplinary cooperative projects, among which the recently finished 4-year European project "The Hand Embodied" (THE). This paper reviews the main insights provided by this framework. Specifically, we provide an overview of neuroscientific bases of hand synergies and introduce how robotics has leveraged the insights from neuroscience for innovative design in hardware and controllers for biomedical engineering applications, including myoelectric hand prostheses, devices for haptics research, and wearable sensing of human hand kinematics. The review also emphasizes how this multidisciplinary collaboration has generated new ways to conceptualize a synergy-based approach for robotics, and provides guidelines and principles for analyzing human behavior and synthesizing artificial robotic systems based on a theory of synergies.

Open-source, low-cost, compliant, modular, underactuated fingers: towards affordable prostheses for partial hand amputations

In this paper we present a series of design directions for the development of affordable, compliant, modular, underactuated robot fingers, that can be used as prostheses by amputees that suffer from various partial hand amputations (index to pinky fingers are considered). Our design is based on parametric models that have been derived from hand anthropometry studies. Various interfaces have been considered in order to control the prosthesis, depending on the type and level of amputation. More precisely: 1) An Electromyography (EMG) based interface is used to control the robot fingers employing the EMG signals of the human forearm muscles 2) A flex sensors based interface is used to record the motion of the intact finger/fingers and predict the motion of the prosthesis implementing a synergistic behavior in an efficient manner, 3) A body powered interface is used for those that want to achieve even lower cost, with robust intuitive operation. Following the proposed design directions, an amputee will be able to replicate our fingers and develop personalized, affordable, light-weight but yet efficient prostheses.

Wearable soft sensing suit for human gait measurement

Wearable robots based on soft materials will augment mobility and performance of the host without restricting natural kinematics. Such wearable robots will need soft sensors to monitor the movement of the wearer and robot outside the lab. Until now wearable soft sensors have not demonstrated significant mechanical robustness nor been systematically characterized for human motion studies of walking and running. Here, we present the design and systematic characterization of a soft sensing suit for monitoring hip, knee, and ankle sagittal plane joint angles. We used hyper-elastic strain sensors based on microchannels of liquid metal embedded within elastomer, but refined their design with the use of discretized stiffness gradients to improve mechanical durability. We found that these robust sensors could stretch up to 396% of their original lengths, would restrict the wearer by less than 0.17% of any given joint’s torque, had gauge factor sensitivities of greater than 2.2, and exhibited less than 2% change in electromechanical specifications through 1500 cycles of loading–unloading. We also evaluated the accuracy and variability of the soft sensing suit by comparing it with joint angle data obtained through optical motion capture. The sensing suit had root mean square (RMS) errors of less than 5° for a walking speed of 0.89 m/s and reached a maximum RMS error of 15° for a running speed of 2.7 m/s. Despite the deviation of absolute measure, the relative repeatability of the sensing suit’s joint angle measurements were statistically equivalent to that of optical motion capture at all speeds. We anticipate that wearable soft sensing will also have applications beyond wearable robotics, such as in medical diagnostics and in human–computer interaction.

Synergy-based hand pose sensing: Optimal glove design

In this paper we study the problem of optimally designing glove-based sensing devices for hand pose reconstruction to maximize their potential for precision. In a companion paper we studied the problem of maximizing the reconstruction accuracy of the hand pose from partial and noisy data provided by any given pose sensing device (a sensorized “glove”) taking into account the knowledge of how humans most frequently use their hands in grasping tasks. In this paper we consider the dual problem of how to design pose sensing devices, i.e. how and where to place sensors on a glove, to get maximum information about the actual hand posture. We study the optimal design of gloves of different nature, according to a classification of current sensing technologies adopted in the domain. The objective is to provide, for given a priori information and fixed number of sensor inputs, the optimal design minimizing the reconstruction error statistics (assuming that optimal reconstruction algorithms are adopted). Finally, an experimental evaluation of the proposed method for optimal design is provided.