A Synergy-Based Optimally Designed Sensing Glove for Functional Grasp Recognition

Association between wrist-worn free-living accelerometry and hand grip strength in middle-aged and older adults

INTRODUCTION

Wrist-worn activity monitors have seen widespread adoption in recent times, particularly in young and sport-oriented cohorts, while their usage among older adults has remained relatively low. The main limitations are in regards to the lack of medical insights that current mainstream activity trackers can provide to older subjects. One of the most important research areas under investigation currently is the possibility of extrapolating clinical information from these wearable devices.

METHODS

The research question of this study is understanding whether accelerometry data collected for 7-days in free-living environments using a consumer-based wristband device, in conjunction with data-driven machine learning algorithms, is able to predict hand grip strength and possible conditions categorized by hand grip strength in a general population consisting of middle-aged and older adults.

RESULTS

The results of the regression analysis reveal that the performance of the developed models is notably superior to a simple mean-predicting dummy regressor. While the improvement in absolute terms may appear modest, the mean absolute error (6.32 kg for males and 4.53 kg for females) falls within the range considered sufficiently accurate for grip strength estimation. The classification models, instead, excel in categorizing individuals as frail/pre-frail, or healthy, depending on the T-score levels applied for frailty/pre-frailty definition. While cut-off values for frailty vary, the results suggest that the models can moderately detect characteristics associated with frailty (AUC-ROC: 0.70 for males, and 0.76 for females) and viably detect characteristics associated with frailty/pre-frailty (AUC-ROC: 0.86 for males, and 0.87 for females).

CONCLUSIONS

The results of this study can enable the adoption of wearable devices as an efficient tool for clinical assessment in older adults with multimorbidities, improving and advancing integrated care, diagnosis and early screening of a number of widespread diseases.

A Multi-Modal Under-Sensorized Wearable System for Optimal Kinematic and Muscular Tracking of Human Upper Limb Motion

Wearable sensing solutions have emerged as a promising paradigm for monitoring human musculoskeletal state in an unobtrusive way. To increase the deployability of these systems, considerations related to cost reduction and enhanced form factor and wearability tend to discourage the number of sensors in use. In our previous work, we provided a theoretical solution to the problem of jointly reconstructing the entire muscular-kinematic state of the upper limb, when only a limited amount of optimally retrieved sensory data are available. However, the effective implementation of these methods in a physical, under-sensorized wearable has never been attempted before. In this work, we propose to bridge this gap by presenting an under-sensorized system based on inertial measurement units (IMUs) and surface electromyography (sEMG) electrodes for the reconstruction of the upper limb musculoskeletal state, focusing on the minimization of the sensors' number. We found that, relying on two IMUs only and eight sEMG sensors, we can conjointly reconstruct all 17 degrees of freedom (five joints, twelve muscles) of the upper limb musculoskeletal state, yielding a median normalized RMS error of 8.5% on the non-measured joints and 2.5% on the non-measured muscles.

Hand Gesture Recognition Using EGaIn-Silicone Soft Sensors

Exploiting hand gestures for non-verbal communication has extraordinary potential in HCI. A data glove is an apparatus widely used to recognize hand gestures. To improve the functionality of the data glove, a highly stretchable and reliable signal-to-noise ratio sensor is indispensable. To do this, the study focused on the development of soft silicone microchannel sensors using a Eutectic Gallium-Indium (EGaIn) liquid metal alloy and a hand gesture recognition system via the proposed data glove using the soft sensor. The EGaIn-silicone sensor was uniquely designed to include two sensing channels to monitor the finger joint movements and to facilitate the EGaIn alloy injection into the meander-type microchannels. We recruited 15 participants to collect hand gesture dataset investigating 12 static hand gestures. The dataset was exploited to estimate the performance of the proposed data glove in hand gesture recognition. Additionally, six traditional classification algorithms were studied. From the results, a random forest shows the highest classification accuracy of 97.3% and a linear discriminant analysis shows the lowest accuracy of 87.4%. The non-linearity of the proposed sensor deteriorated the accuracy of LDA, however, the other classifiers adequately overcame it and performed high accuracies (>90%).

Synergy-Based Sensor Reduction for Recording the Whole Hand Kinematics

Simultaneous measurement of the kinematics of all hand segments is cumbersome due to sensor placement constraints, occlusions, and environmental disturbances. The aim of this study is to reduce the number of sensors required by using kinematic synergies, which are considered the basic building blocks underlying hand motions. Synergies were identified from the public KIN-MUS UJI database (22 subjects, 26 representative daily activities). Ten synergies per subject were extracted as the principal components explaining at least 95% of the total variance of the angles recorded across all tasks. The 220 resulting synergies were clustered, and candidate angles for estimating the remaining angles were obtained from these groups. Different combinations of candidates were tested and the one providing the lowest error was selected, its goodness being evaluated against kinematic data from another dataset (KINE-ADL BE-UJI). Consequently, the original 16 joint angles were reduced to eight: carpometacarpal flexion and abduction of thumb, metacarpophalangeal and interphalangeal flexion of thumb, proximal interphalangeal flexion of index and ring fingers, metacarpophalangeal flexion of ring finger, and palmar arch. Average estimation errors across joints were below 10% of the range of motion of each joint angle for all the activities. Across activities, errors ranged between 3.1% and 16.8%.

Wearable Health Devices in Health Care: Narrative Systematic Review

BACKGROUND

With the rise of mobile medicine, the development of new technologies such as smart sensing, and the popularization of personalized health concepts, the field of smart wearable devices has developed rapidly in recent years. Among them, medical wearable devices have become one of the most promising fields. These intelligent devices not only assist people in pursuing a healthier lifestyle but also provide a constant stream of health care data for disease diagnosis and treatment by actively recording physiological parameters and tracking metabolic status. Therefore, wearable medical devices have the potential to become a mainstay of the future mobile medical market.

OBJECTIVE

Although previous reviews have discussed consumer trends in wearable electronics and the application of wearable technology in recreational and sporting activities, data on broad clinical usefulness are lacking. We aimed to review the current application of wearable devices in health care while highlighting shortcomings for further research. In addition to daily health and safety monitoring, the focus of our work was mainly on the use of wearable devices in clinical practice.

METHODS

We conducted a narrative review of the use of wearable devices in health care settings by searching papers in PubMed, EMBASE, Scopus, and the Cochrane Library published since October 2015. Potentially relevant papers were then compared to determine their relevance and reviewed independently for inclusion.

RESULTS

A total of 82 relevant papers drawn from 960 papers on the subject of wearable devices in health care settings were qualitatively analyzed, and the information was synthesized. Our review shows that the wearable medical devices developed so far have been designed for use on all parts of the human body, including the head, limbs, and torso. These devices can be classified into 4 application areas: (1) health and safety monitoring, (2) chronic disease management, (3) disease diagnosis and treatment, and (4) rehabilitation. However, the wearable medical device industry currently faces several important limitations that prevent further use of wearable technology in medical practice, such as difficulties in achieving user-friendly solutions, security and privacy concerns, the lack of industry standards, and various technical bottlenecks.

CONCLUSIONS

We predict that with the development of science and technology and the popularization of personalized health concepts, wearable devices will play a greater role in the field of health care and become better integrated into people's daily lives. However, more research is needed to explore further applications of wearable devices in the medical field. We hope that this review can provide a useful reference for the development of wearable medical devices.

A Survey on Hand Pose Estimation with Wearable Sensors and Computer-Vision-Based Methods

Real-time sensing and modeling of the human body, especially the hands, is an important research endeavor for various applicative purposes such as in natural human computer interactions. Hand pose estimation is a big academic and technical challenge due to the complex structure and dexterous movement of human hands. Boosted by advancements from both hardware and artificial intelligence, various prototypes of data gloves and computer-vision-based methods have been proposed for accurate and rapid hand pose estimation in recent years. However, existing reviews either focused on data gloves or on vision methods or were even based on a particular type of camera, such as the depth camera. The purpose of this survey is to conduct a comprehensive and timely review of recent research advances in sensor-based hand pose estimation, including wearable and vision-based solutions. Hand kinematic models are firstly discussed. An in-depth review is conducted on data gloves and vision-based sensor systems with corresponding modeling methods. Particularly, this review also discusses deep-learning-based methods, which are very promising in hand pose estimation. Moreover, the advantages and drawbacks of the current hand gesture estimation methods, the applicative scope, and related challenges are also discussed.

Kinematic synergies of hand grasps: a comprehensive study on a large publicly available dataset

Background

Hand grasp patterns require complex coordination. The reduction of the kinematic dimensionality is a key process to study the patterns underlying hand usage and grasping. It allows to define metrics for motor assessment and rehabilitation, to develop assistive devices and prosthesis control methods. Several studies were presented in this field but most of them targeted a limited number of subjects, they focused on postures rather than entire grasping movements and they did not perform separate analysis for the tasks and subjects, which can limit the impact on rehabilitation and assistive applications. This paper provides a comprehensive mapping of synergies from hand grasps targeting activities of daily living. It clarifies several current limits of the field and fosters the development of applications in rehabilitation and assistive robotics.

Methods

In this work, hand kinematic data of 77 subjects, performing up to 20 hand grasps, were acquired with a data glove (a 22-sensor CyberGlove II data glove) and analyzed. Principal Component Analysis (PCA) and hierarchical cluster analysis were used to extract and group kinematic synergies that summarize the coordination patterns available for hand grasps.

Results

Twelve synergies were found to account for > 80% of the overall variation. The first three synergies accounted for more than 50% of the total amount of variance and consisted of: the flexion and adduction of the Metacarpophalangeal joint (MCP) of fingers 3 to 5 (synergy #1), palmar arching and flexion of the wrist (synergy #2) and opposition of the thumb (synergy #3). Further synergies refine movements and have higher variability among subjects.

Conclusion

Kinematic synergies are extracted from a large number of subjects (77) and grasps related to activities of daily living (20). The number of motor modules required to perform the motor tasks is higher than what previously described. Twelve synergies are responsible for most of the variation in hand grasping. The first three are used as primary synergies, while the remaining ones target finer movements (e.g. independence of thumb and index finger). The results generalize the description of hand kinematics, better clarifying several limits of the field and fostering the development of applications in rehabilitation and assistive robotics.

Gesture Prediction Using Wearable Sensing Systems with Neural Networks for Temporal Data Analysis

A human gesture prediction system can be used to estimate human gestures in advance of the actual action to reduce delays in interactive systems. Hand gestures are particularly necessary for human–computer interaction. Therefore, the gesture prediction system must be able to capture hand movements that are both complex and quick. We have already reported a method that allows strain sensors and wearable devices to be fabricated in a simple and easy manner using pyrolytic graphite sheets (PGSs). The wearable electronics could detect various types of human gestures with high sensitivity, high durability, and fast response. In this study, we demonstrated hand gesture prediction by artificial neural networks (ANNs) using gesture data obtained from data gloves based on PGSs. Our experiments entailed measuring the hand gestures of subjects for learning purposes and we used these data to create four-layered ANNs, which enabled the proposed system to successfully predict hand gestures in real time. A comparison of the proposed method with other algorithms using temporal data analysis suggested that the hand gesture prediction system using ANNs would be able to forecast various types of hand gestures using resistance data obtained from wearable devices based on PGSs.

DeepDynamicHand: A Deep Neural Architecture for Labeling Hand Manipulation Strategies in Video Sources Exploiting Temporal Information

Humans are capable of complex manipulation interactions with the environment, relying on the intrinsic adaptability and compliance of their hands. Recently, soft robotic manipulation has attempted to reproduce such an extraordinary behavior, through the design of deformable yet robust end-effectors. To this goal, the investigation of human behavior has become crucial to correctly inform technological developments of robotic hands that can successfully exploit environmental constraint as humans actually do. Among the different tools robotics can leverage on to achieve this objective, deep learning has emerged as a promising approach for the study and then the implementation of neuro-scientific observations on the artificial side. However, current approaches tend to neglect the dynamic nature of hand pose recognition problems, limiting the effectiveness of these techniques in identifying sequences of manipulation primitives underpinning action generation, e.g., during purposeful interaction with the environment. In this work, we propose a vision-based supervised Hand Pose Recognition method which, for the first time, takes into account temporal information to identify meaningful sequences of actions in grasping and manipulation tasks. More specifically, we apply Deep Neural Networks to automatically learn features from hand posture images that consist of frames extracted from grasping and manipulation task videos with objects and external environmental constraints. For training purposes, videos are divided into intervals, each associated to a specific action by a human supervisor. The proposed algorithm combines a Convolutional Neural Network to detect the hand within each video frame and a Recurrent Neural Network to predict the hand action in the current frame, while taking into consideration the history of actions performed in the previous frames. Experimental validation has been performed on two datasets of dynamic hand-centric strategies, where subjects regularly interact with objects and environment. Proposed architecture achieved a very good classification accuracy on both datasets, reaching performance up to 94%, and outperforming state of the art techniques. The outcomes of this study can be successfully applied to robotics, e.g., for planning and control of soft anthropomorphic manipulators.

Recent Data Sets on Object Manipulation: A Survey

Data sets is crucial not only for model learning and evaluation but also to advance knowledge on human behavior, thus fostering mutual inspiration between neuroscience and robotics. However, choosing the right data set to use or creating a new data set is not an easy task, because of the variety of data that can be found in the related literature. The first step to tackle this issue is to collect and organize those that are available. In this work, we take a significant step forward by reviewing data sets that were published in the past 10 years and that are directly related to object manipulation and grasping. We report on modalities, activities, and annotations for each individual data set and we discuss our view on its use for object manipulation. We also compare the data sets and summarize them. Finally, we conclude the survey by providing suggestions and discussing the best practices for the creation of new data sets.