Optimizing Fingernail Imaging Calibration for 3D Force Magnitude Prediction

Estimating Fingertip Forces, Torques, and Local Curvatures from Fingernail Images

The study of dexterous manipulation has provided important insights into human sensorimotor control as well as inspiration for manipulation strategies in robotic hands. Previous work focused on experimental environment with restrictions. Here, we describe a method using the deformation and color distribution of the fingernail and its surrounding skin to estimate the fingertip forces, torques, and contact surface curvatures for various objects, including the shape and material of the contact surfaces and the weight of the objects. The proposed method circumvents limitations associated with sensorized objects, gloves, or fixed contact surface type. In addition, compared with previous single finger estimation in an experimental environment, we extend the approach to multiple finger force estimation, which can be used for applications such as human grasping analysis. Four algorithms are used, c.q., Gaussian process, convolutional neural networks, neural networks with fast dropout, and recurrent neural networks with fast dropout, to model a mapping from images to the corresponding labels. The results further show that the proposed method has high accuracy to predict force, torque, and contact surface.

Hemodynamic Sensing of 3-D Fingertip Force by Using Nonpulsatile and Pulsatile Signals in the Proximal Part

This study proposed a novel sensing method of 3-D contact force at a fingertip by using a photoplethysmogram (PPG) device on the proximal part of a finger. The proposed system detects nonpulsatile and pulsatile components of PPG signals from both sides of the proximal part, extracts 16 feature values related to the contact force, and estimates the 3-D force by using a multiple linear regression model. In the validation experiments, the participants wore a PPG device at the proximal parts of their index fingers and applied a contact force at the fingertips for the 11 types of touch actions. The results indicated that satisfactory agreements are observed between the system outputs and the reference forces by the calibrated force sensor. Moreover, the results revealed that the most effective number of feature values corresponded to six for the higher reproducible sensing. Although the development of the effective calibration method is expected to increase robustness, we realized that the proposed method can potentially be used for a 3-D input user interface.

Estimation of Object Elasticity by Capturing Fingernail Images During Haptic Palpation

In this study, we present a system that performs natural-touch-based elasticity estimation for an object by using a depth camera. To estimate elasticity, which is defined as an object's Young's modulus, a strain-stress curve is obtained from fingernail images during haptic palpation. From a color image, the proposed system detects a fingernail and extracts 10 feature values related to the contact force; then, it estimates the force using a multiple regression model. Deformation of the object was estimated from the finger's three-dimensional position obtained from both color and depth images. Then, a strain-stress curve was determined using the force and deformation data. Evaluation experiments were designed to obtain the strain-stress curves of five objects from 10 participants; then, the estimation performance was investigated. The results show that the reliable range of sensing was within Young's modulus values of 0.12-5.6 MPa and the precision of the measurement was 55 percent of the estimated elasticity.

Ensemble of shape functions and support vector machines for the estimation of discrete arm muscle activation from external biceps 3D point clouds

BACKGROUND

Muscle activation level is currently being captured using impractical and expensive devices which make their use in telemedicine settings extremely difficult. To address this issue, a prototype is presented of a non-invasive, easy-to-install system for the estimation of a discrete level of muscle activation of the biceps muscle from 3D point clouds captured with RGB-D cameras.

METHODS

A methodology is proposed that uses the ensemble of shape functions point cloud descriptor for the geometric characterization of 3D point clouds, together with support vector machines to learn a classifier that, based on this geometric characterization for some points of view of the biceps, provides a model for the estimation of muscle activation for all neighboring points of view. This results in a classifier that is robust to small perturbations in the point of view of the capturing device, greatly simplifying the installation process for end-users.

RESULTS

In the discrimination of five levels of effort with values up to the maximum voluntary contraction (MVC) of the biceps muscle (3800 g), the best variant of the proposed methodology achieved mean absolute errors of about 9.21% MVC - an acceptable performance for telemedicine settings where the electric measurement of muscle activation is impractical.

CONCLUSIONS

The results prove that the correlations between the external geometry of the arm and biceps muscle activation are strong enough to consider computer vision and supervised learning an alternative with great potential for practical applications in tele-physiotherapy.