Decoding Synergy-Based Hand Movements using Electroencephalography

Biomimetic learning of hand gestures in a humanoid robot

Hand gestures are a natural and intuitive form of communication, and integrating this communication method into robotic systems presents significant potential to improve human-robot collaboration. Recent advances in motor neuroscience have focused on replicating human hand movements from synergies also known as movement primitives. Synergies, fundamental building blocks of movement, serve as a potential strategy adapted by the central nervous system to generate and control movements. Identifying how synergies contribute to movement can help in dexterous control of robotics, exoskeletons, prosthetics and extend its applications to rehabilitation. In this paper, 33 static hand gestures were recorded through a single RGB camera and identified in real-time through the MediaPipe framework as participants made various postures with their dominant hand. Assuming an open palm as initial posture, uniform joint angular velocities were obtained from all these gestures. By applying a dimensionality reduction method, kinematic synergies were obtained from these joint angular velocities. Kinematic synergies that explain 98% of variance of movements were utilized to reconstruct new hand gestures using convex optimization. Reconstructed hand gestures and selected kinematic synergies were translated onto a humanoid robot, Mitra, in real-time, as the participants demonstrated various hand gestures. The results showed that by using only few kinematic synergies it is possible to generate various hand gestures, with 95.7% accuracy. Furthermore, utilizing low-dimensional synergies in control of high dimensional end effectors holds promise to enable near-natural human-robot collaboration.

Quantifying Hand Strength and Isometric Pinch Individuation Using a Flexible Pressure Sensor Grid

Modulating force between the thumb and another digit, or isometric pinch individuation, is critical for daily tasks and can be impaired due to central or peripheral nervous system injury. Because surgical and rehabilitative efforts often focus on regaining this dexterous ability, we need to be able to consistently quantify pinch individuation across time and facilities. Currently, a standardized metric for such an assessment does not exist. Therefore, we tested whether we could use a commercially available flexible pressure sensor grid (Tekscan F-Socket [Tekscan Inc., Norwood, MA, USA]) to repeatedly measure isometric pinch individuation and maximum voluntary contraction (MVC) in twenty right-handed healthy volunteers at two visits. We developed a novel equation informed by the prior literature to calculate isometric individuation scores that quantified percentage of force on the grid generated by the indicated digit. MVC intra-class correlation coefficients (ICCs) for the left and right hands were 0.86 (p < 0.0001) and 0.88 (p < 0.0001), respectively, suggesting MVC measurements were consistent over time. However, individuation score ICCs, were poorer (left index ICC 0.41, p = 0.28; right index ICC -0.02, p = 0.51), indicating that this protocol did not provide a sufficiently repeatable individuation assessment. These data support the need to develop novel platforms specifically for repeatable and objective isometric hand dexterity assessments.