Efficient correction of armband rotation for myoelectric-based gesture control interface

Robust gesture recognition based on attention-deep fast convolutional neural network and surface electromyographic signals

The surface electromyographic (sEMG) signals reflect human motor intention and can be utilized for human-machine interfaces (HMI). Comparing to the sparse multi-channel (SMC) electrodes, the high-density (HD) electrodes have a large number of electrodes and compact space between electrodes, which can achieve more sEMG information and have the potential to achieve higher performance in myocontrol. However, when the HD electrodes grid shift or damage, it will affect gesture recognition and reduce recognition accuracy. To minimize the impact resulting from the electrodes shift and damage, we proposed an attention deep fast convolutional neural network (attention-DFCNN) model by utilizing the temporary and spatial characteristics of high-density surface electromyography (HD-sEMG) signals. Contrary to the previous methods, which are mostly base on sEMG temporal features, the attention-DFCNN model can improve the robustness and stability by combining the spatial and temporary features. The performance of the proposed model was compared with other classical method and deep learning methods. We used the dataset provided by The University Medical Center Göttingen. Seven able-bodied subjects and one amputee involved in this work. Each subject executed nine gestures under the electrodes shift (10 mm) and damage (6 channels). As for the electrodes shift 10 mm in four directions (inwards; onwards; upwards; downwards) on seven able-bodied subjects, without any pre-training, the average accuracy of attention-DFCNN (0.942 ± 0.04) is significantly higher than LSDA (0.910 ± 0.04, p < 0.01), CNN (0.920 ± 0.05, p < 0.01), TCN (0.840 ± 0.07, p < 0.01), LSTM (0.864 ± 0.08, p < 0.01), attention-BiLSTM (0.852 ± 0.07, p < 0.01), Transformer (0.903 ± 0.07, p < 0.01) and Swin-Transformer (0.908 ± 0.09, p < 0.01). The proposed attention-DFCNN algorithm and the way of combining the spatial and temporary features of sEMG signals can significantly improve the recognition rate when the HD electrodes grid shift or damage during wear.

Understanding the influence of confounding factors in myoelectric control for discrete gesture recognition

Discrete myoelectric control-based gesture recognition has recently gained interest as a possible input modality for many emerging ubiquitous computing applications. Unlike the continuous control commonly employed in powered prostheses, discrete systems seek to recognize the dynamic sequences associated with gestures to generate event-based inputs. More akin to those used in general-purpose human-computer interaction, these could include, for example, a flick of the wrist to dismiss a phone call or a double tap of the index finger and thumb to silence an alarm. Moelectric control systems have been shown to achieve near-perfect classification accuracy, but in highly constrained offline settings. Real-world, online systems are subject to 'confounding factors' (i.e. factors that hinder the real-world robustness of myoelectric control that are not accounted for during typical offline analyses), which inevitably degrade system performance, limiting their practical use. Although these factors have been widely studied in continuous prosthesis control, there has been little exploration of their impacts on discrete myoelectric control systems for emerging applications and use cases. Correspondingly, this work examines, for the first time, three confounding factors and their effect on the robustness of discrete myoelectric control: (1)limb position variability, (2)cross-day use, and a newly identified confound faced by discrete systems (3)gesture elicitation speed. Results from four different discrete myoelectric control architectures: (1) Majority Vote LDA, (2) Dynamic Time Warping, (3) an LSTM network trained with Cross Entropy, and (4) an LSTM network trained with Contrastive Learning, show that classification accuracy is significantly degraded (p<0.05) as a result of each of these confounds. This work establishes that confounding factors are a critical barrier that must be addressed to enable the real-world adoption of discrete myoelectric control for robust and reliable gesture recognition.

Hand Gesture Interface for Robot Path Definition in Collaborative Applications: Implementation and Comparative Study

The article explores the possibilities of using hand gestures as a control interface for robotic systems in a collaborative workspace. The development of hand gesture control interfaces has become increasingly important in everyday life as well as professional contexts such as manufacturing processes. We present a system designed to facilitate collaboration between humans and robots in manufacturing processes that require frequent revisions of the robot path and that allows direct definition of the waypoints, which differentiates our system from the existing ones. We introduce a novel and intuitive approach to human-robot cooperation through the use of simple gestures. As part of a robotic workspace, a proposed interface was developed and implemented utilising three RGB-D sensors for monitoring the operator's hand movements within the workspace. The system employs distributed data processing through multiple Jetson Nano units, with each unit processing data from a single camera. MediaPipe solution is utilised to localise the hand landmarks in the RGB image, enabling gesture recognition. We compare the conventional methods of defining robot trajectories with their developed gesture-based system through an experiment with 20 volunteers. The experiment involved verification of the system under realistic conditions in a real workspace closely resembling the intended industrial application. Data collected during the experiment included both objective and subjective parameters. The results indicate that the gesture-based interface enables users to define a given path objectively faster than conventional methods. We critically analyse the features and limitations of the developed system and suggest directions for future research. Overall, the experimental results indicate the usefulness of the developed system as it can speed up the definition of the robot's path.

Bio-robotics research for non-invasive myoelectric neural interfaces for upper-limb prosthetic control: a 10-year perspective review

A decade ago, a group of researchers from academia and industry identified a dichotomy between the industrial and academic state-of-the-art in upper-limb prosthesis control, a widely used bio-robotics application. They proposed that four key technical challenges, if addressed, could bridge this gap and translate academic research into clinically and commercially viable products. These challenges are unintuitive control schemes, lack of sensory feedback, poor robustness and single sensor modality. Here, we provide a perspective review on the research effort that occurred in the last decade, aiming at addressing these challenges. In addition, we discuss three research areas essential to the recent development in upper-limb prosthetic control research but were not envisioned in the review 10 years ago: deep learning methods, surface electromyogram decomposition and open-source databases. To conclude the review, we provide an outlook into the near future of the research and development in upper-limb prosthetic control and beyond.

An Extended Spatial Transformer Convolutional Neural Network for Gesture Recognition and Self-Calibration Based on Sparse sEMG Electrodes

sEMG-based gesture recognition is widely applied in human-machine interaction system by its unique advantages. However, the accuracy of recognition drops significantly as electrodes shift. Besides, in applications such as VR, virtual hands should be shown in reasonable posture by self-calibration. We propose an armband fusing sEMG and IMU with autonomously adjustable gain, and an extended spatial transformer convolutional neural network (EST-CNN) with feature enhanced pretreatment (FEP) to accomplish both gesture recognition and self-calibration via a one-shot processing. Different from anthropogenic calibration methods, spatial transformer layers (STL) in EST-CNN automatically learn the transformation relation, and explicitly express the rotational angle for coarse correction. Due to the shape change of feature pattern as rotational shift, we design the fine tuning layer (FTL) which is able to regulate rotational angle within 45°. By combining STL, FTL and IMU-based posture, EST-CNN is able to calculate non-discretized angle, and achieves high resolution of posture estimation based on sparse sEMG electrodes. Experiments collect frequently-used 3 gestures of 4 subjects in equidistant angles to evaluate EST-CNN. The results under electrodes shift show that the accuracy of gesture recognition is 97.06%, which is 5.81% higher than CNN, the fitness between estimated and true rotational angle is 99.44%.

Wrist-worn Hand Gesture Recognition while Walking via Transfer Learning

Walking, one of the most common daily activities, causes unwanted movement artifacts which can significantly deteriorate hand gesture recognition accuracy. However, traditional hand gesture recognition algorithms are typically developed and validated with wrist-worn devices only during static human poses, neglecting the critical importance of dynamic effects on gesture accuracy. Thus, we developed and validated a signal decomposition approach via empirical mode decomposition to accurately segment target gestures from coupled raw signals during dynamic walking and a transfer learning method based on distribution adaptation to enable gesture recognition through domain transfer between dynamic walking and static standing scenarios. Ten healthy subjects performed seven hand gestures during both walking and standing experiments while wearing an IMU wrist-worn device. Experimental results showed that the signal decomposition approach reduced the gesture detection error by 83.8%, and the transfer learning approach (20% transfer rate) improved hand gesture recognition accuracy by 15.1%. This ground-breaking work demonstrates the feasibility of hand gesture recognition while walking via wrist-worn sensing. These findings serve to inform real-life and ubiquitous adoption of wrist-worn hand gesture recognition for intuitive human-machine interaction in dynamic walking situations.

Emerging Wearable Interfaces and Algorithms for Hand Gesture Recognition: A Survey

Hands are vital in a wide range of fundamental daily activities, and neurological diseases that impede hand function can significantly affect quality of life. Wearable hand gesture interfaces hold promise to restore and assist hand function and to enhance human-human and human-computer communication. The purpose of this review is to synthesize current novel sensing interfaces and algorithms for hand gesture recognition, and the scope of applications covers rehabilitation, prosthesis control, sign language recognition, and human-computer interaction. Results showed that electrical, dynamic, acoustical/vibratory, and optical sensing were the primary input modalities in gesture recognition interfaces. Two categories of algorithms were identified: 1) classification algorithms for predefined, fixed hand poses and 2) regression algorithms for continuous finger and wrist joint angles. Conventional machine learning algorithms, including linear discriminant analysis, support vector machines, random forests, and non-negative matrix factorization, have been widely used for a variety of gesture recognition applications, and deep learning algorithms have more recently been applied to further facilitate the complex relationship between sensor signals and multi-articulated hand postures. Future research should focus on increasing recognition accuracy with larger hand gesture datasets, improving reliability and robustness for daily use outside of the laboratory, and developing softer, less obtrusive interfaces.

A novel energy-motion model for continuous sEMG decoding: from muscle energy to motor pattern

At present, sEMG-based gesture recognition requires vast amounts of training data; otherwise it is limited to a few gestures. Objective. This paper presents a novel dynamic energy model that decodes continuous hand actions by training small amounts of sEMG data. Approach. The activation of forearm muscles can set the corresponding fingers in motion or state with movement trends. The moving fingers store kinetic energy, and the fingers with movement trends store potential energy. The kinetic energy and potential energy in each finger are dynamically allocated due to the adaptive-coupling mechanism of five-fingers in actual motion. Meanwhile, the sum of the two energies remains constant at a certain muscle activation. We regarded hand movements with the same direction of acceleration for five-finger as the same in energy mode and divided hand movements into ten energy modes. Independent component analysis and machine learning methods were used to model associations between sEMG signals and energy modes and expressed gestures by energy form adaptively. This theory imitates the self-adapting mechanism in actual tasks. Thus, ten healthy subjects were recruited, and three experiments mimicking activities of daily living were designed to evaluate the interface: (1) the expression of untrained gestures, (2) the decoding of the amount of single-finger energy, and (3) real-time control. Main results. (1) Participants completed the untrained hand movements (100/100, p < 0.0001). (2) The interface performed better than chance in the experiment where participants pricked balloons with a needle tip (779/1000, p < 0.0001). (3) In the experiment where participants punched a hole in the plasticine on the balloon, the success rate was over 95% (97.67 ± 5.04%, p < 0.01). Significance. The model can achieve continuous hand actions with speed or force information by training small amounts of sEMG data, which reduces learning task complexity.

Position Identification for Robust Myoelectric Control Against Electrode Shift

The vulnerability to the electrode shift was one of the key barriers to the wide application of pattern recognition-based (PR-based) myoelectric control systems outside the controlled laboratory conditions. To overcome this challenge, a novel framework named position identification (PI) was proposed. In the PI framework, an anchor gesture performed by the user was first analyzed to identify the current electrode position from a pool of potential electrode shift positions. Next, the classifier calibrated by the data of the identified position would be selected for following myoelectric control tasks. The results of the amputee and able-bodied participants both demonstrated that the differential filter combined with majority voting improved the PI accuracy. With only one second contraction of the chosen anchor gesture (hand close), the subsequent PR-based myoelectric control performance was fully restored from eight different electrode shift scenarios, with 1 cm in either or both perpendicular and parallel directions. The classification accuracies with PI framework were not significant before and after the shift ( 0.001). The advantage of restoring performance fully in just one second made it a practical solution to improve the robustness of PR-based myoelectric control systems in a wide range of real-world applications.