Post-stroke hand gesture recognition via one-shot transfer learning using prototypical networks

BACKGROUND

In-home rehabilitation systems are a promising, potential alternative to conventional therapy for stroke survivors. Unfortunately, physiological differences between participants and sensor displacement in wearable sensors pose a significant challenge to classifier performance, particularly for people with stroke who may encounter difficulties repeatedly performing trials. This makes it challenging to create reliable in-home rehabilitation systems that can accurately classify gestures.

METHODS

Twenty individuals who suffered a stroke performed seven different gestures (mass flexion, mass extension, wrist volar flexion, wrist dorsiflexion, forearm pronation, forearm supination, and rest) related to activities of daily living. They performed these gestures while wearing EMG sensors on the forearm, as well as FMG sensors and an IMU on the wrist. We developed a model based on prototypical networks for one-shot transfer learning, K-Best feature selection, and increased window size to improve model accuracy. Our model was evaluated against conventional transfer learning with neural networks, as well as subject-dependent and subject-independent classifiers: neural networks, LGBM, LDA, and SVM.

RESULTS

Our proposed model achieved 82.2% hand-gesture classification accuracy, which was better (P<0.05) than one-shot transfer learning with neural networks (63.17%), neural networks (59.72%), LGBM (65.09%), LDA (63.35%), and SVM (54.5%). In addition, our model performed similarly to subject-dependent classifiers, slightly lower than SVM (83.84%) but higher than neural networks (81.62%), LGBM (80.79%), and LDA (74.89%). Using K-Best features improved the accuracy in 3 of the 6 classifiers used for evaluation, while not affecting the accuracy in the other classifiers. Increasing the window size improved the accuracy of all the classifiers by an average of 4.28%.

CONCLUSION

Our proposed model showed significant improvements in hand-gesture recognition accuracy in individuals who have had a stroke as compared with conventional transfer learning, neural networks and traditional machine learning approaches. In addition, K-Best feature selection and increased window size can further improve the accuracy. This approach could help to alleviate the impact of physiological differences and create a subject-independent model for stroke survivors that improves the classification accuracy of wearable sensors.

TRIAL REGISTRATION NUMBER

The study was registered in Chinese Clinical Trial Registry with registration number CHiCTR1800017568 in 2018/08/04.

Supervised Myoelectrical Hand Gesture Recognition in Post-Acute Stroke Patients with Upper Limb Paresis on Affected and Non-Affected Sides

In clinical practice, acute post-stroke paresis of the extremities fundamentally complicates timely rehabilitation of motor functions; however, recently, residual and distorted musculoskeletal signals have been used to initiate feedback-driven solutions for establishing motor rehabilitation. Here, we investigate the possibilities of basic hand gesture recognition in acute stroke patients with hand paresis using a novel, acute stroke, four-component multidomain feature set (ASF-4) with feature vector weight additions (ASF-14NP, ASF-24P) and supervised learning algorithms trained only by surface electromyography (sEMG). A total of 19 (65.9 ± 12.4 years old; 12 men, seven women) acute stroke survivors (12.4 ± 6.3 days since onset) with hand paresis (Brunnstrom stage 4 ± 1/4 ± 1, SIAS 3 ± 1/3 ± 2, FMA-UE 40 ± 20) performed 10 repetitive hand movements reflecting basic activities of daily living (ADLs): rest, fist, pinch, wrist flexion, wrist extension, finger spread, and thumb up. Signals were recorded using an eight-channel, portable sEMG device with electrode placement on the forearms and thenar areas of both limbs (four sensors on each extremity). Using data preprocessing, semi-automatic segmentation, and a set of extracted feature vectors, support vector machine (SVM), linear discriminant analysis (LDA), and k-nearest neighbors (k-NN) classifiers for statistical comparison and validity (paired t-tests, p-value < 0.05), we were able to discriminate myoelectrical patterns for each gesture on both paretic and non-paretic sides. Despite any post-stroke conditions, the evaluated total accuracy rate by the 10-fold cross-validation using SVM among four-, five-, six-, and seven-gesture models were 96.62%, 94.20%, 94.45%, and 95.57% for non-paretic and 90.37%, 88.48%, 88.60%, and 89.75% for paretic limbs, respectively. LDA had competitive results using PCA whereas k-NN was a less efficient classifier in gesture prediction. Thus, we demonstrate partial efficacy of the combination of sEMG and supervised learning for upper-limb rehabilitation procedures for early acute stroke motor recovery and various treatment applications.

Adversarial Auto-encoder Domain Adaptation for Cold-start Recommendation with Positive and Negative Hypergraphs

This paper presents a novel model named Adversarial Auto-encoder Domain Adaptation (AADA) to handle the recommendation problem under cold-start settings. Specifically, we divide the hypergraph into two hypergraphs, i.e., a positive hypergraph and a negative one. Below, we adopt the cold-start user recommendation for illustration. After achieving positive and negative hypergraphs, we apply hypergraph auto-encoders to them to obtain positive and negative embeddings of warm users and items. Additionally, we employ a multi-layer perceptron to get warm and cold-start user embeddings called regular embeddings. Subsequently, for warm users, we assign positive and negative pseudo-labels to their positive and negative embeddings, respectively, and treat their positive and regular embeddings as the source and target domain data, respectively. Then, we develop a matching discriminator to jointly minimize the classification loss of the positive and negative warm user embeddings and the distribution gap between the positive and regular warm user embeddings. In this way, warm users’ positive and regular embeddings are connected. Since the positive hypergraph maintains the relations between positive warm user and item embeddings, and the regular warm and cold-start user embeddings follow a similar distribution, the regular cold-start user embedding and positive item embedding are bridged to discover their relationship. The proposed model can be easily extended to handle the cold-start item recommendation by changing inputs. We perform extensive experiments on real-world datasets for both cold-start user and cold-start item recommendations. Promising results in terms of precision, recall, NDCG, and hit rate verify the effectiveness of the proposed method.

A Novel Surface Electromyographic Signal-Based Hand Gesture Prediction Using a Recurrent Neural Network

Surface electromyographic signal (sEMG) is a kind of bioelectrical signal, which records the data of muscle activity intensity. Most sEMG-based hand gesture recognition, which uses machine learning as the classifier, depends on feature extraction of sEMG data. Recently, a deep leaning-based approach such as recurrent neural network (RNN) has provided a choice to automatically learn features from raw data. This paper presents a novel hand gesture prediction method by using an RNN model to learn from raw sEMG data and predict gestures. The sEMG signals of 21 short-term hand gestures of 13 subjects were recorded with a Myo armband, which is a non-intrusive, low cost, commercial portable device. At the start of the gesture, the trained model outputs an instantaneous prediction for the sEMG data. Experimental results showed that the more time steps of data that were known, the higher instantaneous prediction accuracy the proposed model gave. The predicted accuracy reached about 89.6% when the data of 40-time steps (200 ms) were used to predict hand gesture. This means that the gesture could be predicted with a delay of 200 ms after the hand starts to perform the gesture, instead of waiting for the end of the gesture.

Real-Time Hand Gesture Recognition Using Surface Electromyography and Machine Learning: A Systematic Literature Review

Today, daily life is composed of many computing systems, therefore interacting with them in a natural way makes the communication process more comfortable. Human-Computer Interaction (HCI) has been developed to overcome the communication barriers between humans and computers. One form of HCI is Hand Gesture Recognition (HGR), which predicts the class and the instant of execution of a given movement of the hand. One possible input for these models is surface electromyography (EMG), which records the electrical activity of skeletal muscles. EMG signals contain information about the intention of movement generated by the human brain. This systematic literature review analyses the state-of-the-art of real-time hand gesture recognition models using EMG data and machine learning. We selected and assessed 65 primary studies following the Kitchenham methodology. Based on a common structure of machine learning-based systems, we analyzed the structure of the proposed models and standardized concepts in regard to the types of models, data acquisition, segmentation, preprocessing, feature extraction, classification, postprocessing, real-time processing, types of gestures, and evaluation metrics. Finally, we also identified trends and gaps that could open new directions of work for future research in the area of gesture recognition using EMG.