Simultaneous Prediction of Wrist/Hand Motion via Wearable Ultrasound Sensing

Unsupervised Domain Adaptation for Inter-Session Re-Calibration of Ultrasound-Based HMIs

Human-Machine Interfaces (HMIs) have gained popularity as they allow for an effortless and natural interaction between the user and the machine by processing information gathered from a single or multiple sensing modalities and transcribing user intentions to the desired actions. Their operability depends on frequent periodic re-calibration using newly acquired data due to their adaptation needs in dynamic environments, where test-time data continuously change in unforeseen ways, a cause that significantly contributes to their abandonment and remains unexplored by the Ultrasound-based (US-based) HMI community. In this work, we conduct a thorough investigation of Unsupervised Domain Adaptation (UDA) algorithms for the re-calibration of US-based HMIs during within-day sessions, which utilize unlabeled data for re-calibration. Our experimentation led us to the proposal of a CNN-based architecture for simultaneous wrist rotation angle and finger gesture prediction that achieves comparable performance with the state-of-the-art while featuring 87.92% less trainable parameters. According to our findings, DANN (a Domain-Adversarial training algorithm), with proper initialization, offers an average 24.99% classification accuracy performance enhancement when compared to no re-calibration setting. However, our results suggest that in cases where the experimental setup and the UDA configuration may differ, observed enhancements would be rather small or even unnoticeable.

Wearable ultrasound devices: An emerging era for biomedicine and clinical translation

In recent years, personalized diagnosis and treatment have gained significant recognition and rapid development in the biomedicine and healthcare. Due to the flexibility, portability and excellent compatibility, wearable ultrasound (WUS) devices have become emerging personalized medical devices with great potential for development. Currently, with the development of the ongoing advancements in materials and structural design of the ultrasound transducers, WUS devices have improved performance and are increasingly applied in the medical field. In this review, we provide an overview of the design and structure of WUS devices, focusing on their application for diagnosis and treatment of various diseases from a clinical application perspective, and then explore the issues that need to be addressed before clinical translation. Finally, we summarize the progress made in the development of WUS devices, and discuss the current challenges and the future direction of their development. In conclusion, WUS devices usher an emerging era for biomedicine with great clinical promise.

Design of a Wearable Real-Time Hand Motion Tracking System Using an Array of Soft Polymer Acoustic Waveguides

Robust hand motion tracking holds promise for improved human-machine interaction in diverse fields, including virtual reality, and automated sign language translation. However, current wearable hand motion tracking approaches are typically limited in detection performance, wearability, and durability. This article presents a hand motion tracking system using multiple soft polymer acoustic waveguides (SPAWs). The innovative use of SPAWs as strain sensors offers several advantages that address the limitations. SPAWs are easily manufactured by casting a soft polymer shaped as a soft acoustic waveguide and containing a commercially available small ceramic piezoelectric transducer. When used as strain sensors, SPAWs demonstrate high stretchability (up to 100%), high linearity (R2 > 0.996 in all quasi-static, dynamic, and durability tensile tests), negligible hysteresis (<0.7410% under strain of up to 100%), excellent repeatability, and outstanding durability (up to 100,000 cycles). SPAWs also show high accuracy for continuous finger angle estimation (average root-mean-square errors [RMSE] <2.00°) at various flexion-extension speeds. Finally, a hand-tracking system is designed based on a SPAW array. An example application is developed to demonstrate the performance of SPAWs in real-time hand motion tracking in a three-dimensional (3D) virtual environment. To our knowledge, the system detailed in this article is the first to use soft acoustic waveguides to capture human motion. This work is part of an ongoing effort to develop soft sensors using both time and frequency domains, with the goal of extracting decoupled signals from simple sensing structures. As such, it represents a novel and promising path toward soft, simple, and wearable multimodal sensors.

Predicting Wrist Joint Angles from the Kinematics of the Arm: Application to the Control of Upper Limb Prostheses

Automation of wrist rotations in upper limb prostheses allows simplification of the human-machine interface, reducing the user's mental load and avoiding compensatory movements. This study explored the possibility of predicting wrist rotations in pick-and-place tasks based on kinematic information from the other arm joints. To do this, the position and orientation of the hand, forearm, arm, and back were recorded from five subjects during transport of a cylindrical and a spherical object between four different locations on a vertical shelf. The rotation angles in the arm joints were obtained from the records and used to train feed-forward neural networks (FFNNs) and time-delay neural networks (TDNNs) in order to predict wrist rotations (flexion/extension, abduction/adduction, and pronation/supination) based on the angles at the elbow and shoulder. Correlation coefficients between actual and predicted angles of 0.88 for the FFNN and 0.94 for the TDNN were obtained. These correlations improved when object information was added to the network or when it was trained separately for each object (0.94 for the FFNN, 0.96 for the TDNN). Similarly, it improved when the network was trained specifically for each subject. These results suggest that it would be feasible to reduce compensatory movements in prosthetic hands for specific tasks by using motorized wrists and automating their rotation based on kinematic information obtained with sensors appropriately positioned in the prosthesis and the subject's body.

Low-Density sEMG-Based Pattern Recognition of Unrelated Movements Rejection for Wrist Joint Rehabilitation

sEMG-based pattern recognition commonly assumes a limited number of target categories, and the classifiers often predict each target category depending on probability. In wrist rehabilitation training, the patients may make movements that do not belong to the target category unconsciously. However, most pattern recognition methods can only identify limited patterns and are prone to be disturbed by abnormal movement, especially for wrist joint movements. To address the above the problem, a sEMG-based rejection method for unrelated movements is proposed to identify wrist joint unrelated movements using center loss. In this paper, the sEMG signal collected by the Myo armband is used as the input of the sEMG control method. First, the sEMG signal is processed by sliding signal window and image coding. Then, the CNN with center loss and softmax loss is used to describe the spatial information from the sEMG image to extract discriminative features and target movement recognition. Finally, the deep spatial information is used to train the AE to reject unrelated movements based on the reconstruction loss. The results show that the proposed method can realize the target movements recognition and reject unrelated movements with an F-score of 93.4% and a rejection accuracy of 95% when the recall is 0.9, which reveals the effectiveness of the proposed method.

Controlling Upper Limb Prostheses Using Sonomyography (SMG): A Review

This paper presents a critical review and comparison of the results of recently published studies in the fields of human-machine interface and the use of sonomyography (SMG) for the control of upper limb prothesis. For this review paper, a combination of the keywords "Human Machine Interface", "Sonomyography", "Ultrasound", "Upper Limb Prosthesis", "Artificial Intelligence", and "Non-Invasive Sensors" was used to search for articles on Google Scholar and PubMed. Sixty-one articles were found, of which fifty-nine were used in this review. For a comparison of the different ultrasound modes, feature extraction methods, and machine learning algorithms, 16 articles were used. Various modes of ultrasound devices for prosthetic control, various machine learning algorithms for classifying different hand gestures, and various feature extraction methods for increasing the accuracy of artificial intelligence used in their controlling systems are reviewed in this article. The results of the review article show that ultrasound sensing has the potential to be used as a viable human-machine interface in order to control bionic hands with multiple degrees of freedom. Moreover, different hand gestures can be classified by different machine learning algorithms trained with extracted features from collected data with an accuracy of around 95%.

Wearable Real-Time Gesture Recognition Scheme Based on A-Mode Ultrasound

A-mode ultrasound has the advantages of high resolution, easy calculation and low cost in predicting dexterous gestures. In order to accelerate the popularization of A-mode ultrasound gesture recognition technology, we designed a human-machine interface that can interact with the user in real-time. Data processing includes Gaussian filtering, feature extraction and PCA dimensionality reduction. The NB, LDA and SVM algorithms were selected to train machine learning models. The whole process was written in C++ to classify gestures in real-time. This paper conducts offline and real-time experiments based on HMI-A (Human-machine interface based on A-mode ultrasound), including ten subjects and ten common gestures. To demonstrate the effectiveness of HMI-A and avoid accidental interference, the offline experiment collected ten rounds of gestures for each subject for ten-fold cross-validation. The results show that the offline recognition accuracy is 96.92% ± 1.92%. The real-time experiment was evaluated by four online performance metrics: action selection time, action completion time, action completion rate and real-time recognition accuracy. The results show that the action completion rate is 96.0% ± 3.6%, and the real-time recognition accuracy is 83.8% ± 6.9%. This study verifies the great potential of wearable A-mode ultrasound technology, and provides a wider range of application scenarios for gesture recognition.

Simultaneous Prediction of Wrist and Hand Motions via Wearable Ultrasound Sensing for Natural Control of Hand Prostheses

Simultaneous prediction of wrist and hand motions is essential for the natural interaction with hand prostheses. In this paper, we propose a novel multi-out Gaussian process (MOGP) model and a multi-task deep learning (MTDL) algorithm to achieve simultaneous prediction of wrist rotation (pronation/ supination)¹ and finger gestures for transradial amputees via a wearable ultrasound array. We target six finger gestures with concurrent wrist rotation in four transradial amputees. Results show that MOGP outperforms previously reported subclass discriminant analysis for both predictions of discrete finger gestures and continuous wrist rotation. Moreover, we find that MTDL has the potential to improve the accuracy of finger gesture prediction compared to MOGP and classification-specific deep learning, albeit at the expense of reducing the accuracy of wrist rotation prediction. Extended comparative analysis shows the superiority of ultrasound over surface electromyography. This paper prioritizes exploring the performance of wearable ultrasound on the simultaneous prediction of wrist and hand motions for transradial amputees, demonstrating the potential of ultrasound in future prosthetic control. Our ultrasound-based adaptive prosthetic control dataset (UltraPro) will be released to promote the development of the prosthetic community.

Simultaneous control of 2DOF upper-limb prosthesis with body compensations-based control: a multiple cases study

Controlling several joints simultaneously is a common feature of natural arm movements. Robotic prostheses shall offer this possibility to their wearer. Yet, existing approaches to control a robotic upper-limb prosthesis from myoelectric interfaces do not satisfactorily respond to this need: standard methods provide sequential joint-by-joint motion control only; advanced pattern recognition-based approaches allow the control of a limited subset of synchronized multi-joint movements and remain complex to set up. In this paper, we exploit a control method of an upper-limb prosthesis based on body motion measurement called Compensations Cancellation Control (CCC). It offers a straightforward simultaneous control of the intermediate joints, namely the wrist and the elbow. Four transhumeral amputated participants performed the Refined Rolyan Clothespin Test with an experimental prosthesis alternatively running CCC and conventional joint-by-joint myoelectric control. Task performance, joint motions, body compensations and cognitive load were assessed. This experiment shows that CCC restores simultaneity between prosthetic joints while maintaining the level of performance of conventional myoelectric control (used on a daily basis by three participants), without increasing compensatory motions nor cognitive load.

First Demonstration of Functional Task Performance Using a Sonomyographic Prosthesis: A Case Study

Ultrasound-based sensing of muscle deformation, known as sonomyography, has shown promise for accurately classifying the intended hand grasps of individuals with upper limb loss in offline settings. Building upon this previous work, we present the first demonstration of real-time prosthetic hand control using sonomyography to perform functional tasks. An individual with congenital bilateral limb absence was fitted with sockets containing a low-profile ultrasound transducer placed over forearm muscle tissue in the residual limbs. A classifier was trained using linear discriminant analysis to recognize ultrasound images of muscle contractions for three discrete hand configurations (rest, tripod grasp, index finger point) under a variety of arm positions designed to cover the reachable workspace. A prosthetic hand mounted to the socket was then controlled using this classifier. Using this real-time sonomyographic control, the participant was able to complete three functional tasks that required selecting different hand grasps in order to grasp and move one-inch wooden blocks over a broad range of arm positions. Additionally, these tests were successfully repeated without retraining the classifier across 3 hours of prosthesis use and following simulated donning and doffing of the socket. This study supports the feasibility of using sonomyography to control upper limb prostheses in real-world applications.

Evaluating Electromyography and Sonomyography Sensor Fusion to Estimate Lower-Limb Kinematics Using Gaussian Process Regression

Research on robotic lower-limb assistive devices over the past decade has generated autonomous, multiple degree-of-freedom devices to augment human performance during a variety of scenarios. However, the increase in capabilities of these devices is met with an increase in the complexity of the overall control problem and requirement for an accurate and robust sensing modality for intent recognition. Due to its ability to precede changes in motion, surface electromyography (EMG) is widely studied as a peripheral sensing modality for capturing features of muscle activity as an input for control of powered assistive devices. In order to capture features that contribute to muscle contraction and joint motion beyond muscle activity of superficial muscles, researchers have introduced sonomyography, or real-time dynamic ultrasound imaging of skeletal muscle. However, the ability of these sonomyography features to continuously predict multiple lower-limb joint kinematics during widely varying ambulation tasks, and their potential as an input for powered multiple degree-of-freedom lower-limb assistive devices is unknown. The objective of this research is to evaluate surface EMG and sonomyography, as well as the fusion of features from both sensing modalities, as inputs to Gaussian process regression models for the continuous estimation of hip, knee and ankle angle and velocity during level walking, stair ascent/descent and ramp ascent/descent ambulation. Gaussian process regression is a Bayesian nonlinear regression model that has been introduced as an alternative to musculoskeletal model-based techniques. In this study, time-intensity features of sonomyography on both the anterior and posterior thigh along with time-domain features of surface EMG from eight muscles on the lower-limb were used to train and test subject-dependent and task-invariant Gaussian process regression models for the continuous estimation of hip, knee and ankle motion. Overall, anterior sonomyography sensor fusion with surface EMG significantly improved estimation of hip, knee and ankle motion for all ambulation tasks (level ground, stair and ramp ambulation) in comparison to surface EMG alone. Additionally, anterior sonomyography alone significantly improved errors at the hip and knee for most tasks compared to surface EMG. These findings help inform the implementation and integration of volitional control strategies for robotic assistive technologies.

sEMG-Based Motion Recognition of Upper Limb Rehabilitation Using the Improved Yolo-v4 Algorithm

The surface electromyography (sEMG) signal is widely used as a control source of the upper limb exoskeleton rehabilitation robot. However, the traditional way of controlling the exoskeleton robot by the sEMG signal requires one to specially extract and calculate for complex sEMG features. Moreover, due to the huge amount of calculation and individualized difference, the real-time control of the exoskeleton robot cannot be realized. Therefore, this paper proposes a novel method using an improved detection algorithm to recognize limb joint motion and detect joint angle based on sEMG images, aiming to obtain a high-security and fast-processing action recognition strategy. In this paper, MobileNetV2 combined the Ghost module as the feature extraction network to obtain the pretraining model. Then, the target detection network Yolo-V4 was used to estimate the six movement categories of the upper limb joints and to predict the joint movement angles. The experimental results showed that the proposed motion recognition methods were available. Every 100 pictures can be accurately identified in approximately 78 pictures, and the processing speed of every single picture on the PC side was 17.97 ms. For the train data, the mAP@0.5 could reach 82.3%, and mAP@0.5–0.95 could reach 0.42; for the verification data, the average recognition accuracy could reach 80.7%.

Development of a Wearable Human-Machine Interface to Track Forearm Rotation via an Optical Sensor

The goal of this research was to develop an intuitive wearable human-machine interface (HMI), utilizing an optical sensor. The proposed system quantifies wrist pronation and supination using an optical displacement sensor. Compared with existing systems, this HMI ensures intuitiveness by relying on direct measurement of forearm position, minimizes involved sensors, and is expected to be long-lasting. To test for feasibility, the developed HMI was implemented to control a prosthetic wrist based on forearm rotation of able-bodied subjects. Performance of optical sensor system (OSS) prosthesis control was compared to electromyography (EMG) based direct control, for six able-bodied individuals, using a clothespin relocation task. Results showed that the performance of OSS control was comparable to direct control, therefore validating the feasibility of the OSS HMI.

Continuous Multi-DoF Wrist Kinematics Estimation Based on a Human-Machine Interface With Electrical-Impedance-Tomography

This study proposed a multiple degree-of-freedom (DoF) continuous wrist angle estimation approach based on an electrical impedance tomography (EIT) interface. The interface can inspect the spatial information of deep muscles with a soft elastic fabric sensing band, extending the measurement scope of the existing muscle-signal-based sensors. The designed estimation algorithm first extracted the mutual correlation of the EIT regions with a kernel function, and second used a regularization procedure to select the optimal coefficients. We evaluated the method with different features and regression models on 12 healthy subjects when they performed six basic wrist joint motions. The average root-mean-square error of the 3-DoF estimation task was 7.62°, and the average R² was 0.92. The results are comparable to state-of-the-art with sEMG signals in multi-DoF tasks. Future endeavors will be paid in this new direction to get more promising results.

Quantifying Spatial Activation Patterns of Motor Units in Finger Extensor Muscles

The ability to expertly control different fingers contributes to hand dexterity during object manipulation in daily life activities. The macroscopic spatial patterns of muscle activations during finger movements using global surface electromyography (sEMG) have been widely researched. However, the spatial activation patterns of microscopic motor units (MUs) under different finger movements have not been well investigated. The present work aims to quantify MU spatial activation patterns during movement of distinct fingers (index, middle, ring and little finger). Specifically, we focused on extensor muscles during extension contractions. Motor unit action potentials (MUAPs) during movement of each finger were obtained through decomposition of high-density sEMG (HD-sEMG). First, we quantified the spatial activation patterns of MUs for each finger based on 2-dimension (2-D) root-mean-square (RMS) maps of MUAP grids after spike-triggered averaging. We found that these activation patterns under different finger movements are distinct along the distal-proximal direction, but with partial overlap. Second, to further evaluate MU separability, we classified the spatial activation pattern of each individual MU under distinct finger movement and associated each MU with its corresponding finger with Regularized Uncorrelated Multilinear Discriminant Analysis (RUMLDA). A high accuracy of MU-finger classification tested on 12 subjects with a mean of 88.98% was achieved. The quantification of MU spatial activation patterns could be beneficial to studies of neural mechanisms of the hand. To the best of our knowledge, this is the first work which manages to quantify MU behaviors under different finger movements.

Ultrasound Sensing Can Improve Continuous Classification of Discrete Ambulation Modes Compared to Surface Electromyography

Clinical translation of "intelligent" lower-limb assistive technologies relies on robust control interfaces capable of accurately detecting user intent. To date, mechanical sensors and surface electromyography (EMG) have been the primary sensing modalities used to classify ambulation. Ultrasound (US) imaging can be used to detect user-intent by characterizing structural changes of muscle. Our study evaluates wearable US imaging as a new sensing modality for continuous classification of five discrete ambulation modes: level, incline, decline, stair ascent, and stair descent ambulation, and benchmarks performance relative to EMG sensing. Ten able-bodied subjects were equipped with a wearable US scanner and eight unilateral EMG sensors. Time-intensity features were recorded from US images of three thigh muscles. Features from sliding windows of EMG signals were analyzed in two configurations: one including 5 EMG sensors on muscles around the thigh, and another with 3 additional sensors placed on the shank. Linear discriminate analysis was implemented to continuously classify these phase-dependent features of each sensing modality as one of five ambulation modes. US-based sensing statistically improved mean classification accuracy to 99.8% (99.5-100% CI) compared to 8-EMG sensors (85.8%; 84.0-87.6% CI) and 5-EMG sensors (75.3%; 74.5-76.1% CI). Further, separability analyses show the importance of superficial and deep US information for stair classification relative to other modes. These results are the first to demonstrate the ability of US-based sensing to classify discrete ambulation modes, highlighting the potential for improved assistive device control using less widespread, less superficial and higher resolution sensing of skeletal muscle.