Lower Limb Motion Estimation Using Ultrasound Imaging: A Framework for Assistive Device Control

Wearable sensing for understanding and influencing human movement in ecological contexts

Wearable sensors offer a unique opportunity to study movement in ecological contexts - that is, outside the laboratory where movement happens in ordinary life. This article discusses the purpose, means, and impact of using wearable sensors to assess movement context, kinematics, and kinetics during locomotion, and how this information can be used to better understand and influence movement. We outline the types of information wearable sensors can gather and highlight recent developments in sensor technology, data analysis, and applications. We close with a vision for important future research and key questions the field will need to address to bring the potential benefits of wearable sensing to fruition.

Tapping Into Skeletal Muscle Biomechanics for Design and Control of Lower Limb Exoskeletons: A Narrative Review

Lower limb exoskeletons and exosuits ("exos") are traditionally designed with a strong focus on mechatronics and actuation, whereas the "human side" is often disregarded or minimally modeled. Muscle biomechanics principles and skeletal muscle response to robot-delivered loads should be incorporated in design/control of exos. In this narrative review, we summarize the advances in literature with respect to the fusion of muscle biomechanics and lower limb exoskeletons. We report methods to measure muscle biomechanics directly and indirectly and summarize the studies that have incorporated muscle measures for improved design and control of intuitive lower limb exos. Finally, we delve into articles that have studied how the human-exo interaction influences muscle biomechanics during locomotion. To support neurorehabilitation and facilitate everyday use of wearable assistive technologies, we believe that future studies should investigate and predict how exoskeleton assistance strategies would structurally remodel skeletal muscle over time. Real-time mapping of the neuromechanical origin and generation of muscle force resulting in joint torques should be combined with musculoskeletal models to address time-varying parameters such as adaptation to exos and fatigue. Development of smarter predictive controllers that steer rather than assist biological components could result in a synchronized human-machine system that optimizes the biological and electromechanical performance of the combined system.

Design trends in actuated lower-limb prosthetic systems: a narrative review

INTRODUCTION

Actuated lower limb prostheses, including powered (active) and semi-active (quasi-passive) joints, are endowed with controllable power and/or impedance, which can be advantageous to limb impairment individuals by improving locomotion mechanics and reducing the overall metabolic cost of ambulation. However, an increasing number of commercial and research-focused options have made navigating this field a daunting task for users, researchers, clinicians, and professionals.

AREAS COVERED

The present paper provides an overview of the latest trends and developments in the field of actuated lower-limb prostheses and corresponding technologies. Following a gentle summary of essential gait features, we introduce and compare various actuated prosthetic solutions in academia and the market designed to provide assistance at different levels of impairments. Correspondingly, we offer insights into the latest developments of sockets and suspension systems, before finally discussing the established and emerging trends in surgical approaches aimed at improving prosthetic experience through enhanced physical and neural interfaces.

EXPERT OPINION

The ongoing challenges and future research opportunities in the field are summarized for exploring potential avenues for development of next generation of actuated lower limb prostheses. In our opinions, a closer multidisciplinary integration can be found in the field of actuated lower-limb prostheses in the future.

A comparison of point-tracking algorithms in ultrasound videos from the upper limb

Tracking points in ultrasound (US) videos can be especially useful to characterize tissues in motion. Tracking algorithms that analyze successive video frames, such as variations of Optical Flow and Lucas-Kanade (LK), exploit frame-to-frame temporal information to track regions of interest. In contrast, convolutional neural-network (CNN) models process each video frame independently of neighboring frames. In this paper, we show that frame-to-frame trackers accumulate error over time. We propose three interpolation-like methods to combat error accumulation and show that all three methods reduce tracking errors in frame-to-frame trackers. On the neural-network end, we show that a CNN-based tracker, DeepLabCut (DLC), outperforms all four frame-to-frame trackers when tracking tissues in motion. DLC is more accurate than the frame-to-frame trackers and less sensitive to variations in types of tissue movement. The only caveat found with DLC comes from its non-temporal tracking strategy, leading to jitter between consecutive frames. Overall, when tracking points in videos of moving tissue, we recommend using DLC when prioritizing accuracy and robustness across movements in videos, and using LK with the proposed error-correction methods for small movements when tracking jitter is unacceptable.

A clustering-based method for estimating pennation angle from B-mode ultrasound images

B-mode ultrasound (US) is often used to noninvasively measure skeletal muscle architecture, which contains human intent information. Extracted features from B-mode images can help improve closed-loop human-robotic interaction control when using rehabilitation/assistive devices. The traditional manual approach to inferring the muscle structural features from US images is laborious, time-consuming, and subjective among different investigators. This paper proposes a clustering-based detection method that can mimic a well-trained human expert in identifying fascicle and aponeurosis and, therefore, compute the pennation angle. The clustering-based architecture assumes that muscle fibers have tubular characteristics. It is robust for low-frequency image streams. We compared the proposed algorithm to two mature benchmark techniques: UltraTrack and ImageJ. The performance of the proposed approach showed higher accuracy in our dataset (frame frequency is 20 Hz), that is, similar to the human expert. The proposed method shows promising potential in automatic muscle fascicle orientation detection to facilitate implementations in biomechanics modeling, rehabilitation robot control design, and neuromuscular disease diagnosis with low-frequency data stream.

A-Mode Ultrasound-Based Prediction of Transfemoral Amputee Prosthesis Walking Kinematics Via an Artificial Neural Network

Lower-limb powered prostheses can provide users with volitional control of ambulation. To accomplish this goal, they require a sensing modality that reliably interprets user intention to move. Surface electromyography (EMG) has been previously proposed to measure muscle excitation and provide volitional control to upper- and lower-limb powered prosthesis users. Unfortunately, EMG suffers from a low signal to noise ratio and crosstalk between neighboring muscles, often limiting the performance of EMG-based controllers. Ultrasound has been shown to have better resolution and specificity than surface EMG. However, this technology has yet to be integrated into lower-limb prostheses. Here we show that A-mode ultrasound sensing can reliably predict the prosthesis walking kinematics of individuals with a transfemoral amputation. Ultrasound features from the residual limb of 9 transfemoral amputee subjects were recorded with A-mode ultrasound during walking with their passive prosthesis. The ultrasound features were mapped to joint kinematics through a regression neural network. Testing of the trained model against untrained kinematics from an altered walking speed show accurate predictions of knee position, knee velocity, ankle position, and ankle velocity, with a normalized RMSE of 9.0 ± 3.1%, 7.3 ± 1.6%, 8.3 ± 2.3%, and 10.0 ± 2.5% respectively. This ultrasound-based prediction suggests that A-mode ultrasound is a viable sensing technology for recognizing user intent. This study is the first necessary step towards implementation of volitional prosthesis controller based on A-mode ultrasound for individuals with transfemoral amputation.

Controlling Upper Limb Prostheses Using Sonomyography (SMG): A Review

This paper presents a critical review and comparison of the results of recently published studies in the fields of human-machine interface and the use of sonomyography (SMG) for the control of upper limb prothesis. For this review paper, a combination of the keywords "Human Machine Interface", "Sonomyography", "Ultrasound", "Upper Limb Prosthesis", "Artificial Intelligence", and "Non-Invasive Sensors" was used to search for articles on Google Scholar and PubMed. Sixty-one articles were found, of which fifty-nine were used in this review. For a comparison of the different ultrasound modes, feature extraction methods, and machine learning algorithms, 16 articles were used. Various modes of ultrasound devices for prosthetic control, various machine learning algorithms for classifying different hand gestures, and various feature extraction methods for increasing the accuracy of artificial intelligence used in their controlling systems are reviewed in this article. The results of the review article show that ultrasound sensing has the potential to be used as a viable human-machine interface in order to control bionic hands with multiple degrees of freedom. Moreover, different hand gestures can be classified by different machine learning algorithms trained with extracted features from collected data with an accuracy of around 95%.

Discernment on assistive technology for the care and support requirements of older adults and differently-abled individuals

Assistive technology for the differently abled and older adults has made remarkable achievements in providing rehabilitative, adaptive, and assistive devices. It provides huge assistance for people with physical impairments to lead a better self-reliant daily life, in terms of mobility, education, rehabilitation, etc. This technology ranges from simple hand-held devices to complex robotic accessories which promote the individual's independence. This study aimed at identifying the assistance required by differently-abled individuals, and the solutions proposed by different researchers, and reviewed their merits and demerits. It provides a detailed discussion on the state of art assistive technologies, their applications, challenges, types, and their usage for rehabilitation. The study also identifies different unexplored research areas related to assistive technology that can improve the daily life of individuals and advance the field. Despite their high usage, assistive technologies have some limitations which have been briefly described in the study. This review, therefore, can help understand the utilization, and pros and cons of assistive devices in rehabilitation engineering and assistive technologies.

Deep-Learning to Map a Benchmark Dataset of Non-amputee Ambulation for Controlling an Open Source Bionic Leg

Powered lower-limb prosthetic devices may be becoming a promising option for amputation patients. Although various methods have been proposed to produce gait trajectories similar to those of non-disabled individuals, implementing these control methods is still challenging. It remains unclear whether these methods provide appropriate, safe, and intuitive locomotion as intended. This paper proposes the direct mapping of the voluntary movement of a residual limb (i.e., thigh) to the desired impedance parameters for amputated limbs (i.e., knee and ankle). The proposed model was learned from the gait trajectories of intact limb individuals from a publicly available biomechanics dataset, and was applied to control the prosthetic leg without post-tuning the network. Thus, the proposed method does not require training time with individuals with amputation nor configuration time for its use, and it provides a closely resembling gait trajectory of the intact limb. For preliminary testing, three able-bodied subjects participated in bypass tests. The proposed model accomplished intuitive and reliable level-ground walking at three different step lengths: self-selected, long-, and short-step lengths. The results indicate that intact benchmark data with different sensor configurations can be directly used to train the model to control prosthetic legs.

A deep learning method to predict ankle joint moment during walking at different speeds with ultrasound imaging: A framework for assistive devices control

Robotic assistive or rehabilitative devices are promising aids for people with neurological disorders as they help regain normative functions for both upper and lower limbs. However, it remains challenging to accurately estimate human intent or residual efforts non-invasively when using these robotic devices. In this article, we propose a deep learning approach that uses a brightness mode, that is, B-mode, of ultrasound (US) imaging from skeletal muscles to predict the ankle joint net plantarflexion moment while walking. The designed structure of customized deep convolutional neural networks (CNNs) guarantees the convergence and robustness of the deep learning approach. We investigated the influence of the US imaging's region of interest (ROI) on the net plantarflexion moment prediction performance. We also compared the CNN-based moment prediction performance utilizing B-mode US and sEMG spectrum imaging with the same ROI size. Experimental results from eight young participants walking on a treadmill at multiple speeds verified an improved accuracy by using the proposed US imaging + deep learning approach for net joint moment prediction. With the same CNN structure, compared to the prediction performance by using sEMG spectrum imaging, US imaging significantly reduced the normalized prediction root mean square error by 37.55% ( < .001) and increased the prediction coefficient of determination by 20.13% ( < .001). The findings show that the US imaging + deep learning approach personalizes the assessment of human joint voluntary effort, which can be incorporated with assistive or rehabilitative devices to improve clinical performance based on the assist-as-needed control strategy.

A Simultaneous Gesture Classification and Force Estimation Strategy Based on Wearable A-Mode Ultrasound and Cascade Model

The existing Human-Machine Interfaces (HMI) based on gesture recognition using surface electromyography (sEMG) have made significant progress. However, the sEMG has inherent limitations as well as the gesture classification and force estimation have not been effectively combined. There are limitations in applications such as prosthetic control and clinical rehabilitation, etc. In this paper, a grasping gesture and force recognition strategy based on wearable A-mode ultrasound and two-stage cascade model is proposed, which can simultaneously estimate the force while classifying the grasping gesture. This paper experiments five grasping gestures and four force levels (5-50%MVC). The results demonstrate that the performance of the proposed model is significantly better than that of the traditional model both in classification and regression (p < 0.001). Additionally, the two-stage cascade regression model (TSCRM) used the Gaussian Process regression model (GPR) with the mean and standard deviation (MSD) feature obtains excellent results, with normalized root-mean-square error (nRMSE) and correlation coefficient (CC) of 0.10490.0374 and 0.94610.0354, respectively. Besides, the latency of the model meets the requirement of real-time recognition (T < 15ms). Therefore, the research outcomes prove the feasibility of the proposed recognition strategy and provide a reference for the field of prosthetic control, etc.

Simultaneous Prediction of Wrist and Hand Motions via Wearable Ultrasound Sensing for Natural Control of Hand Prostheses

Simultaneous prediction of wrist and hand motions is essential for the natural interaction with hand prostheses. In this paper, we propose a novel multi-out Gaussian process (MOGP) model and a multi-task deep learning (MTDL) algorithm to achieve simultaneous prediction of wrist rotation (pronation/ supination)¹ and finger gestures for transradial amputees via a wearable ultrasound array. We target six finger gestures with concurrent wrist rotation in four transradial amputees. Results show that MOGP outperforms previously reported subclass discriminant analysis for both predictions of discrete finger gestures and continuous wrist rotation. Moreover, we find that MTDL has the potential to improve the accuracy of finger gesture prediction compared to MOGP and classification-specific deep learning, albeit at the expense of reducing the accuracy of wrist rotation prediction. Extended comparative analysis shows the superiority of ultrasound over surface electromyography. This paper prioritizes exploring the performance of wearable ultrasound on the simultaneous prediction of wrist and hand motions for transradial amputees, demonstrating the potential of ultrasound in future prosthetic control. Our ultrasound-based adaptive prosthetic control dataset (UltraPro) will be released to promote the development of the prosthetic community.

Fused ultrasound and electromyography-driven neuromuscular model to improve plantarflexion moment prediction across walking speeds

Background

Improving the prediction ability of a human-machine interface (HMI) is critical to accomplish a bio-inspired or model-based control strategy for rehabilitation interventions, which are of increased interest to assist limb function post neurological injuries. A fundamental role of the HMI is to accurately predict human intent by mapping signals from a mechanical sensor or surface electromyography (sEMG) sensor. These sensors are limited to measuring the resulting limb force or movement or the neural signal evoking the force. As the intermediate mapping in the HMI also depends on muscle contractility, a motivation exists to include architectural features of the muscle as surrogates of dynamic muscle movement, thus further improving the HMI’s prediction accuracy.

Objective

The purpose of this study is to investigate a non-invasive sEMG and ultrasound (US) imaging-driven Hill-type neuromuscular model (HNM) for net ankle joint plantarflexion moment prediction. We hypothesize that the fusion of signals from sEMG and US imaging results in a more accurate net plantarflexion moment prediction than sole sEMG or US imaging.

Methods

Ten young non-disabled participants walked on a treadmill at speeds of 0.50, 0.75, 1.00, 1.25, and 1.50 m/s. The proposed HNM consists of two muscle-tendon units. The muscle activation for each unit was calculated as a weighted summation of the normalized sEMG signal and normalized muscle thickness signal from US imaging. The HNM calibration was performed under both single-speed mode and inter-speed mode, and then the calibrated HNM was validated across all walking speeds.

Results

On average, the normalized moment prediction root mean square error was reduced by 14.58 % (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p=0.012$$\end{document}p=0.012) and 36.79 % (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p<0.001$$\end{document}p<0.001) with the proposed HNM when compared to sEMG-driven and US imaging-driven HNMs, respectively. Also, the calibrated models with data from the inter-speed mode were more robust than those from single-speed modes for the moment prediction.

Conclusions

The proposed sEMG-US imaging-driven HNM can significantly improve the net plantarflexion moment prediction accuracy across multiple walking speeds. The findings imply that the proposed HNM can be potentially used in bio-inspired control strategies for rehabilitative devices due to its superior prediction.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12984-022-01061-z.

Evaluating Electromyography and Sonomyography Sensor Fusion to Estimate Lower-Limb Kinematics Using Gaussian Process Regression

Research on robotic lower-limb assistive devices over the past decade has generated autonomous, multiple degree-of-freedom devices to augment human performance during a variety of scenarios. However, the increase in capabilities of these devices is met with an increase in the complexity of the overall control problem and requirement for an accurate and robust sensing modality for intent recognition. Due to its ability to precede changes in motion, surface electromyography (EMG) is widely studied as a peripheral sensing modality for capturing features of muscle activity as an input for control of powered assistive devices. In order to capture features that contribute to muscle contraction and joint motion beyond muscle activity of superficial muscles, researchers have introduced sonomyography, or real-time dynamic ultrasound imaging of skeletal muscle. However, the ability of these sonomyography features to continuously predict multiple lower-limb joint kinematics during widely varying ambulation tasks, and their potential as an input for powered multiple degree-of-freedom lower-limb assistive devices is unknown. The objective of this research is to evaluate surface EMG and sonomyography, as well as the fusion of features from both sensing modalities, as inputs to Gaussian process regression models for the continuous estimation of hip, knee and ankle angle and velocity during level walking, stair ascent/descent and ramp ascent/descent ambulation. Gaussian process regression is a Bayesian nonlinear regression model that has been introduced as an alternative to musculoskeletal model-based techniques. In this study, time-intensity features of sonomyography on both the anterior and posterior thigh along with time-domain features of surface EMG from eight muscles on the lower-limb were used to train and test subject-dependent and task-invariant Gaussian process regression models for the continuous estimation of hip, knee and ankle motion. Overall, anterior sonomyography sensor fusion with surface EMG significantly improved estimation of hip, knee and ankle motion for all ambulation tasks (level ground, stair and ramp ambulation) in comparison to surface EMG alone. Additionally, anterior sonomyography alone significantly improved errors at the hip and knee for most tasks compared to surface EMG. These findings help inform the implementation and integration of volitional control strategies for robotic assistive technologies.

Plantarflexion Moment Prediction during the Walking Stance Phase with an sEMG-Ultrasound Imaging-Driven Model

Many rehabilitative exoskeletons use non-invasive surface electromyography (sEMG) to measure human volitional intent. However, signals from adjacent muscle groups interfere with sEMG measurements. Further, the inability to measure sEMG signals from deeply located muscles may not accurately measure the volitional intent. In this work, we combined sEMG and ultrasound (US) imaging-derived signals to improve the prediction accuracy of voluntary ankle effort. We used a multivariate linear model (MLM) that combines sEMG and US signals for ankle joint net plantarflexion (PF) moment prediction during the walking stance phase. We hypothesized that the proposed sEMG-US imaging-driven MLM would result in more accurate net PF moment prediction than sEMG-driven and US imaging-driven MLMs. Synchronous measurements including reflective makers coordinates, ground reaction forces, sEMG signals of lateral/medial gastrocnemius (LGS/MGS), and soleus (SOL) muscles, and US imaging of LGS and SOL muscles were collected from five able-bodied participants walking on a treadmill at multiple speeds. The ankle joint net PF moment benchmark was calculated based on inverse dynamics, while the net PF moment prediction was determined by the sEMG-US imaging-driven, sEMG-driven, and US imaging-driven MLMs. The findings show that the sEMG-US imaging-driven MLM can significantly improve the prediction of net PF moment during the walking stance phase at multiple speeds. Potentially, the proposed sEMG-US imaging-driven MLM can be used as a superior joint motion intent model in advanced and intelligent control strategies for rehabilitative exoskeletons.

Ultrasound-Derived Features of Muscle Architecture Provide Unique Temporal Characterization of Volitional Knee Motion

Sonomyography, or dynamic ultrasound imaging of skeletal muscle, has gained significant interest in rehabilitation medicine. Previously, correlations relating sonomyography features of muscle contraction, including muscle thickness, pennation angle, angle between aponeuroses and fascicle length, to muscle force production, strength and joint motion have been established. Additionally, relationships between grayscale image intensity, or echogenicity, with maximum voluntary isometric contraction of muscle have been noted. However, the time relationship between changes in various sonomyography features during volitional motion has yet to be explored, which would highlight if unique information pertaining to muscle contraction and motion can be obtained from this real-time imaging modality. These new insights could inform how we assess muscle function and/or how we use this modality for assistive device control. Thus, our objective was to characterize the time synchronization of changes in five features of rectus femoris contraction extracted from ultrasound images during seated knee extension and flexion. A cross-correlation analysis was performed on data recorded by a handheld ultrasound system as able-bodied subjects completed seated trials of volitional knee extension and flexion. Changes in muscle thickness, angle between aponeuroses, and mean image echogenicity, a change in brightness of the grayscale image, preceded changes in our estimates of pennation angle and fascicle length. The leading nature of these features suggest they could be objective features for early detection of impending joint motion. Finally, multiple sonomyographic features provided unique temporal information associated with this volitional task.Clinical Relevance-This work evaluates the time relationship between five commonly reported features of skeletal muscle architecture during volitional motion, which can be used for targeted clinical assessments and intent detection.

A Dual-Modal Approach Using Electromyography and Sonomyography Improves Prediction of Dynamic Ankle Movement: A Case Study

For decades, surface electromyography (sEMG) has been a popular non-invasive bio-sensing technology for predicting human joint motion. However, cross-talk, interference from adjacent muscles, and its inability to measure deeply located muscles limit its performance in predicting joint motion. Recently, ultrasound (US) imaging has been proposed as an alternative non-invasive technology to predict joint movement due to its high signal-to-noise ratio, direct visualization of targeted tissue, and ability to access deep-seated muscles. This paper proposes a dual-modal approach that combines US imaging and sEMG for predicting volitional dynamic ankle dorsiflexion movement. Three feature sets: 1) a uni-modal set with four sEMG features, 2) a uni-modal set with four US imaging features, and 3) a dual-modal set with four dominant sEMG and US imaging features, together with measured ankle dorsiflexion angles, were used to train multiple machine learning regression models. The experimental results from a seated posture and five walking trials at different speeds, ranging from 0.50 m/s to 1.50 m/s, showed that the dual-modal set significantly reduced the prediction root mean square errors (RMSEs). Compared to the uni-modal sEMG feature set, the dual-modal set reduced RMSEs by up to 47.84% for the seated posture and up to 77.72% for the walking trials. Similarly, when compared to the US imaging feature set, the dual-modal set reduced RMSEs by up to 53.95% for the seated posture and up to 58.39% for the walking trials. The findings show that potentially the dual-modal sensing approach can be used as a superior sensing modality to predict human intent of a continuous motion and implemented for volitional control of clinical rehabilitative and assistive devices.

Ultrasound Sensing Can Improve Continuous Classification of Discrete Ambulation Modes Compared to Surface Electromyography

Clinical translation of "intelligent" lower-limb assistive technologies relies on robust control interfaces capable of accurately detecting user intent. To date, mechanical sensors and surface electromyography (EMG) have been the primary sensing modalities used to classify ambulation. Ultrasound (US) imaging can be used to detect user-intent by characterizing structural changes of muscle. Our study evaluates wearable US imaging as a new sensing modality for continuous classification of five discrete ambulation modes: level, incline, decline, stair ascent, and stair descent ambulation, and benchmarks performance relative to EMG sensing. Ten able-bodied subjects were equipped with a wearable US scanner and eight unilateral EMG sensors. Time-intensity features were recorded from US images of three thigh muscles. Features from sliding windows of EMG signals were analyzed in two configurations: one including 5 EMG sensors on muscles around the thigh, and another with 3 additional sensors placed on the shank. Linear discriminate analysis was implemented to continuously classify these phase-dependent features of each sensing modality as one of five ambulation modes. US-based sensing statistically improved mean classification accuracy to 99.8% (99.5-100% CI) compared to 8-EMG sensors (85.8%; 84.0-87.6% CI) and 5-EMG sensors (75.3%; 74.5-76.1% CI). Further, separability analyses show the importance of superficial and deep US information for stair classification relative to other modes. These results are the first to demonstrate the ability of US-based sensing to classify discrete ambulation modes, highlighting the potential for improved assistive device control using less widespread, less superficial and higher resolution sensing of skeletal muscle.

Identification of muscle morphology with noncontact capacitive sensing: Preliminary study

Human-machine interface with muscle signals serves as an important role in the field of wearable robotics. To compensate for the limitations of the existing surface Electromyography (sEMG) based technologies, we previously proposed a noncontact capacitive sensing approach that could record the limb shape changes. The sensing approach frees the human skin from contacting to the metal electrodes, thus enabling the measurement of muscle signals by dressing the sensing front-ends outside of the clothes. We validated the capacitive sensing in human motion intent recognition tasks with the wearable robots and produced comparable results to existing studies. However, the biological significance of the capacitance signals is still unrevealed, which is an indispensable issue for robot intuitive control. In this study, we address the problems of identifying the relationships between the muscle morphological parameters and the capacitance signals. We constructed a measurement system that recorded the noncon-tact capacitive sensing signals and the muscle ultrasound (US) images simultaneously. With the designed device, five subjects were employed and the US images from the gastrocnemius muscle (GM) and the tibialis anterior (TA) muscle during level walking were sampled. We fitted the calculated muscle morphological parameters (the pinnation angles and the muscle fascicle length) and the capacitance signals of the same gait phases. The results demonstrated that at least one-channel capacitance signal strongly correlated to the muscle morphological parameters (R² > 0.5, quadratic regression). The average R²s of the most correlated channels were up to 0.86 for pinnation angles and 0.83 for the muscle fascicle length changes. The interesting findings in this preliminary study suggest the biological physical significance of the capacitance signals during human locomotion. Future efforts are worth being paid in this new research direction for more promising results.

Towards Sonomyography-Based Real-Time Control of Powered Prosthesis Grasp Synergies

Sonomyography (ultrasound imaging) offers a way of classifying complex muscle activity and configuration, with higher SNR and lower hardware requirements than sEMG, using various supervised learning algorithms. The physiological image obtained from an ultrasound probe can be used to train a classification algorithm which can run on real time ultrasound images. The predicted values can then be mapped onto assistive or teleoperated robots. This paper describes the classification of ultrasound information and its subsequent mapping onto a soft robotic gripper as a step toward direct synergy control. Support Vector Classification algorithm has been used to classify ultrasound information into a set of defined states: open, closed, pinch and hook grasps. Once the model was trained with the ultrasound image data, real time input from the forearm was used to predict these states. The final predicted state output then set joint stiffnesses in the soft actuators, changing their interactions or synergies, to obtain the corresponding soft robotic gripper states. Data collection was carried out on five different test subjects for eight trials each. An average accuracy percentage of 93% was obtained averaged over all data. This real-time ultrasound-based control of a soft robotic gripper constitutes a promising step toward intuitive and robust biosignal-based control methods for robots.

Ultrasound Features of Skeletal Muscle Can Predict Kinematics of Upcoming Lower-Limb Motion

Seamless integration of lower-limb assistive devices with the human body requires an intuitive human-machine interface, which would benefit from predicting the intent of individuals in advance of the upcoming motion. Ultrasound imaging was recently introduced as an intuitive sensing interface. The objective of the present study was to investigate the predictability of joint kinematics using ultrasound features of the rectus femoris muscle during a non-weight-bearing knee extension/flexion. Motion prediction accuracy was evaluated in 67 ms increments, up to 600 ms in time. Statistical analysis was used to evaluate the feasibility of motion prediction, and the linear mixed-effects model was used to determine a prediction time window where the joint angle prediction error is barely perceivable by the sample population, hence clinically reliable. Surprisingly, statistical tests revealed that the prediction accuracy of the joint angle was more sensitive to temporal shifts than the accuracy of the joint angular velocity prediction. Overall, predictability of the upcoming joint kinematics using ultrasound features of skeletal muscle was confirmed, and a time window for a statistically and clinically reliable prediction was found between 133 and 142 ms. A reliable prediction of user intent may provide the time needed for processing, control planning, and actuation of the assistive devices at critical points during ambulation, contributing to the intuitive behavior of lower-limb assistive devices.

Recent Progress in Sensing and Computing Techniques for Human Activity Recognition and Motion Analysis

The recent scientific and technical advances in Internet of Things (IoT) based pervasive sensing and computing have created opportunities for the continuous monitoring of human activities for different purposes. The topic of human activity recognition (HAR) and motion analysis, due to its potentiality in human–machine interaction (HMI), medical care, sports analysis, physical rehabilitation, assisted daily living (ADL), children and elderly care, has recently gained increasing attention. The emergence of some novel sensing devices featuring miniature size, a light weight, and wireless data transmission, the availability of wireless communication infrastructure, the progress of machine learning and deep learning algorithms, and the widespread IoT applications has promised new opportunities for a significant progress in this particular field. Motivated by a great demand for HAR-related applications and the lack of a timely report of the recent contributions to knowledge in this area, this investigation aims to provide a comprehensive survey and in-depth analysis of the recent advances in the diverse techniques and methods of human activity recognition and motion analysis. The focus of this investigation falls on the fundamental theories, the innovative applications with their underlying sensing techniques, data fusion and processing, and human activity classification methods. Based on the state-of-the-art, the technical challenges are identified, and future perspectives on the future rich, sensing, intelligent IoT world are given in order to provide a reference for the research and practices in the related fields.

Evaluation of Non-Invasive Ankle Joint Effort Prediction Methods for Use in Neurorehabilitation Using Electromyography and Ultrasound Imaging

OBJECTIVE

Reliable measurement of voluntary human effort is essential for effective and safe interaction between the wearer and an assistive robot. Existing voluntary effort prediction methods that use surface electromyography (sEMG) are susceptible to prediction inaccuracies due to non-selectivity in measuring muscle responses. This technical challenge motivates an investigation into alternative non-invasive effort prediction methods that directly visualize the muscle response and improve effort prediction accuracy. The paper is a comparative study of ultrasound imaging (US)-derived neuromuscular signals and sEMG signals for their use in predicting isometric ankle dorsiflexion moment. Furthermore, the study evaluates the prediction accuracy of model-based and model-free voluntary effort prediction approaches that use these signals.

METHODS

The study evaluates sEMG signals and three US imaging-derived signals: pennation angle, muscle fascicle length, and echogenicity and three voluntary effort prediction methods: linear regression (LR), feedforward neural network (FFNN), and Hill-type neuromuscular model (HNM).

RESULTS

In all the prediction methods, pennation angle and fascicle length significantly improve the prediction accuracy of dorsiflexion moment, when compared to echogenicity. Also, compared to LR, both FFNN and HNM improve dorsiflexion moment prediction accuracy.

CONCLUSION

The findings indicate FFNN or HNM approach and using pennation angle or fascicle length predict human ankle movement intent with higher accuracy.

SIGNIFICANCE

The accurate ankle effort prediction will pave the path to safe and reliable robotic assistance in patients with drop foot.

Use of Sonomyographic Sensing to Estimate Knee Angular Velocity During Varying Modes of Ambulation

Ultrasound (US) imaging of muscle has been introduced as a promising sensing modality for assistive device control. Ten able-bodied subjects completed level, incline and decline walking on a treadmill in a motion capture laboratory while wearing reflective markers on upper- and lower-body. A wearable US transducer was affixed to subjects' anterior thigh, and time-intensity features were extracted from transverse US images of the knee extensor muscles. These features were used to train and test Gaussian process regression models for continuous estimation of knee flexion/extension angular velocity. Four regression models were evaluated: (1) subject-dependent/task-specific, (2) subject-dependent/pooled-tasks, (3) subject-independent/task-specific, and (4) subject-independent/pooled-tasks. Subject-independent models were "tuned" with up to six strides of the test subject's data to boost performance. A two-factor analysis of variance test was used to assess the effect of each approach on root mean square error (RMSE) of estimated knee angular velocity (α=0.05). Statistical parametric mapping (SPM) was completed to compare actual vs. estimated knee angular velocity as a function of the gait cycle (α=0.05). For incline and level walking, the subject-dependent/pooled-tasks model resulted in the lowest error while the subject-dependent/task-specific model resulted in the lowest error for decline walk. Impressively, the two-factor test revealed no difference between task-specific and pooled-task models. Furthermore, despite capturing many important features of knee velocity across individuals there were, as expected, significant differences between subject-dependent and subject-independent models. Collectively, these results are promising for potential assistive device control with error rates <10% for all regression models that were tested.Clinical Relevance-This work is the first study to demonstrate the feasibility of using ultrasound-based sensing for estimation of knee angular velocity during multiple modes of ambulation.

Ultrasound Measurement of Skeletal Muscle Contractile Parameters Using Flexible and Wearable Single-Element Ultrasonic Sensor

Skeletal muscle is considered as a near-constant volume system, and the contractions of the muscle are related to the changes in tissue thickness. Assessment of the skeletal muscle contractile parameters such as maximum contraction thickness (Th), contraction time (Tc), contraction velocity (Vc), sustain time (Ts), and half-relaxation (Tr) provides valuable information for various medical applications. This paper presents a single-element wearable ultrasonic sensor (WUS) and a method to measure the skeletal muscle contractile parameters in A-mode ultrasonic data acquisition. The developed WUS was made of double-layer polyvinylidene fluoride (PVDF) piezoelectric polymer films with a simple and low-cost fabrication process. A flexible, lightweight, thin, and small size WUS would provide a secure attachment to the skin surface without affecting the muscle contraction dynamics of interest. The developed WUS was employed to monitor the contractions of gastrocnemius (GC) muscle of a human subject. The GC muscle contractions were evoked by the electrical muscle stimulation (EMS) at varying EMS frequencies from 2 Hz up to 30 Hz. The tissue thickness changes due to the muscle contractions were measured by utilizing a time-of-flight method in the ultrasonic through-transmission mode. The developed WUS demonstrated the capability to monitor the tissue thickness changes during the unfused and fused tetanic contractions. The tetanic progression level was quantitatively assessed using the parameter of the fusion index (FI) obtained. In addition, the contractile parameters (Th, Tc, Vc, Ts, and Tr) were successfully extracted from the measured tissue thickness changes. In addition, the unfused and fused tetanus frequencies were estimated from the obtained FI-EMS frequency curve. The WUS and ultrasonic method proposed in this study could be a valuable tool for inexpensive, non-invasive, and continuous monitoring of the skeletal muscle contractile properties.

Comparative Analysis of Wearable A-Mode Ultrasound and sEMG for Muscle-Computer Interface

OBJECTIVE

While surface electromyography (sEMG) is still dominant in the field of muscle-computer interface, ultrasound (US) sensing has been regarded as a promising alternative to sEMG, owing to its ability to precisely monitor muscle deformations. Among different US modalities, A-mode US is more compact and cost-effective for wearable applications against its cumbersome B-mode counterpart. In this article, we conduct a comprehensive comparison of wearable A-mode US and sEMG on gesture recognition and isometric muscle contraction force estimation.

METHODS

We experimented with eight types of gesture, with a range of 0-60% maximum voluntary contraction for each motion.

RESULTS

Results show that A-mode US outperforms sEMG on gesture recognition accuracy, robustness, and discrete force estimation accuracy, while sEMG is superior to US on continuous force estimation accuracy and ease of use in force estimation. Moreover, an extended online experiment demonstrates that the complementary advantages of US and sEMG on gesture recognition and continuous force estimation can be combined for the achievement of multi-class proportional gesture control.

SIGNIFICANCE

This article demonstrates the potential of A-mode US in automated gesture recognition, and the prospect of sEMG/US fusion for proportional gesture interaction.

Prediction of Distal Lower-Limb Motion Using Ultrasound-Derived Features of Proximal Skeletal Muscle

Control of lower-limb assistive devices would benefit from predicting the intent of individuals in advance of upcoming motion, rather than estimating the current states of their motion. Human lower-limb motion estimation using ultrasound (US) image derived features of skeletal muscle has been demonstrated. However, predictability of motion in time remains an open question. The objective of this study was to assess the predictability of distal lower-limb motion using US image features of rectus femoris (RF) muscle during non-weight-bearing knee flexion/extension. A series of time shifts was introduced between the US features and the joint position in 67 ms steps from 0 ms (i.e., estimation, no prediction) up to predicting 467 ms in advance. A US-based algorithm to estimate lower-limb motion was then used to predict the knee joint position in time using the US features after introducing the time shifts. The accuracy of joint motion prediction after each time shift was compared to the accuracy of joint motion estimation. The reliability of the prediction was then assessed using an analysis of variance (ANOVA) test. The motion prediction accuracy was found to be reliable up to 200 ms, where the average root mean square error (RMSE) of prediction across 9 healthy subjects was 0.89 degrees greater than the average RMSE (7.39 degrees) of motion estimation for the same group of subjects. These findings suggest a reliable prediction of upcoming lower-limb motion is feasible using the US features of skeletal muscle up to a certain point. A reliable prediction may provide lower-limb assistive device control systems with a time-window for processing and control planning, and actuation hence improving the volitional control behaviors of lower-limb assistive devices.

Gait Phase Identification During Level, Incline and Decline Ambulation Tasks Using Portable Sonomyographic Sensing

Clinical viability of powered lower-limb assistive devices requires reliable and intuitive control strategies. Stance and swing are the main phases of the gait cycle across different locomotion tasks. Hence, a reliable method to accurately identify these phases can decrease sensing complexity and assist in enabling high-level control of assistive devices. Ultrasound (US) imaging has recently been introduced as a new sensing modality that may provide a solution for intuitive device control. US images of the rectus femoris and vastus intermedius muscles were collected in humans during level, incline, and decline ambulation tasks. Five low-level static (i.e. time-independent) features of US images were measured with respect to a reference image, including correlation coefficient, sum of absolute differences, structural similarity index, sum of squared differences, and image echogenicity. Time-derivatives of the static features were also calculated as temporal features. Support vector machine classifiers were trained using these static features to identify the gait phase both dependent and independent of the ambulation tasks. The results indicate an accuracy of 88.3% in identifying the gait phases for task-independent classifiers when trained using only the static features. Performance of the classifiers improved significantly to 92.8% after using the temporal features (p $\lt0.01)$. The algorithm was efficient and the average processing speed was faster than 100 Hz. This study is the first demonstration on use of US imaging to provide continuous estimates of ambulation phase, and on multiple surfaces. These findings suggest task-independent approaches may reliably identify the main phases of the gait cycle. Advancements in this area of study may provide simpler intuitive strategies for high-level assistive device control and increase their clinical relevance.