Gait Trajectory and Event Prediction from State Estimation for Exoskeletons During Gait

A Data-Driven Approach to Estimate Human Center of Mass State During Perturbed Locomotion Using Simulated Wearable Sensors

Center of mass (COM) state, specifically in a local reference frame (i.e., relative to center of pressure), is an important variable for controlling and quantifying bipedal locomotion. However, this metric is not easily attainable in real time during human locomotion experiments. This information could be valuable when controlling wearable robotic exoskeletons, specifically for stability augmentation where knowledge of COM state could enable step placement planners similar to bipedal robots. Here, we explored the ability of simulated wearable sensor-driven models to rapidly estimate COM state during steady state and perturbed walking, spanning delayed estimates (i.e., estimating past state) to anticipated estimates (i.e., estimating future state). We used various simulated inertial measurement unit (IMU) sensor configurations typically found on lower limb exoskeletons and a temporal convolutional network (TCN) model throughout this analysis. We found comparable COM estimation capabilities across hip, knee, and ankle exoskeleton sensor configurations, where device type did not significantly influence error. We also found that anticipating COM state during perturbations induced a significant increase in error proportional to anticipation time. Delaying COM state estimates significantly increased accuracy for velocity estimates but not position estimates. All tested conditions resulted in models with R2 > 0.85, with a majority resulting in R2 > 0.95, emphasizing the viability of this approach. Broadly, this preliminary work using simulated IMUs supports the efficacy of wearable sensor-driven deep learning approaches to provide real-time COM state estimates for lower limb exoskeleton control or other wearable sensor-based applications, such as mobile data collection or use in real-time biofeedback.

Comparative assessment of heel rise detection for consistent gait phase separation

Background

Accurate identification of gait events is crucial to reliable gait analysis. Heel rise, a key event marking the transition from mid-stance to terminal stance, poses challenges in precise detection due to its gradual nature. This leads to variability in accuracy across studies utilizing diverse measuring techniques.

Research question

How do different HR detection methods compare when assessed against the underlying heel motion pattern and visual detection across varying speed, footwear conditions, and individuals?

Methods

Leveraging data from over 10,000 strides in diverse scenarios with 15 healthy subjects, we evaluated methods based on measurements from optical motion capture (OMC), force plates, and shank-mounted inertial measurement units (IMUs). The evaluation of these methods included an assessment of their precision and consistency with the heel marker's motion pattern and agreement with visually detected heel rise.

Results

OMC-based heel rise detection methods, utilizing the heel marker's vertical acceleration and jerk, consistently identified the same point in the heel motion pattern, outperforming velocity-based methods and our new position-based method resembling traditional footswitch-based heel rise detection. Variability in velocity and position-based methods derives from subtle heel rise variations after mid-stance, exhibiting individual differences. Our proposed IMU-based methods show promise by closely matching OMC-based accuracy.

Significance

The results have significant implications for gait analysis, providing insights into heel rise event detection's complexities. Accurate HR identification is crucial for gait phase separation, and our findings, especially with the robust heel marker's jerk-based method, enhance precision and consistency across walking conditions. Moreover, our successful development and validation of IMU-based algorithm offer cost-effective and mobile alternative for HR detection, expanding their potential use in comprehensive gait analysis.

Implementing Gait Kinematic Trajectory Forecasting Models on an Embedded System

Smart algorithms for gait kinematic motion prediction in wearable assistive devices including prostheses, bionics, and exoskeletons can ensure safer and more effective device functionality. Although embedded systems can support the use of smart algorithms, there are important limitations associated with computational load. This poses a tangible barrier for models with increased complexity that demand substantial computational resources for superior performance. Forecasting through Recurrent Topology (FReT) represents a computationally lightweight time-series data forecasting algorithm with the ability to update and adapt to the input data structure that can predict complex dynamics. Here, we deployed FReT on an embedded system and evaluated its accuracy, computational time, and precision to forecast gait kinematics from lower-limb motion sensor data from fifteen subjects. FReT was compared to pretrained hyperparameter-optimized NNET and deep-NNET (D-NNET) model architectures, both with static model weight parameters and iteratively updated model weight parameters to enable adaptability to evolving data structures. We found that FReT was not only more accurate than all the network models, reducing the normalized root-mean-square error by almost half on average, but that it also provided the best balance between accuracy, computational time, and precision when considering the combination of these performance variables. The proposed FReT framework on an embedded system, with its improved performance, represents an important step towards the development of new sensor-aided technologies for assistive ambulatory devices.

Gait Prediction and Variable Admittance Control for Lower Limb Exoskeleton With Measurement Delay and Extended-State-Observer

The measurement delay of the feedback control system is a universal problem in industrial engineering, which will degrade output performance, especially causing undesirable chatter responses. In this study, a deep-Gaussian-process (DGP)-based method for operator's gait prediction is proposed to estimate the real-time motion intention and to compensate for the measurement delay of the inertial measurement unit (IMU). On the basis of these gait prediction uncertainties quantified by the DGP method, a variable admittance controller is designed to reduce real-time human-exoskeleton interaction torque. The reference trajectory is generated by the admittance controller, which is smoothed by the two-order Bessel interpolation. Meanwhile, the admittance parameters are self-regulated based on the defined uncertainty index of gait prediction. The extend-state observer (ESO) with backstepping iteration is adopted to compensate unmeasured system state, model uncertainties, and unmodeled dynamics of lower limb exoskeleton. The effectiveness of the proposed gait prediction and control scheme is verified by both the comparative simulations and experimental results of the human-exoskeleton cooperative motion.

Gait Trajectory Prediction on an Embedded Microcontroller Using Deep Learning

Achieving a normal gait trajectory for an amputee's active prosthesis is challenging due to its kinematic complexity. Accordingly, lower limb gait trajectory kinematics and gait phase segmentation are essential parameters in controlling an active prosthesis. Recently, the most practiced algorithm in gait trajectory generation is the neural network. Deploying such a complex Artificial Neural Network (ANN) algorithm on an embedded system requires performing the calculations on an external computational device; however, this approach lacks mobility and reliability. In this paper, more simple and reliable ANNs are investigated to be deployed on a single low-cost Microcontroller (MC) and hence provide system mobility. Two neural network configurations were studied: Multi-Layered Perceptron (MLP) and Convolutional Neural Network (CNN); the models were trained on shank and foot IMU data. The data were collected from four subjects and tested on a fifth to predict the trajectory of 200 ms ahead. The prediction was made for two cases: with and without providing the current phase of the gait. Then, the models were deployed on a low-cost microcontroller (ESP32). It was found that with fewer data (excluding the current gait phase), CNN achieved a better correlation coefficient of 0.973 when compared to 0.945 for MLP; when including the current phase, both network configurations achieved better correlation coefficients of nearly 0.98. However, when comparing the execution time required for the prediction on the intended MC, MLP was much faster than CNN, with an execution time of 2.4 ms and 142 ms, respectively. In summary, it was found that when training data are scarce, CNN is more efficient within the acceptable execution time, while MLP achieves relative accuracy with low execution time with enough data.

Performance of Deep Learning Models in Forecasting Gait Trajectories of Children with Neurological Disorders

Forecasted gait trajectories of children could be used as feedforward input to control lower limb robotic devices, such as exoskeletons and actuated orthotic devices (e.g., Powered Ankle Foot Orthosis—PAFO). Several studies have forecasted healthy gait trajectories, but, to the best of our knowledge, none have forecasted gait trajectories of children with pathological gait yet. These exhibit higher inter- and intra-subject variability compared to typically developing gait of healthy subjects. Pathological trajectories represent the typical gait patterns that rehabilitative exoskeletons and actuated orthoses would target. In this study, we implemented two deep learning models, a Long-Term Short Memory (LSTM) and a Convolutional Neural Network (CNN), to forecast hip, knee, and ankle trajectories in terms of corresponding Euler angles in the pitch, roll, and yaw form for children with neurological disorders, up to 200 ms in the future. The deep learning models implemented in our study are trained on data (available online) from children with neurological disorders collected by Gillette Children’s Speciality Healthcare over the years 1994–2017. The children’s ages range from 4 to 19 years old and the majority of them had cerebral palsy (73%), while the rest were a combination of neurological, developmental, orthopaedic, and genetic disorders (27%). Data were recorded with a motion capture system (VICON) with a sampling frequency of 120 Hz while walking for 15 m. We investigated a total of 35 combinations of input and output time-frames, with window sizes for input vectors ranging from 50–1000 ms, and output vectors from 8.33–200 ms. Results show that LSTMs outperform CNNs, and the gap in performance becomes greater the larger the input and output window sizes are. The maximum difference between the Mean Absolute Errors (MAEs) of the CNN and LSTM networks was 0.91 degrees. Results also show that the input size has no significant influence on mean prediction errors when the output window is 50 ms or smaller. For output window sizes greater than 50 ms, the larger the input window, the lower the error. Overall, we obtained MAEs ranging from 0.095–2.531 degrees for the LSTM network, and from 0.129–2.840 degrees for the CNN. This study establishes the feasibility of forecasting pathological gait trajectories of children which could be integrated with exoskeleton control systems and experimentally explores the characteristics of such intelligent systems under varying input and output window time-frames.

A Novel sEMG-Based Gait Phase-Kinematics-Coupled Predictor and Its Interaction With Exoskeletons

The interaction between human and exoskeletons increasingly relies on the precise decoding of human motion. One main issue of the current motion decoding algorithms is that seldom algorithms provide both discrete motion patterns (e.g., gait phases) and continuous motion parameters (e.g., kinematics). In this paper, we propose a novel algorithm that uses the surface electromyography (sEMG) signals that are generated prior to their corresponding motions to perform both gait phase recognition and lower-limb kinematics prediction. Particularly, we first propose an end-to-end architecture that uses the gait phase and EMG signals as the priori of the kinematics predictor. In so doing, the prediction of kinematics can be enhanced by the ahead-of-motion property of sEMG and quasi-periodicity of gait phases. Second, we propose to select the optimal muscle set and reduce the number of sensors according to the muscle effects in a gait cycle. Finally, we experimentally investigate how the assistance of exoskeletons can affect the motion intent predictor, and we propose a novel paradigm to make the predictor adapt to the change of data distribution caused by the exoskeleton assistance. The experiments on 10 subjects demonstrate the effectiveness of our algorithm and reveal the interaction between assistance and the kinematics predictor. This study would aid the design of exoskeleton-oriented motion-decoding and human–machine interaction methods.

Prediction of gait trajectories based on the Long Short Term Memory neural networks

The forecasting of lower limb trajectories can improve the operation of assistive devices and minimise the risk of tripping and balance loss. The aim of this work was to examine four Long Short Term Memory (LSTM) neural network architectures (Vanilla, Stacked, Bidirectional and Autoencoders) in predicting the future trajectories of lower limb kinematics, i.e. Angular Velocity (AV) and Linear Acceleration (LA). Kinematics data of foot, shank and thigh (LA and AV) were collected from 13 male and 3 female participants (28 ± 4 years old, 1.72 ± 0.07 m in height, 66 ± 10 kg in mass) who walked for 10 minutes at preferred walking speed (4.34 ± 0.43 km.h-1) and at an imposed speed (5km.h-1, 15.4% ± 7.6% faster) on a 0% gradient treadmill. The sliding window technique was adopted for training and testing the LSTM models with total kinematics time-series data of 10,500 strides. Results based on leave-one-out cross validation, suggested that the LSTM autoencoders is the top predictor of the lower limb kinematics trajectories (i.e. up to 0.1s). The normalised mean squared error was evaluated on trajectory predictions at each time-step and it obtained 2.82-5.31% for the LSTM autoencoders. The ability to predict future lower limb motions may have a wide range of applications including the design and control of bionics allowing improved human-machine interface and mitigating the risk of falls and balance loss.

A Muscle Synergy-Inspired Method of Detecting Human Movement Intentions Based on Wearable Sensor Fusion

Detecting human movement intentions is fundamental to neural control of robotic exoskeletons, as it is essential for achieving seamless transitions between different locomotion modes. In this study, we enhanced a muscle synergy-inspired method of locomotion mode identification by fusing the electromyography data with two types of data from wearable sensors (inertial measurement units), namely linear acceleration and angular velocity. From the finite state machine perspective, the enhanced method was used to systematically identify 2 static modes, 7 dynamic modes, and 27 transitions among them. In addition to the five broadly studied modes (level ground walking, ramps ascent/descent, stairs ascent/descent), we identified the transition between different walking speeds and modes of ramp walking at different inclination angles. Seven combinations of sensor fusion were conducted, on experimental data from 8 able-bodied adult subjects, and their classification accuracy and prediction time were compared. Prediction based on a fusion of electromyography and gyroscope (angular velocity) data predicted transitions earlier and with higher accuracy. All transitions and modes were identified with a total average classification accuracy of 94.5% with fused sensor data. For nearly all transitions, we were able to predict the next locomotion mode 300-500ms prior to the step into that mode.

Gait Trajectory and Gait Phase Prediction Based on an LSTM Network

Lower body segment trajectory and gait phase prediction is crucial for the control of assistance-as-needed robotic devices, such as exoskeletons. In order for a powered exoskeleton with phase-based control to determine and provide proper assistance to the wearer during gait, we propose an approach to predict segment trajectories up to 200 ms ahead (angular velocity of the thigh, shank and foot segments) and five gait phases (loading response, mid-stance, terminal stance, preswing and swing), based on collected data from inertial measurement units placed on the thighs, shanks, and feet. The approach we propose is a long-short term memory (LSTM)-based network, a modified version of recurrent neural networks, which can learn order dependence in sequence prediction problems. The algorithm proposed has a weighted discount loss function that places more weight in predicting the next three to five time frames but also contributes to an overall prediction performance for up to 10 time frames. The LSTM model was designed to learn lower limb segment trajectories using training samples and was tested for generalization across participants. All predicted trajectories were strongly correlated with the measured trajectories, with correlation coefficients greater than 0.98. The proposed LSTM approach can also accurately predict the five gait phases, particularly swing phase with 95% accuracy in inter-subject implementation. The ability of the LSTM network to predict future gait trajectories and gait phases can be applied in designing exoskeleton controllers that can better compensate for system delays to smooth the transition between gait phases.

Lower Limb Kinematics Trajectory Prediction Using Long Short-Term Memory Neural Networks

This study determined whether the kinematics of lower limb trajectories during walking could be extrapolated using long short-term memory (LSTM) neural networks. It was hypothesised that LSTM auto encoders could reliably forecast multiple time-step trajectories of the lower limb kinematics, specifically linear acceleration (LA) and angular velocity (AV). Using 3D motion capture, lower limb position-time coordinates were sampled (100 Hz) from six male participants (age 22 ± 2 years, height 1.77 ± 0.02 m, body mass 82 ± 4 kg) who walked for 10 min at 5 km/h on a 0% gradient motor-driven treadmill. These data were fed into an LSTM model with a sliding window of four kinematic variables with 25 samples or time steps: LA and AV for thigh and shank. The LSTM was tested to forecast five samples (i.e., time steps) of the four kinematic input variables. To attain generalisation, the model was trained on a dataset of 2,665 strides from five participants and evaluated on a test set of 1 stride from a sixth participant. The LSTM model learned the lower limb kinematic trajectories using the training samples and tested for generalisation across participants. The forecasting horizon suggested higher model reliability in predicting earlier future trajectories. The mean absolute error (MAE) was evaluated on each variable across the single tested stride, and for the five-sample forecast, it obtained 0.047 m/s2 thigh LA, 0.047 m/s2 shank LA, 0.028 deg/s thigh AV and 0.024 deg/s shank AV. All predicted trajectories were highly correlated with the measured trajectories, with correlation coefficients greater than 0.98. The motion prediction model may have a wide range of applications, such as mitigating the risk of falls or balance loss and improving the human-machine interface for wearable assistive devices.