Real-Time Activity Recognition With Instantaneous Characteristic Features of Thigh Kinematics

Automatic Stub Avoidance for a Powered Prosthetic Leg Over Stairs and Obstacles

Passive prosthetic legs require undesirable compensations from amputee users to avoid stubbing obstacles and stairsteps. Powered prostheses can reduce those compensations by restoring normative joint biomechanics, but the absence of user proprioception and volitional control combined with the absence of environmental awareness by the prosthesis increases the risk of collisions. This article presents a novel stub avoidance controller that automatically adjusts prosthetic knee/ankle kinematics based on suprasensory measurements of environmental distance from a small, lightweight, low-power, low-cost ultrasonic sensor mounted above the prosthetic ankle. In a case study with two transfemoral amputee participants, this control method reduced the stub rate during stair ascent by 89.95% and demonstrated an 87.50% avoidance rate for crossing different obstacles on level ground. No thigh kinematic compensation was required to achieve these results. These findings demonstrate a practical perception solution for powered prostheses to avoid collisions with stairs and obstacles while restoring normative biomechanics during daily activities.

Swing-phase detection of locomotive mode transitions for smooth multi-functional robotic lower-limb prosthesis control

Robotic lower-limb prostheses, with their actively powered joints, may significantly improve amputee users' mobility and enable them to obtain healthy-like gait in various modes of locomotion in daily life. However, timely recognition of the amputee users' locomotive mode and mode transition still remains a major challenge in robotic lower-limb prosthesis control. In the paper, the authors present a new multi-dimensional dynamic time warping (mDTW)-based intent recognizer to provide high-accuracy recognition of the locomotion mode/mode transition sufficiently early in the swing phase, such that the prosthesis' joint-level motion controller can operate in the correct locomotive mode and assist the user to complete the desired (and often power-demanding) motion in the stance phase. To support the intent recognizer development, the authors conducted a multi-modal gait data collection study to obtain the related sensor signal data in various modes of locomotion. The collected data were then segmented into individual cycles, generating the templates used in the mDTW classifier. Considering the large number of sensor signals available, we conducted feature selection to identify the most useful sensor signals as the input to the mDTW classifier. We also augmented the standard mDTW algorithm with a voting mechanism to make full use of the data generated from the multiple subjects. To validate the proposed intent recognizer, we characterized its performance using the data cumulated at different percentages of progression into the gait cycle (starting from the beginning of the swing phase). It was shown that the mDTW classifier was able to recognize three locomotive mode/mode transitions (walking, walking to stair climbing, and walking to stair descending) with 99.08% accuracy at 30% progression into the gait cycle, well before the stance phase starts. With its high performance, low computational load, and easy personalization (through individual template generation), the proposed mDTW intent recognizer may become a highly useful building block of a prosthesis control system to facilitate the robotic prostheses' real-world use among lower-limb amputees.

A SE-DenseNet-LSTM model for locomotion mode recognition in lower limb exoskeleton

Locomotion mode recognition in humans is fundamental for flexible control in wearable-powered exoskeleton robots. This article proposes a hybrid model that combines a dense convolutional network (DenseNet) and long short-term memory (LSTM) with a channel attention mechanism (SENet) for locomotion mode recognition. DenseNet can automatically extract deep-level features from data, while LSTM effectively captures long-dependent information in time series. To evaluate the validity of the hybrid model, inertial measurement units (IMUs) and pressure sensors were used to obtain motion data from 15 subjects. Five locomotion modes were tested for the hybrid model, such as level ground walking, stair ascending, stair descending, ramp ascending, and ramp descending. Furthermore, the data features of the ramp were inconspicuous, leading to large recognition errors. To address this challenge, the SENet module was incorporated, which improved recognition rates to some extent. The proposed model automatically extracted the features and achieved an average recognition rate of 97.93%. Compared with known algorithms, the proposed model has substantial recognition results and robustness. This work holds promising potential for applications such as limb support and weight bearing.

Improving Amputee Endurance over Activities of Daily Living with a Robotic Knee-Ankle Prosthesis: A Case Study

Robotic knee-ankle prostheses have often fallen short relative to passive microprocessor prostheses in time-based clinical outcome tests. User ambulation endurance is an alternative clinical outcome metric that may better highlight the benefits of robotic prostheses. However, previous studies were unable to show endurance benefits due to inaccurate high-level classification, discretized mid-level control, and insufficiently difficult ambulation tasks. In this case study, we present a phase-based mid-level prosthesis controller which yields biomimetic joint kinematics and kinetics that adjust to suit a continuum of tasks. We enrolled an individual with an above-knee amputation and challenged him to perform repeated, rapid laps of a circuit comprising activities of daily living with both his passive prosthesis and a robotic prosthesis. The participant demonstrated improved endurance with the robotic prosthesis and our mid-level controller compared to his passive prosthesis, completing over twice as many total laps before fatigue and muscle discomfort required him to stop. We also show that time-based outcome metrics fail to capture this endurance improvement, suggesting that alternative metrics related to endurance and fatigue may better highlight the clinical benefits of robotic prostheses.

Data-Driven Variable Impedance Control of a Powered Knee-Ankle Prosthesis for Adaptive Speed and Incline Walking

Most impedance-based walking controllers for powered knee-ankle prostheses use a finite state machine with dozens of user-specific parameters that require manual tuning by technical experts. These parameters are only appropriate near the task (e.g., walking speed and incline) at which they were tuned, necessitating many different parameter sets for variable-task walking. In contrast, this paper presents a data-driven, phase-based controller for variable-task walking that uses continuously-variable impedance control during stance and kinematic control during swing to enable biomimetic locomotion. After generating a data-driven model of variable joint impedance with convex optimization, we implement a novel task-invariant phase variable and real-time estimates of speed and incline to enable autonomous task adaptation. Experiments with above-knee amputee participants (N=2) show that our data-driven controller 1) features highly-linear phase estimates and accurate task estimates, 2) produces biomimetic kinematic and kinetic trends as task varies, leading to low errors relative to able-bodied references, and 3) produces biomimetic joint work and cadence trends as task varies. We show that the presented controller meets and often exceeds the performance of a benchmark finite state machine controller for our two participants, without requiring manual impedance tuning.

Real-Time Human Activity Recognition with IMU and Encoder Sensors in Wearable Exoskeleton Robot via Deep Learning Networks

Wearable exoskeleton robots have become a promising technology for supporting human motions in multiple tasks. Activity recognition in real-time provides useful information to enhance the robot's control assistance for daily tasks. This work implements a real-time activity recognition system based on the activity signals of an inertial measurement unit (IMU) and a pair of rotary encoders integrated into the exoskeleton robot. Five deep learning models have been trained and evaluated for activity recognition. As a result, a subset of optimized deep learning models was transferred to an edge device for real-time evaluation in a continuous action environment using eight common human tasks: stand, bend, crouch, walk, sit-down, sit-up, and ascend and descend stairs. These eight robot wearer's activities are recognized with an average accuracy of 97.35% in real-time tests, with an inference time under 10 ms and an overall latency of 0.506 s per recognition using the selected edge device.

IMU-Based Classification of Locomotion Modes, Transitions, and Gait Phases with Convolutional Recurrent Neural Networks

This paper focuses on the classification of seven locomotion modes (sitting, standing, level ground walking, ramp ascent and descent, stair ascent and descent), the transitions among these modes, and the gait phases within each mode, by only using data in the frequency domain from one or two inertial measurement units. Different deep neural network configurations are investigated and compared by combining convolutional and recurrent layers. The results show that a system composed of a convolutional neural network followed by a long short-term memory network is able to classify with a mean F1-score of 0.89 and 0.91 for ten healthy subjects, and of 0.92 and 0.95 for one osseointegrated transfemoral amputee subject (excluding the gait phases because they are not labeled in the data-set), using one and two inertial measurement units, respectively, with a 5-fold cross-validation. The promising results obtained in this study pave the way for using deep learning for the control of transfemoral prostheses with a minimum number of inertial measurement units.

Modeling the Transitional Kinematics Between Variable-Incline Walking and Stair Climbing

Although emerging powered prostheses can enable people with lower-limb amputation to walk and climb stairs over different task conditions (e.g., speeds and inclines), the control architecture typically uses a finite-state machine to switch between activity-specific controllers. Because these controllers focus on steady-state locomotion, powered prostheses abruptly switch between controllers during gait transitions rather than continuously adjusting leg biomechanics in synchrony with the users. This paper introduces a new framework for powered prosthesis control by modeling the lower-limb joint kinematics over a continuum of variable-incline walking and stair climbing, including steady-state and transitional gaits. Steady-state models for walking and stair climbing represent joint kinematics as continuous functions of gait phase, forward speed, and incline. Transition models interpolate kinematics as convex combinations of the two steady-state models, with an additional term to account for kinematics that fall outside their convex hull. The coefficients of this convex combination denote the similarity of the transitional kinematics to each steady-state mode, providing insight into how able-bodied individuals continuously transition between ambulation modes. Cross-validation demonstrates that the model predictions of untrained kinematics have errors within the range of physiological variability for all joints. Simulation results demonstrate the model's robustness to incline estimation and mode classification errors.

Stair Ascent Phase-Variable Control of a Powered Knee-Ankle Prosthesis

Passive prostheses cannot provide the net positive work required at the knee and ankle for step-over stair ascent. Powered prostheses can provide this net positive work, but user synchronization of joint motion and power input are critical to enabling natural stair ascent gaits. In this work, we build on previous phase variable-based control methods for walking and propose a stair ascent controller driven by the motion of the user's residual thigh. We use reference kinematics from an able-bodied dataset to produce knee and ankle joint trajectories parameterized by gait phase. We redefine the gait cycle to begin at the point of maximum hip flexion instead of heel strike to improve the phase estimate. Able-bodied bypass adapter experiments demonstrate that the phase variable controller replicates normative able-bodied kinematic trajectories with a root mean squared error of 12.66° and 2.64° for the knee and ankle, respectively. The knee and ankle joints provided on average 0.39 J/kg and 0.21 J/kg per stride, compared to the normative averages of 0.34 J/kg and 0.21 J/kg, respectively. Thus, this controller allows powered knee-ankle prostheses to perform net positive mechanical work to assist stair ascent.