Online Reinforcement Learning Control for the Personalization of a Robotic Knee Prosthesis

Experiment-free exoskeleton assistance via learning in simulation

Exoskeletons have enormous potential to improve human locomotive performance1-3. However, their development and broad dissemination are limited by the requirement for lengthy human tests and handcrafted control laws2. Here we show an experiment-free method to learn a versatile control policy in simulation. Our learning-in-simulation framework leverages dynamics-aware musculoskeletal and exoskeleton models and data-driven reinforcement learning to bridge the gap between simulation and reality without human experiments. The learned controller is deployed on a custom hip exoskeleton that automatically generates assistance across different activities with reduced metabolic rates by 24.3%, 13.1% and 15.4% for walking, running and stair climbing, respectively. Our framework may offer a generalizable and scalable strategy for the rapid development and widespread adoption of a variety of assistive robots for both able-bodied and mobility-impaired individuals.

Intuitive control of additional prosthetic joints via electro-neuromuscular constructs improves functional and disability outcomes during home use-a case study

Objective.The advent of surgical reconstruction techniques has enabled the recreation of myoelectric controls sites that were previously lost due to amputation. This advancement is particularly beneficial for individuals with higher-level arm amputations, who were previously constrained to using a single degree of freedom (DoF) myoelectric prostheses due to the limited number of available muscles from which control signals could be extracted. In this study, we explore the use of surgically created electro-neuromuscular constructs to intuitively control multiple bionic joints during daily life with a participant who was implanted with a neuromusculoskeletal prosthetic interface.Approach.We sequentially increased the number of controlled joints, starting at a single DoF allowing to open and close the hand, subsequently adding control of the wrist (2 DoF) and elbow (3 DoF).Main results.We found that the surgically created electro-neuromuscular constructs allow for intuitive simultaneous and proportional control of up to three degrees of freedom using direct control. Extended home-use and the additional bionic joints resulted in improved prosthesis functionality and disability outcomes.Significance.Our findings indicate that electro-neuromuscular constructs can aid in restoring lost functionality and thereby support a person who lost their arm in daily-life tasks.

Human-in-the-Loop Optimization of Wearable Robotic Devices to Improve Human-Robot Interaction: A Systematic Review

This article presents a systematic review on wearable robotic devices that use human-in-the-loop optimization (HILO) strategies to improve human-robot interaction. A total of 46 HILO studies were identified and divided into upper and lower limb robotic devices. The main aspects from HILO were identified, reviewed, and classified in four areas: 1) human-machine systems; 2) optimization methods; 3) control strategies; and 4) experimental protocols. A variety of objective functions (physiological, biomechanical, and subjective), optimization strategies, and optimized control parameters configurations used in different control strategies are presented and analyzed. An overview of experimental protocols is provided, including metrics, tasks, and conditions tested. Moreover, the relevance given to training or adaptation periods was explored. We outline an HILO framework that includes current wearable robots, optimization strategies, objective functions, control strategies, and experimental protocols. We conclude by highlighting current research gaps and defining future directions to improve the development of advanced HILO strategies in upper and lower limb wearable robots.

Tracking Control for a Lower Extremity Exoskeleton Based on Adaptive Dynamic Programing

The utilization of lower extremity exoskeletons has witnessed a growing presence across diverse domains such as the military, medical treatment, and rehabilitation. This paper introduces a novel design of a lower extremity exoskeleton specifically tailored for individuals engaged in heavy object carrying tasks. The exoskeleton incorporates an impressive 12 degrees of freedom (DOF), with four of them being effectively controlled through hydraulic cylinders. To achieve optimal control of this intricate lower extremity exoskeleton system, the authors propose an adaptive dynamic programming (ADP) algorithm. Several crucial components are established to implement this control scheme. These include the formulation of the state equation for the lower extremity exoskeleton system, which is well-suited for the ADP algorithm. Additionally, a corresponding performance index function based on the tracking error is devised, along with the game algebraic Riccati equation. By employing the value iteration ADP scheme, the lower extremity exoskeleton demonstrates highly effective tracking control. This research not only highlights the potential of the proposed control approach but also showcases its ability to enhance the overall performance and functionality of lower extremity exoskeletons, particularly in scenarios involving heavy object carrying. Overall, this study contributes to the advancement of lower extremity exoskeleton technology and offers valuable insights into the application of ADP algorithms for achieving precise and efficient control in demanding tasks.

A Review of Current State-of-the-Art Control Methods for Lower-Limb Powered Prostheses

Lower-limb prostheses aim to restore ambulatory function for individuals with lower-limb amputations. While the design of lower-limb prostheses is important, this paper focuses on the complementary challenge - the control of lower-limb prostheses. Specifically, we focus on powered prostheses, a subset of lower-limb prostheses, which utilize actuators to inject mechanical power into the walking gait of a human user. In this paper, we present a review of existing control strategies for lower-limb powered prostheses, including the control objectives, sensing capabilities, and control methodologies. We separate the various control methods into three main tiers of prosthesis control: high-level control for task and gait phase estimation, mid-level control for desired torque computation (both with and without the use of reference trajectories), and low-level control for enforcing the computed torque commands on the prosthesis. In particular, we focus on the high- and mid-level control approaches in this review. Additionally, we outline existing methods for customizing the prosthetic behavior for individual human users. Finally, we conclude with a discussion on future research directions for powered lower-limb prostheses based on the potential of current control methods and open problems in the field.

Graph-Attention-Based Casual Discovery With Trust Region-Navigated Clipping Policy Optimization

In many domains of empirical sciences, discovering the causal structure within variables remains an indispensable task. Recently, to tackle unoriented edges or latent assumptions violation suffered by conventional methods, researchers formulated a reinforcement learning (RL) procedure for causal discovery and equipped a REINFORCE algorithm to search for the best rewarded directed acyclic graph. The two keys to the overall performance of the procedure are the robustness of RL methods and the efficient encoding of variables. However, on the one hand, REINFORCE is prone to local convergence and unstable performance during training. Neither trust region policy optimization, being computationally expensive, nor proximal policy optimization (PPO), suffering from aggregate constraint deviation, is a decent alternative for combinatory optimization problems with considerable individual subactions. We propose a trust region-navigated clipping policy optimization method for causal discovery that guarantees both better search efficiency and steadiness in policy optimization, in comparison with REINFORCE, PPO, and our prioritized sampling-guided REINFORCE implementation. On the other hand, to boost the efficient encoding of variables, we propose a refined graph attention encoder called SDGAT that can grasp more feature information without priori neighborhood information. With these improvements, the proposed method outperforms the former RL method in both synthetic and benchmark datasets in terms of output results and optimization robustness.

Double Sparse Deep Reinforcement Learning via Multilayer Sparse Coding and Nonconvex Regularized Pruning

Deep reinforcement learning (DRL), which highly depends on the data representation, has shown its potential in many practical decision-making problems. However, the process of acquiring representations in DRL is easily affected by interference from models, and moreover leaves unnecessary parameters, leading to control performance reduction. In this article, we propose a double sparse DRL via multilayer sparse coding and nonconvex regularized pruning. To alleviate interference in DRL, we propose a multilayer sparse-coding-structural network to obtain deep sparse representation for control in reinforcement learning. Furthermore, we employ a nonconvex log regularizer to promote strong sparsity, efficiently removing the unnecessary weights with a regularizer-based pruning scheme. Hence, a double sparse DRL algorithm is developed, which can not only learn deep sparse representation to reduce the interference but also remove redundant weights while keeping the robust performance. The experimental results in five benchmark environments of the deep q network (DQN) architecture demonstrate that the proposed method with deep sparse representations from the multilayer sparse-coding structure can outperform existing sparse-coding-based DRL in control, for example, completing Mountain Car with 140.81 steps, achieving near 10% reward increase from the single-layer sparse-coding DRL algorithm, and obtaining 286.08 scores in Catcher, which are over two times the rewards of the other algorithms. Moreover, the proposed algorithm can reduce over 80% parameters while keeping performance improvements from deep sparse representations.

Addressing Hindsight Bias in Multigoal Reinforcement Learning

Multigoal reinforcement learning (RL) extends the typical RL with goal-conditional value functions and policies. One efficient multigoal RL algorithm is the hindsight experience replay (HER). By treating a hindsight goal from failed experiences as the original goal, HER enables the agent to receive rewards frequently. However, a key assumption of HER is that the hindsight goals do not change the likelihood of the sampled transitions and trajectories used in training, which is not the fact according to our analysis. More specifically, we show that using hindsight goals changes such a likelihood and results in a biased learning objective for multigoal RL. We analyze the hindsight bias due to this use of hindsight goals and propose the bias-corrected HER (BHER), an efficient algorithm that corrects the hindsight bias in training. We further show that BHER outperforms several state-of-the-art multigoal RL approaches in challenging robotics tasks.

Continuous mode adaptation for cable-driven rehabilitation robot using reinforcement learning

Continuous mode adaptation is very important and useful to satisfy the different user rehabilitation needs and improve human-robot interaction (HRI) performance for rehabilitation robots. Hence, we propose a reinforcement-learning-based optimal admittance control (RLOAC) strategy for a cable-driven rehabilitation robot (CDRR), which can realize continuous mode adaptation between passive and active working mode. To obviate the requirement of the knowledge of human and robot dynamics model, a reinforcement learning algorithm was employed to obtain the optimal admittance parameters by minimizing a cost function composed of trajectory error and human voluntary force. Secondly, the contribution weights of the cost function were modulated according to the human voluntary force, which enabled the CDRR to achieve continuous mode adaptation between passive and active working mode. Finally, simulation and experiments were conducted with 10 subjects to investigate the feasibility and effectiveness of the RLOAC strategy. The experimental results indicated that the desired performances could be obtained; further, the tracking error and energy per unit distance of the RLOAC strategy were notably lower than those of the traditional admittance control method. The RLOAC strategy is effective in improving the tracking accuracy and robot compliance. Based on its performance, we believe that the proposed RLOAC strategy has potential for use in rehabilitation robots.

Human-prosthesis cooperation: combining adaptive prosthesis control with visual feedback guided gait

BACKGROUND

Personalizing prosthesis control is often structured as human-in-the-loop optimization. However, gait performance is influenced by both human control and intelligent prosthesis control. Hence, we need to consider both human and prosthesis control, and their cooperation, to achieve desired gait patterns. In this study, we developed a novel paradigm that engages human gait control via user-fed visual feedback (FB) of stance time to cooperate with automatic prosthesis control tuning. Three initial questions were studied: (1) does user control of gait timing (via visual FB) help the prosthesis tuning algorithm to converge faster? (2) in turn, does the prosthesis control influence the user's ability to reach and maintain the target stance time defined by the feedback? and (3) does the prosthesis control parameters tuned with extended stance time on prosthesis side allow the user to maintain this potentially beneficial behavior even after feedback is removed (short- and long-term retention)?

METHODS

A reinforcement learning algorithm was used to achieve prosthesis control to meet normative knee kinematics in walking. A visual FB system cued the user to control prosthesis-side stance time to facilitate the prosthesis tuning goal. Seven individuals without amputation (AB) and four individuals with transfemoral amputation (TFA) walked with a powered knee prosthesis on a treadmill. Participants completed prosthesis auto-tuning with three visual feedback conditions: no FB, self-selected stance time FB (SS FB), and increased stance time FB (Inc FB). The retention of FB effects was studied by comparing the gait performance across three different prosthesis controls, tuned with different visual FB.

RESULTS

(1) Human control of gait timing reduced the tuning duration in individuals without amputation, but not for individuals with TFA. (2) The change of prosthesis control did not influence users' ability to reach and maintain the visual FB goal. (3) All participants increased their prosthesis-side stance time with the feedback and maintain it right after feedback was removed. However, in the post-test, the prosthesis control parameters tuned with visual FB only supported a few participants with longer stance time and better stance time symmetry.

CONCLUSIONS

The study provides novel insights on human-prosthesis interaction when cooperating in walking, which may guide the future successful adoption of this paradigm in prosthesis control personalization or human-in-the-loop optimization to improve the prosthesis user's gait performance.

Deep Reinforcement Learning for Solving the Heterogeneous Capacitated Vehicle Routing Problem

Existing deep reinforcement learning (DRL)-based methods for solving the capacitated vehicle routing problem (CVRP) intrinsically cope with a homogeneous vehicle fleet, in which the fleet is assumed as repetitions of a single vehicle. Hence, their key to construct a solution solely lies in the selection of the next node (customer) to visit excluding the selection of vehicle. However, vehicles in real-world scenarios are likely to be heterogeneous with different characteristics that affect their capacity (or travel speed), rendering existing DRL methods less effective. In this article, we tackle heterogeneous CVRP (HCVRP), where vehicles are mainly characterized by different capacities. We consider both min-max and min-sum objectives for HCVRP, which aim to minimize the longest or total travel time of the vehicle(s) in the fleet. To solve those problems, we propose a DRL method based on the attention mechanism with a vehicle selection decoder accounting for the heterogeneous fleet constraint and a node selection decoder accounting for the route construction, which learns to construct a solution by automatically selecting both a vehicle and a node for this vehicle at each step. Experimental results based on randomly generated instances show that, with desirable generalization to various problem sizes, our method outperforms the state-of-the-art DRL method and most of the conventional heuristics, and also delivers competitive performance against the state-of-the-art heuristic method, that is, slack induction by string removal. In addition, the results of extended experiments demonstrate that our method is also able to solve CVRPLib instances with satisfactory performance.

Deep-Learning to Map a Benchmark Dataset of Non-amputee Ambulation for Controlling an Open Source Bionic Leg

Powered lower-limb prosthetic devices may be becoming a promising option for amputation patients. Although various methods have been proposed to produce gait trajectories similar to those of non-disabled individuals, implementing these control methods is still challenging. It remains unclear whether these methods provide appropriate, safe, and intuitive locomotion as intended. This paper proposes the direct mapping of the voluntary movement of a residual limb (i.e., thigh) to the desired impedance parameters for amputated limbs (i.e., knee and ankle). The proposed model was learned from the gait trajectories of intact limb individuals from a publicly available biomechanics dataset, and was applied to control the prosthetic leg without post-tuning the network. Thus, the proposed method does not require training time with individuals with amputation nor configuration time for its use, and it provides a closely resembling gait trajectory of the intact limb. For preliminary testing, three able-bodied subjects participated in bypass tests. The proposed model accomplished intuitive and reliable level-ground walking at three different step lengths: self-selected, long-, and short-step lengths. The results indicate that intact benchmark data with different sensor configurations can be directly used to train the model to control prosthetic legs.

Reinforcement Learning Control of Robotic Knee With Human-in-the-Loop by Flexible Policy Iteration

We are motivated by the real challenges presented in a human-robot system to develop new designs that are efficient at data level and with performance guarantees, such as stability and optimality at system level. Existing approximate/adaptive dynamic programming (ADP) results that consider system performance theoretically are not readily providing practically useful learning control algorithms for this problem, and reinforcement learning (RL) algorithms that address the issue of data efficiency usually do not have performance guarantees for the controlled system. This study fills these important voids by introducing innovative features to the policy iteration algorithm. We introduce flexible policy iteration (FPI), which can flexibly and organically integrate experience replay and supplemental values from prior experience into the RL controller. We show system-level performances, including convergence of the approximate value function, (sub)optimality of the solution, and stability of the system. We demonstrate the effectiveness of the FPI via realistic simulations of the human-robot system. It is noted that the problem we face in this study may be difficult to address by design methods based on classical control theory as it is nearly impossible to obtain a customized mathematical model of a human-robot system either online or offline. The results we have obtained also indicate the great potential of RL control to solving realistic and challenging problems with high-dimensional control inputs.

Feasibility study of personalized speed adaptation method based on mental state for teleoperated robots

The teleoperated robotic system can support humans to complete tasks in high-risk, high-precision and difficult special environments. Because this kind of special working environment is easy to cause stress, high mental workload, fatigue and other mental states of the operator, which will reduce the quality of operation and even cause safety accidents, so the mental state of the people in this system has received extensive attention. However, the existence of individual differences and mental state diversity is often ignored, so that most of the existing adjustment strategy is out of a match between mental state and adaptive decision, which cannot effectively improve operational quality and safety. Therefore, a personalized speed adaptation (PSA) method based on policy gradient reinforcement learning was proposed in this paper. It can use electroencephalogram and electro-oculogram to accurately perceive the operator’s mental state, and adjust the speed of the robot individually according to the mental state of different operators, in order to perform teleoperation tasks efficiently and safely. The experimental results showed that the PSA method learns the mapping between the mental state and the robot’s speed regulation action by means of rewards and punishments, and can adjust the speed of the robot individually according to the mental state of different operators, thereby improving the operating quality of the system. And the feasibility and superiority of this method were proved. It is worth noting that the PSA method was validated on 6 real subjects rather than a simulation model. To the best of our knowledge, the PSA method is the first implementation of online reinforcement learning control of teleoperated robots involving human subjects.

Online Reinforcement Learning Control by Direct Heuristic Dynamic Programming: From Time-Driven to Event-Driven

In this work, time-driven learning refers to the machine learning method that updates parameters in a prediction model continuously as new data arrives. Among existing approximate dynamic programming (ADP) and reinforcement learning (RL) algorithms, the direct heuristic dynamic programming (dHDP) has been shown an effective tool as demonstrated in solving several complex learning control problems. It continuously updates the control policy and the critic as system states continuously evolve. It is therefore desirable to prevent the time-driven dHDP from updating due to insignificant system event such as noise. Toward this goal, we propose a new event-driven dHDP. By constructing a Lyapunov function candidate, we prove the uniformly ultimately boundedness (UUB) of the system states and the weights in the critic and the control policy networks. Consequently, we show the approximate control and cost-to-go function approaching Bellman optimality within a finite bound. We also illustrate how the event-driven dHDP algorithm works in comparison to the original time-driven dHDP.

Data-Mined Continuous Hip-knee Coordination Mapping with Motion Lag for Lower-limb Prosthesis Control

Trajectory planning of the knee joint plays an essential role in controlling the lower limb prosthesis. Nowadays, the idea of mapping the trajectory of the healthy limb to the motion trajectory of the prosthetic joint has begun to emerge. However, establishing a simple and intuitive coordination mapping is still challenging. This paper employs the method of experimental data mining to explore such a coordination mapping. The coordination indexes, i.e., the mean absolute relative phase (MARP) and the deviation phase (DP), are obtained from experimental data. Statistical results covering different subjects indicate that the hip motion possesses a stable phase difference with the knee, inspiring us to construct a hip-knee Motion-Lagged Coordination Mapping (MLCM). The MLCM first introduces a time lag to the hip motion to avoid conventional integral or differential calculations. The model in polynomials, which is proved more efficient than Gaussian process regression and neural network learning, is then constructed to represent the mapping from the lagged hip motion to the knee motion. In addition, a strong linear correlation between hip-knee MARP and hip-knee motion lag is discovered for the first time. By using the MLCM, one can generate the knee trajectory for the prosthesis control only via the hip motion of the healthy limb, indicating less sensing and better robustness. Numerical simulations show that the prosthesis can achieve normal gaits at different walking speeds.

Neural H₂ Control Using Continuous-Time Reinforcement Learning

In this article, we discuss continuous-time H₂ control for the unknown nonlinear system. We use differential neural networks to model the system, then apply the H₂ tracking control based on the neural model. Since the neural H₂ control is very sensitive to the neural modeling error, we use reinforcement learning to improve the control performance. The stabilities of the neural modeling and the H₂ tracking control are proven. The convergence of the approach is also given. The proposed method is validated with two benchmark control problems.

Nondominated Policy-Guided Learning in Multi-Objective Reinforcement Learning

Control intelligence is a typical field where there is a trade-off between target objectives, and researchers in this field have longed for artificial intelligence that achieves the target objectives. Multi-objective deep reinforcement learning was sufficient to satisfy this need. In particular, multi-objective deep reinforcement learning methods based on policy optimization are leading the optimization of control intelligence. However, multi-objective reinforcement learning has difficulties when finding various Pareto optimals of multi-objectives due to the greedy nature of reinforcement learning. We propose a method of policy assimilation to solve this problem. This method was applied to MO-V-MPO, one of preference-based multi-objective reinforcement learning, to increase diversity. The performance of this method has been verified through experiments in a continuous control environment.

Improving IMU-based prediction of lower limb kinematics in natural environments using egocentric optical flow

We seek to predict knee and ankle motion using wearable sensors. These predictions could serve as target trajectories for a lower limb prosthesis. In this manuscript, we investigate the use of egocentric vision for improving performance over kinematic wearable motion capture. We present an out-of-the-lab dataset of 23 healthy subjects navigating public classrooms, a large atrium, and stairs for a total of almost 12 hours of recording. The prediction task is difficult because the movements include avoiding obstacles, other people, idiosyncratic movements such as traversing doors, and individual choices in selecting the future path. We demonstrate that using vision improves the quality of the predicted knee and ankle trajectories, especially in congested spaces and when the visual environment provides information that does not appear simply in the movements of the body. Overall, including vision results in 7.9% and 7.0% improvement in root mean squared error of knee and ankle angle predictions respectively. The improvement in Pearson Correlation Coefficient for knee and ankle predictions is 1.5% and 12.3% respectively. We discuss particular moments where vision greatly improved, or failed to improve, the prediction performance. We also find that the benefits of vision can be enhanced with more data. Lastly, we discuss challenges of continuous estimation of gait in natural, out-of-the-lab datasets.

Model-Based Mid-Level Regulation for Assist-As-Needed Hierarchical Control of Wearable Robots: A Computational Study of Human-Robot Adaptation

The closed-loop human–robot system requires developing an effective robotic controller that considers models of both the human and the robot, as well as human adaptation to the robot. This paper develops a mid-level controller providing assist-as-needed (AAN) policies in a hierarchical control setting using two novel methods: model-based and fuzzy logic rule. The goal of AAN is to provide the required extra torque because of the robot’s dynamics and external load compared to the human limb free movement. The human–robot adaptation is simulated using a nonlinear model predictive controller (NMPC) as the human central nervous system (CNS) for three conditions of initial (the initial session of wearing the robot, without any previous experience), short-term (the entire first session, e.g., 45 min), and long-term experiences. The results showed that the two methods (model-based and fuzzy logic) outperform the traditional proportional method in providing AAN by considering distinctive human and robot models. Additionally, the CNS actuator model has difficulty in the initial experience and activates both antagonist and agonist muscles to reduce movement oscillations. In the long-term experience, the simulation shows no oscillation when the CNS NMPC learns the robot model and modifies its weights to simulate realistic human behavior. We found that the desired strength of the robot should be increased gradually to ignore unexpected human–robot interactions (e.g., robot vibration, human spasticity). The proposed mid-level controllers can be used for wearable assistive devices, exoskeletons, and rehabilitation robots.

Control Framework for Sloped Walking With a Powered Transfemoral Prosthesis

User customization of a lower-limb powered Prosthesis controller remains a challenge to this date. Controllers adopting impedance control strategies mandate tedious tuning for every joint, terrain condition, and user. Moreover, no relationship is known to exist between the joint control parameters and the slope condition. We present a control framework composed of impedance control and trajectory tracking, with the transitioning between the two strategies facilitated by Bezier curves. The impedance (stiffness and damping) functions vary as polynomials during the stance phase for both the knee and ankle. These functions were derived through least squares optimization with healthy human sloped walking data. The functions derived for each slope condition were simplified using principal component analysis. The weights of the resulting basis functions were found to obey monotonic trends within upslope and downslope walking, proving the existence of a relationship between the joint parameter functions and the slope angle. Using these trends, one can now design a controller for any given slope angle. Amputee and able-bodied walking trials with a powered transfemoral prosthesis revealed the controller to generate a healthy human gait. The observed kinematic and kinetic trends with the slope angle were similar to those found in healthy walking.

Seamless and intuitive control of a powered prosthetic leg using deep neural network for transfemoral amputees

Powered prosthetic legs are becoming a promising option for amputee patients. However, developing safe, robust, and intuitive control strategies for powered legs remains one of the greatest challenges. Although a variety of control strategies have been proposed, creating and fine-tuning the system parameters is time-intensive and complicated when more activities need to be restored. In this study, we developed a deep neural network (DNN) model that facilitates seamless and intuitive gait generation and transitions across five ambulation modes: level-ground walking, ascending/descending ramps, and ascending/descending stairs. The combination of latent and time sequence features generated the desired impedance parameters within the ambulation modes and allowed seamless transitions between ambulation modes. The model was applied to the open-source bionic leg and tested on unilateral transfemoral users. It achieved the overall coefficient of determination of 0.72 with the state machine-based impedance parameters in the offline testing session. In addition, users were able to perform in-laboratory ambulation modes with an overall success rate of 96% during the online testing session. The results indicate that the DNN model is a promising candidate for subject-independent and tuning-free prosthetic leg control for transfemoral amputees.

Taking Both Sides: Seeking Symbiosis Between Intelligent Prostheses and Human Motor Control during Locomotion

Robotic lower-limb prostheses aim to replicate the power-generating capability of biological joints during locomotion to empower individuals with lower-limb loss. However, recent clinical trials have not demonstrated clear advantages of these devices over traditional passive devices. We believe this is partly because the current designs of robotic prothesis controllers and clinical methods for fitting and training individuals to use them do not ensure good coordination between the prosthesis and user. Accordingly, we advocate for new holistic approaches in which human motor control and intelligent prosthesis control function as one system (defined as human-prosthesis symbiosis). We hope engineers and clinicians will work closely to achieve this symbiosis, thereby improving the functionality and acceptance of robotic prostheses and users' quality of life.

Assistive Sliding Mode Control of a Rehabilitation Robot with Automatic Weight Adjustment

There are approximately 13 million new stroke cases worldwide each year. Research has shown that robotics can provide practical and efficient solutions for expediting post-stroke patient recovery. This simulation study aimed to design a sliding mode controller (SMC) for an end-effector-based rehabilitation robot. A genetic algorithm (GA) was designed for automatic controller weight adjustment. The optimal weights were obtained by minimizing a cost function comprising the end-effector position error, robot input, robot input-rate, and patient input. To promote safe tuner optimization, a model of the human arm was incorporated to generate the human joint torque. A computed-torque proportional derivative controller (CTPD) was designed for the human arm to approximate the central nervous system (CNS) motor control. This controller was adjusted to simulate rehabilitation effects and patient adaptation. The tuner was optimized for a trajectory tracking task with an assistive high-level control scheme. The simulation results showed lower cost compared to seven manual weight settings. The optimal weights provided good tracking performance and suitable robot inputs. This research provides a framework to conduct various simulations before testing our controller on human subjects. The preliminary results of this study will be used as the starting point for online adaptive controller tuning, which will be examined in our future research.

Knee Exoskeletons Design Approaches to Boost Strength Capability: A Review

There are different devices to increase the strength capacity of people with walking problems. These devices can be classified into exoskeletons, orthotics, and braces. This review aims to identify the state of the art in the design of these medical devices, based on an analysis of patents and literature. However, there are some difficulties in processing the records due to the lack of filters and standardization in the names, generating discrepancies between the search engines, among others. Concerning the patents, 74 patents were analyzed using search engines such as Google Patents, Derwent, The Lens, Patentscope, and Espacenet over the past ten years. A bibliometric analysis was performed using 63 scientific reports from Web of Science and The Lens in the same period for scientific communications. The results show a trend to use the mechanical design of exoskeletons based on articulated rigid structures and elements that provide force to move the structure. These are generally two types: (a) elastic elements and (b) electromechanical elements. The United States accounts for 32% of the technological patents reviewed. The results suggest that the use of exoskeletons or orthoses customized to the users’ needs will continue to increase over the years due to the worldwide growth in disability, particularly related to mobility difficulties and technologies related to the combined use of springs and actuators.

Direct continuous electromyographic control of a powered prosthetic ankle for improved postural control after guided physical training: A case study

Despite the promise of powered lower limb prostheses, existing controllers do not assist many daily activities that require continuous control of prosthetic joints according to human states and environments. The objective of this case study was to investigate the feasibility of direct, continuous electromyographic (dEMG) control of a powered ankle prosthesis, combined with physical therapist-guided training, for improved standing postural control in an individual with transtibial amputation. Specifically, EMG signals of the residual antagonistic muscles (i.e. lateral gastrocnemius and tibialis anterior) were used to proportionally drive pneumatical artificial muscles to move a prosthetic ankle. Clinical-based activities were used in the training and evaluation protocol of the control paradigm. We quantified the EMG signals in the bilateral shank muscles as well as measures of postural control and stability. Compared to the participant's daily passive prosthesis, the dEMG-controlled ankle, combined with the training, yielded improved clinical balance scores and reduced compensation from intact joints. Cross-correlation coefficient of bilateral center of pressure excursions, a metric for quantifying standing postural control, increased to .83(±.07) when using dEMG ankle control (passive device: .39(±.29)). We observed synchronized activation of homologous muscles, rapid improvement in performance on the first day of the training for load transfer tasks, and further improvement in performance across training days (p = .006). This case study showed the feasibility of this dEMG control paradigm of a powered prosthetic ankle to assist postural control. This study lays the foundation for future study to extend these results through the inclusion of more participants and activities.

Myoelectric control of robotic lower limb prostheses: a review of electromyography interfaces, control paradigms, challenges and future directions

Objective.Advanced robotic lower limb prostheses are mainly controlled autonomously. Although the existing control can assist cyclic movements during locomotion of amputee users, the function of these modern devices is still limited due to the lack of neuromuscular control (i.e. control based on human efferent neural signals from the central nervous system to peripheral muscles for movement production). Neuromuscular control signals can be recorded from muscles, called electromyographic (EMG) or myoelectric signals. In fact, using EMG signals for robotic lower limb prostheses control has been an emerging research topic in the field for the past decade to address novel prosthesis functionality and adaptability to different environments and task contexts. The objective of this paper is to review robotic lower limb Prosthesis control via EMG signals recorded from residual muscles in individuals with lower limb amputations.Approach.We performed a literature review on surgical techniques for enhanced EMG interfaces, EMG sensors, decoding algorithms, and control paradigms for robotic lower limb prostheses.Main results.This review highlights the promise of EMG control for enabling new functionalities in robotic lower limb prostheses, as well as the existing challenges, knowledge gaps, and opportunities on this research topic from human motor control and clinical practice perspectives.Significance.This review may guide the future collaborations among researchers in neuromechanics, neural engineering, assistive technologies, and amputee clinics in order to build and translate true bionic lower limbs to individuals with lower limb amputations for improved motor function.

A Subvision System for Enhancing the Environmental Adaptability of the Powered Transfemoral Prosthesis

Visual information is indispensable to human locomotion in complex environments. Although amputees can perceive the environmental information by eyes, they cannot transmit the neural signals to prostheses directly. To augment human-prosthesis interaction, this article introduces a subvision system that can perceive environments actively, assist to control the powered prosthesis predictively, and accordingly reconstruct a complete vision-locomotion loop for transfemoral amputees. By using deep learning, the subvision system can classify common static terrains (e.g., level ground, stairs, and ramps) and estimate corresponding motion intents of amputees with high accuracy (98%). After applying the subvision system to the locomotion control system, the powered prosthesis can help amputees to achieve nonrhythmic locomotion naturally, including switching between different locomotion modes and crossing the obstacle. The subvision system can also recognize dynamic objects, such as an unexpected obstacle approaching the amputee, and assist in generating an agile obstacle-avoidance reflex movement. The experimental results demonstrate that the subvision system can cooperate with the powered prosthesis to reconstruct a complete vision-locomotion loop, which enhances the environmental adaptability of the amputees.

Using Reinforcement Learning to Estimate Human Joint Moments From Electromyography or Joint Kinematics: An Alternative Solution to Musculoskeletal-Based Biomechanics

Reinforcement learning (RL) has potential to provide innovative solutions to existing challenges in estimating joint moments in motion analysis, such as kinematic or electromyography (EMG) noise and unknown model parameters. Here, we explore feasibility of RL to assist joint moment estimation for biomechanical applications. Forearm and hand kinematics and forearm EMGs from four muscles during free finger and wrist movement were collected from six healthy subjects. Using the proximal policy optimization approach, we trained two types of RL agents that estimated joint moment based on measured kinematics or measured EMGs, respectively. To quantify the performance of trained RL agents, the estimated joint moment was used to drive a forward dynamic model for estimating kinematics, which was then compared with measured kinematics using Pearson correlation coefficient. The results demonstrated that both trained RL agents are feasible to estimate joint moment for wrist and metacarpophalangeal (MCP) joint motion prediction. The correlation coefficients between predicted and measured kinematics, derived from the kinematics-driven agent and subject-specific EMG-driven agents, were 98% ± 1% and 94% ± 3% for the wrist, respectively, and were 95% ± 2% and 84% ± 6% for the metacarpophalangeal joint, respectively. In addition, a biomechanically reasonable joint moment-angle-EMG relationship (i.e., dependence of joint moment on joint angle and EMG) was predicted using only 15 s of collected data. In conclusion, this study illustrates that an RL approach can be an alternative technique to conventional inverse dynamic analysis in human biomechanics study and EMG-driven human-machine interfacing applications.

Cognitive measures during walking with and without lower-limb prosthesis: protocol for a scoping review

INTRODUCTION

Tuning of lower-limb (LL) robotic prosthesis control is necessary to provide personalised assistance to each human wearer during walking. Prostheses wearers' adaptation processes are subjective and the efficiency largely depends on one's mental processes. Therefore, beyond physical motor performance, prosthesis personalisation should consider the wearer's preference and cognitive performance during walking. As a first step, it is necessary to examine the current measures of cognitive performance when a wearer walks with an LL prosthesis, identify the gaps and methodological considerations, and explore additional measures in a walking setting. In this protocol, we outlined a scoping review that will systematically summarise and evaluate the measures of cognitive performance during walking with and without LL prosthesis.

METHODS AND ANALYSIS

The review process will be guided and documented by CADIMA, an open-access online data management portal for evidence synthesis. Keyword searches will be conducted in seven databases (Web of Science, MEDLINE, BIOSIS, SciELO Citation Index, ProQuest, CINAHL and PsycINFO) up to 2020 supplemented with grey literature searches. Retrieved records will be screened by at least two independent reviewers on the title-and-abstract level and then the full-text level. Selected studies will be evaluated for reporting bias. Data on sample characteristics, type of cognitive function, characteristics of cognitive measures, task prioritisation, experimental design and walking setting will be extracted.

ETHICS AND DISSEMINATION

This scoping review will evaluate the measures used in previously published studies thus does not require ethical approval. The results will contribute to the advancement of prosthesis tuning processes by reviewing the application status of cognitive measures during walking with and without prosthesis and laying the foundation for developing needed measures for cognitive assessment during walking. The results will be disseminated through conferences and journals.

Learning for a Robot: Deep Reinforcement Learning, Imitation Learning, Transfer Learning

Dexterous manipulation of the robot is an important part of realizing intelligence, but manipulators can only perform simple tasks such as sorting and packing in a structured environment. In view of the existing problem, this paper presents a state-of-the-art survey on an intelligent robot with the capability of autonomous deciding and learning. The paper first reviews the main achievements and research of the robot, which were mainly based on the breakthrough of automatic control and hardware in mechanics. With the evolution of artificial intelligence, many pieces of research have made further progresses in adaptive and robust control. The survey reveals that the latest research in deep learning and reinforcement learning has paved the way for highly complex tasks to be performed by robots. Furthermore, deep reinforcement learning, imitation learning, and transfer learning in robot control are discussed in detail. Finally, major achievements based on these methods are summarized and analyzed thoroughly, and future research challenges are proposed.

Reinforcement Q-Learning Control With Reward Shaping Function for Swing Phase Control in a Semi-active Prosthetic Knee

In this study, we investigated a control algorithm for a semi-active prosthetic knee based on reinforcement learning (RL). Model-free reinforcement Q-learning control with a reward shaping function was proposed as the voltage controller of a magnetorheological damper based on the prosthetic knee. The reward function was designed as a function of the performance index that accounts for the trajectory of the subject-specific knee angle. We compared our proposed reward function to a conventional single reward function under the same random initialization of a Q-matrix. We trained this control algorithm to adapt to several walking speed datasets under one control policy and subsequently compared its performance with that of other control algorithms. The results showed that our proposed reward function performed better than the conventional single reward function in terms of the normalized root mean squared error and also showed a faster convergence trend. Furthermore, our control strategy converged within our desired performance index and could adapt to several walking speeds. Our proposed control structure has also an overall better performance compared to user-adaptive control, while some of its walking speeds performed better than the neural network predictive control from existing studies.

Bicycling Phase Recognition for Lower Limb Amputees Using Support Vector Machine Optimized by Particle Swarm Optimization

A novel method for recognizing the phases in bicycling of lower limb amputees using support vector machine (SVM) optimized by particle swarm optimization (PSO) is proposed in this paper. The method is essential for enhanced prosthetic knee joint control for lower limb amputees in carrying out bicycling activity. Some wireless wearable accelerometers and a knee joint angle sensor are installed in the prosthesis to obtain data on the knee joint and ankle joint horizontal, vertical acceleration signal and knee joint angle. In order to overcome the problem of high noise content in the collected data, a soft-hard threshold filter was used to remove the noise caused by the vibration. The filtered information is then used to extract the multi-dimensional feature vector for the training of SVM for performing bicycling phase recognition. The SVM is optimized by PSO to enhance its classification accuracy. The recognition accuracy of the PSO-SVM classification model on testing data is 93%, which is much higher than those of BP, SVM and PSO-BP classification models.

Perceiving amputee gait from biological motion: kinematics cues and effect of experience level

Physical therapists (PT) and clinicians must be skilled in identifying gait features through observation to assess motor deficits in patients and intervene appropriately. Inconsistent results in the literature have led researchers to question how clinical experience influences PT's gait perception and to seek the key kinematic features that should be trained to enhance PT's skill. Thus, this study investigated (1) what are the informative kinematic features that allow gait-deviation perception in amputee gait and (2) whether there are differences in observational gait skills between PT and individuals with less clinical experience (PT students [PTS] and Novices). We introduced a new method that combines biological motion and principal component analysis to gradually mesh amputee and typical walking patterns. Our analysis showed that on average the accuracy rate in identifying gait deviations between PT and PTS was similar and better than Novices. Also, we found that PT's experience was demonstrated by their better perception of gait asymmetry. The extracted principal components demonstrated that the major gait deviation of amputees was the medial-lateral body sway and spatial gait asymmetry.

Pavlovian control of intraspinal microstimulation to produce over-ground walking

OBJECTIVE

Neuromodulation technologies are increasingly used for improving function after neural injury. To achieve a symbiotic relationship between device and user, the device must augment remaining function, and independently adapt to day-to-day changes in function. The goal of this study was to develop predictive control strategies to produce over-ground walking in a model of hemisection spinal cord injury (SCI) using intraspinal microstimulation (ISMS).

APPROACH

Eight cats were anaesthetized and placed in a sling over a walkway. The residual function of a hemisection SCI was mimicked by manually moving one hind-limb through the walking cycle. ISMS targeted motor networks in the lumbosacral enlargement to activate muscles in the other, presumably 'paralyzed' limb, using low levels of current (<130 μA). Four people took turns to move the 'intact' limb, generating four different walking styles. Two control strategies, which used ground reaction force and angular velocity information about the manually moved 'intact' limb to control the timing of the transitions of the 'paralyzed' limb through the step cycle, were compared. The first strategy used thresholds on the raw sensor values to initiate transitions. The second strategy used reinforcement learning and Pavlovian control to learn predictions about the sensor values. Thresholds on the predictions were then used to initiate transitions.

MAIN RESULTS

Both control strategies were able to produce alternating, over-ground walking. Transitions based on raw sensor values required manual tuning of thresholds for each person to produce walking, whereas Pavlovian control did not. Learning occurred quickly during walking: predictions of the sensor signals were learned rapidly, initiating correct transitions after ≤4 steps. Pavlovian control was resilient to different walking styles and different cats, and recovered from induced mistakes during walking.

SIGNIFICANCE

This work demonstrates, for the first time, that Pavlovian control can augment remaining function and facilitate personalized walking with minimal tuning requirements.

Wearer-Prosthesis Interaction for Symmetrical Gait: A Study Enabled by Reinforcement Learning Prosthesis Control

With advances in robotic prostheses, rese-archers attempt to improve amputee's gait performance (e.g., gait symmetry) beyond restoring normative knee kinematics/kinetics. Yet, little is known about how the prosthesis mechanics/control influence wearer-prosthesis' gait performance, such as gait symmetry, stability, etc. This study aimed to investigate the influence of robotic transfemoral prosthesis mechanics on human wearers' gait symmetry. The investigation was enabled by our previously designed reinforcement learning (RL) supplementary control, which simultaneously tuned 12 control parameters that determined the prosthesis mechanics throughout a gait cycle. The RL control design facilitated safe explorations of prosthesis mechanics with the human in the loop. Subjects were recruited and walked with a robotic transfemoral prosthesis on a treadmill while the RL controller tuned the control parameters. Stance time symmetry, step length symmetry, and bilateral anteroposterior (AP) impulses were measured. The data analysis showed that changes in robotic knee mechanics led to movement variations in both lower limbs and therefore gait temporal-spatial symmetry measures. Consistent across all the subjects, inter-limb AP impulse measurements explained gait symmetry: the stance time symmetry was significantly correlated with the net inter-limb AP impulse, and the step length symmetry was significantly correlated with braking and propulsive impulse symmetry. The results suggest that it is possible to personalize transfemoral prosthesis control for improved temporal-spatial gait symmetry. However, adjusting prosthesis mechanics alone was insufficient to maximize the gait symmetry. Rather, achieving gait symmetry may require coordination between the wearer's motor control of the intact limb and adaptive control of the prosthetic joints. The results also indicated that the RL-based prosthesis tuning system was a potential tool for studying wearer-prosthesis interactions.

Effects of extended powered knee prosthesis stance time via visual feedback on gait symmetry of individuals with unilateral amputation: a preliminary study

BACKGROUND

Establishing gait symmetry is a major aim of amputee rehabilitation and may be more attainable with powered prostheses. Though, based on previous work, we postulate that users transfer a previously-learned motor pattern across devices, limiting the functionality of more advanced prostheses. The objective of this study was to preliminarily investigate the effect of increased stance time via visual feedback on amputees' gait symmetry using powered and passive knee prostheses.

METHODS

Five individuals with transfemoral amputation or knee disarticulation walked at their self-selected speed on a treadmill. Visual feedback was used to promote an increase in the amputated-limb stance time. Individuals were fit with a commercially-available powered prosthesis by a certified prosthetist and practiced walking during a prior visit. The same protocol was completed with a passive knee and powered knee prosthesis on separate days. We used repeated-measures, two-way ANOVA (alpha = 0.05) to test for significant effects of the feedback and device factors. Our main outcome measures were stance time asymmetry, peak anterior-posterior ground reaction forces, and peak anterior propulsion asymmetry.

RESULTS

Increasing the amputated-limb stance time via visual feedback significantly improved the stance time symmetry (p = 0.012) and peak propulsion symmetry (p = 0.036) of individuals walking with both prostheses. With the powered knee prosthesis, the highest feedback target elicited 36% improvement in stance time symmetry, 22% increase in prosthesis-side peak propulsion, and 47% improvement in peak propulsion symmetry compared to a no feedback condition. The changes with feedback were not different with the passive prosthesis, and the main effects of device/ prosthesis type were not statistically different. However, subject by device interactions were significant, indicating individuals did not respond consistently with each device (e.g. prosthesis-side propulsion remained comparable to or was greater with the powered versus passive prosthesis for different subjects). Overall, prosthesis-side peak propulsion averaged across conditions was 31% greater with the powered prosthesis and peak propulsion asymmetry improved by 48% with the powered prosthesis.

CONCLUSIONS

Increasing prosthesis-side stance time via visual feedback favorably improved individuals' temporal and propulsive symmetry. The powered prosthesis commonly enabled greater propulsion, but individuals adapted to each device with varying behavior, requiring further investigation.

A Phase Variable Approach for Improved Rhythmic and Non-Rhythmic Control of a Powered Knee-Ankle Prosthesis

Although there has been recent progress in control of multi-joint prosthetic legs for rhythmic tasks such as walking, control of these systems for non-rhythmic motions and general real-world maneuvers is still an open problem. In this article, we develop a new controller that is capable of both rhythmic (constant-speed) walking, transitions between speeds and/or tasks, and some common volitional leg motions. We introduce a new piecewise holonomic phase variable, which, through a finite state machine, forms the basis of our controller. The phase variable is constructed by measuring the thigh angle, and the transitions in the finite state machine are formulated through sensing foot contact along with attributes of a nominal reference gait trajectory. The controller was implemented on a powered knee-ankle prosthesis and tested with a transfemoral amputee subject, who successfully performed a wide range of rhythmic and non-rhythmic tasks, including slow and fast walking, quick start and stop, backward walking, walking over obstacles, and kicking a soccer ball. Use of the powered leg resulted in clinically significant reductions in amputee compensations for rhythmic tasks (including vaulting and hip circumduction) when compared to use of the take-home passive leg. In addition, considerable improvements were also observed in the performance for non-rhythmic tasks. The proposed approach is expected to provide a better understanding of rhythmic and non-rhythmic motions in a unified framework, which in turn can lead to more reliable control of multi-joint prostheses for a wider range of real-world tasks.

A Comparison of Control Strategies in Commercial and Research Knee Prostheses

GOAL

To provide an overview of control strategies in commercial and research microprocessor-controlled prosthetic knees (MPKs).

METHODS

Five commercially available MPKs described in patents, and five research MPKs reported in scientific literature were compared. Their working principles, intent recognition, and walking controller were analyzed. Speed and slope adaptability of the walking controller was considered as well.

RESULTS

Whereas commercial MPKs are mostly passive, i.e., do not inject energy in the system, and employ heuristic rule-based intent classifiers, research MPKs are all powered and often utilize machine learning algorithms for intention detection. Both commercial and research MPKs rely on finite state machine impedance controllers for walking. Yet while commercial MPKs require a prosthetist to adjust impedance settings, scientific research is focused on reducing the tunable parameter space and developing unified controllers, independent of subject anthropometrics, walking speed, and ground slope.

CONCLUSION

The main challenges in the field of powered, active MPKs (A-MPKs) to boost commercial viability are first to demonstrate the benefit of A-MPKs compared to passive MPKs or mechanical non-microprocessor knees using biomechanical, performance-based and patient-reported metrics. Second, to evaluate control strategies and intent recognition in an uncontrolled environment, preferably outside the laboratory setting. And third, even though research MPKs favor sophisticated algorithms, to maintain the possibility of practical and comprehensible tuning of control parameters, considering optimal control cannot be known a priori.

SIGNIFICANCE

This review identifies main challenges in the development of A-MPKs, which have thus far hindered their broad availability on the market.