Reinforcement Learning and Control of a Lower Extremity Exoskeleton for Squat Assistance

Comparison of Empirical and Reinforcement Learning (RL)-Based Control Based on Proximal Policy Optimization (PPO) for Walking Assistance: Does AI Always Win?

The use of wearable assistive devices is growing in both industrial and medical fields. Combining human expertise and artificial intelligence (AI), e.g., in human-in-the-loop-optimization, is gaining popularity for adapting assistance to individuals. Amidst prevailing assertions that AI could surpass human capabilities in customizing every facet of support for human needs, our study serves as an initial step towards such claims within the context of human walking assistance. We investigated the efficacy of the Biarticular Thigh Exosuit, a device designed to aid human locomotion by mimicking the action of the hamstrings and rectus femoris muscles using Serial Elastic Actuators. Two control strategies were tested: an empirical controller based on human gait knowledge and empirical data and a control optimized using Reinforcement Learning (RL) on a neuromuscular model. The performance results of these controllers were assessed by comparing muscle activation in two assisted and two unassisted walking modes. Results showed that both controllers reduced hamstring muscle activation and improved the preferred walking speed, with the empirical controller also decreasing gastrocnemius muscle activity. However, the RL-based controller increased muscle activity in the vastus and rectus femoris, indicating that RL-based enhancements may not always improve assistance without solid empirical support.

Human-Exoskeleton Coupling Simulation for Lifting Tasks with Shoulder, Spine, and Knee-Joint Powered Exoskeletons

In this study, we introduce a two-dimensional (2D) human skeletal model coupled with knee, spine, and shoulder exoskeletons. The primary purpose of this model is to predict the optimal lifting motion and provide torque support from the exoskeleton through the utilization of inverse dynamics optimization. The kinematics and dynamics of the human model are expressed using the Denavit-Hartenberg (DH) representation. The lifting optimization formulation integrates the electromechanical dynamics of the DC motors in the exoskeletons of the knee, spine, and shoulder. The design variables for this study include human joint angle profiles and exoskeleton motor current profiles. The optimization objective is to minimize the squared normalized human joint torques, subject to physical and task-specific lifting constraints. We solve this optimization problem using the gradient-based optimizer SNOPT. Our results include a comparison of predicted human joint angle profiles, joint torque profiles, and ground reaction force (GRF) profiles between lifting tasks with and without exoskeleton assistance. We also explore various combinations of exoskeletons for the knee, spine, and shoulder. By resolving the lifting optimization problems, we designed the optimal torques for the exoskeletons located at the knee, spine, and shoulder. It was found that the support from the exoskeletons substantially lowers the torque levels in human joints. Additionally, we conducted experiments only on the knee exoskeleton. Experimental data indicated that using the knee exoskeleton decreases the muscle activation peaks by 35.00%, 10.03%, 22.12%, 30.14%, 16.77%, and 25.71% for muscles of the erector spinae, latissimus dorsi, vastus medialis, vastus lateralis, rectus femoris, and biceps femoris, respectively.

Use of Artificial Intelligence Techniques to Assist Individuals with Physical Disabilities

Assistive technologies (AT) enable people with disabilities to perform activities of daily living more independently, have greater access to community and healthcare services, and be more productive performing educational and/or employment tasks. Integrating artificial intelligence (AI) with various agents, including electronics, robotics, and software, has revolutionized AT, resulting in groundbreaking technologies such as mind-controlled exoskeletons, bionic limbs, intelligent wheelchairs, and smart home assistants. This article provides a review of various AI techniques that have helped those with physical disabilities, including brain-computer interfaces, computer vision, natural language processing, and human-computer interaction. The current challenges and future directions for AI-powered advanced technologies are also addressed.

Experiment-free exoskeleton assistance via learning in simulation

Exoskeletons have enormous potential to improve human locomotive performance1-3. However, their development and broad dissemination are limited by the requirement for lengthy human tests and handcrafted control laws2. Here we show an experiment-free method to learn a versatile control policy in simulation. Our learning-in-simulation framework leverages dynamics-aware musculoskeletal and exoskeleton models and data-driven reinforcement learning to bridge the gap between simulation and reality without human experiments. The learned controller is deployed on a custom hip exoskeleton that automatically generates assistance across different activities with reduced metabolic rates by 24.3%, 13.1% and 15.4% for walking, running and stair climbing, respectively. Our framework may offer a generalizable and scalable strategy for the rapid development and widespread adoption of a variety of assistive robots for both able-bodied and mobility-impaired individuals.

AI-based methodologies for exoskeleton-assisted rehabilitation of the lower limb: a review

Over the past few years, there has been a noticeable surge in efforts to design novel tools and approaches that incorporate Artificial Intelligence (AI) into rehabilitation of persons with lower-limb impairments, using robotic exoskeletons. The potential benefits include the ability to implement personalized rehabilitation therapies by leveraging AI for robot control and data analysis, facilitating personalized feedback and guidance. Despite this, there is a current lack of literature review specifically focusing on AI applications in lower-limb rehabilitative robotics. To address this gap, our work aims at performing a review of 37 peer-reviewed papers. This review categorizes selected papers based on robotic application scenarios or AI methodologies. Additionally, it uniquely contributes by providing a detailed summary of input features, AI model performance, enrolled populations, exoskeletal systems used in the validation process, and specific tasks for each paper. The innovative aspect lies in offering a clear understanding of the suitability of different algorithms for specific tasks, intending to guide future developments and support informed decision-making in the realm of lower-limb exoskeleton and AI applications.

Gait Analysis in Neurorehabilitation: From Research to Clinical Practice

When brain damage occurs, gait and balance are often impaired. Evaluation of the gait cycle, therefore, has a pivotal role during the rehabilitation path of subjects who suffer from neurological disorders. Gait analysis can be performed through laboratory systems, non-wearable sensors (NWS), and/or wearable sensors (WS). Using these tools, physiotherapists and neurologists have more objective measures of motion function and can plan tailored and specific gait and balance training early to achieve better outcomes and improve patients' quality of life. However, most of these innovative tools are used for research purposes (especially the laboratory systems and NWS), although they deserve more attention in the rehabilitation field, considering their potential in improving clinical practice. In this narrative review, we aimed to summarize the most used gait analysis systems in neurological patients, shedding some light on their clinical value and implications for neurorehabilitation practice.

Robust walking control of a lower limb rehabilitation exoskeleton coupled with a musculoskeletal model via deep reinforcement learning

BACKGROUND

Few studies have systematically investigated robust controllers for lower limb rehabilitation exoskeletons (LLREs) that can safely and effectively assist users with a variety of neuromuscular disorders to walk with full autonomy. One of the key challenges for developing such a robust controller is to handle different degrees of uncertain human-exoskeleton interaction forces from the patients. Consequently, conventional walking controllers either are patient-condition specific or involve tuning of many control parameters, which could behave unreliably and even fail to maintain balance.

METHODS

We present a novel, deep neural network, reinforcement learning-based robust controller for a LLRE based on a decoupled offline human-exoskeleton simulation training with three independent networks, which aims to provide reliable walking assistance against various and uncertain human-exoskeleton interaction forces. The exoskeleton controller is driven by a neural network control policy that acts on a stream of the LLRE's proprioceptive signals, including joint kinematic states, and subsequently predicts real-time position control targets for the actuated joints. To handle uncertain human interaction forces, the control policy is trained intentionally with an integrated human musculoskeletal model and realistic human-exoskeleton interaction forces. Two other neural networks are connected with the control policy network to predict the interaction forces and muscle coordination. To further increase the robustness of the control policy to different human conditions, we employ domain randomization during training that includes not only randomization of exoskeleton dynamics properties but, more importantly, randomization of human muscle strength to simulate the variability of the patient's disability. Through this decoupled deep reinforcement learning framework, the trained controller of LLREs is able to provide reliable walking assistance to patients with different degrees of neuromuscular disorders without any control parameter tuning.

RESULTS AND CONCLUSION

A universal, RL-based walking controller is trained and virtually tested on a LLRE system to verify its effectiveness and robustness in assisting users with different disabilities such as passive muscles (quadriplegic), muscle weakness, or hemiplegic conditions without any control parameter tuning. Analysis of the RMSE for joint tracking, CoP-based stability, and gait symmetry shows the effectiveness of the controller. An ablation study also demonstrates the strong robustness of the control policy under large exoskeleton dynamic property ranges and various human-exoskeleton interaction forces. The decoupled network structure allows us to isolate the LLRE control policy network for testing and sim-to-real transfer since it uses only proprioception information of the LLRE (joint sensory state) as the input. Furthermore, the controller is shown to be able to handle different patient conditions without the need for patient-specific control parameter tuning.

Predicting the metabolic cost of exoskeleton-assisted squatting using foot pressure features and machine learning

Introduction: Recent studies found that wearable exoskeletons can reduce physical effort and fatigue during squatting. In particular, subject-specific assistance helped to significantly reduce physical effort, shown by reduced metabolic cost, using human-in-the-loop optimization of the exoskeleton parameters. However, measuring metabolic cost using respiratory data has limitations, such as long estimation times, presence of noise, and user discomfort. A recent study suggests that foot contact forces can address those challenges and be used as an alternative metric to the metabolic cost to personalize wearable robot assistance during walking. Methods: In this study, we propose that foot center of pressure (CoP) features can be used to estimate the metabolic cost of squatting using a machine learning method. Five subjects' foot pressure and metabolic cost data were collected as they performed squats with an ankle exoskeleton at different assistance conditions in our prior study. In this study, we extracted statistical features from the CoP squat trajectories and fed them as input to a random forest model, with the metabolic cost as the output. Results: The model predicted the metabolic cost with a mean error of 0.55 W/kg on unseen test data, with a high correlation (r = 0.89, p < 0.01) between the true and predicted cost. The features of the CoP trajectory in the medial-lateral direction of the foot (xCoP), which relate to ankle eversion-inversion, were found to be important and highly correlated with metabolic cost. Conclusion: Our findings indicate that increased ankle eversion (outward roll of the ankle), which reflects a suboptimal squatting strategy, results in higher metabolic cost. Higher ankle eversion has been linked with the etiology of chronic lower limb injuries. Hence, a CoP-based cost function in human-in-the-loop optimization could offer several advantages, such as reduced estimation time, injury risk mitigation, and better user comfort.

Myoelectric Control Systems for Upper Limb Wearable Robotic Exoskeletons and Exosuits-A Systematic Review

In recent years, myoelectric control systems have emerged for upper limb wearable robotic exoskeletons to provide movement assistance and/or to restore motor functions in people with motor disabilities and to augment human performance in able-bodied individuals. In myoelectric control, electromyographic (EMG) signals from muscles are utilized to implement control strategies in exoskeletons and exosuits, improving adaptability and human-robot interactions during various motion tasks. This paper reviews the state-of-the-art myoelectric control systems designed for upper-limb wearable robotic exoskeletons and exosuits, and highlights the key focus areas for future research directions. Here, different modalities of existing myoelectric control systems were described in detail, and their advantages and disadvantages were summarized. Furthermore, key design aspects (i.e., supported degrees of freedom, portability, and intended application scenario) and the type of experiments conducted to validate the efficacy of the proposed myoelectric controllers were also discussed. Finally, the challenges and limitations of current myoelectric control systems were analyzed, and future research directions were suggested.

A Survey on Design and Control of Lower Extremity Exoskeletons for Bipedal Walking

Exoskeleton robots are electrically, pneumatically, or hydraulically actuated devices that externally support the bones and cartilage of the human body while trying to mimic the human movement capabilities and augment muscle power. The lower extremity exoskeleton device may support specific human joints such as hip, knee, and ankle, or provide support to carry and balance the weight of the full upper body. Their assistive functionality for physically-abled and disabled humans is demanded in medical, industrial, military, safety applications, and other related fields. The vision of humans walking with an exoskeleton without external support is the prospect of the robotics and artificial intelligence working groups. This paper presents a survey on the design and control of lower extremity exoskeletons for bipedal walking. First, a historical view on the development of walking exoskeletons is presented and various lower body exoskeleton designs are categorized in different application areas. Then, these designs are studied from design, modeling, and control viewpoints. Finally, a discussion on future research directions is provided.