Deep reinforcement learning for modeling human locomotion control in neuromechanical simulation

Comparison of synergy extrapolation and static optimization for estimating multiple unmeasured muscle activations during walking

BACKGROUND

Calibrated electromyography (EMG)-driven musculoskeletal models can provide insight into internal quantities (e.g., muscle forces) that are difficult or impossible to measure experimentally. However, the need for EMG data from all involved muscles presents a significant barrier to the widespread application of EMG-driven modeling methods. Synergy extrapolation (SynX) is a computational method that can estimate a single missing EMG signal with reasonable accuracy during the EMG-driven model calibration process, yet its performance in estimating a larger number of missing EMG signals remains unknown.

METHODS

This study assessed the accuracy with which SynX can use eight measured EMG signals to estimate muscle activations and forces associated with eight missing EMG signals in the same leg during walking while simultaneously performing EMG-driven model calibration. Experimental gait data collected from two individuals post-stroke, including 16 channels of EMG data per leg, were used to calibrate an EMG-driven musculoskeletal model, providing "gold standard" muscle activations and forces for evaluation purposes. SynX was then used to predict the muscle activations and forces associated with the eight missing EMG signals while simultaneously calibrating EMG-driven model parameter values. Due to its widespread use, static optimization (SO) applied to a scaled generic musculoskeletal model was also utilized to estimate the same muscle activations and forces. Estimation accuracy for SynX and SO was evaluated using root mean square errors (RMSE) to quantify amplitude errors and correlation coefficient r values to quantify shape similarity, each calculated with respect to "gold standard" muscle activations and forces.

RESULTS

On average, compared to SO, SynX with simultaneous model calibration produced significantly more accurate amplitude and shape estimates for unmeasured muscle activations (RMSE 0.08 vs. 0.15, r value 0.55 vs. 0.12) and forces (RMSE 101.3 N vs. 174.4 N, r value 0.53 vs. 0.07). SynX yielded calibrated Hill-type muscle-tendon model parameter values for all muscles and activation dynamics model parameter values for measured muscles that were similar to "gold standard" calibrated model parameter values.

CONCLUSIONS

These findings suggest that SynX could make it possible to calibrate EMG-driven musculoskeletal models for all important lower-extremity muscles with as few as eight carefully chosen EMG signals and eventually contribute to the design of personalized rehabilitation and surgical interventions for mobility impairments.

Ethnokinesiology: towards a neuromechanical understanding of cultural differences in movement

Each individual's movements are sculpted by constant interactions between sensorimotor and sociocultural factors. A theoretical framework grounded in motor control mechanisms articulating how sociocultural and biological signals converge to shape movement is currently missing. Here, we propose a framework for the emerging field of ethnokinesiology aiming to provide a conceptual space and vocabulary to help bring together researchers at this intersection. We offer a first-level schema for generating and testing hypotheses about cultural differences in movement to bridge gaps between the rich observations of cross-cultural movement variations and neurophysiological and biomechanical accounts of movement. We explicitly dissociate two interacting feedback loops that determine culturally relevant movement: one governing sensorimotor tasks regulated by neural signals internal to the body, the other governing ecological tasks generated through actions in the environment producing ecological consequences. A key idea is the emergence of individual-specific and culturally influenced motor concepts in the nervous system, low-dimensional functional mappings between sensorimotor and ecological task spaces. Motor accents arise from perceived differences in motor concept topologies across cultural contexts. We apply the framework to three examples: speech, gait and grasp. Finally, we discuss how ethnokinesiological studies may inform personalized motor skill training and rehabilitation, and challenges moving forward.This article is part of the theme issue 'Minds in movement: embodied cognition in the age of artificial intelligence'.

Acquiring musculoskeletal skills with curriculum-based reinforcement learning

Efficient musculoskeletal simulators and powerful learning algorithms provide computational tools to tackle the grand challenge of understanding biological motor control. Our winning solution for the inaugural NeurIPS MyoChallenge leverages an approach mirroring human skill learning. Using a novel curriculum learning approach, we trained a recurrent neural network to control a realistic model of the human hand with 39 muscles to rotate two Baoding balls in the palm of the hand. In agreement with data from human subjects, the policy uncovers a small number of kinematic synergies, even though it is not explicitly biased toward low-dimensional solutions. However, selectively inactivating parts of the control signal, we found that more dimensions contribute to the task performance than suggested by traditional synergy analysis. Overall, our work illustrates the emerging possibilities at the interface of musculoskeletal physics engines, reinforcement learning, and neuroscience to advance our understanding of biological motor control.

NeuroMotion: Open-source platform with neuromechanical and deep network modules to generate surface EMG signals during voluntary movement

Neuromechanical studies investigate how the nervous system interacts with the musculoskeletal (MSK) system to generate volitional movements. Such studies have been supported by simulation models that provide insights into variables that cannot be measured experimentally and allow a large number of conditions to be tested before the experimental analysis. However, current simulation models of electromyography (EMG), a core physiological signal in neuromechanical analyses, remain either limited in accuracy and conditions or are computationally heavy to apply. Here, we provide a computational platform to enable future work to overcome these limitations by presenting NeuroMotion, an open-source simulator that can modularly test a variety of approaches to the full-spectrum synthesis of EMG signals during voluntary movements. We demonstrate NeuroMotion using three sample modules. The first module is an upper-limb MSK model with OpenSim API to estimate the muscle fibre lengths and muscle activations during movements. The second module is BioMime, a deep neural network-based EMG generator that receives nonstationary physiological parameter inputs, like the afore-estimated muscle fibre lengths, and efficiently outputs motor unit action potentials (MUAPs). The third module is a motor unit pool model that transforms the muscle activations into discharge timings of motor units. The discharge timings are convolved with the output of BioMime to simulate EMG signals during the movement. We first show how MUAP waveforms change during different levels of physiological parameter variations and different movements. We then show that the synthetic EMG signals during two-degree-of-freedom hand and wrist movements can be used to augment experimental data for regressing joint angles. Ridge regressors trained on the synthetic dataset were directly used to predict joint angles from experimental data. In this way, NeuroMotion was able to generate full-spectrum EMG for the first use-case of human forearm electrophysiology during voluntary hand, wrist, and forearm movements. All intermediate variables are available, which allows the user to study cause-effect relationships in the complex neuromechanical system, fast iterate algorithms before collecting experimental data, and validate algorithms that estimate non-measurable parameters in experiments. We expect this modular platform will enable validation of generative EMG models, complement experimental approaches and empower neuromechanical research.

Reinforcement Learning May Demystify the Limited Human Motor Learning Efficacy Due to Visual-Proprioceptive Mismatch

Vision and proprioception have fundamental sensory mismatches in delivering locational information, and such mismatches are critical factors limiting the efficacy of motor learning. However, it is still not clear how and to what extent this mismatch limits motor learning outcomes. To further the understanding of the effect of sensory mismatch on motor learning outcomes, a reinforcement learning algorithm and the simplified biomechanical elbow joint model were employed to mimic the motor learning process in a computational environment. By applying a reinforcement learning algorithm to the motor learning of elbow joint flexion task, simulation results successfully explained how visual-proprioceptive mismatch limits motor learning outcomes in terms of motor control accuracy and task completion speed. The larger the perceived angular offset between the two sensory modalities, the lower the motor control accuracy. Also, the more similar the peak reward amplitude of the two sensory modalities, the lower the motor control accuracy. In addition, simulation results suggest that insufficient exploration rate limits task completion speed, and excessive exploration rate limits motor control accuracy. Such a speed-accuracy trade-off shows that a moderate exploration rate could serve as another important factor in motor learning.

Experiment-free exoskeleton assistance via learning in simulation

Exoskeletons have enormous potential to improve human locomotive performance1-3. However, their development and broad dissemination are limited by the requirement for lengthy human tests and handcrafted control laws2. Here we show an experiment-free method to learn a versatile control policy in simulation. Our learning-in-simulation framework leverages dynamics-aware musculoskeletal and exoskeleton models and data-driven reinforcement learning to bridge the gap between simulation and reality without human experiments. The learned controller is deployed on a custom hip exoskeleton that automatically generates assistance across different activities with reduced metabolic rates by 24.3%, 13.1% and 15.4% for walking, running and stair climbing, respectively. Our framework may offer a generalizable and scalable strategy for the rapid development and widespread adoption of a variety of assistive robots for both able-bodied and mobility-impaired individuals.

Effect of hyperthermia on simulated muscle activation in female when crossing obstacle

It is well known that hyperthermia greatly impairs neuromuscular function and dynamic balance. However, whether a greater level of hyperthermia could potentially alter the lower limb simulated muscle activation when crossing an obstacle in female participants remains unknown. Therefore we examined the effect of a systematic increase in oral temperature on lower limb simulated muscle activation when crossing an obstacle in female participants. Eighteen female participants were recruited where they underwent a control trial (Con) and two progressive passive heating trials with Δ 1°C and Δ 2°C increase of oral temperature (Toral) using a 45°C water bath. In each trial, we assessed lower limb simulated muscle activation when crossing an obstacle height of 10%, 20%, and 30% of the participant's leg length and toe-off, toe-above-obstacle and heel-strike events were identified and analyzed. In all events, the lower limb simulated muscle activation were greater in Δ2°C than Δ1°C and Con when both leading and trailing limbs crossed the obstacle height of 20% and 30% leg length (all p < 0.001). However, the lower limb simulated muscle activation were not different between Δ1°C and Con across all obstacle heights (p > 0.05). This study concluded that a greater level of hyperthermia resulted in a greater lower limb simulated muscle activation to ensure safety and stability when females cross an obstacle height of 20% leg length or higher.

NSF DARE-Transforming modeling in neurorehabilitation: Four threads for catalyzing progress

We present an overview of the Conference on Transformative Opportunities for Modeling in Neurorehabilitation held in March 2023. It was supported by the Disability and Rehabilitation Engineering (DARE) program from the National Science Foundation's Engineering Biology and Health Cluster. The conference brought together experts and trainees from around the world to discuss critical questions, challenges, and opportunities at the intersection of computational modeling and neurorehabilitation to understand, optimize, and improve clinical translation of neurorehabilitation. We organized the conference around four key, relevant, and promising Focus Areas for modeling: Adaptation & Plasticity, Personalization, Human-Device Interactions, and Modeling 'In-the-Wild'. We identified four common threads across the Focus Areas that, if addressed, can catalyze progress in the short, medium, and long terms. These were: (i) the need to capture and curate appropriate and useful data necessary to develop, validate, and deploy useful computational models (ii) the need to create multi-scale models that span the personalization spectrum from individuals to populations, and from cellular to behavioral levels (iii) the need for algorithms that extract as much information from available data, while requiring as little data as possible from each client (iv) the insistence on leveraging readily available sensors and data systems to push model-driven treatments from the lab, and into the clinic, home, workplace, and community. The conference archive can be found at (dare2023.usc.edu). These topics are also extended by three perspective papers prepared by trainees and junior faculty, clinician researchers, and federal funding agency representatives who attended the conference.

Design of Low-Cost Modular Bio-Inspired Electric-Pneumatic Actuator (EPA)-Driven Legged Robots

Exploring the fundamental mechanisms of locomotion extends beyond mere simulation and modeling. It necessitates the utilization of physical test benches to validate hypotheses regarding real-world applications of locomotion. This study introduces cost-effective modular robotic platforms designed specifically for investigating the intricacies of locomotion and control strategies. Expanding upon our prior research in electric-pneumatic actuation (EPA), we present the mechanical and electrical designs of the latest developments in the EPA robot series. These include EPA Jumper, a human-sized segmented monoped robot, and its extension EPA Walker, a human-sized bipedal robot. Both replicate the human weight and inertia distributions, featuring co-actuation through electrical motors and pneumatic artificial muscles. These low-cost modular platforms, with considerations for degrees of freedom and redundant actuation, (1) provide opportunities to study different locomotor subfunctions-stance, swing, and balance; (2) help investigate the role of actuation schemes in tasks such as hopping and walking; and (3) allow testing hypotheses regarding biological locomotors in real-world physical test benches.

NSF DARE-transforming modeling in neurorehabilitation: a patient-in-the-loop framework

In 2023, the National Science Foundation (NSF) and the National Institute of Health (NIH) brought together engineers, scientists, and clinicians by sponsoring a conference on computational modelling in neurorehabiilitation. To facilitate multidisciplinary collaborations and improve patient care, in this perspective piece we identify where and how computational modelling can support neurorehabilitation. To address the where, we developed a patient-in-the-loop framework that uses multiple and/or continual measurements to update diagnostic and treatment model parameters, treatment type, and treatment prescription, with the goal of maximizing clinically-relevant functional outcomes. This patient-in-the-loop framework has several key features: (i) it includes diagnostic and treatment models, (ii) it is clinically-grounded with the International Classification of Functioning, Disability and Health (ICF) and patient involvement, (iii) it uses multiple or continual data measurements over time, and (iv) it is applicable to a range of neurological and neurodevelopmental conditions. To address the how, we identify state-of-the-art and highlight promising avenues of future research across the realms of sensorimotor adaptation, neuroplasticity, musculoskeletal, and sensory & pain computational modelling. We also discuss both the importance of and how to perform model validation, as well as challenges to overcome when implementing computational models within a clinical setting. The patient-in-the-loop approach offers a unifying framework to guide multidisciplinary collaboration between computational and clinical stakeholders in the field of neurorehabilitation.

Higher coactivations of lower limb muscles increase stability during walking on slippery ground in forward dynamics musculoskeletal simulation

The energy efficiency theory of human bipedal locomotion has been widely accepted as a neuro-musculoskeletal control method. However, coactivation of agonist and antagonist muscles in the lower limb has been observed during various limb movements, including walking. The emergence of this coactivation cannot be explained solely by the energy efficiency theory and remains a subject of debate. To shed light on this, we investigated the role of muscle coactivations in walking stability using a forward dynamics musculoskeletal simulation combined with neural-network-based gait controllers. Our study revealed that a gait controller with minimal muscle activations had a high probability of falls under challenging gait conditions such as slippery ground and uneven terrain. Lower limb muscle coactivations emerged in the process of gait controller training on slippery ground. Controllers with physiological coactivation levels demonstrated a significantly reduced probability of falls. Our results suggest that achieving stable walking requires muscle coactivations beyond the minimal level of muscle energy. This study implies that coactivations likely emerge to maintain gait stability under challenging conditions, and both coactivation and energy optimization of lower limb muscles should be considered when exploring the foundational control mechanisms of human walking.

Simulating human walking: a model-based reinforcement learning approach with musculoskeletal modeling

Introduction

Recent advancements in reinforcement learning algorithms have accelerated the development of control models with high-dimensional inputs and outputs that can reproduce human movement. However, the produced motion tends to be less human-like if algorithms do not involve a biomechanical human model that accounts for skeletal and muscle-tendon properties and geometry. In this study, we have integrated a reinforcement learning algorithm and a musculoskeletal model including trunk, pelvis, and leg segments to develop control modes that drive the model to walk.

Methods

We simulated human walking first without imposing target walking speed, in which the model was allowed to settle on a stable walking speed itself, which was 1.45 m/s. A range of other speeds were imposed for the simulation based on the previous self-developed walking speed. All simulations were generated by solving the Markov decision process problem with covariance matrix adaptation evolution strategy, without any reference motion data.

Results

Simulated hip and knee kinematics agreed well with those in experimental observations, but ankle kinematics were less well-predicted.

Discussion

We finally demonstrated that our reinforcement learning framework also has the potential to model and predict pathological gait that can result from muscle weakness.

Modeling Human Suboptimal Control: A Review

This review paper provides an overview of the approaches to model neuromuscular control, focusing on methods to identify nonoptimal control strategies typical of populations with neuromuscular disorders or children. Where possible, the authors tightened the description of the methods to the mechanisms behind the underlying biomechanical and physiological rationale. They start by describing the first and most simplified approach, the reductionist approach, which splits the role of the nervous and musculoskeletal systems. Static optimization and dynamic optimization methods and electromyography-based approaches are summarized to highlight their limitations and understand (the need for) their developments over time. Then, the authors look at the more recent stochastic approach, introduced to explore the space of plausible neural solutions, thus implementing the uncontrolled manifold theory, according to which the central nervous system only controls specific motions and tasks to limit energy consumption while allowing for some degree of adaptability to perturbations. Finally, they explore the literature covering the explicit modeling of the coupling between the nervous system (acting as controller) and the musculoskeletal system (the actuator), which may be employed to overcome the split characterizing the reductionist approach.

Ten steps to becoming a musculoskeletal simulation expert: A half-century of progress and outlook for the future

Over the past half-century, musculoskeletal simulations have deepened our knowledge of human and animal movement. This article outlines ten steps to becoming a musculoskeletal simulation expert so you can contribute to the next half-century of technical innovation and scientific discovery. We advocate looking to the past, present, and future to harness the power of simulations that seek to understand and improve mobility. Instead of presenting a comprehensive literature review, we articulate a set of ideas intended to help researchers use simulations effectively and responsibly by understanding the work on which today's musculoskeletal simulations are built, following established modeling and simulation principles, and branching out in new directions.

Modern three-dimensional digital methods for studying locomotor biomechanics in tetrapods

Here, we review the modern interface of three-dimensional (3D) empirical (e.g. motion capture) and theoretical (e.g. modelling and simulation) approaches to the study of terrestrial locomotion using appendages in tetrapod vertebrates. These tools span a spectrum from more empirical approaches such as XROMM, to potentially more intermediate approaches such as finite element analysis, to more theoretical approaches such as dynamic musculoskeletal simulations or conceptual models. These methods have much in common beyond the importance of 3D digital technologies, and are powerfully synergistic when integrated, opening a wide range of hypotheses that can be tested. We discuss the pitfalls and challenges of these 3D methods, leading to consideration of the problems and potential in their current and future usage. The tools (hardware and software) and approaches (e.g. methods for using hardware and software) in the 3D analysis of tetrapod locomotion have matured to the point where now we can use this integration to answer questions we could never have tackled 20 years ago, and apply insights gleaned from them to other fields.

Inertia-Constrained Reinforcement Learning to Enhance Human Motor Control Modeling

Locomotor impairment is a highly prevalent and significant source of disability and significantly impacts the quality of life of a large portion of the population. Despite decades of research on human locomotion, challenges remain in simulating human movement to study the features of musculoskeletal drivers and clinical conditions. Most recent efforts to utilize reinforcement learning (RL) techniques are promising in the simulation of human locomotion and reveal musculoskeletal drives. However, these simulations often fail to mimic natural human locomotion because most reinforcement strategies have yet to consider any reference data regarding human movement. To address these challenges, in this study, we designed a reward function based on the trajectory optimization rewards (TOR) and bio-inspired rewards, which includes the rewards obtained from reference motion data captured by a single Inertial Moment Unit (IMU) sensor. The sensor was equipped on the participants' pelvis to capture reference motion data. We also adapted the reward function by leveraging previous research on walking simulations for TOR. The experimental results showed that the simulated agents with the modified reward function performed better in mimicking the collected IMU data from participants, which means that the simulated human locomotion was more realistic. As a bio-inspired defined cost, IMU data enhanced the agent's capacity to converge during the training process. As a result, the models' convergence was faster than those developed without reference motion data. Consequently, human locomotion can be simulated more quickly and in a broader range of environments, with a better simulation performance.

Low limb prostheses and complex human prosthetic interaction: A systematic literature review

A few years ago, powered prostheses triggered new technological advances in diverse areas such as mobility, comfort, and design, which have been essential to improving the quality of life of individuals with lower limb disability. The human body is a complex system involving mental and physical health, meaning a dependant relationship between its organs and lifestyle. The elements used in the design of these prostheses are critical and related to lower limb amputation level, user morphology and human-prosthetic interaction. Hence, several technologies have been employed to accomplish the end user's needs, for example, advanced materials, control systems, electronics, energy management, signal processing, and artificial intelligence. This paper presents a systematic literature review on such technologies, to identify the latest advances, challenges, and opportunities in developing lower limb prostheses with the analysis on the most significant papers. Powered prostheses for walking in different terrains were illustrated and examined, with the kind of movement the device should perform by considering the electronics, automatic control, and energy efficiency. Results show a lack of a specific and generalised structure to be followed by new developments, gaps in energy management and improved smoother patient interaction. Additionally, Human Prosthetic Interaction (HPI) is a term introduced in this paper since no other research has integrated this interaction in communication between the artificial limb and the end-user. The main goal of this paper is to provide, with the found evidence, a set of steps and components to be followed by new researchers and experts looking to improve knowledge in this field.

Experiment protocols for brain-body imaging of locomotion: A systematic review

Introduction

Human locomotion is affected by several factors, such as growth and aging, health conditions, and physical activity levels for maintaining overall health and well-being. Notably, impaired locomotion is a prevalent cause of disability, significantly impacting the quality of life of individuals. The uniqueness and high prevalence of human locomotion have led to a surge of research to develop experimental protocols for studying the brain substrates, muscle responses, and motion signatures associated with locomotion. However, from a technical perspective, reproducing locomotion experiments has been challenging due to the lack of standardized protocols and benchmarking tools, which impairs the evaluation of research quality and the validation of previous findings.

Methods

This paper addresses the challenges by conducting a systematic review of existing neuroimaging studies on human locomotion, focusing on the settings of experimental protocols, such as locomotion intensity, duration, distance, adopted brain imaging technologies, and corresponding brain activation patterns. Also, this study provides practical recommendations for future experiment protocols.

Results

The findings indicate that EEG is the preferred neuroimaging sensor for detecting brain activity patterns, compared to fMRI, fNIRS, and PET. Walking is the most studied human locomotion task, likely due to its fundamental nature and status as a reference task. In contrast, running has received little attention in research. Additionally, cycling on an ergometer at a speed of 60 rpm using fNIRS has provided some research basis. Dual-task walking tasks are typically used to observe changes in cognitive function. Moreover, research on locomotion has primarily focused on healthy individuals, as this is the scenario most closely resembling free-living activity in real-world environments.

Discussion

Finally, the paper outlines the standards and recommendations for setting up future experiment protocols based on the review findings. It discusses the impact of neurological and musculoskeletal factors, as well as the cognitive and locomotive demands, on the experiment design. It also considers the limitations imposed by the sensing techniques used, including the acceptable level of motion artifacts in brain-body imaging experiments and the effects of spatial and temporal resolutions on brain sensor performance. Additionally, various experiment protocol constraints that need to be addressed and analyzed are explained.

Emergent behaviour and neural dynamics in artificial agents tracking odour plumes

Tracking an odour plume to locate its source under variable wind and plume statistics is a complex task. Flying insects routinely accomplish such tracking, often over long distances, in pursuit of food or mates. Several aspects of this remarkable behaviour and its underlying neural circuitry have been studied experimentally. Here we take a complementary in silico approach to develop an integrated understanding of their behaviour and neural computations. Specifically, we train artificial recurrent neural network agents using deep reinforcement learning to locate the source of simulated odour plumes that mimic features of plumes in a turbulent flow. Interestingly, the agents' emergent behaviours resemble those of flying insects, and the recurrent neural networks learn to compute task-relevant variables with distinct dynamic structures in population activity. Our analyses put forward a testable behavioural hypothesis for tracking plumes in changing wind direction, and we provide key intuitions for memory requirements and neural dynamics in odour plume tracking.

Generating Human Arm Kinematics Using Reinforcement Learning to Train Active Muscle Behavior in Automotive Research

Computational human body models (HBMs) are important tools for predicting human biomechanical responses under automotive crash environments. In many scenarios, the prediction of the occupant response will be improved by incorporating active muscle control into the HBMs to generate biofidelic kinematics during different vehicle maneuvers. In this study, we have proposed an approach to develop an active muscle controller based on reinforcement learning (RL). The RL muscle activation control (RL-MAC) approach is a shift from using traditional closed-loop feedback controllers, which can mimic accurate active muscle behavior under a limited range of loading conditions for which the controller has been tuned. Conversely, the RL-MAC uses an iterative training approach to generate active muscle forces for desired joint motion and is analogous to how a child develops gross motor skills. In this study, the ability of a deep deterministic policy gradient (DDPG) RL controller to generate accurate human kinematics is demonstrated using a multibody model of the human arm. The arm model was trained to perform goal-directed elbow rotation by activating the responsible muscles and investigated using two recruitment schemes: as independent muscles or as antagonistic muscle groups. Simulations with the trained controller show that the arm can move to the target position in the presence or absence of externally applied loads. The RL-MAC trained under constant external loads was able to maintain the desired elbow joint angle under a simplified automotive impact scenario, implying the robustness of the motor control approach.

Learning to Ascend Stairs and Ramps: Deep Reinforcement Learning for a Physics-Based Human Musculoskeletal Model

This paper proposes to use deep reinforcement learning to teach a physics-based human musculoskeletal model to ascend stairs and ramps. The deep reinforcement learning architecture employs the proximal policy optimization algorithm combined with imitation learning and is trained with experimental data of a public dataset. The human model is developed in the open-source simulation software OpenSim, together with two objects (i.e., the stairs and ramp) and the elastic foundation contact dynamics. The model can learn to ascend stairs and ramps with muscle forces comparable to healthy subjects and with a forward dynamics comparable to the experimental training data, achieving an average correlation of 0.82 during stair ascent and of 0.58 during ramp ascent across both the knee and ankle joints.

From motor control to team play in simulated humanoid football

Learning to combine control at the level of joint torques with longer-term goal-directed behavior is a long-standing challenge for physically embodied artificial agents. Intelligent behavior in the physical world unfolds across multiple spatial and temporal scales: Although movements are ultimately executed at the level of instantaneous muscle tensions or joint torques, they must be selected to serve goals that are defined on much longer time scales and that often involve complex interactions with the environment and other agents. Recent research has demonstrated the potential of learning-based approaches applied to the respective problems of complex movement, long-term planning, and multiagent coordination. However, their integration traditionally required the design and optimization of independent subsystems and remains challenging. In this work, we tackled the integration of motor control and long-horizon decision-making in the context of simulated humanoid football, which requires agile motor control and multiagent coordination. We optimized teams of agents to play simulated football via reinforcement learning, constraining the solution space to that of plausible movements learned using human motion capture data. They were trained to maximize several environment rewards and to imitate pretrained football-specific skills if doing so led to improved performance. The result is a team of coordinated humanoid football players that exhibit complex behavior at different scales, quantified by a range of analysis and statistics, including those used in real-world sport analytics. Our work constitutes a complete demonstration of learned integrated decision-making at multiple scales in a multiagent setting.

Policy Design for an Ankle-Foot Orthosis Using Simulated Physical Human-Robot Interaction via Deep Reinforcement Learning

This paper presents a novel approach for designing a robotic orthosis controller considering physical human-robot interaction (pHRI). Computer simulation for this human-robot system can be advantageous in terms of time and cost due to the laborious nature of designing a robot controller that effectively assists humans with the appropriate magnitude and phase. Therefore, we propose a two-stage policy training framework based on deep reinforcement learning (deep RL) to design a robot controller using human-robot dynamic simulation. In Stage 1, the optimal policy of generating human gaits is obtained from deep RL-based imitation learning on a healthy subject model using the musculoskeletal simulation in OpenSim-RL. In Stage 2, human models in which the right soleus muscle is weakened to a certain severity are created by modifying the human model obtained from Stage 1. A robotic orthosis is then attached to the right ankle of these models. The orthosis policy that assists walking with optimal torque is then trained on these models. Here, the elastic foundation model is used to predict the pHRI in the coupling part between the human and robotic orthosis. Comparative analysis of kinematic and kinetic simulation results with the experimental data shows that the derived human musculoskeletal model imitates a human walking. It also shows that the robotic orthosis policy obtained from two-stage policy training can assist the weakened soleus muscle. The proposed approach was validated by applying the learned policy to ankle orthosis, conducting a gait experiment, and comparing it with the simulation results.

Evaluating anticipatory control strategies for their capability to cope with step-down perturbations in computer simulations of human walking

Previous simulation studies investigated the role of reflexes and central pattern generators to explain the kinematic and dynamic adaptations in reaction to step-down perturbations. However, experiments also show preparatory adaptations in humans based on visual anticipation of a perturbation. In this study, we propose a high-level anticipatory strategy augmenting a low-level muscle-reflex control. This strategy directly changes the gain of the reflex control exclusively during the last contact prior to a drop in ground level. Our simulations show that especially the anticipatory reduction of soleus activity and the increase of hamstrings activity result in higher robustness. The best results were obtained when the change in stimulation of the soleus muscle occurred 300 ms after the heel strike of the contralateral leg. This enabled the model to descend perturbation heights up to - 0.21 m and the resulting kinematic and dynamic adaptations are similar to the experimental observations. This proves that the anticipatory strategy observed in experiments has the purpose of increasing robustness. Furthermore, this strategy outperforms other reactive strategies, e.g., pure feedback control or combined feedback and feed-forward control, with maximum perturbation heights of - 0.03 and - 0.07 m, respectively.

Reinforcement learning coupled with finite element modeling for facial motion learning

BACKGROUND AND OBJECTIVE

Facial palsy patients or patients with facial transplantation have abnormal facial motion due to altered facial muscle functions and nerve damage. Computer-aided system and physics-based models have been developed to provide objective and quantitative information. However, the predictive capacity of these solutions is still limited to explore the facial motion patterns with emerging properties. The present study aims to couple the reinforcement learning and the finite element modeling for facial motion learning and prediction.

METHODS

A novel modeling workflow for learning facial motion was developed. A physically-based model of the face within the Artisynth modeling platform was used. Information exchange protocol was proposed to link reinforcement learning and rigid multi-bodies dynamics outcomes. Two reinforcement learning algorithms (deep deterministic policy gradient (DDPG) and Twin-delayed DDPG (TD3)) were used and implemented to drive the simulations of symmetry-oriented and smile movements. Numerical outcomes were compared to experimental observations (Bosphorus database) for evaluation and validation purposes.

RESULTS

As result, after more than 100 episodes of exploring the environment, the agent starts to learn from previous trials and can find the optimal policy after more than 300 episodes of training. Regarding the symmetry-oriented motion, the muscle excitations predicted by the trained agent help to increase the value of reward from R = -2.06 to R = -0.23, which counts for ∼89% improvement of the symmetry value of the face. For smile-oriented motion, two points at the edge of the mouth move up 0.35 cm, which is within the range of movements estimated from the Bosphorus database (0.4 ± 0.32 cm).

CONCLUSIONS

The present study explored the muscle excitation patterns by coupling reinforcement learning with a detailed finite element model of the face. We developed, for the first time, a novel coupling scheme to integrate the finite element simulation into the reinforcement learning process for facial motion learning. As perspectives, this present workflow will be applied for facial palsy and facial transplantation patients to guide and optimize the functional rehabilitation program.

Leaving flatland: Advances in 3D behavioral measurement

Animals move in three dimensions (3D). Thus, 3D measurement is necessary to report the true kinematics of animal movement. Existing 3D measurement techniques draw on specialized hardware, such as motion capture or depth cameras, as well as deep multi-view and monocular computer vision. Continued advances at the intersection of deep learning and computer vision will facilitate 3D tracking across more anatomical features, with less training data, in additional species, and within more natural, occlusive environments. 3D behavioral measurement enables unique applications in phenotyping, investigating the neural basis of behavior, and designing artificial agents capable of imitating animal behavior.