Reinforcement learning control of a biomechanical model of the upper extremity

Synergy quality assessment of muscle modules for determining learning performance using a realistic musculoskeletal model

How our central nervous system efficiently controls our complex musculoskeletal system is still debated. The muscle synergy hypothesis is proposed to simplify this complex system by assuming the existence of functional neural modules that coordinate several muscles. Modularity based on muscle synergies can facilitate motor learning without compromising task performance. However, the effectiveness of modularity in motor control remains debated. This ambiguity can, in part, stem from overlooking that the performance of modularity depends on the mechanical aspects of modules of interest, such as the torque the modules exert. To address this issue, this study introduces two criteria to evaluate the quality of module sets based on commonly used performance metrics in motor learning studies: the accuracy of torque production and learning speed. One evaluates the regularity in the direction of mechanical torque the modules exert, while the other evaluates the evenness of its magnitude. For verification of our criteria, we simulated motor learning of torque production tasks in a realistic musculoskeletal system of the upper arm using feed-forward neural networks while changing the control conditions. We found that the proposed criteria successfully explain the tendency of learning performance in various control conditions. These result suggest that regularity in the direction of and evenness in magnitude of mechanical torque of utilized modules are significant factor for determining learning performance. Although the criteria were originally conceived for an error-based learning scheme, the approach to pursue which set of modules is better for motor control can have significant implications in other studies of modularity in general.

Motor Interaction Control Based on Muscle Force Model and Depth Reinforcement Strategy

The current motion interaction model has the problems of insufficient motion fidelity and lack of self-adaptation to complex environments. To address this problem, this study proposed to construct a human motion control model based on the muscle force model and stage particle swarm, and based on this, this study utilized the deep deterministic gradient strategy algorithm to construct a motion interaction control model based on the muscle force model and the deep reinforcement strategy. Empirical analysis of the human motion control model proposed in this study revealed that the joint trajectory correlation and muscle activity correlation of the model were higher than those of other comparative models, and its joint trajectory correlation was up to 0.90, and its muscle activity correlation was up to 0.84. In addition, this study validated the effectiveness of the motion interaction control model using the depth reinforcement strategy and found that in the mixed-obstacle environment, the model's desired results were obtained by training 1.1 × 103 times, and the walking distance was 423 m, which was better than other models. In summary, the proposed motor interaction control model using the muscle force model and deep reinforcement strategy has higher motion fidelity and can realize autonomous decision making and adaptive control in the face of complex environments. It can provide a theoretical reference for improving the effect of motion control and realizing intelligent motion interaction.

Tracing curves in the plane: Geometric-invariant learning from human demonstrations

The empirical laws governing human-curvilinear movements have been studied using various relationships, including minimum jerk, the 2/3 power law, and the piecewise power law. These laws quantify the speed-curvature relationships of human movements during curve tracing using critical speed and curvature as regressors. In this work, we provide a reservoir computing-based framework that can learn and reproduce human-like movements. Specifically, the geometric invariance of the observations, i.e., lateral distance from the closest point on the curve, instantaneous velocity, and curvature, when viewed from the moving frame of reference, are exploited to train the reservoir system. The artificially produced movements are evaluated using the power law to assess whether they are indistinguishable from their human counterparts. The generalisation capabilities of the trained reservoir to curves that have not been used during training are also shown.

Computer simulation on the cueing movements in cue sports: a validation study

Background

Simulation models have been applied to analyze daily living activities and some sports movements. However, it is unknown whether the current upper extremity musculoskeletal models can be utilized for investigating cue sports movements to generate corresponding kinematic and muscle activation profiles. This study aimed to test the feasibility of applying simulation models to investigate cue sports players' cueing movements with OpenSim. Preliminary muscle forces would be calculated once the model is validated.

Methods

A previously customized and validated unimanual upper extremity musculoskeletal model with six degrees of freedom at the scapula, shoulder, elbow, and wrist, as well as muscles was used in this study. Two types of cueing movements were simulated: (1) the back spin shot, and (2) 9-ball break shot. Firstly, kinematic data of the upper extremity joints were collected with a 3D motion capture system. Using the experimental marker trajectories of the back spin shot on 10 male cue sports players, the simulation on the cueing movements was executed. The model was then validated by comparing the model-generated joint angles against the experimental results using statistical parametric mapping (SPM1D) to examine the entire angle-time waveform as well as t-tests to compare the discrete variables (e.g., joint range of motion). Secondly, simulation of the break shot was run with the experimental marker trajectories and electromyographic (EMG) data of two male cue sports players as the model inputs. A model-estimated muscle activation calculation was performed accordingly for the upper extremity muscles.

Results

The OpenSim-generated joint angles for the back spin shot corresponded well with the experimental results for the elbow, while the model outputs of the shoulder deviated from the experimental data. The discrepancy in shoulder joint angles could be due to the insufficient kinematic inputs for the shoulder joint. In the break shot simulation, the preliminary findings suggested that great shoulder muscle forces could primarily contribute to the forward swing in a break shot. This suggests that strengthening the shoulder muscles may be a viable strategy to improve the break shot performance.

Conclusion

It is feasible to cater simulation modeling in OpenSim for biomechanical investigations of the upper extremity movements in cue sports. Model outputs can help better understand the contributions of individual muscle forces when performing cueing movements.

Towards AI-Controlled Movement Restoration: Learning FES-Cycling Stimulation with Reinforcement Learning

Functional electrical stimulation (FES) has been increasingly integrated with other rehabilitation devices, including rehabilitation robots. FES cycling is one of the common FES applications in rehabilitation, which is performed by stimulating leg muscles in a certain pattern. The appropriate pattern varies across individuals and requires manual tuning which can be time-consuming and challenging for the individual user. Here, we present an AI-based method for finding the patterns, which requires no extra hardware or sensors. Our method starts with finding model-based patterns using reinforcement learning (RL) and customised cycling models. Next, our method fine-tunes the pattern using real cycling data and offline RL. We test our method both in simulation and experimentally on a stationary tricycle. Our method can robustly deliver model-based patterns for different cycling configurations. In the experimental evaluation, the model-based pattern can induce higher cycling speed than an EMG-based pattern. And by using just 100 seconds of cycling data, our method can deliver a fine-tuned pattern with better cycling performance. Beyond FES cycling, this work is a case study, displaying the feasibility and potential of human-in-the-loop AI in real-world rehabilitation.

Generating Human Arm Kinematics Using Reinforcement Learning to Train Active Muscle Behavior in Automotive Research

Computational human body models (HBMs) are important tools for predicting human biomechanical responses under automotive crash environments. In many scenarios, the prediction of the occupant response will be improved by incorporating active muscle control into the HBMs to generate biofidelic kinematics during different vehicle maneuvers. In this study, we have proposed an approach to develop an active muscle controller based on reinforcement learning (RL). The RL muscle activation control (RL-MAC) approach is a shift from using traditional closed-loop feedback controllers, which can mimic accurate active muscle behavior under a limited range of loading conditions for which the controller has been tuned. Conversely, the RL-MAC uses an iterative training approach to generate active muscle forces for desired joint motion and is analogous to how a child develops gross motor skills. In this study, the ability of a deep deterministic policy gradient (DDPG) RL controller to generate accurate human kinematics is demonstrated using a multibody model of the human arm. The arm model was trained to perform goal-directed elbow rotation by activating the responsible muscles and investigated using two recruitment schemes: as independent muscles or as antagonistic muscle groups. Simulations with the trained controller show that the arm can move to the target position in the presence or absence of externally applied loads. The RL-MAC trained under constant external loads was able to maintain the desired elbow joint angle under a simplified automotive impact scenario, implying the robustness of the motor control approach.

An Intelligent TCP Congestion Control Method Based on Deep Q Network

To optimize the data migration performance between different supercomputing centers in China, we present TCP-DQN, which is an intelligent TCP congestion control method based on DQN (Deep Q network). The TCP congestion control process is abstracted as a partially observed Markov decision process. In this process, an agent is constructed to interact with the network environment. The agent adjusts the size of the congestion window by observing the characteristics of the network state. The network environment feeds back the reward to the agent, and the agent tries to maximize the expected reward in an episode. We designed a weighted reward function to balance the throughput and delay. Compared with traditional Q-learning, DQN uses double-layer neural networks and experience replay to reduce the oscillation problem that may occur in gradient descent. We implemented the TCP-DQN method and compared it with mainstream congestion control algorithms such as cubic, Highspeed and NewReno. The results show that the throughput of TCP-DQN can reach more than 2 times of the comparison method while the latency is close to the three compared methods.