Learning to use Muscles

Acquiring musculoskeletal skills with curriculum-based reinforcement learning

Efficient musculoskeletal simulators and powerful learning algorithms provide computational tools to tackle the grand challenge of understanding biological motor control. Our winning solution for the inaugural NeurIPS MyoChallenge leverages an approach mirroring human skill learning. Using a novel curriculum learning approach, we trained a recurrent neural network to control a realistic model of the human hand with 39 muscles to rotate two Baoding balls in the palm of the hand. In agreement with data from human subjects, the policy uncovers a small number of kinematic synergies, even though it is not explicitly biased toward low-dimensional solutions. However, selectively inactivating parts of the control signal, we found that more dimensions contribute to the task performance than suggested by traditional synergy analysis. Overall, our work illustrates the emerging possibilities at the interface of musculoskeletal physics engines, reinforcement learning, and neuroscience to advance our understanding of biological motor control.

A computational study of how an α- to γ-motoneurone collateral can mitigate velocity-dependent stretch reflexes during voluntary movement

The primary motor cortex does not uniquely or directly produce alpha motoneurone (α-MN) drive to muscles during voluntary movement. Rather, α-MN drive emerges from the synthesis and competition among excitatory and inhibitory inputs from multiple descending tracts, spinal interneurons, sensory inputs, and proprioceptive afferents. One such fundamental input is velocity-dependent stretch reflexes in lengthening muscles, which should be inhibited to enable voluntary movement. It remains an open question, however, the extent to which unmodulated stretch reflexes disrupt voluntary movement, and whether and how they are inhibited in limbs with numerous multiarticular muscles. We used a computational model of a Rhesus Macaque arm to simulate movements with feedforward α-MN commands only, and with added velocity-dependent stretch reflex feedback. We found that velocity-dependent stretch reflex caused movement-specific, typically large and variable disruptions to arm movements. These disruptions were greatly reduced when modulating velocity-dependent stretch reflex feedback (i) as per the commonly proposed (but yet to be clarified) idealized alpha-gamma (α-γ) coactivation or (ii) an alternative α-MN collateral projection to homonymous γ-MNs. We conclude that such α-MN collaterals are a physiologically tenable propriospinal circuit in the mammalian fusimotor system. These collaterals could still collaborate with α-γ coactivation, and the few skeletofusimotor fibers (β-MNs) in mammals, to create a flexible fusimotor ecosystem to enable voluntary movement. By locally and automatically regulating the highly nonlinear neuro-musculo-skeletal mechanics of the limb, these collaterals could be a critical low-level enabler of learning, adaptation, and performance via higher-level brainstem, cerebellar, and cortical mechanisms.

MotorNet, a Python toolbox for controlling differentiable biomechanical effectors with artificial neural networks

Artificial neural networks (ANNs) are a powerful class of computational models for unravelling neural mechanisms of brain function. However, for neural control of movement, they currently must be integrated with software simulating biomechanical effectors, leading to limiting impracticalities: (1) researchers must rely on two different platforms and (2) biomechanical effectors are not generally differentiable, constraining researchers to reinforcement learning algorithms despite the existence and potential biological relevance of faster training methods. To address these limitations, we developed MotorNet, an open-source Python toolbox for creating arbitrarily complex, differentiable, and biomechanically realistic effectors that can be trained on user-defined motor tasks using ANNs. MotorNet is designed to meet several goals: ease of installation, ease of use, a high-level user-friendly application programming interface, and a modular architecture to allow for flexibility in model building. MotorNet requires no dependencies outside Python, making it easy to get started with. For instance, it allows training ANNs on typically used motor control models such as a two joint, six muscle, planar arm within minutes on a typical desktop computer. MotorNet is built on PyTorch and therefore can implement any network architecture that is possible using the PyTorch framework. Consequently, it will immediately benefit from advances in artificial intelligence through PyTorch updates. Finally, it is open source, enabling users to create and share their own improvements, such as new effector and network architectures or custom task designs. MotorNet's focus on higher-order model and task design will alleviate overhead cost to initiate computational projects for new researchers by providing a standalone, ready-to-go framework, and speed up efforts of established computational teams by enabling a focus on concepts and ideas over implementation.

Synergy quality assessment of muscle modules for determining learning performance using a realistic musculoskeletal model

How our central nervous system efficiently controls our complex musculoskeletal system is still debated. The muscle synergy hypothesis is proposed to simplify this complex system by assuming the existence of functional neural modules that coordinate several muscles. Modularity based on muscle synergies can facilitate motor learning without compromising task performance. However, the effectiveness of modularity in motor control remains debated. This ambiguity can, in part, stem from overlooking that the performance of modularity depends on the mechanical aspects of modules of interest, such as the torque the modules exert. To address this issue, this study introduces two criteria to evaluate the quality of module sets based on commonly used performance metrics in motor learning studies: the accuracy of torque production and learning speed. One evaluates the regularity in the direction of mechanical torque the modules exert, while the other evaluates the evenness of its magnitude. For verification of our criteria, we simulated motor learning of torque production tasks in a realistic musculoskeletal system of the upper arm using feed-forward neural networks while changing the control conditions. We found that the proposed criteria successfully explain the tendency of learning performance in various control conditions. These result suggest that regularity in the direction of and evenness in magnitude of mechanical torque of utilized modules are significant factor for determining learning performance. Although the criteria were originally conceived for an error-based learning scheme, the approach to pursue which set of modules is better for motor control can have significant implications in other studies of modularity in general.

NSF DARE-Transforming modeling in neurorehabilitation: Four threads for catalyzing progress

We present an overview of the Conference on Transformative Opportunities for Modeling in Neurorehabilitation held in March 2023. It was supported by the Disability and Rehabilitation Engineering (DARE) program from the National Science Foundation's Engineering Biology and Health Cluster. The conference brought together experts and trainees from around the world to discuss critical questions, challenges, and opportunities at the intersection of computational modeling and neurorehabilitation to understand, optimize, and improve clinical translation of neurorehabilitation. We organized the conference around four key, relevant, and promising Focus Areas for modeling: Adaptation & Plasticity, Personalization, Human-Device Interactions, and Modeling 'In-the-Wild'. We identified four common threads across the Focus Areas that, if addressed, can catalyze progress in the short, medium, and long terms. These were: (i) the need to capture and curate appropriate and useful data necessary to develop, validate, and deploy useful computational models (ii) the need to create multi-scale models that span the personalization spectrum from individuals to populations, and from cellular to behavioral levels (iii) the need for algorithms that extract as much information from available data, while requiring as little data as possible from each client (iv) the insistence on leveraging readily available sensors and data systems to push model-driven treatments from the lab, and into the clinic, home, workplace, and community. The conference archive can be found at (dare2023.usc.edu). These topics are also extended by three perspective papers prepared by trainees and junior faculty, clinician researchers, and federal funding agency representatives who attended the conference.

The spinal cord facilitates cerebellar upper limb motor learning and control; inputs from neuromusculoskeletal simulation

Complex interactions between brain regions and the spinal cord (SC) govern body motion, which is ultimately driven by muscle activation. Motor planning or learning are mainly conducted at higher brain regions, whilst the SC acts as a brain-muscle gateway and as a motor control centre providing fast reflexes and muscle activity regulation. Thus, higher brain areas need to cope with the SC as an inherent and evolutionary older part of the body dynamics. Here, we address the question of how SC dynamics affects motor learning within the cerebellum; in particular, does the SC facilitate cerebellar motor learning or constitute a biological constraint? We provide an exploratory framework by integrating biologically plausible cerebellar and SC computational models in a musculoskeletal upper limb control loop. The cerebellar model, equipped with the main form of cerebellar plasticity, provides motor adaptation; whilst the SC model implements stretch reflex and reciprocal inhibition between antagonist muscles. The resulting spino-cerebellar model is tested performing a set of upper limb motor tasks, including external perturbation studies. A cerebellar model, lacking the implemented SC model and directly controlling the simulated muscles, was also tested in the same. The performances of the spino-cerebellar and cerebellar models were then compared, thus allowing directly addressing the SC influence on cerebellar motor adaptation and learning, and on handling external motor perturbations. Performance was assessed in both joint and muscle space, and compared with kinematic and EMG recordings from healthy participants. The differences in cerebellar synaptic adaptation between both models were also studied. We conclude that the SC facilitates cerebellar motor learning; when the SC circuits are in the loop, faster convergence in motor learning is achieved with simpler cerebellar synaptic weight distributions. The SC is also found to improve robustness against external perturbations, by better reproducing and modulating muscle cocontraction patterns.

Remembrance of things perceived: Adding thalamocortical function to artificial neural networks

Recent research has illuminated the complexity and importance of the thalamocortical system but it has been difficult to identify what computational functions it performs. Meanwhile, deep-learning artificial neural networks (ANNs) based on bio-inspired models of purely cortical circuits have achieved surprising success solving sophisticated cognitive problems associated historically with human intelligence. Nevertheless, the limitations and shortcomings of artificial intelligence (AI) based on such ANNs are becoming increasingly clear. This review considers how the addition of thalamocortical connectivity and its putative functions related to cortical attention might address some of those shortcomings. Such bio-inspired models are now providing both testable theories of biological cognition and improved AI technology, much of which is happening outside the usual academic venues.

Developing Intelligent Robots that Grasp Affordance

Humans and robots operating in unstructured environments both need to classify objects through haptic exploration and use them in various tasks, but currently they differ greatly in their strategies for acquiring such capabilities. This review explores nascent technologies that promise more convergence. A novel form of artificial intelligence classifies objects according to sensory percepts during active exploration and decides on efficient sequences of exploratory actions to identify objects. Representing objects according to the collective experience of manipulating them provides a substrate for discovering causality and affordances. Such concepts that generalize beyond explicit training experiences are an important aspect of human intelligence that has eluded robots. For robots to acquire such knowledge, they will need an extended period of active exploration and manipulation similar to that employed by infants. The efficacy, efficiency and safety of such behaviors depends on achieving smooth transitions between movements that change quickly from exploratory to executive to reflexive. Animals achieve such smoothness by using a hierarchical control scheme that is fundamentally different from those of conventional robotics. The lowest level of that hierarchy, the spinal cord, starts to self-organize during spontaneous movements in the fetus. This allows its connectivity to reflect the mechanics of the musculoskeletal plant, a bio-inspired process that could be used to adapt spinal-like middleware for robots. Implementation of these extended and essential stages of fetal and infant development is impractical, however, for mechatronic hardware that does not heal and replace itself like biological tissues. Instead such development can now be accomplished in silico and then cloned into physical robots, a strategy that could transcend human performance.

Heksor: The CNS substrate of an adaptive behavior

Over the past half‐century, the largely hardwired central nervous system (CNS) of 1970 has become the ubiquitously plastic CNS of today, in which change is the rule not the exception. This transformation complicates a central question in neuroscience: how are adaptive behaviours – behaviours that serve the needs of the individual – acquired and maintained through life? It poses a more basic question: how do many adaptive behaviours share the ubiquitously plastic CNS? This question compels neuroscience to adopt a new paradigm. The core of this paradigm is a CNS entity with unique properties, here given the name heksor from the Greek hexis. A heksor is a distributed network of neurons and synapses that changes itself as needed to maintain the key features of an adaptive behaviour, the features that make the behaviour satisfactory. Through their concurrent changes, the numerous heksors that share the CNS negotiate the properties of the neurons and synapses that they all use. Heksors keep the CNS in a state of negotiated equilibrium that enables each heksor to maintain the key features of its behaviour. The new paradigm based on heksors and the negotiated equilibrium they create is supported by animal and human studies of interactions among new and old adaptive behaviours, explains otherwise inexplicable results, and underlies promising new approaches to restoring behaviours impaired by injury or disease. Furthermore, the paradigm offers new and potentially important answers to extant questions, such as the generation and function of spontaneous neuronal activity, the aetiology of muscle synergies, and the control of homeostatic plasticity.

A Model for Self-Organization of Sensorimotor Function: Spinal Interneuronal Integration

Control of musculoskeletal systems depends on integration of voluntary commands and somatosensory feedback in the complex neural circuits of the spinal cord. Particular connectivity patterns have been identified experimentally, and it has been suggested that these may result from the wide variety of transcriptional types that have been observed in spinal interneurons. We ask instead whether the details of these connectivity patterns (and perhaps many others) can arise as a consequence of Hebbian adaptation during early development. We constructed an anatomically simplified model plant system with realistic muscles and sensors and connected it to a recurrent, random neuronal network consisting of both excitatory and inhibitory neurons endowed with Hebbian learning rules. We then generated a wide set of randomized muscle twitches typical of those described during fetal development and allowed the network to learn. Multiple simulations consistently resulted in diverse and stable patterns of activity and connectivity that included subsets of the interneurons that were similar to 'archetypical' interneurons described in the literature. We also found that such learning led to an increased degree of cooperativity between interneurons when performing larger limb movements on which it had not been trained. Hebbian learning gives rise to rich sets of diverse interneurons whose connectivity reflects the mechanical properties of the plant. At least some of the transcriptomic diversity may reflect the effects of this process rather than the cause of the connectivity. Such a learning process seems better suited to respond to the musculoskeletal mutations that underlie the evolution of new species.

Neural Control of Stopping and Stabilizing the Arm

Stopping is a crucial yet under-studied action for planning and producing meaningful and efficient movements. In this review, we discuss classical human psychophysics studies as well as those using engineered systems that aim to develop models of motor control of the upper limb. We present evidence for a hybrid model of motor control, which has an evolutionary advantage due to division of labor between cerebral hemispheres. Stopping is a fundamental aspect of movement that deserves more attention in research than it currently receives. Such research may provide a basis for understanding arm stabilization deficits that can occur following central nervous system (CNS) damage.

Muscle Synergies and Clinical Outcome Measures Describe Different Factors of Upper Limb Motor Function in Stroke Survivors Undergoing Rehabilitation in a Virtual Reality Environment

Recent studies have investigated muscle synergies as biomarkers for stroke, but it remains controversial if muscle synergies and clinical observation convey the same information on motor impairment. We aim to identify whether muscle synergies and clinical scales convey the same information or not. Post-stroke patients were administered an upper limb treatment. Before (T0) and after (T1) treatment, we assessed motor performance with clinical scales and motor output with EMG-derived muscle synergies. We implemented an exploratory factor analysis (EFA) and a confirmatory factor analysis (CFA) to identify the underlying relationships among all variables, at T0 and T1, and a general linear regression model to infer any relationships between the similarity between the affected and unaffected synergies (Median-sp) and clinical outcomes at T0. Clinical variables improved with rehabilitation whereas muscle-synergy parameters did not show any significant change. EFA and CFA showed that clinical variables and muscle-synergy parameters (except Median-sp) were grouped into different factors. Regression model showed that Median-sp could be well predicted by clinical scales. The information underlying clinical scales and muscle synergies are therefore different. However, clinical scales well predicted the similarity between the affected and unaffected synergies. Our results may have implications on personalizing rehabilitation protocols.

A Hand Motor Skills Rehabilitation for the Injured Implemented on a Social Robot

In this work, we introduce HaReS, a hand rehabilitation system. Our proposal integrates a series of exercises, jointly developed with a foundation for those with motor and cognitive injuries, that are aimed at improving the skills of patients and the adherence to the rehabilitation plan. Our system takes advantage of a low-cost hand-tracking device to provide a quantitative analysis of the performance of the patient. It also integrates a low-cost surface electromyography (sEMG) sensor in order to provide insight about which muscles are being activated while completing the exercises. It is also modular and can be deployed on a social robot. We tested our proposal in two different facilities for rehabilitation with high success. The therapists and patients felt more motivation while using HaReS, which improved the adherence to the rehabilitation plan. In addition, the therapists were able to provide services to more patients than when they used their traditional methodology.