1
|
Wood JM, Kim HE, Morton SM. Reinforcement Learning during Locomotion. eNeuro 2024; 11:ENEURO.0383-23.2024. [PMID: 38438263 PMCID: PMC10946027 DOI: 10.1523/eneuro.0383-23.2024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/03/2023] [Revised: 02/20/2024] [Accepted: 02/23/2024] [Indexed: 03/06/2024] Open
Abstract
When learning a new motor skill, people often must use trial and error to discover which movement is best. In the reinforcement learning framework, this concept is known as exploration and has been linked to increased movement variability in motor tasks. For locomotor tasks, however, increased variability decreases upright stability. As such, exploration during gait may jeopardize balance and safety, making reinforcement learning less effective. Therefore, we set out to determine if humans could acquire and retain a novel locomotor pattern using reinforcement learning alone. Young healthy male and female participants walked on a treadmill and were provided with binary reward feedback (indicated by a green checkmark on the screen) that was tied to a fixed monetary bonus, to learn a novel stepping pattern. We also recruited a comparison group who walked with the same novel stepping pattern but did so by correcting for target error, induced by providing real-time veridical visual feedback of steps and a target. In two experiments, we compared learning, motor variability, and two forms of motor memories between the groups. We found that individuals in the binary reward group did, in fact, acquire the new walking pattern by exploring (increasing motor variability). Additionally, while reinforcement learning did not increase implicit motor memories, it resulted in more accurate explicit motor memories compared with the target error group. Overall, these results demonstrate that humans can acquire new walking patterns with reinforcement learning and retain much of the learning over 24 h.
Collapse
Affiliation(s)
- Jonathan M Wood
- Department of Physical Therapy, University of Delaware, Newark, Delaware 19713
- Interdisciplinary Graduate Program in Biomechanics & Movement Science, University of Delaware, Newark, Delaware 19713
| | - Hyosub E Kim
- Department of Physical Therapy, University of Delaware, Newark, Delaware 19713
- Interdisciplinary Graduate Program in Biomechanics & Movement Science, University of Delaware, Newark, Delaware 19713
- Department of Psychological and Brain Sciences, University of Delaware, Newark, Delaware 19716
- School of Kinesiology, University of British Columbia, Vancouver, British Columbia V6T 1Z1, Canada
| | - Susanne M Morton
- Department of Physical Therapy, University of Delaware, Newark, Delaware 19713
- Interdisciplinary Graduate Program in Biomechanics & Movement Science, University of Delaware, Newark, Delaware 19713
| |
Collapse
|
2
|
Cashaback JGA, Allen JL, Chou AHY, Lin DJ, Price MA, Secerovic NK, Song S, Zhang H, Miller HL. NSF DARE-transforming modeling in neurorehabilitation: a patient-in-the-loop framework. J Neuroeng Rehabil 2024; 21:23. [PMID: 38347597 PMCID: PMC10863253 DOI: 10.1186/s12984-024-01318-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/10/2023] [Accepted: 01/25/2024] [Indexed: 02/15/2024] Open
Abstract
In 2023, the National Science Foundation (NSF) and the National Institute of Health (NIH) brought together engineers, scientists, and clinicians by sponsoring a conference on computational modelling in neurorehabiilitation. To facilitate multidisciplinary collaborations and improve patient care, in this perspective piece we identify where and how computational modelling can support neurorehabilitation. To address the where, we developed a patient-in-the-loop framework that uses multiple and/or continual measurements to update diagnostic and treatment model parameters, treatment type, and treatment prescription, with the goal of maximizing clinically-relevant functional outcomes. This patient-in-the-loop framework has several key features: (i) it includes diagnostic and treatment models, (ii) it is clinically-grounded with the International Classification of Functioning, Disability and Health (ICF) and patient involvement, (iii) it uses multiple or continual data measurements over time, and (iv) it is applicable to a range of neurological and neurodevelopmental conditions. To address the how, we identify state-of-the-art and highlight promising avenues of future research across the realms of sensorimotor adaptation, neuroplasticity, musculoskeletal, and sensory & pain computational modelling. We also discuss both the importance of and how to perform model validation, as well as challenges to overcome when implementing computational models within a clinical setting. The patient-in-the-loop approach offers a unifying framework to guide multidisciplinary collaboration between computational and clinical stakeholders in the field of neurorehabilitation.
Collapse
Affiliation(s)
- Joshua G A Cashaback
- Biomedical Engineering, Mechanical Engineering, Kinesiology and Applied Physiology, Biome chanics and Movement Science Program, Interdisciplinary Neuroscience Graduate Program, University of Delaware, 540 S College Ave, Newark, DE, 19711, USA.
| | - Jessica L Allen
- Department of Mechanical Engineering, University of Florida, Gainesville, USA
| | | | - David J Lin
- Division of Neurocritical Care and Stroke Service, Department of Neurology, Center for Neurotechnology and Neurorecovery, Massachusetts General Hospital, Harvard Medical School, Boston, USA
- Department of Veterans Affairs, Center for Neurorestoration and Neurotechnology, Rehabilitation Research and Development Service, Providence, USA
| | - Mark A Price
- Department of Mechanical and Industrial Engineering, Department of Kinesiology, University of Massachusetts Amherst, Amherst, USA
| | - Natalija K Secerovic
- School of Electrical Engineering, The Mihajlo Pupin Institute, University of Belgrade, Belgrade, Serbia
- Laboratory for Neuroengineering, Institute for Robotics and Intelligent Systems ETH Zürich, Zurich, Switzerland
| | - Seungmoon Song
- Mechanical and Industrial Engineering, Northeastern University, Boston, USA
| | - Haohan Zhang
- Department of Mechanical Engineering, University of Utah, Salt Lake City, USA
| | - Haylie L Miller
- School of Kinesiology, University of Michigan, 830 N University Ave, Ann Arbor, MI, 48109, USA.
| |
Collapse
|