1
|
Tsay JS, Kim HE, McDougle SD, Taylor JA, Haith A, Avraham G, Krakauer JW, Collins AGE, Ivry RB. Fundamental processes in sensorimotor learning: Reasoning, refinement, and retrieval. eLife 2024; 13:e91839. [PMID: 39087986 PMCID: PMC11293869 DOI: 10.7554/elife.91839] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/14/2023] [Accepted: 07/22/2024] [Indexed: 08/02/2024] Open
Abstract
Motor learning is often viewed as a unitary process that operates outside of conscious awareness. This perspective has led to the development of sophisticated models designed to elucidate the mechanisms of implicit sensorimotor learning. In this review, we argue for a broader perspective, emphasizing the contribution of explicit strategies to sensorimotor learning tasks. Furthermore, we propose a theoretical framework for motor learning that consists of three fundamental processes: reasoning, the process of understanding action-outcome relationships; refinement, the process of optimizing sensorimotor and cognitive parameters to achieve motor goals; and retrieval, the process of inferring the context and recalling a control policy. We anticipate that this '3R' framework for understanding how complex movements are learned will open exciting avenues for future research at the intersection between cognition and action.
Collapse
Affiliation(s)
- Jonathan S Tsay
- Department of Psychology, Carnegie Mellon UniversityPittsburghUnited States
- Neuroscience Institute, Carnegie Mellon UniversityPittsburgUnited States
| | - Hyosub E Kim
- School of Kinesiology, University of British ColumbiaVancouverCanada
| | | | - Jordan A Taylor
- Department of Psychology, Princeton UniversityPrincetonUnited States
| | - Adrian Haith
- Department of Neurology, Johns Hopkins UniversityBaltimoreUnited States
| | - Guy Avraham
- Department of Psychology, University of California BerkeleyBerkeleyUnited States
- Helen Wills Neuroscience Institute, University of California BerkeleyBerkeleyUnited States
| | - John W Krakauer
- Department of Neurology, Johns Hopkins UniversityBaltimoreUnited States
- Department of Neuroscience, Johns Hopkins UniversityBaltimoreUnited States
- Santa Fe InstituteSanta FeUnited States
| | - Anne GE Collins
- Department of Psychology, University of California BerkeleyBerkeleyUnited States
- Helen Wills Neuroscience Institute, University of California BerkeleyBerkeleyUnited States
| | - Richard B Ivry
- Department of Psychology, University of California BerkeleyBerkeleyUnited States
- Helen Wills Neuroscience Institute, University of California BerkeleyBerkeleyUnited States
| |
Collapse
|
2
|
Chen Y, Abram SJ, Ivry RB, Tsay JS. Motor adaptation is reduced by symbolic compared to sensory feedback. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.06.28.601293. [PMID: 39005305 PMCID: PMC11244888 DOI: 10.1101/2024.06.28.601293] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/16/2024]
Abstract
Motor adaptation - the process of reducing motor errors through feedback and practice - is an essential feature of human competence, allowing us to move accurately in dynamic and novel environments. Adaptation typically results from sensory feedback, with most learning driven by visual and proprioceptive feedback that arises with the movement. In humans, motor adaptation can also be driven by symbolic feedback. In the present study, we examine how implicit and explicit components of motor adaptation are modulated by symbolic feedback. We conducted three reaching experiments involving over 400 human participants to compare sensory and symbolic feedback using a task in which both types of learning processes could be operative (Experiment 1) or tasks in which learning was expected to be limited to only an explicit process (Experiments 2 and 3). Adaptation with symbolic feedback was dominated by explicit strategy use, with minimal evidence of implicit recalibration. Even when matched in terms of information content, adaptation to rotational and mirror reversal perturbations was slower in response to symbolic feedback compared to sensory feedback. Our results suggest that the abstract and indirect nature of symbolic feedback disrupts strategic reasoning and/or refinement, deepening our understanding of how feedback type influences the mechanisms of sensorimotor learning.
Collapse
Affiliation(s)
- Yifei Chen
- Department of Psychology, University of California, Berkeley
- Helen Wills Neuroscience Institute, University of California, Berkeley
| | - Sabrina J. Abram
- Department of Psychology, University of California, Berkeley
- Helen Wills Neuroscience Institute, University of California, Berkeley
| | - Richard B. Ivry
- Department of Psychology, University of California, Berkeley
- Helen Wills Neuroscience Institute, University of California, Berkeley
| | | |
Collapse
|
3
|
Forano M, Franklin DW. Reward actively engages both implicit and explicit components in dual force field adaptation. J Neurophysiol 2024; 132:1-22. [PMID: 38717332 DOI: 10.1152/jn.00307.2023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/16/2023] [Revised: 04/08/2024] [Accepted: 05/06/2024] [Indexed: 06/26/2024] Open
Abstract
Motor learning occurs through multiple mechanisms, including unsupervised, supervised (error based), and reinforcement (reward based) learning. Although studies have shown that reward leads to an overall better motor adaptation, the specific processes by which reward influences adaptation are still unclear. Here, we examine how the presence of reward affects dual adaptation to novel dynamics and distinguish its influence on implicit and explicit learning. Participants adapted to two opposing force fields in an adaptation/deadaptation/error-clamp paradigm, where five levels of reward (a score and a digital face) were provided as participants reduced their lateral error. Both reward and control (no reward provided) groups simultaneously adapted to both opposing force fields, exhibiting a similar final level of adaptation, which was primarily implicit. Triple-rate models fit to the adaptation process found higher learning rates in the fast and slow processes and a slightly increased fast retention rate for the reward group. Whereas differences in the slow learning rate were only driven by implicit learning, the large difference in the fast learning rate was mainly explicit. Overall, we confirm previous work showing that reward increases learning rates, extending this to dual-adaptation experiments and demonstrating that reward influences both implicit and explicit adaptation. Specifically, we show that reward acts primarily explicitly on the fast learning rate and implicitly on the slow learning rates.NEW & NOTEWORTHY Here we show that rewarding participants' performance during dual force field adaptation primarily affects the initial rate of learning and the early timescales of adaptation, with little effect on the final adaptation level. However, reward affects both explicit and implicit components of adaptation. Whereas the learning rate of the slow process is increased implicitly, the fast learning and retention rates are increased through both implicit components and the use of explicit strategies.
Collapse
Affiliation(s)
- Marion Forano
- Neuromuscular Diagnostics, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany
- Department of Orthopaedics and Sports Orthopaedics, Klinikum Rechts der Isar, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany
| | - David W Franklin
- Neuromuscular Diagnostics, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany
- Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University of Munich, Munich, Germany
- Munich Data Science Institute (MDSI), Technical University of Munich, Munich, Germany
| |
Collapse
|
4
|
Nick Q, Gale DJ, Areshenkoff C, De Brouwer A, Nashed J, Wammes J, Zhu T, Flanagan R, Smallwood J, Gallivan J. Reconfigurations of cortical manifold structure during reward-based motor learning. eLife 2024; 12:RP91928. [PMID: 38916598 PMCID: PMC11198988 DOI: 10.7554/elife.91928] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 06/26/2024] Open
Abstract
Adaptive motor behavior depends on the coordinated activity of multiple neural systems distributed across the brain. While the role of sensorimotor cortex in motor learning has been well established, how higher-order brain systems interact with sensorimotor cortex to guide learning is less well understood. Using functional MRI, we examined human brain activity during a reward-based motor task where subjects learned to shape their hand trajectories through reinforcement feedback. We projected patterns of cortical and striatal functional connectivity onto a low-dimensional manifold space and examined how regions expanded and contracted along the manifold during learning. During early learning, we found that several sensorimotor areas in the dorsal attention network exhibited increased covariance with areas of the salience/ventral attention network and reduced covariance with areas of the default mode network (DMN). During late learning, these effects reversed, with sensorimotor areas now exhibiting increased covariance with DMN areas. However, areas in posteromedial cortex showed the opposite pattern across learning phases, with its connectivity suggesting a role in coordinating activity across different networks over time. Our results establish the neural changes that support reward-based motor learning and identify distinct transitions in the functional coupling of sensorimotor to transmodal cortex when adapting behavior.
Collapse
Affiliation(s)
- Qasem Nick
- Centre for Neuroscience Studies, Queen’s UniversityKingstonCanada
- Department of Psychology, Queen’s UniversityKingstonCanada
| | - Daniel J Gale
- Centre for Neuroscience Studies, Queen’s UniversityKingstonCanada
| | - Corson Areshenkoff
- Centre for Neuroscience Studies, Queen’s UniversityKingstonCanada
- Department of Psychology, Queen’s UniversityKingstonCanada
| | - Anouk De Brouwer
- Centre for Neuroscience Studies, Queen’s UniversityKingstonCanada
| | - Joseph Nashed
- Centre for Neuroscience Studies, Queen’s UniversityKingstonCanada
- Department of Medicine, Queen's UniversityKingstonCanada
| | - Jeffrey Wammes
- Centre for Neuroscience Studies, Queen’s UniversityKingstonCanada
- Department of Psychology, Queen’s UniversityKingstonCanada
| | - Tianyao Zhu
- Centre for Neuroscience Studies, Queen’s UniversityKingstonCanada
| | - Randy Flanagan
- Centre for Neuroscience Studies, Queen’s UniversityKingstonCanada
- Department of Psychology, Queen’s UniversityKingstonCanada
| | - Jonny Smallwood
- Centre for Neuroscience Studies, Queen’s UniversityKingstonCanada
- Department of Psychology, Queen’s UniversityKingstonCanada
| | - Jason Gallivan
- Centre for Neuroscience Studies, Queen’s UniversityKingstonCanada
- Department of Psychology, Queen’s UniversityKingstonCanada
- Department of Biomedical and Molecular Sciences, Queen’s UniversityKingstonCanada
| |
Collapse
|
5
|
White CM, Snow EC, Therrien AS. Reinforcement Motor Learning After Cerebellar Damage Is Related to State Estimation. CEREBELLUM (LONDON, ENGLAND) 2024; 23:1061-1073. [PMID: 37828231 DOI: 10.1007/s12311-023-01615-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 10/03/2023] [Indexed: 10/14/2023]
Abstract
Recent work showed that individuals with cerebellar degeneration could leverage intact reinforcement learning (RL) to alter their movement. However, there was marked inter-individual variability in learning, and the factors underlying it were unclear. Cerebellum-dependent sensory prediction may contribute to RL in motor contexts by enhancing body state estimates, which are necessary to solve the credit-assignment problem. The objective of this study was to test the relationship between the predictive component of state estimation and RL in individuals with cerebellar degeneration. Individuals with cerebellar degeneration and neurotypical control participants completed two tasks: an RL task that required them to alter the angle of reaching movements and a state estimation task that tested the somatosensory perception of active and passive movement. The state estimation task permitted the calculation of the active benefit shown by each participant, which is thought to reflect the cerebellum-dependent predictive component of state estimation. We found that the cerebellar and control groups showed similar magnitudes of learning with reinforcement and active benefit on average, but there was substantial variability across individuals. Using multiple regression, we assessed potential predictors of RL. Our analysis included active benefit, somatosensory acuity, clinical ataxia severity, movement variability, movement speed, and age. We found a significant relationship in which greater active benefit predicted better learning with reinforcement in the cerebellar, but not the control group. No other variables showed significant relationships with learning. Overall, our results support the hypothesis that the integrity of sensory prediction is a strong predictor of RL after cerebellar damage.
Collapse
Affiliation(s)
- Christopher M White
- Moss Rehabilitation Research Institute, Medical Arts Building, Suite 100, 50 Township Line Rd, Elkins Park, PA, USA
| | - Evan C Snow
- Moss Rehabilitation Research Institute, Medical Arts Building, Suite 100, 50 Township Line Rd, Elkins Park, PA, USA
| | - Amanda S Therrien
- Moss Rehabilitation Research Institute, Medical Arts Building, Suite 100, 50 Township Line Rd, Elkins Park, PA, USA.
- Department of Rehabilitation Medicine, Thomas Jefferson University, Philadelphia, PA, USA.
| |
Collapse
|
6
|
Kim K, Oblak E, Manella K, Sulzer J. Simulated operant reflex conditioning environment reveals effects of feedback parameters. PLoS One 2024; 19:e0300338. [PMID: 38512998 PMCID: PMC10956789 DOI: 10.1371/journal.pone.0300338] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/10/2023] [Accepted: 02/26/2024] [Indexed: 03/23/2024] Open
Abstract
Operant conditioning of neural activation has been researched for decades in humans and animals. Many theories suggest two parallel learning processes, implicit and explicit. The degree to which feedback affects these processes individually remains to be fully understood and may contribute to a large percentage of non-learners. Our goal is to determine the explicit decision-making processes in response to feedback representing an operant conditioning environment. We developed a simulated operant conditioning environment based on a feedback model of spinal reflex excitability, one of the simplest forms of neural operant conditioning. We isolated the perception of the feedback signal from self-regulation of an explicit unskilled visuomotor task, enabling us to quantitatively examine feedback strategy. Our hypothesis was that feedback type, biological variability, and reward threshold affect operant conditioning performance and operant strategy. Healthy individuals (N = 41) were instructed to play a web application game using keyboard inputs to rotate a virtual knob representative of an operant strategy. The goal was to align the knob with a hidden target. Participants were asked to "down-condition" the amplitude of the virtual feedback signal, which was achieved by placing the knob as close as possible to the hidden target. We varied feedback type (knowledge of performance, knowledge of results), biological variability (low, high), and reward threshold (easy, moderate, difficult) in a factorial design. Parameters were extracted from real operant conditioning data. Our main outcomes were the feedback signal amplitude (performance) and the mean change in dial position (operant strategy). We observed that performance was modulated by variability, while operant strategy was modulated by feedback type. These results show complex relations between fundamental feedback parameters and provide the principles for optimizing neural operant conditioning for non-responders.
Collapse
Affiliation(s)
- Kyoungsoon Kim
- University of Texas at Austin, Austin, Texas, United States of America
| | - Ethan Oblak
- RIKEN Center for Brain Science, Saitama, Japan
| | - Kathleen Manella
- Nova Southeastern University, Clearwater, Florida, United States of America
| | - James Sulzer
- MetroHealth Hospital and Case Western Reserve University, Cleveland, Ohio, United States of America
| |
Collapse
|
7
|
Roth AM, Lokesh R, Tang J, Buggeln JH, Smith C, Calalo JA, Sullivan SR, Ngo T, Germain LS, Carter MJ, Cashaback JGA. Punishment Leads to Greater Sensorimotor Learning But Less Movement Variability Compared to Reward. Neuroscience 2024; 540:12-26. [PMID: 38220127 PMCID: PMC10922623 DOI: 10.1016/j.neuroscience.2024.01.004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/18/2023] [Revised: 01/05/2024] [Accepted: 01/09/2024] [Indexed: 01/16/2024]
Abstract
When a musician practices a new song, hitting a correct note sounds pleasant while striking an incorrect note sounds unpleasant. Such reward and punishment feedback has been shown to differentially influence the ability to learn a new motor skill. Recent work has suggested that punishment leads to greater movement variability, which causes greater exploration and faster learning. To further test this idea, we collected 102 participants over two experiments. Unlike previous work, in Experiment 1 we found that punishment did not lead to faster learning compared to reward (n = 68), but did lead to a greater extent of learning. Surprisingly, we also found evidence to suggest that punishment led to less movement variability, which was related to the extent of learning. We then designed a second experiment that did not involve adaptation, allowing us to further isolate the influence of punishment feedback on movement variability. In Experiment 2, we again found that punishment led to significantly less movement variability compared to reward (n = 34). Collectively our results suggest that punishment feedback leads to less movement variability. Future work should investigate whether punishment feedback leads to a greater knowledge of movement variability and or increases the sensitivity of updating motor actions.
Collapse
Affiliation(s)
- Adam M Roth
- Department of Mechanical Engineering, University of Delaware, United States
| | - Rakshith Lokesh
- Department of Biomedical Engineering, University of Delaware, United States
| | - Jiaqiao Tang
- Department of Kinesiology, McMaster University, Canada
| | - John H Buggeln
- Department of Biomedical Engineering, University of Delaware, United States
| | - Carly Smith
- Department of Biomedical Engineering, University of Delaware, United States
| | - Jan A Calalo
- Department of Mechanical Engineering, University of Delaware, United States
| | - Seth R Sullivan
- Department of Biomedical Engineering, University of Delaware, United States
| | - Truc Ngo
- Department of Biomedical Engineering, University of Delaware, United States
| | | | | | - Joshua G A Cashaback
- Department of Mechanical Engineering, University of Delaware, United States; Department of Biomedical Engineering, University of Delaware, United States; Kinesiology and Applied Physiology, University of Delaware, United States; Interdisciplinary Neuroscience Graduate Program, University of Delaware, United States; Biomechanics and Movement Science Program, University of Delaware, United States; Department of Kinesiology, McMaster University, Canada.
| |
Collapse
|
8
|
Wood JM, Kim HE, Morton SM. Reinforcement Learning during Locomotion. eNeuro 2024; 11:ENEURO.0383-23.2024. [PMID: 38438263 PMCID: PMC10946027 DOI: 10.1523/eneuro.0383-23.2024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/03/2023] [Revised: 02/20/2024] [Accepted: 02/23/2024] [Indexed: 03/06/2024] Open
Abstract
When learning a new motor skill, people often must use trial and error to discover which movement is best. In the reinforcement learning framework, this concept is known as exploration and has been linked to increased movement variability in motor tasks. For locomotor tasks, however, increased variability decreases upright stability. As such, exploration during gait may jeopardize balance and safety, making reinforcement learning less effective. Therefore, we set out to determine if humans could acquire and retain a novel locomotor pattern using reinforcement learning alone. Young healthy male and female participants walked on a treadmill and were provided with binary reward feedback (indicated by a green checkmark on the screen) that was tied to a fixed monetary bonus, to learn a novel stepping pattern. We also recruited a comparison group who walked with the same novel stepping pattern but did so by correcting for target error, induced by providing real-time veridical visual feedback of steps and a target. In two experiments, we compared learning, motor variability, and two forms of motor memories between the groups. We found that individuals in the binary reward group did, in fact, acquire the new walking pattern by exploring (increasing motor variability). Additionally, while reinforcement learning did not increase implicit motor memories, it resulted in more accurate explicit motor memories compared with the target error group. Overall, these results demonstrate that humans can acquire new walking patterns with reinforcement learning and retain much of the learning over 24 h.
Collapse
Affiliation(s)
- Jonathan M Wood
- Department of Physical Therapy, University of Delaware, Newark, Delaware 19713
- Interdisciplinary Graduate Program in Biomechanics & Movement Science, University of Delaware, Newark, Delaware 19713
| | - Hyosub E Kim
- Department of Physical Therapy, University of Delaware, Newark, Delaware 19713
- Interdisciplinary Graduate Program in Biomechanics & Movement Science, University of Delaware, Newark, Delaware 19713
- Department of Psychological and Brain Sciences, University of Delaware, Newark, Delaware 19716
- School of Kinesiology, University of British Columbia, Vancouver, British Columbia V6T 1Z1, Canada
| | - Susanne M Morton
- Department of Physical Therapy, University of Delaware, Newark, Delaware 19713
- Interdisciplinary Graduate Program in Biomechanics & Movement Science, University of Delaware, Newark, Delaware 19713
| |
Collapse
|
9
|
Cashaback JGA, Allen JL, Chou AHY, Lin DJ, Price MA, Secerovic NK, Song S, Zhang H, Miller HL. NSF DARE-transforming modeling in neurorehabilitation: a patient-in-the-loop framework. J Neuroeng Rehabil 2024; 21:23. [PMID: 38347597 PMCID: PMC10863253 DOI: 10.1186/s12984-024-01318-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/10/2023] [Accepted: 01/25/2024] [Indexed: 02/15/2024] Open
Abstract
In 2023, the National Science Foundation (NSF) and the National Institute of Health (NIH) brought together engineers, scientists, and clinicians by sponsoring a conference on computational modelling in neurorehabiilitation. To facilitate multidisciplinary collaborations and improve patient care, in this perspective piece we identify where and how computational modelling can support neurorehabilitation. To address the where, we developed a patient-in-the-loop framework that uses multiple and/or continual measurements to update diagnostic and treatment model parameters, treatment type, and treatment prescription, with the goal of maximizing clinically-relevant functional outcomes. This patient-in-the-loop framework has several key features: (i) it includes diagnostic and treatment models, (ii) it is clinically-grounded with the International Classification of Functioning, Disability and Health (ICF) and patient involvement, (iii) it uses multiple or continual data measurements over time, and (iv) it is applicable to a range of neurological and neurodevelopmental conditions. To address the how, we identify state-of-the-art and highlight promising avenues of future research across the realms of sensorimotor adaptation, neuroplasticity, musculoskeletal, and sensory & pain computational modelling. We also discuss both the importance of and how to perform model validation, as well as challenges to overcome when implementing computational models within a clinical setting. The patient-in-the-loop approach offers a unifying framework to guide multidisciplinary collaboration between computational and clinical stakeholders in the field of neurorehabilitation.
Collapse
Affiliation(s)
- Joshua G A Cashaback
- Biomedical Engineering, Mechanical Engineering, Kinesiology and Applied Physiology, Biome chanics and Movement Science Program, Interdisciplinary Neuroscience Graduate Program, University of Delaware, 540 S College Ave, Newark, DE, 19711, USA.
| | - Jessica L Allen
- Department of Mechanical Engineering, University of Florida, Gainesville, USA
| | | | - David J Lin
- Division of Neurocritical Care and Stroke Service, Department of Neurology, Center for Neurotechnology and Neurorecovery, Massachusetts General Hospital, Harvard Medical School, Boston, USA
- Department of Veterans Affairs, Center for Neurorestoration and Neurotechnology, Rehabilitation Research and Development Service, Providence, USA
| | - Mark A Price
- Department of Mechanical and Industrial Engineering, Department of Kinesiology, University of Massachusetts Amherst, Amherst, USA
| | - Natalija K Secerovic
- School of Electrical Engineering, The Mihajlo Pupin Institute, University of Belgrade, Belgrade, Serbia
- Laboratory for Neuroengineering, Institute for Robotics and Intelligent Systems ETH Zürich, Zurich, Switzerland
| | - Seungmoon Song
- Mechanical and Industrial Engineering, Northeastern University, Boston, USA
| | - Haohan Zhang
- Department of Mechanical Engineering, University of Utah, Salt Lake City, USA
| | - Haylie L Miller
- School of Kinesiology, University of Michigan, 830 N University Ave, Ann Arbor, MI, 48109, USA.
| |
Collapse
|
10
|
Leow LA, Bernheine L, Carroll TJ, Dux PE, Filmer HL. Dopamine Increases Accuracy and Lengthens Deliberation Time in Explicit Motor Skill Learning. eNeuro 2024; 11:ENEURO.0360-23.2023. [PMID: 38238069 PMCID: PMC10849023 DOI: 10.1523/eneuro.0360-23.2023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/17/2023] [Revised: 11/23/2023] [Accepted: 11/28/2023] [Indexed: 01/23/2024] Open
Abstract
Although animal research implicates a central role for dopamine in motor skill learning, a direct causal link has yet to be established in neurotypical humans. Here, we tested if a pharmacological manipulation of dopamine alters motor learning, using a paradigm which engaged explicit, goal-directed strategies. Participants (27 females; 11 males; aged 18-29 years) first consumed either 100 mg of levodopa (n = 19), a dopamine precursor that increases dopamine availability, or placebo (n = 19). Then, during training, participants learnt the explicit strategy of aiming away from presented targets by instructed angles of varying sizes. Targets jumped mid-movement by the instructed aiming angle. Task success was thus contingent upon aiming accuracy and not speed. The effect of the dopamine manipulations on skill learning was assessed during training and after an overnight follow-up. Increasing dopamine availability at training improved aiming accuracy and lengthened reaction times, particularly for larger, more difficult aiming angles, both at training and, importantly, at follow-up, despite prominent session-by-session performance improvements in both accuracy and speed. Exogenous dopamine thus seems to result in a learnt, persistent propensity to better adhere to task goals. Results support the proposal that dopamine is important in engagement of instrumental motivation to optimize adherence to task goals, particularly when learning to execute goal-directed strategies in motor skill learning.
Collapse
Affiliation(s)
- Li-Ann Leow
- School of Psychology, The University of Queensland, St Lucia, 4072, Australia
- Centre for Sensorimotor Performance, School of Human Movement & Nutrition Sciences, St Lucia, 4067, Australia
| | - Lena Bernheine
- Centre for Sensorimotor Performance, School of Human Movement & Nutrition Sciences, St Lucia, 4067, Australia
- School of Sport Science Faculty of Sport Governance and Event Management, University of Bayreuth, 95447 Bayreuth, Germany
| | - Timothy J Carroll
- Centre for Sensorimotor Performance, School of Human Movement & Nutrition Sciences, St Lucia, 4067, Australia
| | - Paul E Dux
- School of Psychology, The University of Queensland, St Lucia, 4072, Australia
| | - Hannah L Filmer
- School of Psychology, The University of Queensland, St Lucia, 4072, Australia
| |
Collapse
|
11
|
Kunavar T, Cheng X, Franklin DW, Burdet E, Babič J. Explicit learning based on reward prediction error facilitates agile motor adaptations. PLoS One 2023; 18:e0295274. [PMID: 38055714 DOI: 10.1371/journal.pone.0295274] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/02/2023] [Accepted: 11/17/2023] [Indexed: 12/08/2023] Open
Abstract
Error based motor learning can be driven by both sensory prediction error and reward prediction error. Learning based on sensory prediction error is termed sensorimotor adaptation, while learning based on reward prediction error is termed reward learning. To investigate the characteristics and differences between sensorimotor adaptation and reward learning, we adapted a visuomotor paradigm where subjects performed arm movements while presented with either the sensory prediction error, signed end-point error, or binary reward. Before each trial, perturbation indicators in the form of visual cues were presented to inform the subjects of the presence and direction of the perturbation. To analyse the interconnection between sensorimotor adaptation and reward learning, we designed a computational model that distinguishes between the two prediction errors. Our results indicate that subjects adapted to novel perturbations irrespective of the type of prediction error they received during learning, and they converged towards the same movement patterns. Sensorimotor adaptations led to a pronounced aftereffect, while adaptation based on reward consequences produced smaller aftereffects suggesting that reward learning does not alter the internal model to the same degree as sensorimotor adaptation. Even though all subjects had learned to counteract two different perturbations separately, only those who relied on explicit learning using reward prediction error could timely adapt to the randomly changing perturbation. The results from the computational model suggest that sensorimotor and reward learning operate through distinct adaptation processes and that only sensorimotor adaptation changes the internal model, whereas reward learning employs explicit strategies that do not result in aftereffects. Additionally, we demonstrate that when humans learn motor tasks, they utilize both learning processes to successfully adapt to the new environments.
Collapse
Affiliation(s)
- Tjasa Kunavar
- Laboratory for Neuromechanics and Biorobotics, Department of Automatics, Biocybernetics, and Robotics, Jožef Stefan Institute, Ljubljana, Slovenia
- Jožef Stefan International Postgraduate School, Ljubljana, Slovenia
| | - Xiaoxiao Cheng
- Department of Bioengineering, Imperial College of Science, Technology and Medicine, London, United Kingdom
| | - David W Franklin
- Neuromuscular Diagnostics, Department Health and Sport Sciences, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany
- Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University of Munich, Munich, Germany
- Munich Data Science Institute (MDSI), Technical University of Munich, Munich, Germany
| | - Etienne Burdet
- Department of Bioengineering, Imperial College of Science, Technology and Medicine, London, United Kingdom
| | - Jan Babič
- Laboratory for Neuromechanics and Biorobotics, Department of Automatics, Biocybernetics, and Robotics, Jožef Stefan Institute, Ljubljana, Slovenia
| |
Collapse
|
12
|
Yin C, Li B, Gao T. Differential effects of reward and punishment on reinforcement-based motor learning and generalization. J Neurophysiol 2023; 130:1150-1161. [PMID: 37791387 DOI: 10.1152/jn.00242.2023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/19/2023] [Revised: 09/13/2023] [Accepted: 09/28/2023] [Indexed: 10/05/2023] Open
Abstract
Reward and punishment have long been recognized as potent modulators of human behavior. Although reinforcement learning is a significant motor learning process, the exact mechanisms underlying how the brain learns movements through reward and punishment are not yet fully understood. Beyond the memory of specific examples, investigating the ability to generalize to new situations offers a better understanding of motor learning. This study hypothesizes that reward and punishment engage qualitatively different motivational systems with different neurochemical and neuroanatomical substrates, which would have differential effects on reinforcement-based motor learning and generalization. To test this hypothesis, two groups of participants learn a motor task in one direction and then relearn the same task in a new direction, receiving only performance-based reward or punishment score feedback. Our findings support our hypothesis, showing that reward led to slower learning but promoted generalization. On the other hand, punishment led to faster learning but impaired generalization. These behavioral differences may be due to different tendencies of movement variability in each group. The punishment group tended to explore more actively than the reward group during the initial learning phase, possibly due to loss aversion. In contrast, the reward group tended to explore more actively than the initial learning phase during the generalization test phase, seemingly recalling the strategy that led to the reward. These results suggest that reward and punishment may engage different neural mechanisms during reinforcement-based motor learning and generalization, with important implications for practical applications such as sports training and motor rehabilitation.NEW & NOTEWORTHY Although reinforcement learning is a significant motor learning process, the mechanisms underlying how the brain learns movements through reward and punishment are not fully understood. We modified a well-established motor adaptation task and used savings (faster relearning) to measure generalization. We found reward led to slower learning but promoted generalization, whereas punishment led to faster learning but impaired generalization, suggesting that reward and punishment may engage different neural mechanisms during reinforcement-based motor learning and generalization.
Collapse
Affiliation(s)
- Cong Yin
- School of Kinesiology and Health, Capital University of Physical Education and Sports, Beijing, People's Republic of China
| | - Biao Li
- School of Kinesiology and Health, Capital University of Physical Education and Sports, Beijing, People's Republic of China
| | - Tian Gao
- School of Kinesiology and Health, Capital University of Physical Education and Sports, Beijing, People's Republic of China
| |
Collapse
|
13
|
Roth AM, Calalo JA, Lokesh R, Sullivan SR, Grill S, Jeka JJ, van der Kooij K, Carter MJ, Cashaback JGA. Reinforcement-based processes actively regulate motor exploration along redundant solution manifolds. Proc Biol Sci 2023; 290:20231475. [PMID: 37848061 PMCID: PMC10581769 DOI: 10.1098/rspb.2023.1475] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2023] [Accepted: 09/06/2023] [Indexed: 10/19/2023] Open
Abstract
From a baby's babbling to a songbird practising a new tune, exploration is critical to motor learning. A hallmark of exploration is the emergence of random walk behaviour along solution manifolds, where successive motor actions are not independent but rather become serially dependent. Such exploratory random walk behaviour is ubiquitous across species' neural firing, gait patterns and reaching behaviour. The past work has suggested that exploratory random walk behaviour arises from an accumulation of movement variability and a lack of error-based corrections. Here, we test a fundamentally different idea-that reinforcement-based processes regulate random walk behaviour to promote continual motor exploration to maximize success. Across three human reaching experiments, we manipulated the size of both the visually displayed target and an unseen reward zone, as well as the probability of reinforcement feedback. Our empirical and modelling results parsimoniously support the notion that exploratory random walk behaviour emerges by utilizing knowledge of movement variability to update intended reach aim towards recently reinforced motor actions. This mechanism leads to active and continuous exploration of the solution manifold, currently thought by prominent theories to arise passively. The ability to continually explore muscle, joint and task redundant solution manifolds is beneficial while acting in uncertain environments, during motor development or when recovering from a neurological disorder to discover and learn new motor actions.
Collapse
Affiliation(s)
- Adam M. Roth
- Department of Mechanical Engineering, University of Delaware, Newark, DE 19716, USA
| | - Jan A. Calalo
- Department of Mechanical Engineering, University of Delaware, Newark, DE 19716, USA
| | - Rakshith Lokesh
- Department of Biomedical Engineering, University of Delaware, Newark, DE 19716, USA
| | - Seth R. Sullivan
- Department of Biomedical Engineering, University of Delaware, Newark, DE 19716, USA
| | - Stephen Grill
- Kinesiology and Applied Physiology, University of Delaware, Newark, DE 19716, USA
| | - John J. Jeka
- Kinesiology and Applied Physiology, University of Delaware, Newark, DE 19716, USA
- Interdisciplinary Neuroscience Graduate Program, University of Delaware, Newark, DE 19716, USA
- Biomechanics and Movement Science Program, University of Delaware, Newark, DE 19716, USA
| | - Katinka van der Kooij
- Faculty of Behavioural and Movement Science, Vrije University Amsterdam, Amsterdam, 1081HV, The Netherlands
| | - Michael J. Carter
- Department of Kinesiology, McMaster University, Room 203, Ivor Wynne Centre, Hamilton, L8S 4L8, Ontario, Canada
| | - Joshua G. A. Cashaback
- Department of Mechanical Engineering, University of Delaware, Newark, DE 19716, USA
- Department of Biomedical Engineering, University of Delaware, Newark, DE 19716, USA
- Kinesiology and Applied Physiology, University of Delaware, Newark, DE 19716, USA
- Interdisciplinary Neuroscience Graduate Program, University of Delaware, Newark, DE 19716, USA
- Biomechanics and Movement Science Program, University of Delaware, Newark, DE 19716, USA
| |
Collapse
|
14
|
van Mastrigt NM, Tsay JS, Wang T, Avraham G, Abram SJ, van der Kooij K, Smeets JBJ, Ivry RB. Implicit reward-based motor learning. Exp Brain Res 2023; 241:2287-2298. [PMID: 37580611 PMCID: PMC10471724 DOI: 10.1007/s00221-023-06683-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/05/2023] [Accepted: 08/02/2023] [Indexed: 08/16/2023]
Abstract
Binary feedback, providing information solely about task success or failure, can be sufficient to drive motor learning. While binary feedback can induce explicit adjustments in movement strategy, it remains unclear if this type of feedback also induces implicit learning. We examined this question in a center-out reaching task by gradually moving an invisible reward zone away from a visual target to a final rotation of 7.5° or 25° in a between-group design. Participants received binary feedback, indicating if the movement intersected the reward zone. By the end of the training, both groups modified their reach angle by about 95% of the rotation. We quantified implicit learning by measuring performance in a subsequent no-feedback aftereffect phase, in which participants were told to forgo any adopted movement strategies and reach directly to the visual target. The results showed a small, but robust (2-3°) aftereffect in both groups, highlighting that binary feedback elicits implicit learning. Notably, for both groups, reaches to two flanking generalization targets were biased in the same direction as the aftereffect. This pattern is at odds with the hypothesis that implicit learning is a form of use-dependent learning. Rather, the results suggest that binary feedback can be sufficient to recalibrate a sensorimotor map.
Collapse
Affiliation(s)
- Nina M van Mastrigt
- Department of Human Movement Sciences, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands.
| | | | | | | | | | - Katinka van der Kooij
- Department of Human Movement Sciences, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
| | - Jeroen B J Smeets
- Department of Human Movement Sciences, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
| | | |
Collapse
|
15
|
Al-Fawakhiri N, Ma A, Taylor JA, Kim OA. Exploring the role of task success in implicit motor adaptation. J Neurophysiol 2023; 130:332-344. [PMID: 37403601 PMCID: PMC10396223 DOI: 10.1152/jn.00061.2023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/06/2023] [Revised: 06/08/2023] [Accepted: 06/30/2023] [Indexed: 07/06/2023] Open
Abstract
Although implicit motor adaptation is driven by sensory-prediction errors (SPEs), recent work has shown that task success modulates this process. Task success has typically been defined as hitting a target, which signifies the goal of the movement. Visuomotor adaptation tasks are uniquely situated to experimentally manipulate task success independently from SPE by changing the target size or the location of the target. These two, distinct manipulations may influence implicit motor adaptation in different ways, so, over four experiments, we sought to probe the efficacy of each manipulation. We found that changes in target size, which caused the target to fully envelop the cursor, only affected implicit adaptation for a narrow range of SPE sizes, while jumping the target to overlap with the cursor more reliably and robustly affected implicit adaptation. Taken together, our data indicate that, while task success exerts a small effect on implicit adaptation, these effects are susceptible to methodological variations. Future investigations of the effect of task success on implicit adaptation could benefit from employing target jump manipulations instead of target size manipulations.NEW & NOTEWORTHY Recent work has suggested that task success, namely, hitting a target, influences implicit motor adaptation. Here, we observed that implicit adaptation is modulated by target jump manipulations, where the target abruptly "jumps" to meet the cursor; however, implicit adaptation was only weakly modulated by target size manipulations, where a static target either envelops or excludes the cursor. We discuss how these manipulations may exert their effects through different mechanisms.
Collapse
Affiliation(s)
- Naser Al-Fawakhiri
- Department of Psychology, Yale University, New Haven, Connecticut, United States
| | - Ambri Ma
- Department of Psychology, Princeton University, Princeton, New Jersey, United States
| | - Jordan A Taylor
- Department of Psychology, Princeton University, Princeton, New Jersey, United States
| | - Olivia A Kim
- Department of Psychology, Princeton University, Princeton, New Jersey, United States
| |
Collapse
|
16
|
Malone LA, Hill NM, Tripp H, Wolpert DM, Bastian AJ. A novel video game for remote studies of motor adaptation in children. Physiol Rep 2023; 11:e15764. [PMID: 37434268 PMCID: PMC10336020 DOI: 10.14814/phy2.15764] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/08/2023] [Revised: 06/19/2023] [Accepted: 06/22/2023] [Indexed: 07/13/2023] Open
Abstract
Here we designed a motor adaptation video game that could be played remotely (at home) through a web browser. This required the child to adapt to a visuomotor rotation between their hand movement and a ball displayed in the game. The task had several novel features, specifically designed to allow the study of the developmental trajectory of adaptation across a wide range of ages. We test the concurrent validity by comparing children's performance on our remote task to the same task performed in the laboratory. All participants remained engaged and completed the task. We quantified feedforward and feedback control during this task. Feedforward control, a key measure of adaptation, was similar at home and in the laboratory. All children could successfully use feedback control to guide the ball to a target. Traditionally, motor learning studies are performed in a laboratory to obtain high quality kinematic data. However, here we demonstrate concurrent validity of kinematic behavior when conducted at home. Our online platform provides the flexibility and ease of collecting data that will enable future studies with large sample sizes, longitudinal experiments, and the study of children with rare diseases.
Collapse
Affiliation(s)
- Laura A. Malone
- Kennedy Krieger InstituteBaltimoreMarylandUSA
- Department of NeurologyJohns Hopkins School of MedicineBaltimoreMarylandUSA
- Department of Physical Medicine and RehabilitationJohns Hopkins School of MedicineBaltimoreMarylandUSA
| | - Nayo M. Hill
- Kennedy Krieger InstituteBaltimoreMarylandUSA
- Department of NeuroscienceJohns Hopkins School of MedicineBaltimoreMarylandUSA
| | - Haley Tripp
- Kennedy Krieger InstituteBaltimoreMarylandUSA
| | - Daniel M. Wolpert
- Mortimer B. Zuckerman Mind Brain Behavior InstituteColumbia UniversityNew YorkNew YorkUSA
- Department of NeuroscienceColumbia UniversityNew YorkNew YorkUSA
| | - Amy J. Bastian
- Kennedy Krieger InstituteBaltimoreMarylandUSA
- Department of NeuroscienceJohns Hopkins School of MedicineBaltimoreMarylandUSA
| |
Collapse
|
17
|
van Mastrigt NM, Tsay JS, Wang T, Avraham G, Abram SJ, van der Kooij K, Smeets JBJ, Ivry RB. Implicit reward-based motor learning. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.06.27.546738. [PMID: 37425740 PMCID: PMC10327077 DOI: 10.1101/2023.06.27.546738] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/11/2023]
Abstract
Binary feedback, providing information solely about task success or failure, can be sufficient to drive motor learning. While binary feedback can induce explicit adjustments in movement strategy, it remains unclear if this type of feedback also induce implicit learning. We examined this question in a center-out reaching task by gradually moving an invisible reward zone away from a visual target to a final rotation of 7.5° or 25° in a between-group design. Participants received binary feedback, indicating if the movement intersected the reward zone. By the end of the training, both groups modified their reach angle by about 95% of the rotation. We quantified implicit learning by measuring performance in a subsequent no-feedback aftereffect phase, in which participants were told to forgo any adopted movement strategies and reach directly to the visual target. The results showed a small, but robust (2-3°) aftereffect in both groups, highlighting that binary feedback elicits implicit learning. Notably, for both groups, reaches to two flanking generalization targets were biased in the same direction as the aftereffect. This pattern is at odds with the hypothesis that implicit learning is a form of use-dependent learning. Rather, the results suggest that binary feedback can be sufficient to recalibrate a sensorimotor map.
Collapse
Affiliation(s)
- Nina M van Mastrigt
- Vrije Universiteit Amsterdam, Department of Human Movement Sciences, Amsterdam, The Netherlands
| | - Jonathan S Tsay
- UC Berkeley, CognAc lab, Berkeley, California, United States
| | - Tianhe Wang
- UC Berkeley, CognAc lab, Berkeley, California, United States
| | - Guy Avraham
- UC Berkeley, CognAc lab, Berkeley, California, United States
| | - Sabrina J Abram
- UC Berkeley, CognAc lab, Berkeley, California, United States
| | - Katinka van der Kooij
- Vrije Universiteit Amsterdam, Department of Human Movement Sciences, Amsterdam, The Netherlands
| | - Jeroen B J Smeets
- Vrije Universiteit Amsterdam, Department of Human Movement Sciences, Amsterdam, The Netherlands
| | - Richard B Ivry
- UC Berkeley, CognAc lab, Berkeley, California, United States
| |
Collapse
|
18
|
Al-Fawakhiri N, Ma A, Taylor JA, Kim OA. Exploring the role of task success in implicit motor adaptation. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.02.01.526533. [PMID: 36778277 PMCID: PMC9915693 DOI: 10.1101/2023.02.01.526533] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 02/06/2023]
Abstract
We learn to improve our motor skills using different forms of feedback: sensory-prediction error, task success, and reward/punishment. While implicit motor adaptation is driven by sensory-prediction errors, recent work has shown that task success modulates this process. Task success is often confounded with reward, so we sought to determine if the effects of these two signals on adaptation can be dissociated. To address this question, we conducted five experiments that isolated implicit learning using error-clamp visuomotor reach adaptation paradigms. Task success was manipulated by changing the size and position of the target relative to the cursor providing visual feedback, and reward expectation was established using monetary cues and auditory feedback. We found that neither monetary cues nor auditory feedback affected implicit adaptation, suggesting that task success influences implicit adaptation via mechanisms distinct from conventional reward-related processes. Additionally, we found that changes in target size, which caused the target to either exclude or fully envelop the cursor, only affected implicit adaptation for a narrow range of error sizes, while jumping the target to overlap with the cursor more reliably and robustly affected implicit adaptation. Taken together, our data indicate that, while task success exerts a small effect on implicit adaptation, these effects are susceptible to methodological variations and unlikely to be mediated by reward.
Collapse
Affiliation(s)
| | - Ambri Ma
- Department of Psychology, Princeton University, Princeton, NJ 08544
| | - Jordan A Taylor
- Department of Psychology, Princeton University, Princeton, NJ 08544
| | - Olivia A Kim
- Department of Psychology, Princeton University, Princeton, NJ 08544
| |
Collapse
|
19
|
Wiegel P, Elizabeth Spedden M, Ramsenthaler C, Malling Beck M, Lundbye-Jensen J. Trial-to-trial Variability and Cortical Processing Depend on Recent Outcomes During Human Reinforcement Motor Learning. Neuroscience 2022; 501:85-102. [PMID: 35970424 DOI: 10.1016/j.neuroscience.2022.08.012] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/24/2022] [Revised: 07/22/2022] [Accepted: 08/09/2022] [Indexed: 10/15/2022]
Abstract
The history of our actions and their outcomes represent important information, informing choices and efficiently guiding future behavior. While unsuccessful (S-) outcomes are expected to lead to more explorative motor states and increased behavioral variability, successful (S+) outcomes are expected to reinforce the use of the previous action. Here, we show that humans attribute different values to previous actions during reinforcement motor learning when they experience S- compared to S+ outcomes. Behavioral variability after an S- outcome is influenced more by the previous outcome than after S+ outcomes. Using electroencephalography, we show that theta band oscillations of the prefrontal cortex are most prominent during changes in two consecutive outcomes, potentially reflecting the need for enhanced cognitive control. Our results suggest that S+ experiences 'overwrite' previous motor states to a greater extent than S- experiences and that modulations in neural oscillations in the prefrontal cortex play a potential role in encoding changes in movement variability state during reinforcement motor learning.
Collapse
Affiliation(s)
- Patrick Wiegel
- Movement & Neuroscience, Department of Nutrition, Exercise & Sports, University of Copenhagen, Denmark.
| | - Meaghan Elizabeth Spedden
- Movement & Neuroscience, Department of Nutrition, Exercise & Sports, University of Copenhagen, Denmark
| | - Christina Ramsenthaler
- Clinic for Palliative Care, University Medical Center Freiburg, Freiburg, Germany; Wolfson Palliative Care Research Centre, Hull & York Medical School, University of Hull, Hull, UK; Cicely Saunders Institute, Department of Palliative Care, Policy & Rehabilitation, King's College London, London, UK
| | - Mikkel Malling Beck
- Movement & Neuroscience, Department of Nutrition, Exercise & Sports, University of Copenhagen, Denmark
| | - Jesper Lundbye-Jensen
- Movement & Neuroscience, Department of Nutrition, Exercise & Sports, University of Copenhagen, Denmark
| |
Collapse
|
20
|
Sporn S, Chen X, Galea JM. The dissociable effects of reward on sequential motor behavior. J Neurophysiol 2022; 128:86-104. [PMID: 35642849 PMCID: PMC9291426 DOI: 10.1152/jn.00467.2021] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/14/2023] Open
Abstract
Reward has consistently been shown to enhance motor behavior; however, its beneficial effects appear to be largely unspecific. For example, reward is associated with both rapid and training-dependent improvements in performance, with a mechanistic account of these effects currently lacking. Here we tested the hypothesis that these distinct reward-based improvements are driven by dissociable reward types: monetary incentive and performance feedback. Whereas performance feedback provides information on how well a motor task has been completed (knowledge of performance), monetary incentive increases the motivation to perform optimally without providing a performance-based learning signal. Experiment 1 showed that groups who received monetary incentive rapidly improved movement times (MTs), using a novel sequential reaching task. In contrast, only groups with correct performance-based feedback showed learning-related improvements. Importantly, pairing both maximized MT performance gains and accelerated movement fusion. Fusion describes an optimization process during which neighboring sequential movements blend together to form singular actions. Results from experiment 2 served as a replication and showed that fusion led to enhanced performance speed while also improving movement efficiency through increased smoothness. Finally, experiment 3 showed that these improvements in performance persist for 24 h even without reward availability. This highlights the dissociable impact of monetary incentive and performance feedback, with their combination maximizing performance gains and leading to stable improvements in the speed and efficiency of sequential actions.NEW & NOTEWORTHY Our work provides a mechanistic framework for how reward influences motor behavior. Specifically, we show that rapid improvements in speed and accuracy are driven by reward presented in the form of money, whereas knowledge of performance through performance feedback leads to training-based improvements. Importantly, combining both maximized performance gains and led to improvements in movement quality through fusion, which describes an optimization process during which sequential movements blend into a single action.
Collapse
Affiliation(s)
- Sebastian Sporn
- 1School of Psychology and Centre for Human Brain Health, grid.6572.6University of Birmingham, Birmingham, United Kingdom,2Department of Clinical and Movement Neuroscience, Queens Square Institute of Neurology, grid.83440.3bUniversity College London, London, United Kingdom
| | - Xiuli Chen
- 1School of Psychology and Centre for Human Brain Health, grid.6572.6University of Birmingham, Birmingham, United Kingdom
| | - Joseph M. Galea
- 1School of Psychology and Centre for Human Brain Health, grid.6572.6University of Birmingham, Birmingham, United Kingdom
| |
Collapse
|
21
|
Ikegami T, Flanagan JR, Wolpert DM. Reach adaption to a visuomotor gain with terminal error feedback involves reinforcement learning. PLoS One 2022; 17:e0269297. [PMID: 35648778 PMCID: PMC9159621 DOI: 10.1371/journal.pone.0269297] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/28/2022] [Accepted: 05/17/2022] [Indexed: 11/18/2022] Open
Abstract
Motor adaptation can be achieved through error-based learning, driven by sensory prediction errors, or reinforcement learning, driven by reward prediction errors. Recent work on visuomotor adaptation has shown that reinforcement learning leads to more persistent adaptation when visual feedback is removed, compared to error-based learning in which continuous visual feedback of the movement is provided. However, there is evidence that error-based learning with terminal visual feedback of the movement (provided at the end of movement) may be driven by both sensory and reward prediction errors. Here we examined the influence of feedback on learning using a visuomotor adaptation task in which participants moved a cursor to a single target while the gain between hand and cursor movement displacement was gradually altered. Different groups received either continuous error feedback (EC), terminal error feedback (ET), or binary reinforcement feedback (success/fail) at the end of the movement (R). Following adaptation we tested generalization to targets located in different directions and found that generalization in the ET group was intermediate between the EC and R groups. We then examined the persistence of adaptation in the EC and ET groups when the cursor was extinguished and only binary reward feedback was provided. Whereas performance was maintained in the ET group, it quickly deteriorated in the EC group. These results suggest that terminal error feedback leads to a more robust form of learning than continuous error feedback. In addition our findings are consistent with the view that error-based learning with terminal feedback involves both error-based and reinforcement learning.
Collapse
Affiliation(s)
- Tsuyoshi Ikegami
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY, United States of America
- Department of Neuroscience, Columbia University, New York, NY, United States of America
- Center for Information and Neural Networks (CiNet), National Institute of Information and Communications Technology, Suita City, Osaka, Japan
- Graduate School of Frontier Biosciences, Osaka University, Osaka, Japan
| | - J. Randall Flanagan
- Department of Psychology, Queen’s University, Kingston, Ontario, Canada
- Centre for Neuroscience Studies, Queen’s University, Kingston, Ontario, Canada
| | - Daniel M. Wolpert
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY, United States of America
- Department of Neuroscience, Columbia University, New York, NY, United States of America
| |
Collapse
|
22
|
Vandevoorde K, Vollenkemper L, Schwan C, Kohlhase M, Schenck W. Using Artificial Intelligence for Assistance Systems to Bring Motor Learning Principles into Real World Motor Tasks. SENSORS 2022; 22:s22072481. [PMID: 35408094 PMCID: PMC9002555 DOI: 10.3390/s22072481] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/18/2022] [Revised: 03/18/2022] [Accepted: 03/20/2022] [Indexed: 11/03/2022]
Abstract
Humans learn movements naturally, but it takes a lot of time and training to achieve expert performance in motor skills. In this review, we show how modern technologies can support people in learning new motor skills. First, we introduce important concepts in motor control, motor learning and motor skill learning. We also give an overview about the rapid expansion of machine learning algorithms and sensor technologies for human motion analysis. The integration between motor learning principles, machine learning algorithms and recent sensor technologies has the potential to develop AI-guided assistance systems for motor skill training. We give our perspective on this integration of different fields to transition from motor learning research in laboratory settings to real world environments and real world motor tasks and propose a stepwise approach to facilitate this transition.
Collapse
|
23
|
Johnson BP, Cohen LG. Reward and plasticity: Implications for neurorehabilitation. HANDBOOK OF CLINICAL NEUROLOGY 2022; 184:331-340. [PMID: 35034746 DOI: 10.1016/b978-0-12-819410-2.00018-7] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
Abstract
Neuroplasticity follows nervous system injury in the presence or absence of rehabilitative treatments. Rehabilitative interventions can be used to modulate adaptive neuroplasticity, reducing motor impairment and improving activities of daily living in patients with brain lesions. Learning principles guide some rehabilitative interventions. While basic science research has shown that reward combined with training enhances learning, this principle has been only recently explored in the context of neurorehabilitation. Commonly used reinforcers may be more or less rewarding depending on the individual or the context in which the task is performed. Studies in healthy humans showed that both reward and punishment can enhance within-session motor performance; but reward, and not punishment, improves consolidation and retention of motor skills. On the other hand, neurorehabilitative training after brain lesions involves complex tasks (e.g., walking and activities of daily living). The contribution of reward to neurorehabilitation is incompletely understood. Here, we discuss recent research on the role of reward in neurorehabilitation and the needed directions of future research.
Collapse
Affiliation(s)
- Brian P Johnson
- Human Cortical Physiology and Neurorehabilitation Section, National Institute of Neurological Disorders and Stroke, National Institutes of Health, Bethesda, MD, United States
| | - Leonardo G Cohen
- Human Cortical Physiology and Neurorehabilitation Section, National Institute of Neurological Disorders and Stroke, National Institutes of Health, Bethesda, MD, United States.
| |
Collapse
|
24
|
Therrien AS, Wong AL. Mechanisms of Human Motor Learning Do Not Function Independently. Front Hum Neurosci 2022; 15:785992. [PMID: 35058767 PMCID: PMC8764186 DOI: 10.3389/fnhum.2021.785992] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/29/2021] [Accepted: 12/13/2021] [Indexed: 11/13/2022] Open
Abstract
Human motor learning is governed by a suite of interacting mechanisms each one of which modifies behavior in distinct ways and rely on different neural circuits. In recent years, much attention has been given to one type of motor learning, called motor adaptation. Here, the field has generally focused on the interactions of three mechanisms: sensory prediction error SPE-driven, explicit (strategy-based), and reinforcement learning. Studies of these mechanisms have largely treated them as modular, aiming to model how the outputs of each are combined in the production of overt behavior. However, when examined closely the results of some studies also suggest the existence of additional interactions between the sub-components of each learning mechanism. In this perspective, we propose that these sub-component interactions represent a critical means through which different motor learning mechanisms are combined to produce movement; understanding such interactions is critical to advancing our knowledge of how humans learn new behaviors. We review current literature studying interactions between SPE-driven, explicit, and reinforcement mechanisms of motor learning. We then present evidence of sub-component interactions between SPE-driven and reinforcement learning as well as between SPE-driven and explicit learning from studies of people with cerebellar degeneration. Finally, we discuss the implications of interactions between learning mechanism sub-components for future research in human motor learning.
Collapse
|
25
|
French MA, Cohen ML, Pohlig RT, Reisman DS. Fluid Cognition Relates to Locomotor Switching in Neurotypical Adults, Not Individuals After Stroke. J Neurol Phys Ther 2022; 46:3-10. [PMID: 34507340 PMCID: PMC8692381 DOI: 10.1097/npt.0000000000000373] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/03/2023]
Abstract
BACKGROUND AND PURPOSE The ability to switch between walking patterns (ie, locomotor switching) is vital for successful community navigation and may be impacted by poststroke impairments. Thus, the purpose of this work was to examine locomotor switching and the relationship between locomotor switching and fluid cognition in individuals after stroke compared with neurotypical adults. METHODS Twenty-nine individuals more than 6 months after stroke and 18 neurotypical adults participated in a 2-day study. On day 1, participants were taught a new walking pattern on the treadmill and then locomotor switching was assessed by instructing participants to switch between the new walking pattern and their usual walking pattern. The change between these 2 patterns was calculated as the switching index. On day 2, the NIH Toolbox Cognition Battery was administered to obtain the Fluid Cognition Composite Score (FCCS), which reflected fluid cognition. The switching index was compared between groups using an analysis of covariance, and the relationship between locomotor switching and fluid cognition was assessed with regression. RESULTS Individuals after stroke had significantly lower switching indexes compared with neurotypical adults (P = 0.03). The regression showed a significant interaction between group and FCCS (P = 0.002), with the FCCS predicting the switching index in neurotypical adults but not in individuals after stroke. DISCUSSION AND CONCLUSIONS Individuals after stroke appear to have deficits in locomotor switching compared with neurotypical adults. The relationship between fluid cognition and locomotor switching was significant in neurotypical adults but not in individuals after stroke. Future work to understand the relationship between specific cognitive domains and locomotor switching is needed (see the Video, Supplemental Digital Content 1, available at: http://links.lww.com/JNPT/A361).
Collapse
Affiliation(s)
- Margaret A French
- Departments of Physical Therapy (M.A.F., D.S.R.) and Communication Sciences and Disorders (M.L.C.), University of Delaware, Newark; Biomechanics and Movement Science Program, University of Delaware, Newark (M.A.F., D.S.R.); and College of Health Sciences Biostatistics Core Facility, University of Delaware, Newark (R.T.P.)
| | | | | | | |
Collapse
|
26
|
de Brouwer AJ, Areshenkoff CN, Rashid MR, Flanagan JR, Poppenk J, Gallivan JP. Human Variation in Error-Based and Reinforcement Motor Learning Is Associated With Entorhinal Volume. Cereb Cortex 2021; 32:3423-3440. [PMID: 34963128 PMCID: PMC9376876 DOI: 10.1093/cercor/bhab424] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/08/2021] [Revised: 11/01/2021] [Accepted: 11/03/2021] [Indexed: 12/31/2022] Open
Abstract
Error-based and reward-based processes are critical for motor learning and are thought to be mediated via distinct neural pathways. However, recent behavioral work in humans suggests that both learning processes can be bolstered by the use of cognitive strategies, which may mediate individual differences in motor learning ability. It has been speculated that medial temporal lobe regions, which have been shown to support motor sequence learning, also support the use of cognitive strategies in error-based and reinforcement motor learning. However, direct evidence in support of this idea remains sparse. Here we first show that better overall learning during error-based visuomotor adaptation is associated with better overall learning during the reward-based shaping of reaching movements. Given the cognitive contribution to learning in both of these tasks, these results support the notion that strategic processes, associated with better performance, drive intersubject variation in both error-based and reinforcement motor learning. Furthermore, we show that entorhinal cortex volume is larger in better learning individuals-characterized across both motor learning tasks-compared with their poorer learning counterparts. These results suggest that individual differences in learning performance during error and reinforcement learning are related to neuroanatomical differences in entorhinal cortex.
Collapse
Affiliation(s)
- Anouk J de Brouwer
- Centre for Neuroscience Studies, Queen’s University, Kingston, ON K7L 3N6, Canada
| | - Corson N Areshenkoff
- Centre for Neuroscience Studies, Queen’s University, Kingston, ON K7L 3N6, Canada,Department of Psychology, Queen’s University, Kingston, ON K7L 3N6, Canada
| | - Mohammad R Rashid
- School of Computing, Queen’s University, Kingston, ON K7L 3N6, Canada
| | - J Randall Flanagan
- Centre for Neuroscience Studies, Queen’s University, Kingston, ON K7L 3N6, Canada,Department of Psychology, Queen’s University, Kingston, ON K7L 3N6, Canada
| | - Jordan Poppenk
- Centre for Neuroscience Studies, Queen’s University, Kingston, ON K7L 3N6, Canada,Department of Psychology, Queen’s University, Kingston, ON K7L 3N6, Canada,School of Computing, Queen’s University, Kingston, ON K7L 3N6, Canada
| | - Jason P Gallivan
- Address correspondence to Jason P. Gallivan, Queen’s University, Kingston, Ontario K7L 3N6, Canada.
| |
Collapse
|
27
|
Yoo AH, Collins AGE. How Working Memory and Reinforcement Learning Are Intertwined: A Cognitive, Neural, and Computational Perspective. J Cogn Neurosci 2021; 34:551-568. [PMID: 34942642 DOI: 10.1162/jocn_a_01808] [Citation(s) in RCA: 21] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022]
Abstract
Reinforcement learning and working memory are two core processes of human cognition and are often considered cognitively, neuroscientifically, and algorithmically distinct. Here, we show that the brain networks that support them actually overlap significantly and that they are less distinct cognitive processes than often assumed. We review literature demonstrating the benefits of considering each process to explain properties of the other and highlight recent work investigating their more complex interactions. We discuss how future research in both computational and cognitive sciences can benefit from one another, suggesting that a key missing piece for artificial agents to learn to behave with more human-like efficiency is taking working memory's role in learning seriously. This review highlights the risks of neglecting the interplay between different processes when studying human behavior (in particular when considering individual differences). We emphasize the importance of investigating these dynamics to build a comprehensive understanding of human cognition.
Collapse
|
28
|
Sidarta A, Komar J, Ostry DJ. Clustering analysis of movement kinematics in reinforcement learning. J Neurophysiol 2021; 127:341-353. [PMID: 34936514 PMCID: PMC8816628 DOI: 10.1152/jn.00229.2021] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/22/2022] Open
Abstract
Reinforcement learning has been used as an experimental model of motor skill acquisition, where at times movements are successful and thus reinforced. One fundamental problem is to understand how humans select exploration over exploitation during learning. The decision could be influenced by factors such as task demands and reward availability. In this study, we applied a clustering algorithm to examine how a change in the accuracy requirements of a task affected the choice of exploration over exploitation. Participants made reaching movements to an unseen target using a planar robot arm and received reward after each successful movement. For one group of participants, the width of the hidden target decreased after every other training block. For a second group, it remained constant. The clustering algorithm was applied to the kinematic data to characterize motor learning on a trial-to-trial basis as a sequence of movements, each belonging to one of the identified clusters. By the end of learning, movement trajectories across all participants converged primarily to a single cluster with the greatest number of successful trials. Within this analysis framework, we defined exploration and exploitation as types of behavior in which two successive trajectories belong to different or similar clusters, respectively. The frequency of each mode of behavior was evaluated over the course of learning. It was found that by reducing the target width, participants used a greater variety of different clusters and displayed more exploration than exploitation. Excessive exploration relative to exploitation was found to be detrimental to subsequent motor learning. NEW & NOTEWORTHY The choice of exploration versus exploitation is a fundamental problem in learning new motor skills through reinforcement. In this study, we employed a data-driven approach to characterize movements on a trial-by-trial basis with an unsupervised clustering algorithm. Using this technique, we found that changes in task demands and, in particular, in the required accuracy of movements, influenced the ratio of exploration to exploitation. This analysis framework provides an attractive tool to investigate mechanisms of explorative and exploitative behavior while studying motor learning.
Collapse
Affiliation(s)
- Ananda Sidarta
- Rehabilitation Research Institute of Singapore, Nanyang Technological University, Singapore
| | - John Komar
- National Institute of Education, Nanyang Technological University, Singapore
| | - David J Ostry
- Department of Psychology, McGill University, Montreal, Quebec, Canada.,Haskins Laboratories, New Haven, CT, United States
| |
Collapse
|
29
|
Edwards EM, Fritz NE, Therrien AS. Cerebellar Dysfunction in Multiple Sclerosis: Considerations for Research and Rehabilitation Therapy. Neurorehabil Neural Repair 2021; 36:103-106. [DOI: 10.1177/15459683211065442] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
Introduction. Cerebellar pathology is common among persons with multiple sclerosis (PwMS). The cerebellum is well recognized for its role in motor control and motor learning and cerebellar pathology in multiple sclerosis is associated with enhanced motor impairment and disability progression. The Problem. To mitigate motor disability progression, PwMS are commonly prescribed exercise and task-specific rehabilitation training. Yet, whether cerebellar dysfunction differentially affects rehabilitation outcomes in this population remains unknown. Furthermore, we lack rehabilitation interventions targeting cerebellar dysfunction. The Solution. Here, we summarize the current understanding of the impact of cerebellar dysfunction on motor control, motor training, and rehabilitation in persons with multiple sclerosis. Recommendations. Additionally, we highlight critical knowledge gaps and propose that these guide future research studying cerebellar dysfunction in persons with multiple sclerosis.
Collapse
Affiliation(s)
- Erin M. Edwards
- Translational Neuroscience Program, Wayne State University, Detroit, MI, USA
| | - Nora E. Fritz
- Translational Neuroscience Program, Wayne State University, Detroit, MI, USA
- Physical Therapy Program, Wayne State University, Detroit, MI, USA
- Department of Neurology, Wayne State University, Detroit, MI, USA
| | | |
Collapse
|
30
|
Leech KA, Roemmich RT, Gordon J, Reisman DS, Cherry-Allen KM. Updates in Motor Learning: Implications for Physical Therapist Practice and Education. Phys Ther 2021; 102:6409654. [PMID: 34718787 PMCID: PMC8793168 DOI: 10.1093/ptj/pzab250] [Citation(s) in RCA: 35] [Impact Index Per Article: 11.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/16/2021] [Revised: 08/12/2021] [Accepted: 10/01/2021] [Indexed: 11/29/2022]
Abstract
Over the past 3 decades, the volume of human motor learning research has grown enormously. As such, the understanding of motor learning (ie, sustained change in motor behavior) has evolved. It has been learned that there are multiple mechanisms through which motor learning occurs, each with distinctive features. These mechanisms include use-dependent, instructive, reinforcement, and sensorimotor adaptation-based motor learning. It is now understood that these different motor learning mechanisms contribute in parallel or in isolation to drive desired changes in movement, and each mechanism is thought to be governed by distinct neural substrates. This expanded understanding of motor learning mechanisms has important implications for physical therapy. It has the potential to facilitate the development of new, more precise treatment approaches that physical therapists can leverage to improve human movement. This Perspective describes scientific advancements related to human motor learning mechanisms and discusses the practical implications of this work for physical therapist practice and education.
Collapse
Affiliation(s)
- Kristan A Leech
- Division of Biokinesiology and Physical Therapy, University of Southern California, Los Angeles, California, USA
- Address all correspondence to Dr Leech at:
| | - Ryan T Roemmich
- Department of Physical Medicine and Rehabilitation, Johns Hopkins University, Baltimore, Maryland, USA
- Center for Movement Studies, Kennedy Krieger Institute, Baltimore, Maryland, USA
| | - James Gordon
- Division of Biokinesiology and Physical Therapy, University of Southern California, Los Angeles, California, USA
| | - Darcy S Reisman
- Physical Therapy Department, University of Delaware, Newark, Delaware, USA
| | - Kendra M Cherry-Allen
- Department of Physical Medicine and Rehabilitation, Johns Hopkins University, Baltimore, Maryland, USA
| |
Collapse
|
31
|
Hamel R, De La Fontaine É, Lepage JF, Bernier PM. Punishments and rewards both modestly impair visuomotor memory retention. Neurobiol Learn Mem 2021; 185:107532. [PMID: 34592470 DOI: 10.1016/j.nlm.2021.107532] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2021] [Revised: 08/31/2021] [Accepted: 09/24/2021] [Indexed: 01/21/2023]
Abstract
While the effects of rewards on memory appear well documented, the effects of punishments remain uncertain. Based on neuroimaging data, this study tested the hypothesis that, as compared to a neutral condition, a context allowing successful punishment avoidance would enhance memory to a similar extent as rewards. In a fully within-subject and counter-balanced design, participants (n = 18) took part in 3 distinct learning sessions during which the delivery of performance-contingent monetary punishments and rewards was manipulated. Specifically, participants had to reach towards visual targets while compensating for a gradually introduced visual deviation. Accuracy at achieving targets was either punished (Hit: "+0$"; Miss: "-0.5$), rewarded (Hit: "+0.5$"; Miss: "-0$"), or associated with neutral binary feedback (Hit: "Hit"; Miss: "Miss"). Retention was assessed through reach aftereffects both immediately and 24 h after initial acquisition. The results disconfirmed the hypothesis by showing that the punishment and reward learning sessions both impaired retention as compared to the neutral session, suggesting that both types of incentives similarly impaired memory formation and consolidation. Two alternative but complementary interpretations are discussed. One interpretation is that the presence of punishments and rewards induced a negative learning context, which - based on neurobiological data - could have been sufficient to interfere with memory formation and consolidation. Another interpretation is that punishments and rewards emphasized the disrupting effects of target hits on implicit learning processes, therefore yielding retention impairments. Altogether, these results suggest that incentives can have counter-productive effects on memory.
Collapse
Affiliation(s)
- R Hamel
- Département de kinanthropologie, Faculté des sciences de l'activité physique, Université de Sherbrooke, Canada; Département de pédiatrie, Faculté de médecine et des sciences de la santé, Université de Sherbrooke, Centre de Recherche du Centre Hospitalier Universitaire de Sherbrooke, Canada
| | - É De La Fontaine
- Département de kinanthropologie, Faculté des sciences de l'activité physique, Université de Sherbrooke, Canada
| | - J F Lepage
- Département de pédiatrie, Faculté de médecine et des sciences de la santé, Université de Sherbrooke, Centre de Recherche du Centre Hospitalier Universitaire de Sherbrooke, Canada
| | - P M Bernier
- Département de kinanthropologie, Faculté des sciences de l'activité physique, Université de Sherbrooke, Canada.
| |
Collapse
|
32
|
Hooyman A, Gordon J, Winstein C. Unique behavioral strategies in visuomotor learning: Hope for the non-learner. Hum Mov Sci 2021; 79:102858. [PMID: 34392189 DOI: 10.1016/j.humov.2021.102858] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/14/2021] [Revised: 05/06/2021] [Accepted: 08/02/2021] [Indexed: 10/20/2022]
Abstract
The existence of individual differences in motor learning capability is well known but the behaviors or strategies that contribute to this variability have been vastly understudied. What performance characteristics distinguish an expert level performer from individuals who experience little to no success, those labeled non-learners? We designed a rule-based visuomotor task which requires identification (discovery) and then exploitation of specific explicit and implicit task components that requires a specific movement pattern, the task rule, for goal achievement. When participants first attempt the task, they are informed about the goal, but are naïve to the task rule. Therefore, the purpose of this experiment is to determine how acquisition of both implicit and explicit task components, the inherent elements of the task rule, reveals differing strategies associated with performance and task success. We test the hypothesis that an examination of performance will reveal sub-groups with varying levels of success. Further, for each subgroup, we expect to find a unique relationship between visual Time-in-Target feedback (a measure of success) and subsequent updating of each task component. Out of 32 non-disabled adults, we identified three distinct sub-groups: (Low Performer/Non-Learner (LP, N = 9), Moderate Performer (MP, N = 12) and High Performer (HP, N = 11)). A quantitative analysis of behavioral patterns reveals three findings: First, the LP sub-group demonstrated significantly lower task success which was associated with difficulty identifying the explicit component of the task. Second, the HP sub-group acquired the two task components in parallel over practice. Third, when both explicit and implicit component performance is plotted across sub-groups, a task component continuum emerges that seamlessly progresses from low to moderate to high performer groups. An exploratory analysis reveals that self-reported level of prior lifetime accumulation of video game and physical activity experience is a significant predictor of individual task performance (R2 = 0.50). In summary, what appears to be a key distinction between varying levels of human rule-based motor learning is the process by which feedback is used to update performance of inherent elements of the task rule. Evidence of a performance continuum and limited prior experience suggests that Low Performer/Non-Learners are generally inexperienced with these kinds of tasks, although the role of genetics and other innate learning capabilities in visuomotor learning is still largely unknown. These findings provoke new research directions toward probing the differential performance strategies associated with expertise and the development of interventions aimed to convert non-learners into learners.
Collapse
Affiliation(s)
- Andrew Hooyman
- School of Biological and Health Systems Engineering, Arizona State University, United States of America.
| | - James Gordon
- Division of Biokinesiology and Physical Therapy, Herman Ostrow School of Dentistry, and Department of Neurology, Keck School of Medicine, University of Southern California, United States of America
| | - Carolee Winstein
- Division of Biokinesiology and Physical Therapy, Herman Ostrow School of Dentistry, and Department of Neurology, Keck School of Medicine, University of Southern California, United States of America
| |
Collapse
|
33
|
van Mastrigt NM, van der Kooij K, Smeets JBJ. Pitfalls in quantifying exploration in reward-based motor learning and how to avoid them. BIOLOGICAL CYBERNETICS 2021; 115:365-382. [PMID: 34341885 PMCID: PMC8382626 DOI: 10.1007/s00422-021-00884-8] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 03/22/2021] [Accepted: 06/23/2021] [Indexed: 06/13/2023]
Abstract
When learning a movement based on binary success information, one is more variable following failure than following success. Theoretically, the additional variability post-failure might reflect exploration of possibilities to obtain success. When average behavior is changing (as in learning), variability can be estimated from differences between subsequent movements. Can one estimate exploration reliably from such trial-to-trial changes when studying reward-based motor learning? To answer this question, we tried to reconstruct the exploration underlying learning as described by four existing reward-based motor learning models. We simulated learning for various learner and task characteristics. If we simply determined the additional change post-failure, estimates of exploration were sensitive to learner and task characteristics. We identified two pitfalls in quantifying exploration based on trial-to-trial changes. Firstly, performance-dependent feedback can cause correlated samples of motor noise and exploration on successful trials, which biases exploration estimates. Secondly, the trial relative to which trial-to-trial change is calculated may also contain exploration, which causes underestimation. As a solution, we developed the additional trial-to-trial change (ATTC) method. By moving the reference trial one trial back and subtracting trial-to-trial changes following specific sequences of trial outcomes, exploration can be estimated reliably for the three models that explore based on the outcome of only the previous trial. Since ATTC estimates are based on a selection of trial sequences, this method requires many trials. In conclusion, if exploration is a binary function of previous trial outcome, the ATTC method allows for a model-free quantification of exploration.
Collapse
Affiliation(s)
- Nina M van Mastrigt
- Department of Human Movement Sciences, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands.
| | - Katinka van der Kooij
- Department of Human Movement Sciences, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
| | - Jeroen B J Smeets
- Department of Human Movement Sciences, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
| |
Collapse
|
34
|
Hill NM, Sukal-Moulton T, Dewald JPA. Between Limb Muscle Co-activation Patterns in the Paretic Arm During Non-paretic Arm Tasks in Hemiparetic Cerebral Palsy. Front Neurosci 2021; 15:666697. [PMID: 34393702 PMCID: PMC8358604 DOI: 10.3389/fnins.2021.666697] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/10/2021] [Accepted: 07/07/2021] [Indexed: 11/16/2022] Open
Abstract
Tasks of daily life require the independent use of the arms and hands. Individuals with hemiparetic cerebral palsy (HCP) often experience difficulty with fine motor tasks demonstrating mirrored movements between the arms. In this study, bilateral muscle activations were quantified during single arm isometric maximum efforts and submaximal reaching tasks. The magnitude and direction of mirrored activation was examined in 14 individuals with HCP and 9 age-matched controls. Participants generated maximum voluntary torques (MVTs) in five different directions and completed ballistic reaches while producing up to 80% of shoulder abduction MVT. Electromyography (EMG) signals were recorded from six upper extremity muscles bilaterally. Participants with HCP demonstrated more mirrored activation when volitionally contracting the non-paretic (NP) arm than the paretic arm (F = 83.543, p < 0.001) in isometric efforts. Increased EMG activation during reach acceleration resulted in a larger increase in rest arm co-activation when reaching with the NP arm compared to the paretic arm in the HCP group (t = 8.425, p < 0.001). Mirrored activation is more pronounced when driving the NP arm and scales with effort level. This directionality of mirroring is indicative of the use of ipsilaterally terminating projections of the corticospinal tract (CST) originating in the non-lesioned hemisphere. Peripheral measures of muscle activation provide insight into the descending pathways available for control of the upper extremity after early unilateral brain injury.
Collapse
Affiliation(s)
- Nayo M Hill
- Department of Physical Therapy and Human Movement Sciences, Northwestern University, Chicago, IL, United States.,Department of Biomedical Engineering, Northwestern University, Evanston, IL, United States
| | - Theresa Sukal-Moulton
- Department of Physical Therapy and Human Movement Sciences, Northwestern University, Chicago, IL, United States.,Department of Pediatrics, Northwestern University, Chicago, IL, United States
| | - Julius P A Dewald
- Department of Physical Therapy and Human Movement Sciences, Northwestern University, Chicago, IL, United States.,Department of Biomedical Engineering, Northwestern University, Evanston, IL, United States.,Department of Physical Medicine and Rehabilitation, Northwestern University, Chicago, IL, United States
| |
Collapse
|
35
|
Orozco SP, Albert ST, Shadmehr R. Adaptive control of movement deceleration during saccades. PLoS Comput Biol 2021; 17:e1009176. [PMID: 34228710 PMCID: PMC8284628 DOI: 10.1371/journal.pcbi.1009176] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/12/2020] [Revised: 07/16/2021] [Accepted: 06/13/2021] [Indexed: 11/19/2022] Open
Abstract
As you read this text, your eyes make saccades that guide your fovea from one word to the next. Accuracy of these movements require the brain to monitor and learn from visual errors. A current model suggests that learning is supported by two different adaptive processes, one fast (high error sensitivity, low retention), and the other slow (low error sensitivity, high retention). Here, we searched for signatures of these hypothesized processes and found that following experience of a visual error, there was an adaptive change in the motor commands of the subsequent saccade. Surprisingly, this adaptation was not uniformly expressed throughout the movement. Rather, after experience of a single error, the adaptive response in the subsequent trial was limited to the deceleration period. After repeated exposure to the same error, the acceleration period commands also adapted, and exhibited resistance to forgetting during set-breaks. In contrast, the deceleration period commands adapted more rapidly, but suffered from poor retention during these same breaks. State-space models suggested that acceleration and deceleration periods were supported by a shared adaptive state which re-aimed the saccade, as well as two separate processes which resembled a two-state model: one that learned slowly and contributed primarily via acceleration period commands, and another that learned rapidly but contributed primarily via deceleration period commands.
Collapse
Affiliation(s)
- Simon P. Orozco
- Laboratory for Computational Motor Control, Dept. of Biomedical Engineering, Johns Hopkins School of Medicine, Baltimore, Maryland, United States of America
| | - Scott T. Albert
- Laboratory for Computational Motor Control, Dept. of Biomedical Engineering, Johns Hopkins School of Medicine, Baltimore, Maryland, United States of America
| | - Reza Shadmehr
- Laboratory for Computational Motor Control, Dept. of Biomedical Engineering, Johns Hopkins School of Medicine, Baltimore, Maryland, United States of America
| |
Collapse
|
36
|
Maresch J, Mudrik L, Donchin O. Measures of explicit and implicit in motor learning: what we know and what we don't. Neurosci Biobehav Rev 2021; 128:558-568. [PMID: 34214514 DOI: 10.1016/j.neubiorev.2021.06.037] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/24/2021] [Revised: 06/21/2021] [Accepted: 06/25/2021] [Indexed: 11/19/2022]
Abstract
Adaptation tasks are a key tool in characterizing the contribution of explicit and implicit processes to sensorimotor learning. However, different assumptions and ideas underlie methods used to measure these processes, leading to inconsistencies between studies. For instance, it is still unclear explicit and implicit combine additively. Cognitive studies of explicit and implicit processes show how non-additivity and bias in measurement can distort results. We argue that to understand explicit and implicit processes in visuomotor adaptation, we need a stronger characterization of the phenomenology and a richer set of models to test it on.
Collapse
Affiliation(s)
- Jana Maresch
- Department of Brain and Cognitive Sciences, Ben Gurion University of the Negev, Be'er Sheva, Israel.
| | - Liad Mudrik
- Sagol School of Neuroscience and School of Psychological Sciences, Tel Aviv University, PO Box 39040, Tel Aviv, 69978, Israel.
| | - Opher Donchin
- Department of Biomedical Engineering and Zlotowski Center for Neuroscience, Ben-Gurion University of the Negev, P.O.B. 653, Be'er Sheva, 8410501, Israel.
| |
Collapse
|
37
|
Yang CS, Cowan NJ, Haith AM. De novo learning versus adaptation of continuous control in a manual tracking task. eLife 2021; 10:e62578. [PMID: 34169838 PMCID: PMC8266385 DOI: 10.7554/elife.62578] [Citation(s) in RCA: 25] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/29/2020] [Accepted: 06/22/2021] [Indexed: 12/20/2022] Open
Abstract
How do people learn to perform tasks that require continuous adjustments of motor output, like riding a bicycle? People rely heavily on cognitive strategies when learning discrete movement tasks, but such time-consuming strategies are infeasible in continuous control tasks that demand rapid responses to ongoing sensory feedback. To understand how people can learn to perform such tasks without the benefit of cognitive strategies, we imposed a rotation/mirror reversal of visual feedback while participants performed a continuous tracking task. We analyzed behavior using a system identification approach, which revealed two qualitatively different components of learning: adaptation of a baseline controller and formation of a new, task-specific continuous controller. These components exhibited different signatures in the frequency domain and were differentially engaged under the rotation/mirror reversal. Our results demonstrate that people can rapidly build a new continuous controller de novo and can simultaneously deploy this process with adaptation of an existing controller.
Collapse
Affiliation(s)
- Christopher S Yang
- Department of Neuroscience, Johns Hopkins UniversityBaltimoreUnited States
| | - Noah J Cowan
- Department of Mechanical Engineering, Laboratory for Computational Sensing and Robotics, Johns Hopkins UniversityBaltimoreUnited States
| | - Adrian M Haith
- Department of Neurology, Johns Hopkins UniversityBaltimoreUnited States
| |
Collapse
|
38
|
Palidis DJ, McGregor HR, Vo A, MacDonald PA, Gribble PL. Null effects of levodopa on reward- and error-based motor adaptation, savings, and anterograde interference. J Neurophysiol 2021; 126:47-67. [PMID: 34038228 DOI: 10.1152/jn.00696.2020] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/22/2022] Open
Abstract
Dopamine signaling is thought to mediate reward-based learning. We tested for a role of dopamine in motor adaptation by administering the dopamine precursor levodopa to healthy participants in two experiments involving reaching movements. Levodopa has been shown to impair reward-based learning in cognitive tasks. Thus, we hypothesized that levodopa would selectively impair aspects of motor adaptation that depend on the reinforcement of rewarding actions. In the first experiment, participants performed two separate tasks in which adaptation was driven either by visual error-based feedback of the hand position or binary reward feedback. We used EEG to measure event-related potentials evoked by task feedback. We hypothesized that levodopa would specifically diminish adaptation and the neural responses to feedback in the reward learning task. However, levodopa did not affect motor adaptation in either task nor did it diminish event-related potentials elicited by reward outcomes. In the second experiment, participants learned to compensate for mechanical force field perturbations applied to the hand during reaching. Previous exposure to a particular force field can result in savings during subsequent adaptation to the same force field or interference during adaptation to an opposite force field. We hypothesized that levodopa would diminish savings and anterograde interference, as previous work suggests that these phenomena result from a reinforcement learning process. However, we found no reliable effects of levodopa. These results suggest that reward-based motor adaptation, savings, and interference may not depend on the same dopaminergic mechanisms that have been shown to be disrupted by levodopa during various cognitive tasks.NEW & NOTEWORTHY Motor adaptation relies on multiple processes including reinforcement of successful actions. Cognitive reinforcement learning is impaired by levodopa-induced disruption of dopamine function. We administered levodopa to healthy adults who participated in multiple motor adaptation tasks. We found no effects of levodopa on any component of motor adaptation. This suggests that motor adaptation may not depend on the same dopaminergic mechanisms as cognitive forms or reinforcement learning that have been shown to be impaired by levodopa.
Collapse
Affiliation(s)
- Dimitrios J Palidis
- Brain and Mind Institute, Western University, London, Ontario, Canada.,Department of Psychology, Western University, London, Ontario, Canada.,Graduate Program in Neuroscience, Schulich School of Medicine and Dentistry, Western University, London, Ontario, Canada
| | - Heather R McGregor
- Department of Applied Physiology and Kinesiology, University of Florida, Gainesville, Florida
| | - Andrew Vo
- Department of Neurology and Neurosurgery, Montreal Neurological Institute, McGill University, Montreal, Quebec, Canada
| | - Penny A MacDonald
- Brain and Mind Institute, Western University, London, Ontario, Canada.,Department of Psychology, Western University, London, Ontario, Canada.,Department of Physiology and Pharmacology, Schulich School of Medicine and Dentistry, Western University, London, Ontario, Canada.,Department of Clinical Neurological Sciences, University of Western Ontario, London, Ontario, Canada
| | - Paul L Gribble
- Brain and Mind Institute, Western University, London, Ontario, Canada.,Department of Psychology, Western University, London, Ontario, Canada.,Department of Physiology and Pharmacology, Schulich School of Medicine and Dentistry, Western University, London, Ontario, Canada.,Haskins Laboratories, New Haven, Connecticut
| |
Collapse
|
39
|
van der Kooij K, van Mastrigt NM, Crowe EM, Smeets JBJ. Learning a reach trajectory based on binary reward feedback. Sci Rep 2021; 11:2667. [PMID: 33514779 PMCID: PMC7846559 DOI: 10.1038/s41598-020-80155-x] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/05/2019] [Accepted: 12/15/2020] [Indexed: 11/09/2022] Open
Abstract
Binary reward feedback on movement success is sufficient for learning some simple sensorimotor mappings in a reaching task, but not for some other tasks in which multiple kinematic factors contribute to performance. The critical condition for learning in more complex tasks remains unclear. Here, we investigate whether reward-based motor learning is possible in a multi-dimensional trajectory matching task and whether simplifying the task by providing feedback on one factor at a time ('factorized feedback') can improve learning. In two experiments, participants performed a trajectory matching task in which learning was measured as a reduction in the error. In Experiment 1, participants matched a straight trajectory slanted in depth. We factorized the task by providing feedback on the slant error, the length error, or on their composite. In Experiment 2, participants matched a curved trajectory, also slanted in depth. In this experiment, we factorized the feedback by providing feedback on the slant error, the curvature error, or on the integral difference between the matched and target trajectory. In Experiment 1, there was anecdotal evidence that participants learnt the multidimensional task. Factorization did not improve learning. In Experiment 2, there was anecdotal evidence the multidimensional task could not be learnt. We conclude that, within a complexity range, multiple kinematic factors can be learnt in parallel.
Collapse
Affiliation(s)
- Katinka van der Kooij
- Department of Human Movement Sciences, Vrije Universiteit Amsterdam, 1081 BT, Amsterdam, The Netherlands.
| | - Nina M van Mastrigt
- Department of Human Movement Sciences, Vrije Universiteit Amsterdam, 1081 BT, Amsterdam, The Netherlands
| | - Emily M Crowe
- Department of Human Movement Sciences, Vrije Universiteit Amsterdam, 1081 BT, Amsterdam, The Netherlands
| | - Jeroen B J Smeets
- Department of Human Movement Sciences, Vrije Universiteit Amsterdam, 1081 BT, Amsterdam, The Netherlands.
| |
Collapse
|
40
|
Prolonged response time helps eliminate residual errors in visuomotor adaptation. Psychon Bull Rev 2021; 28:834-844. [PMID: 33483935 PMCID: PMC8219572 DOI: 10.3758/s13423-020-01865-x] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 12/11/2020] [Indexed: 12/18/2022]
Abstract
One persistent curiosity in visuomotor adaptation tasks is that participants often do not reach maximal performance. This incomplete asymptote has been explained as a consequence of obligatory computations within the implicit adaptation system, such as an equilibrium between learning and forgetting. A body of recent work has shown that in standard adaptation tasks, cognitive strategies operate alongside implicit learning. We reasoned that incomplete learning in adaptation tasks may primarily reflect a speed-accuracy tradeoff on time-consuming motor planning. Across three experiments, we find evidence supporting this hypothesis, showing that hastened motor planning may primarily lead to under-compensation. When an obligatory waiting period was administered before movement start, participants were able to fully counteract imposed perturbations (Experiment 1). Inserting the same delay between trials – rather than during movement planning – did not induce full compensation, suggesting that the motor planning interval influences the learning asymptote (Experiment 2). In the last experiment (Experiment 3), we asked participants to continuously report their movement intent. We show that emphasizing explicit re-aiming strategies (and concomitantly increasing planning time) also lead to complete asymptotic learning. Findings from all experiments support the hypothesis that incomplete adaptation is, in part, the result of an intrinsic speed-accuracy tradeoff, perhaps related to cognitive strategies that require parametric attentional reorienting from the visual target to the goal.
Collapse
|
41
|
French MA, Morton SM, Reisman DS. Use of explicit processes during a visually guided locomotor learning task predicts 24-h retention after stroke. J Neurophysiol 2021; 125:211-222. [PMID: 33174517 PMCID: PMC8087382 DOI: 10.1152/jn.00340.2020] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/08/2020] [Revised: 11/02/2020] [Accepted: 11/03/2020] [Indexed: 01/19/2023] Open
Abstract
Implicit and explicit processes can occur within a single locomotor learning task. The combination of these learning processes may impact how individuals acquire/retain the task. Because these learning processes rely on distinct neural pathways, neurological conditions may selectively impact the processes that occur, thus, impacting learning and retention. Thus, our purpose was to examine the contribution of implicit and explicit processes during a visually guided walking task and characterize the relationship between explicit processes and performance/retention in stroke survivors and age-matched healthy adults. Twenty chronic stroke survivors and twenty healthy adults participated in a 2-day treadmill study. Day 1 included baseline, acquisition1, catch, acquisition2, and immediate retention phases, and day 2 included 24-h retention. During acquisition phases, subjects learned to take a longer step with one leg through distorted visual feedback. During catch and retention phases, visual feedback was removed and subjects were instructed to walk normally (catch) or how they walked during the acquisition phases (retention). Change in step length from baseline to catch represented implicit processes. Change in step length from catch to the end of acquisition2 represented explicit processes. A mixed ANOVA found no difference in the type of learning between groups (P = 0.74). There was a significant relationship between explicit processes and 24-h retention in stroke survivors (r = 0.47, P = 0.04) but not in healthy adults (r = 0.34, P = 0.15). These results suggest that stroke may not affect the underlying learning mechanisms used during locomotor learning, but that these mechanisms impact how well stroke survivors retain the new walking pattern.NEW & NOTEWORTHY This study found that stroke survivors used implicit and explicit processes similar to age-matched healthy adults during a visually guided locomotion learning task. The amount of explicit processes was related to how well stroke survivors retained the new walking pattern but not to how well they performed during the task. This work illustrates the importance of understanding the underlying learning mechanisms to maximize retention of a newly learned motor behavior.
Collapse
Affiliation(s)
- Margaret A French
- Department of Physical Therapy, University of Delaware, Newark, Delaware
- Biomechanics and Movement Science Program, University of Delaware, Newark, Delaware
| | - Susanne M Morton
- Department of Physical Therapy, University of Delaware, Newark, Delaware
- Biomechanics and Movement Science Program, University of Delaware, Newark, Delaware
| | - Darcy S Reisman
- Department of Physical Therapy, University of Delaware, Newark, Delaware
- Biomechanics and Movement Science Program, University of Delaware, Newark, Delaware
| |
Collapse
|
42
|
Ranganathan R, Tomlinson AD, Lokesh R, Lin TH, Patel P. A tale of too many tasks: task fragmentation in motor learning and a call for model task paradigms. Exp Brain Res 2020; 239:1-19. [PMID: 33170341 DOI: 10.1007/s00221-020-05908-6] [Citation(s) in RCA: 14] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2020] [Accepted: 08/17/2020] [Indexed: 12/28/2022]
Abstract
Motor learning encompasses a broad set of phenomena that requires a diverse set of experimental paradigms. However, excessive variation in tasks across studies creates fragmentation that can adversely affect the collective advancement of knowledge. Here, we show that motor learning studies tend toward extreme fragmentation in the choice of tasks, with almost no overlap between task paradigms across studies. We argue that this extreme level of task fragmentation poses serious theoretical and methodological barriers to advancing the field. To address these barriers, we propose the need for developing common 'model' task paradigms which could be widely used across labs. Combined with the open sharing of methods and data, we suggest that these model task paradigms could be an important step in increasing the robustness of the motor learning literature and facilitate the cumulative process of science.
Collapse
Affiliation(s)
- Rajiv Ranganathan
- Department of Kinesiology, Michigan State University, 308 W Circle Dr, East Lansing, MI, 48824, USA.
| | - Aimee D Tomlinson
- Department of Kinesiology, Michigan State University, 308 W Circle Dr, East Lansing, MI, 48824, USA
| | - Rakshith Lokesh
- Department of Kinesiology, Michigan State University, 308 W Circle Dr, East Lansing, MI, 48824, USA
| | - Tzu-Hsiang Lin
- Department of Kinesiology, Michigan State University, 308 W Circle Dr, East Lansing, MI, 48824, USA
| | - Priya Patel
- Department of Kinesiology, Michigan State University, 308 W Circle Dr, East Lansing, MI, 48824, USA
| |
Collapse
|
43
|
Shiller DM, Mitsuya T, Max L. Exposure to Auditory Feedback Delay while Speaking Induces Perceptual Habituation but does not Mitigate the Disruptive Effect of Delay on Speech Auditory-motor Learning. Neuroscience 2020; 446:213-224. [PMID: 32738430 PMCID: PMC7530077 DOI: 10.1016/j.neuroscience.2020.07.041] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2020] [Revised: 05/31/2020] [Accepted: 07/21/2020] [Indexed: 01/17/2023]
Abstract
Perceiving the sensory consequences of our actions with a delay alters the interpretation of these afferent signals and impacts motor learning. For reaching movements, delayed visual feedback of hand position reduces the rate and extent of visuomotor adaptation, but substantial adaptation still occurs. Moreover, the detrimental effect of visual feedback delay on reach motor learning-selectively affecting its implicit component-can be mitigated by prior habituation to the delay. Auditory-motor learning for speech has been reported to be more sensitive to feedback delay, and it remains unknown whether habituation to auditory delay reduces its negative impact on learning. We investigated whether 30 min of exposure to auditory delay during speaking (a) affects the subjective perception of delay, and (b) mitigates its disruptive effect on speech auditory-motor learning. During a speech adaptation task with real-time perturbation of vowel spectral properties, participants heard this frequency-shifted feedback with no delay, 75 ms delay, or 115 ms delay. In the delay groups, 50% of participants had been exposed to the delay throughout a preceding 30-minute block of speaking whereas the remaining participants completed this block without delay. Although habituation minimized awareness of the delay, no improvement in adaptation to the spectral perturbation was observed. Thus, short-term habituation to auditory feedback delays is not effective in reducing the negative impact of delay on speech auditory-motor adaptation. Combined with previous findings, the strong negative effect of delay and the absence of an influence of delay awareness suggest the involvement of predominantly implicit learning mechanisms in speech.
Collapse
Affiliation(s)
- Douglas M Shiller
- École d'orthophonie et d'audiologie, Universite de Montréal, Montreal, Canada; CHU Sainte-Justine Research Centre, Montreal, Canada; Centre for Research on Brain, Language, and Music, Montreal, Canada
| | - Takashi Mitsuya
- Department of Speech and Hearing Sciences, University of Washington, Seattle, USA
| | - Ludo Max
- Department of Speech and Hearing Sciences, University of Washington, Seattle, USA; Haskins Laboratories, New Haven, USA.
| |
Collapse
|
44
|
Kim HE, Avraham G, Ivry RB. The Psychology of Reaching: Action Selection, Movement Implementation, and Sensorimotor Learning. Annu Rev Psychol 2020; 72:61-95. [PMID: 32976728 DOI: 10.1146/annurev-psych-010419-051053] [Citation(s) in RCA: 32] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
The study of motor planning and learning in humans has undergone a dramatic transformation in the 20 years since this journal's last review of this topic. The behavioral analysis of movement, the foundational approach for psychology, has been complemented by ideas from control theory, computer science, statistics, and, most notably, neuroscience. The result of this interdisciplinary approach has been a focus on the computational level of analysis, leading to the development of mechanistic models at the psychological level to explain how humans plan, execute, and consolidate skilled reaching movements. This review emphasizes new perspectives on action selection and motor planning, research that stands in contrast to the previously dominant representation-based perspective of motor programming, as well as an emerging literature highlighting the convergent operation of multiple processes in sensorimotor learning.
Collapse
Affiliation(s)
- Hyosub E Kim
- Departments of Physical Therapy, Psychological and Brain Sciences, and Biomedical Engineering, University of Delaware, Newark, Delaware 19716, USA
| | - Guy Avraham
- Department of Psychology and Helen Wills Neuroscience Institute, University of California, Berkeley, California 94720, USA;
| | - Richard B Ivry
- Department of Psychology and Helen Wills Neuroscience Institute, University of California, Berkeley, California 94720, USA;
| |
Collapse
|
45
|
Reinforcement Signaling Can Be Used to Reduce Elements of Cerebellar Reaching Ataxia. THE CEREBELLUM 2020; 20:62-73. [PMID: 32880848 DOI: 10.1007/s12311-020-01183-x] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/23/2022]
Abstract
Damage to the cerebellum causes a disabling movement disorder called ataxia, which is characterized by poorly coordinated movement. Arm ataxia causes dysmetria (over- or under-shooting of targets) with many corrective movements. As a result, people with cerebellar damage exhibit reaching movements with highly irregular and prolonged movement paths. Cerebellar patients are also impaired in error-based motor learning, which may impede rehabilitation interventions. However, we have recently shown that cerebellar patients can learn a simple reaching task using a binary reinforcement paradigm, in which feedback is based on participants' mean performance. Here, we present a pilot study that examined whether patients with cerebellar damage can use this reinforcement training to learn a more complex motor task-to decrease the path length of their reaches. We compared binary reinforcement training to a control condition of massed practice without reinforcement feedback. In both conditions, participants made target-directed reaches in 3-dimensional space while vision of their movement was occluded. In the reinforcement training condition, reaches with a path length below participants' mean were reinforced with an auditory stimulus at reach endpoint. We found that patients were able to use reinforcement signaling to significantly reduce their reach paths. Massed practice produced no systematic change in patients' reach performance. Overall, our results suggest that binary reinforcement training can improve reaching movements in patients with cerebellar damage and the benefit cannot be attributed solely to repetition or reduced visual control.
Collapse
|
46
|
Hill CM, Stringer M, Waddell DE, Del Arco A. Punishment Feedback Impairs Memory and Changes Cortical Feedback-Related Potentials During Motor Learning. Front Hum Neurosci 2020; 14:294. [PMID: 32848669 PMCID: PMC7419689 DOI: 10.3389/fnhum.2020.00294] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/24/2020] [Accepted: 06/30/2020] [Indexed: 01/29/2023] Open
Abstract
Reward and punishment have demonstrated dissociable effects on motor learning and memory, which suggests that these reinforcers are differently processed by the brain. To test this possibility, we use electroencephalography to record cortical neural activity after the presentation of reward and punishment feedback during a visuomotor rotation task. Participants were randomly placed into Reward, Punishment, or Control groups and performed the task under different conditions to assess the adaptation (learning) and retention (memory) of the motor task. These conditions featured an incongruent position between the cursor and the target, with the cursor trajectory, rotated 30° counterclockwise, requiring the participant to adapt their movement to hit the target. Feedback based on error magnitude was provided during the Adaptation condition in the form of a positive number (Reward) or negative number (Punishment), each representing a monetary gain or loss, respectively. No reinforcement or visual feedback was provided during the No Vision condition (retention). Performance error and event-related potentials (ERPs) time-locked to feedback presentation were calculated for each participant during both conditions. Punishment feedback reduced performance error and promoted faster learning during the Adaptation condition. In contrast, punishment feedback increased performance error during the No Vision condition compared to Control and Reward groups, which suggests a diminished motor memory. Moreover, the Punishment group showed a significant decrease in the amplitude of ERPs during the No Vision condition compared to the Adaptation condition. The amplitude of ERPs did not change in the other two groups. These results suggest that punishment feedback impairs motor retention by altering the neural processing involved in memory encoding. This study provides a neurophysiological underpinning for the dissociative effects of punishment feedback on motor learning.
Collapse
Affiliation(s)
- Christopher M. Hill
- Kinesiology and Physical Education, Northern Illinois University, Dekalb, IL, United States
- Health, Exercise Science, and Recreation Management, University of Mississippi, Oxford, MS, United States
| | - Mason Stringer
- Biomedical Engineering, University of Mississippi, Oxford, MS, United States
| | - Dwight E. Waddell
- Biomedical Engineering, University of Mississippi, Oxford, MS, United States
| | - Alberto Del Arco
- Health, Exercise Science, and Recreation Management, University of Mississippi, Oxford, MS, United States
- Department of Neurobiology and Anatomical Sciences, School of Medicine, University of Mississippi Medical Campus, Jackson, MS, United States
| |
Collapse
|
47
|
Song Y, Adams S, Legon W. Intermittent theta burst stimulation of the right dorsolateral prefrontal cortex accelerates visuomotor adaptation with delayed feedback. Cortex 2020; 129:376-389. [PMID: 32574841 DOI: 10.1016/j.cortex.2020.04.033] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/12/2020] [Revised: 03/09/2020] [Accepted: 04/27/2020] [Indexed: 01/08/2023]
Abstract
Implicit adaptation to visual rotations during fast reaching is a well-recognized function of the cerebellum. However, there is still no well-established understanding of the neural underpinnings that support explicit processes during visuomotor adaptation. We tested the causative involvement of dorsolateral prefrontal cortex (DLPFC) in an adaptive reaching task by employing excitatory intermittent theta burst stimulation (iTBS) to left or right DLPFC during learning to adapt to a sudden large visual rotation with delayed terminal feedback. Spontaneous resting-state electroencephalography (EEG) signals were recorded before and immediately after the administration of iTBS. iTBS to right DLPFC, compared to left DLPFC or control, induced faster adaptation to the rotation and had a greater adjustment of aiming directions in early adaptation trials. Moreover, resting-state functional connectivity of EEG of the frontal cortex after iTBS predicted subsequent adaptation rate. These results suggest a critical role of right DLPFC in supporting explicit learning in the adaptive reaching task.
Collapse
Affiliation(s)
- Yanlong Song
- Department of Neurological Surgery, School of Medicine, University of Virginia, Charlottesville, VA, United States.
| | - Sarah Adams
- Department of Neurological Surgery, School of Medicine, University of Virginia, Charlottesville, VA, United States
| | - Wynn Legon
- Department of Neurological Surgery, School of Medicine, University of Virginia, Charlottesville, VA, United States
| |
Collapse
|
48
|
Intention to learn modulates the impact of reward and punishment on sequence learning. Sci Rep 2020; 10:8906. [PMID: 32483289 PMCID: PMC7264311 DOI: 10.1038/s41598-020-65853-w] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/10/2019] [Accepted: 04/20/2020] [Indexed: 11/09/2022] Open
Abstract
In real-world settings, learning is often characterised as intentional: learners are aware of the goal during the learning process, and the goal of learning is readily dissociable from the awareness of what is learned. Recent evidence has shown that reward and punishment (collectively referred to as valenced feedback) are important factors that influence performance during learning. Presently, however, studies investigating the impact of valenced feedback on skill learning have only considered unintentional learning, and therefore the interaction between intentionality and valenced feedback has not been systematically examined. The present study investigated how reward and punishment impact behavioural performance when participants are instructed to learn in a goal-directed fashion (i.e. intentionally) rather than unintentionally. In Experiment 1, participants performed the serial response time task with reward, punishment, or control feedback and were instructed to ignore the presence of the sequence, i.e., learn unintentionally. Experiment 2 followed the same design, but participants were instructed to intentionally learn the sequence. We found that punishment significantly benefitted performance during learning only when participants learned unintentionally, and we observed no effect of punishment when participants learned intentionally. Thus, the impact of feedback on performance may be influenced by goal of the learner.
Collapse
|
49
|
Schween R, McDougle SD, Hegele M, Taylor JA. Assessing explicit strategies in force field adaptation. J Neurophysiol 2020; 123:1552-1565. [PMID: 32208878 PMCID: PMC7191530 DOI: 10.1152/jn.00427.2019] [Citation(s) in RCA: 25] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/12/2019] [Revised: 02/12/2020] [Accepted: 03/19/2020] [Indexed: 02/06/2023] Open
Abstract
In recent years, it has become increasingly clear that a number of learning processes are at play in visuomotor adaptation tasks. In addition to implicitly adapting to a perturbation, learners can develop explicit knowledge allowing them to select better actions in responding to it. Advances in visuomotor rotation experiments have underscored the important role of such "explicit learning" in shaping adaptation to kinematic perturbations. Yet, in adaptation to dynamic perturbations, its contribution has been largely overlooked. We therefore sought to approach the assessment of explicit learning in adaptation to dynamic perturbations, by developing two novel modifications of a force field experiment. First, we asked learners to abandon any cognitive strategy before selected force channel trials to expose consciously accessible parts of overall learning. Here, learners indeed reduced compensatory force compared with standard Catch channels. Second, we instructed a group of learners to mimic their right hand's adaptation by moving with their naïve left hand. While a control group displayed negligible left hand force compensation, the mimicking group reported forces that approximated right hand adaptation but appeared to under-report the velocity component of the force field in favor of a more position-based component. Our results highlight the viability of explicit learning as a potential contributor to force field adaptation, though the fraction of learning under participants' deliberate control on average remained considerably smaller than that of implicit learning, despite task conditions favoring explicit learning. The methods we employed provide a starting point for investigating the contribution of explicit strategies to force field adaptation.NEW & NOTEWORTHY While the contribution of explicit learning has been increasingly studied in visuomotor adaptation, its contribution to force field adaptation has not been studied extensively. We employed two novel methods to assay explicit learning in a force field adaptation task and found that learners can voluntarily control aspects of compensatory force production and manually report it with their untrained limb. This supports the general viability of the contribution of explicit learning also in force field adaptation.
Collapse
Affiliation(s)
- Raphael Schween
- Neuromotor Behavior Laboratory, Department of Psychology & Sport Science, Justus-Liebig-University Giessen, Giessen, Germany
| | - Samuel D McDougle
- Department of Psychology, University of California, Berkeley, California
| | - Mathias Hegele
- Neuromotor Behavior Laboratory, Department of Psychology & Sport Science, Justus-Liebig-University Giessen, Giessen, Germany
- Center for Mind, Brain and Behavior, Universities of Marburg and Giessen, Marburg, Germany
| | - Jordan A Taylor
- Intelligent Performance and Adaptation Laboratory, Department of Psychology, Princeton University, Princeton, New Jersey
| |
Collapse
|
50
|
Holland P, Codol O, Oxley E, Taylor M, Hamshere E, Joseph S, Huffer L, Galea JM. Domain-Specific Working Memory, But Not Dopamine-Related Genetic Variability, Shapes Reward-Based Motor Learning. J Neurosci 2019; 39:9383-9396. [PMID: 31604835 PMCID: PMC6867814 DOI: 10.1523/jneurosci.0583-19.2019] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/20/2019] [Revised: 09/17/2019] [Accepted: 09/24/2019] [Indexed: 11/21/2022] Open
Abstract
The addition of rewarding feedback to motor learning tasks has been shown to increase the retention of learning, spurring interest in its possible utility for rehabilitation. However, motor tasks using rewarding feedback have repeatedly been shown to lead to great interindividual variability in performance. Understanding the causes of such variability is vital for maximizing the potential benefits of reward-based motor learning. Thus, using a large human cohort of both sexes (n = 241), we examined whether spatial (SWM), verbal, and mental rotation (RWM) working memory capacity and dopamine-related genetic profiles were associated with performance in two reward-based motor tasks. The first task assessed the participant's ability to follow a slowly shifting reward region based on hit/miss (binary) feedback. The second task investigated the participant's capacity to preserve performance with binary feedback after adapting to the rotation with full visual feedback. Our results demonstrate that higher SWM is associated with greater success and an enhanced capacity to reproduce a successful motor action, measured as change in reach angle following reward. In contrast, higher RWM was predictive of an increased propensity to express an explicit strategy when required to make large reach angle adjustments. Therefore, SWM and RWM were reliable, but dissociable, predictors of success during reward-based motor learning. Change in reach direction following failure was also a strong predictor of success rate, although we observed no consistent relationship with working memory. Surprisingly, no dopamine-related genotypes predicted performance. Therefore, working memory capacity plays a pivotal role in determining individual ability in reward-based motor learning.SIGNIFICANCE STATEMENT Reward-based motor learning tasks have repeatedly been shown to lead to idiosyncratic behaviors that cause varying degrees of task success. Yet, the factors determining an individual's capacity to use reward-based feedback are unclear. Here, we assessed a wide range of possible candidate predictors, and demonstrate that domain-specific working memory plays an essential role in determining individual capacity to use reward-based feedback. Surprisingly, genetic variations in dopamine availability were not found to play a role. This is in stark contrast with seminal work in the reinforcement and decision-making literature, which show strong and replicated effects of the same dopaminergic genes in decision-making. Therefore, our results provide novel insights into reward-based motor learning, highlighting a key role for domain-specific working memory capacity.
Collapse
Affiliation(s)
- Peter Holland
- School of Psychology and Centre for Human Brain Health, University of Birmingham, Edgbaston, Birmingham, B15 2TT, United Kingdom
| | - Olivier Codol
- School of Psychology and Centre for Human Brain Health, University of Birmingham, Edgbaston, Birmingham, B15 2TT, United Kingdom
| | - Elizabeth Oxley
- School of Psychology and Centre for Human Brain Health, University of Birmingham, Edgbaston, Birmingham, B15 2TT, United Kingdom
| | - Madison Taylor
- School of Psychology and Centre for Human Brain Health, University of Birmingham, Edgbaston, Birmingham, B15 2TT, United Kingdom
| | - Elizabeth Hamshere
- School of Psychology and Centre for Human Brain Health, University of Birmingham, Edgbaston, Birmingham, B15 2TT, United Kingdom
| | - Shadiq Joseph
- School of Psychology and Centre for Human Brain Health, University of Birmingham, Edgbaston, Birmingham, B15 2TT, United Kingdom
| | - Laura Huffer
- School of Psychology and Centre for Human Brain Health, University of Birmingham, Edgbaston, Birmingham, B15 2TT, United Kingdom
| | - Joseph M Galea
- School of Psychology and Centre for Human Brain Health, University of Birmingham, Edgbaston, Birmingham, B15 2TT, United Kingdom
| |
Collapse
|