Structure learning in action

Novelty seeking is neither necessary nor sufficient for curiosity or creativity, instead both curiosity and creativity may reflect an epistemic drive

Novelty is neither necessary nor sufficient to link curiosity and creativity as stated in the target article. We point out the article's logical shortcomings, outline preconditions that may link curiosity and creativity, and suggest that curiosity and creativity may be expressions of a common epistemic drive.

Feature variability determines specificity and transfer in multiorientation feature detection learning

Historically, in many perceptual learning experiments, only a single stimulus is practiced, and learning is often specific to the trained feature. Our prior work has demonstrated that multi-stimulus learning (e.g., training-plus-exposure procedure) has the potential to achieve generalization. Here, we investigated two important characteristics of multi-stimulus learning, namely, roving and feature variability, and their impacts on multi-stimulus learning and generalization. We adopted a feature detection task in which an oddly oriented target bar differed by 16° from the background bars. The stimulus onset asynchrony threshold between the target and the mask was measured with a staircase procedure. Observers were trained with four target orientation search stimuli, either with a 5° deviation (30°-35°-40°-45°) or with a 45° deviation (30°-75°-120°-165°), and the four reference stimuli were presented in a roving manner. The transfer of learning to the swapped target-background orientations was evaluated after training. We found that multi-stimulus training with a 5° deviation resulted in significant learning improvement, but learning failed to transfer to the swapped target-background orientations. In contrast, training with a 45° deviation slowed learning but produced a significant generalization to swapped orientations. Furthermore, a modified training-plus-exposure procedure, in which observers were trained with four orientation search stimuli with a 5° deviation and simultaneously passively exposed to orientations with high feature variability (45° deviation), led to significant orientation learning generalization. Learning transfer also occurred when the four orientation search stimuli with a 5° deviation were presented in separate blocks. These results help us to specify the condition under which multistimuli learning produces generalization, which holds potential for real-world applications of perceptual learning, such as vision rehabilitation and expert training.

Bayesian reinforcement learning: A basic overview

We and other animals learn because there is some aspect of the world about which we are uncertain. This uncertainty arises from initial ignorance, and from changes in the world that we do not perfectly know; the uncertainty often becomes evident when our predictions about the world are found to be erroneous. The Rescorla-Wagner learning rule, which specifies one way that prediction errors can occasion learning, has been hugely influential as a characterization of Pavlovian conditioning and, through its equivalence to the delta rule in engineering, in a much wider class of learning problems. Here, we review the embedding of the Rescorla-Wagner rule in a Bayesian context that is precise about the link between uncertainty and learning, and thereby discuss extensions to such suggestions as the Kalman filter, structure learning, and beyond, that collectively encompass a wider range of uncertainties and accommodate a wider assortment of phenomena in conditioning.

Brain mechanism of foraging: Reward-dependent synaptic plasticity versus neural integration of values

During foraging behavior, action values are persistently encoded in neural activity and updated depending on the history of choice outcomes. What is the neural mechanism for action value maintenance and updating? Here, we explore two contrasting network models: synaptic learning of action value versus neural integration. We show that both models can reproduce extant experimental data, but they yield distinct predictions about the underlying biological neural circuits. In particular, the neural integrator model but not the synaptic model requires that reward signals are mediated by neural pools selective for action alternatives and their projections are aligned with linear attractor axes in the valuation system. We demonstrate experimentally observable neural dynamical signatures and feasible perturbations to differentiate the two contrasting scenarios, suggesting that the synaptic model is a more robust candidate mechanism. Overall, this work provides a modeling framework to guide future experimental research on probabilistic foraging.

Learning to stand with sensorimotor delays generalizes across directions and from hand to leg effectors

Humans receive sensory information from the past, requiring the brain to overcome delays to perform daily motor skills such as standing upright. Because delays vary throughout the body and change over a lifetime, it would be advantageous to generalize learned control policies of balancing with delays across contexts. However, not all forms of learning generalize. Here, we use a robotic simulator to impose delays into human balance. When delays are imposed in one direction of standing, participants are initially unstable but relearn to balance by reducing the variability of their motor actions and transfer balance improvements to untrained directions. Upon returning to normal standing, aftereffects from learning are observed as small oscillations in control, yet they do not destabilize balance. Remarkably, when participants train to balance with delays using their hand, learning transfers to standing with the legs. Our findings establish that humans use experience to broadly update their neural control to balance with delays.

Curriculum learning inspired by behavioral shaping trains neural networks to adopt animal-like decision making strategies

Recurrent neural networks (RNN) are ubiquitously used in neuroscience to capture both neural dynamics and behaviors of living systems. However, when it comes to complex cognitive tasks, traditional methods for training RNNs can fall short in capturing crucial aspects of animal behavior. To address this challenge, we take inspiration from a commonly used (though rarely appreciated) approach from the experimental neuroscientist's toolkit: behavioral shaping. Our solution leverages task compositionality and models the animal's relevant learning experiences prior to the task. Taking as target a temporal wagering task previously studied in rats, we designed a pretraining curriculum of simpler cognitive tasks that are prerequisites for performing it well. These pretraining tasks are not just simplified versions of the temporal wagering task, but reflect relevant sub-computations. We show that this approach is required for RNNs to adopt similar strategies as rats, including long-timescale inference of latent states, which conventional pretraining approaches fail to capture. Mechanistically, our pretraining supports the development of key dynamical systems features needed for implementing both inference and value-based decision making. Overall, our approach addresses a gap in neural network model training by incorporating inductive biases of animals, which is important when modeling complex behaviors that rely on computational abilities acquired from past experiences.

Proficiency Barrier in Track and Field: Adaptation and Generalization Processes

The literature on motor development and training assumes a hierarchy for learning skills-learning the "fundamentals"-that has yet to be empirically demonstrated. The present study addressed this issue by verifying (1) whether this strong hierarchy (i.e., the proficiency barrier) holds between three fundamental skills and three sport skills and (2) considering different transfer processes (generalization/adaptation) that would occur as a result of the existence of this strong hierarchy. Twenty-seven children/adolescents participated in performing the countermovement jump, standing long jump, leap, high jump, long jump, and hurdle transposition. We identified the proficiency barrier in two pairs of tasks (between the countermovement jump and high jump and between the standing long jump and long jump). Nonetheless, the transfer processes were not related to the proficiency barrier. We conclude that the proposed learning hierarchy holds for some tasks. The underlying reason for this is still unknown.

Transfer of strategic task components across unique tasks that share some common structures

Flexible, adaptive behaviour depends on the application of prior learning to novel contexts (transfer). Transfer can take many forms, but the focus of the present study was on "task schemas"-learning strategies that guide the earliest stages of engaging in a novel task. The central aim was to examine the architecture of task schemas and determine whether strategic task components can expedite learning novel tasks that share some structural components with the training tasks. Groups of participants across two experiments were exposed to different training regimes centred around multiple unique tasks that shared some/all/none of the structural task components (the kinds of stimuli, classifications, and/or responses) but none of the surface features (the specific stimuli, classifications, and/or responses) with the test task (a dot-pattern classification task). Initial test performance was improved (to a degree) in all groups relative to a control group whose training did not include any of the structural components relevant to the test task. The strongest evidence of transfer was found in the motoric, perceptual + categorization, and full schema training groups. This observation indicates that training with some (or all) strategic task components expedited learning of a novel task that shared those components. That is, task schemas were found to be componential and were able to expedite learning a novel task where similar (learning) strategies could be applied to specific elements of the test task.

Age-related impairments and influence of visual feedback when learning to stand with unexpected sensorimotor delays

Background

While standing upright, the brain must accurately accommodate for delays between sensory feedback and self-generated motor commands. Natural aging may limit adaptation to sensorimotor delays due to age-related decline in sensory acuity, neuromuscular capacity and cognitive function. This study examined balance learning in young and older adults as they stood with robot-induced sensorimotor delays.

Methods

A cohort of community dwelling young (mean = 23.6 years, N = 20) and older adults (mean = 70.1 years, N = 20) participated in this balance learning study. Participants stood on a robotic balance simulator which was used to artificially impose a 250 ms delay into their control of standing. Young and older adults practiced to balance with the imposed delay either with or without visual feedback (i.e., eyes open or closed), resulting in four training groups. We assessed their balance behavior and performance (i.e., variability in postural sway and ability to maintain upright posture) before, during and after training. We further evaluated whether training benefits gained in one visual condition transferred to the untrained condition.

Results

All participants, regardless of age or visual training condition, improved their balance performance through training to stand with the imposed delay. Compared to young adults, however, older adults had larger postural oscillations at all stages of the experiments, exhibited less relative learning to balance with the delay and had slower rates of balance improvement. Visual feedback was not required to learn to stand with the imposed delay, but it had a modest effect on the amount of time participants could remain upright. For all groups, balance improvements gained from training in one visual condition transferred to the untrained visual condition.

Conclusion

Our study reveals that while advanced age partially impairs balance learning, the older nervous system maintains the ability to recalibrate motor control to stand with initially destabilizing sensorimotor delays under differing visual feedback conditions.

The emergence of task-relevant representations in a nonlinear decision-making task

This paper describes the relationship between performance in a decision-making task and the emergence of task-relevant representations. Participants learnt two tasks in which the appropriate response depended on multiple relevant stimuli and the underlying stimulus-outcome associations were governed by a latent feature that participants could discover. We divided participants into good and bad performers based on their overall classification rate and computed behavioural accuracy for each feature value. We found that participants with better performance had a better representation of the latent feature space. We then used representation similarity analysis on Electroencephalographic (EEG) data to identify when these representations emerge. We were able to decode task-relevant representations in a time window emerging 700 ms after stimulus presentation, but only for participants with good task performance. Our findings suggest that, in order to make good decisions, it is necessary to create and extract a low-dimensional representation of the task at hand.

Goalkeepers' plasticity during learning of a whole-body visuomotor rotation in a stable or variable environment

Postural adjustments performed in anticipation of uncertain visual events is a common sensorimotor control problem in open sport skills. In this study, we examined how expert soccer goalkeepers and non-athletes learn a whole body visuomotor rotation during postural tracking of constant and variable visual target motions. Twenty-one (21) soccer goalkeepers (18 ± 15 years, 75 ± 12 kg) and 25 age-matched non-athletes (18 ± 12 years, 75 ± 15 kg) practiced lateral weight shifting on a dual force platform while tracking the motion of a constant (11 goalkeepers and 12 non-athletes) or a variable (10 goalkeepers and 13 non-athletes) visual target with provision of online visual feedback (VF). After 40s of tracking (baseline), the visual presentation of the VF signal reversed direction relative to the participant's motion (180° visuo-motor rotation) for 60s (adaptation) and then returned to its veridical direction for another 20s (washout). During adaptation, goalkeepers reduced the spatiotemporal error to baseline levels at an earlier time block (3rd block) compared to non-athletes (6th block), but this difference was significant only for groups tracking of the constant and not the variable target motion. Only the groups tracking the constant target increased the spatiotemporal error during the 1st washout block demonstrating a significant aftereffect. It is concluded that goalkeepers adapt faster to the feedback rotation due to their prior field knowledge of relevant visuomotor transformations in anticipation of deceptive visual cues. This expertise advantage however is present only in a stable visual environment possibly because learning is compromised when tracking uncertain motion cues requiring closed loop control.HighlightsWe examined how expert goalkeepers and non-athletes adopt to a novel whole body visuomotor rotation when tracking a constantly or variably moving targetGoalkeepers adopted faster to the visuomotor rotation than non-athletesExpertise related differences were evident only for groups tracking the constant target motionGroups tracking the variable target motion did not learn the visuomotor rotation.

Generalization of procedural motor sequence learning after a single practice trial

When humans begin learning new motor skills, they typically display early rapid performance improvements. It is not well understood how knowledge acquired during this early skill learning period generalizes to new, related skills. Here, we addressed this question by investigating factors influencing generalization of early learning from a skill A to a different, but related skill B. Early skill generalization was tested over four experiments (N = 2095). Subjects successively learned two related motor sequence skills (skills A and B) over different practice schedules. Skill A and B sequences shared ordinal (i.e., matching keypress locations), transitional (i.e., ordered keypress pairs), parsing rule (i.e., distinct sequence events like repeated keypresses that can be used as a breakpoint for segmenting the sequence into smaller units) structures, or possessed no structure similarities. Results showed generalization for shared parsing rule structure between skills A and B after only a single 10-second practice trial of skill A. Manipulating the initial practice exposure to skill A (1 to 12 trials) and inter-practice rest interval (0-30 s) between skills A and B had no impact on parsing rule structure generalization. Furthermore, this generalization was not explained by stronger sensorimotor mapping between individual keypress actions and their symbolic representations. In contrast, learning from skill A did not generalize to skill B during early learning when the sequences shared only ordinal or transitional structure features. These results document sequence structure that can be very rapidly generalized during initial learning to facilitate generalization of skill.

Neural representations for multi-context visuomotor adaptation and the impact of common representation on multi-task performance: a multivariate decoding approach

The human brain's remarkable motor adaptability stems from the formation of context representations and the use of a common context representation (e.g., an invariant task structure across task contexts) derived from structural learning. However, direct evaluation of context representations and structural learning in sensorimotor tasks remains limited. This study aimed to rigorously distinguish neural representations of visual, movement, and context levels crucial for multi-context visuomotor adaptation and investigate the association between representation commonality across task contexts and adaptation performance using multivariate decoding analysis with fMRI data. Here, we focused on three distinct task contexts, two of which share a rotation structure (i.e., visuomotor rotation contexts with -90° and +90° rotations, in which the mouse cursor's movement was rotated 90 degrees counterclockwise and clockwise relative to the hand-movement direction, respectively) and the remaining one does not (i.e., mirror-reversal context where the horizontal movement of the computer mouse was inverted). This study found that visual representations (i.e., visual direction) were decoded in the occipital area, while movement representations (i.e., hand-movement direction) were decoded across various visuomotor-related regions. These findings are consistent with prior research and the widely recognized roles of those areas. Task-context representations (i.e., either -90° rotation, +90° rotation, or mirror-reversal) were also distinguishable in various brain regions. Notably, these regions largely overlapped with those encoding visual and movement representations. This overlap suggests a potential intricate dependency of encoding visual and movement directions on the context information. Moreover, we discovered that higher task performance is associated with task-context representation commonality, as evidenced by negative correlations between task performance and task-context-decoding accuracy in various brain regions, potentially supporting structural learning. Importantly, despite limited similarities between tasks (e.g., rotation and mirror-reversal contexts), such association was still observed, suggesting an efficient mechanism in the brain that extracts commonalities from different task contexts (such as visuomotor rotations or mirror-reversal) at multiple structural levels, from high-level abstractions to lower-level details. In summary, while illuminating the intricate interplay between visuomotor processing and context information, our study highlights the efficiency of learning mechanisms, thereby paving the way for future exploration of the brain's versatile motor ability.

Schema formation in a neural population subspace underlies learning-to-learn in flexible sensorimotor problem-solving

Learning-to-learn, a progressive speedup of learning while solving a series of similar problems, represents a core process of knowledge acquisition that draws attention in both neuroscience and artificial intelligence. To investigate its underlying brain mechanism, we trained a recurrent neural network model on arbitrary sensorimotor mappings known to depend on the prefrontal cortex. The network displayed an exponential time course of accelerated learning. The neural substrate of a schema emerges within a low-dimensional subspace of population activity; its reuse in new problems facilitates learning by limiting connection weight changes. Our work highlights the weight-driven modifications of the vector field, which determines the population trajectory of a recurrent network and behavior. Such plasticity is especially important for preserving and reusing the learned schema in spite of undesirable changes of the vector field due to the transition to learning a new problem; the accumulated changes across problems account for the learning-to-learn dynamics.

Visuospatial information foraging describes search behavior in learning latent environmental features

In the real world, making sequences of decisions to achieve goals often depends upon the ability to learn aspects of the environment that are not directly perceptible. Learning these so-called latent features requires seeking information about them. Prior efforts to study latent feature learning often used single decisions, used few features, and failed to distinguish between reward-seeking and information-seeking. To overcome this, we designed a task in which humans and monkeys made a series of choices to search for shapes hidden on a grid. On our task, the effects of reward and information outcomes from uncovering parts of shapes could be disentangled. Members of both species adeptly learned the shapes and preferred to select tiles expected to be informative earlier in trials than previously rewarding ones, searching a part of the grid until their outcomes dropped below the average information outcome-a pattern consistent with foraging behavior. In addition, how quickly humans learned the shapes was predicted by how well their choice sequences matched the foraging pattern, revealing an unexpected connection between foraging and learning. This adaptive search for information may underlie the ability in humans and monkeys to learn latent features to support goal-directed behavior in the long run.

Thermodynamic fluctuation theorems govern human sensorimotor learning

The application of thermodynamic reasoning in the study of learning systems has a long tradition. Recently, new tools relating perfect thermodynamic adaptation to the adaptation process have been developed. These results, known as fluctuation theorems, have been tested experimentally in several physical scenarios and, moreover, they have been shown to be valid under broad mathematical conditions. Hence, although not experimentally challenged yet, they are presumed to apply to learning systems as well. Here we address this challenge by testing the applicability of fluctuation theorems in learning systems, more specifically, in human sensorimotor learning. In particular, we relate adaptive movement trajectories in a changing visuomotor rotation task to fully adapted steady-state behavior of individual participants. We find that human adaptive behavior in our task is generally consistent with fluctuation theorem predictions and discuss the merits and limitations of the approach.

Contextual inference in learning and memory

Context is widely regarded as a major determinant of learning and memory across numerous domains, including classical and instrumental conditioning, episodic memory, economic decision-making, and motor learning. However, studies across these domains remain disconnected due to the lack of a unifying framework formalizing the concept of context and its role in learning. Here, we develop a unified vernacular allowing direct comparisons between different domains of contextual learning. This leads to a Bayesian model positing that context is unobserved and needs to be inferred. Contextual inference then controls the creation, expression, and updating of memories. This theoretical approach reveals two distinct components that underlie adaptation, proper and apparent learning, respectively referring to the creation and updating of memories versus time-varying adjustments in their expression. We review a number of extensions of the basic Bayesian model that allow it to account for increasingly complex forms of contextual learning.

Inconsistent attentional contexts impair re-learning following gradual visuomotor adaptation

One of the brain's primary functions is to promote actions in dynamic, distracting environments. Because distractions divert attention away from our primary goals, we learn to maintain accurate actions under sensory and cognitive distractions. Visuomotor adaptation refers to learning processes that restore performance when sensorimotor capacities or environmental conditions are abruptly or gradually altered. Prior work showed that learning to counteract an abrupt perturbation while performing either a single or dual task, referred to as the attentional context, led to better and faster re-learning when the same attentional context was reinstated at recall. This suggested that the attentional context was associated with visuomotor adaptation and used as a contextual cue during recall. The current study investigated whether attentional context was associated with visuomotor adaptation to a gradual perturbation, which limits awareness of errors. During adaptation, participants reached to targets while learning to counteract a visuomotor rotation that increased from 0 to 45 deg by 0.3 deg each trial, with or without performing a secondary task. Re-learning was impaired when the attentional context changed between adaptation and recall (Experiment 1), even compared to when the secondary task was only performed during the early or late half of adaptation (Experiment 2). Changing the secondary task between adaptation and recall did not impair re-learning, indicating that the effect was attentional-context-dependent, rather than task-specific (Experiment 3). These findings further highlight the importance of cognitive factors, such as attention, in visuomotor adaptation and have implications for learning and rehabilitation paradigms.

Tracking the contribution of inductive bias to individualised internal models

Internal models capture the regularities of the environment and are central to understanding how humans adapt to environmental statistics. In general, the correct internal model is unknown to observers, instead they rely on an approximate model that is continually adapted throughout learning. However, experimenters assume an ideal observer model, which captures stimulus structure but ignores the diverging hypotheses that humans form during learning. We combine non-parametric Bayesian methods and probabilistic programming to infer rich and dynamic individualised internal models from response times. We demonstrate that the approach is capable of characterizing the discrepancy between the internal model maintained by individuals and the ideal observer model and to track the evolution of the contribution of the ideal observer model to the internal model throughout training. In particular, in an implicit visuomotor sequence learning task the identified discrepancy revealed an inductive bias that was consistent across individuals but varied in strength and persistence.

Instead of mapping stimuli directly to response, humans and other complex organisms are thought to maintain internal models of the environment. These internal models represent parts of the environment that are most relevant for deciding how to act in a given situation and therefore are key to explaining human behaviour. In behavioural experiments it is often assumed that the internal model in the subject’s brain matches the true model that governs the experiment. However this assumption can be violated due to a variety of reasons, such as insufficient training. Furthermore, the deviation of the internal model from the true model is not uniform across individuals, and therefore it summarizes the subjective beliefs of humans. In this paper, we provide a method to reverse engineer the internal model for individual subjects by analysing trial by trial behavioural measurements such as reaction times. We then track and analyse these reverse engineered models over the course of the experiment to see how participants trade off between an early inductive bias towards Markovian dynamics and the model that reflects the evidence that humans accumulate during learning about the actual statistics of the stimuli.

Statistical Learning in Vision

Vision and learning have long been considered to be two areas of research linked only distantly. However, recent developments in vision research have changed the conceptual definition of vision from a signal-evaluating process to a goal-oriented interpreting process, and this shift binds learning, together with the resulting internal representations, intimately to vision. In this review, we consider various types of learning (perceptual, statistical, and rule/abstract) associated with vision in the past decades and argue that they represent differently specialized versions of the fundamental learning process, which must be captured in its entirety when applied to complex visual processes. We show why the generalized version of statistical learning can provide the appropriate setup for such a unified treatment of learning in vision, what computational framework best accommodates this kind of statistical learning, and what plausible neural scheme could feasibly implement this framework. Finally, we list the challenges that the field of statistical learning faces in fulfilling the promise of being the right vehicle for advancing our understanding of vision in its entirety. Expected final online publication date for the Annual Review of Vision Science, Volume 8 is September 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Learning a new balance task: the influence of prior motor practice on training adaptations

Prior motor experience is thought to aid in the acquisition of new skills. However, studies have shown that balance training does not promote learning of a subsequent balance task. These results stand in contrast to the learning-to-learn paradigm, which is well described for other tasks. We therefore tested if a coordinative affinity between tasks is needed to achieve a learning-to-learn for balance control.Three groups trained different motor tasks during training phase1 (coordination ladder (COOR); bipedal wobble board (2WB); single-leg wobble board (1WB)). During training phase2, all groups trained a tiltboard balance task. Task-specific and transfer effects were evaluated for phase1. A potential learning-to-learn effect was evaluated by comparing the acquisition rates from phase2 for the tiltboard task that was used for training and testing.The results indicate task-specific adaptations after phase1 for 1WB. In contrast, 2WB showed similar improvements than 1WB and COOR (effect sizes: -0.31 to -0.38) when tested on the wobble board with bipedal stance indicating no task-specific improvement for 2WB. For phase2, the linear regression analysis showed larger adaptations for 1WB and 2WB when compared to COOR. This effect implies some uncertainty due to overlapping confidence intervals.Task-specific adaptations after phase1 were found for 1WB but not 2WB. It is discussed that the difficulty of the training task could explain these contrasting results. During phase2, larger adaptations were found for both groups that trained balance tasks during phase1. Thus, despite some uncertainty, prior balance training appears to promote adaptations of a subsequently learned balance task.

Always Pay Attention to Which Model of Motor Learning You Are Using

This critical review considers the epistemological and historical background of the theoretical construct of motor learning for a more differentiated understanding. More than simply reflecting critically on the models that are used to solve problems-whether they are applied in therapy, physical education, or training practice-this review seeks to respond constructively to the recent discussion caused by the replication crisis in life sciences. To this end, an in-depth review of contemporary motor learning approaches is provided, with a pragmatism-oriented clarification of the researcher's intentions on fundamentals (what?), subjects (for whom?), time intervals (when?), and purpose (for what?). The complexity in which the processes of movement acquisition, learning, and refinement take place removes their predictable and linear character and therefore, from an applied point of view, invites a great deal of caution when trying to make generalization claims. Particularly when we attempt to understand and study these phenomena in unpredictable and dynamic contexts, it is recommended that scientists and practitioners seek to better understand the central role that the individual and their situatedness plays in the system. In this way, we will be closer to making a meaningful and authentic contribution to the advancement of knowledge, and not merely for the sake of renaming inventions.

The Pursuit of Effective Working Memory Training: a Pre-registered Randomised Controlled Trial with a Novel Varied Training Protocol

Working memory (WM) training, typically entailing repetitive practice with one or two tasks, has mostly yielded only limited task-specific transfer effects. We developed and tested a new WM training approach where the task paradigm, stimulus type, and predictability of the stimulus sequence were constantly altered during the 4-week training period. We expected that this varied training protocol would generate more extensive transfer by facilitating the use of more general strategies that could be applied to a range of WM tasks. Pre-post transfer effects following varied training (VT group, n = 60) were compared against traditional training (TT group, training a single adaptive WM task, n = 63), and active controls (AC, n = 65). As expected, TT evidenced strong task-specific near transfer as compared to AC. In turn, VT exhibited task-specific near transfer only on one of the measures, and only as compared to the TT group. Critically, no evidence for task-general near transfer or far transfer effects was observed. In sum, the present form of VT failed to demonstrate broader transfer. Nevertheless, as VT has met with success in other cognitive domains, future studies should probe if and how it would be possible to design WM training protocols that promote structural learning where common features of specific tasks would be identified and utilised when selecting strategies for novel memory tasks.

Computational mechanisms of distributed value representations and mixed learning strategies

Learning appropriate representations of the reward environment is challenging in the real world where there are many options, each with multiple attributes or features. Despite existence of alternative solutions for this challenge, neural mechanisms underlying emergence and adoption of value representations and learning strategies remain unknown. To address this, we measure learning and choice during a multi-dimensional probabilistic learning task in humans and trained recurrent neural networks (RNNs) to capture our experimental observations. We find that human participants estimate stimulus-outcome associations by learning and combining estimates of reward probabilities associated with the informative feature followed by those of informative conjunctions. Through analyzing representations, connectivity, and lesioning of the RNNs, we demonstrate this mixed learning strategy relies on a distributed neural code and opponency between excitatory and inhibitory neurons through value-dependent disinhibition. Together, our results suggest computational and neural mechanisms underlying emergence of complex learning strategies in naturalistic settings.

Motor memories of object dynamics are categorically organized

The ability to predict the dynamics of objects, linking applied force to motion, underlies our capacity to perform many of the tasks we carry out on a daily basis. Thus, a fundamental question is how the dynamics of the myriad objects we interact with are organized in memory. Using a custom-built three-dimensional robotic interface that allowed us to simulate objects of varying appearance and weight, we examined how participants learned the weights of sets of objects that they repeatedly lifted. We find strong support for the novel hypothesis that motor memories of object dynamics are organized categorically, in terms of families, based on covariation in their visual and mechanical properties. A striking prediction of this hypothesis, supported by our findings and not predicted by standard associative map models, is that outlier objects with weights that deviate from the family-predicted weight will never be learned despite causing repeated lifting errors.

Reinforcement learning and its connections with neuroscience and psychology

Reinforcement learning methods have recently been very successful at performing complex sequential tasks like playing Atari games, Go and Poker. These algorithms have outperformed humans in several tasks by learning from scratch, using only scalar rewards obtained through interaction with their environment. While there certainly has been considerable independent innovation to produce such results, many core ideas in reinforcement learning are inspired by phenomena in animal learning, psychology and neuroscience. In this paper, we comprehensively review a large number of findings in both neuroscience and psychology that evidence reinforcement learning as a promising candidate for modeling learning and decision making in the brain. In doing so, we construct a mapping between various classes of modern RL algorithms and specific findings in both neurophysiological and behavioral literature. We then discuss the implications of this observed relationship between RL, neuroscience and psychology and its role in advancing research in both AI and brain science.

Bayes, predictive processing, and the cognitive architecture of motor control

Despite their popularity, relatively scant attention has been paid to the upshot of Bayesian and predictive processing models of cognition for views of overall cognitive architecture. Many of these models are hierarchical; they posit generative models at multiple distinct "levels," whose job is to predict the consequences of sensory input at lower levels. I articulate one possible position that could be implied by these models, namely, that there is a continuous hierarchy of perception, cognition, and action control comprising levels of generative models. I argue that this view is not entailed by a general Bayesian/predictive processing outlook. Bayesian approaches are compatible with distinct formats of mental representation. Focusing on Bayesian approaches to motor control, I argue that the junctures between different types of mental representation are places where the transitivity of hierarchical prediction may be broken, and I consider the upshot of this conclusion for broader discussions of cognitive architecture.

Brain oscillatory correlates of visuomotor adaptive learning

Sensorimotor adaptation involves the recalibration of the mapping between motor command and sensory feedback in response to movement errors. Although adaptation operates within individual movements on a trial-to-trial basis, it can also undergo learning when adaptive responses improve over the course of many trials. Brain oscillatory activities related to these "adaptation" and "learning" processes remain unclear. The main reason for this is that previous studies principally focused on the beta band, which confined the outcome message to trial-to-trial adaptation. To provide a wider understanding of adaptive learning, we decoded visuomotor tasks with constant, random or no perturbation from EEG recordings in different bandwidths and brain regions using a multiple kernel learning approach. These different experimental tasks were intended to separate trial-to-trial adaptation from the formation of the new visuomotor mapping across trials. We found changes in EEG power in the post-movement period during the course of the visuomotor-constant rotation task, in particular an increased (i) theta power in prefrontal region, (ii) beta power in supplementary motor area, and (iii) gamma power in motor regions. Classifying the visuomotor task with constant rotation versus those with random or no rotation, we were able to relate power changes in beta band mainly to trial-to-trial adaptation to error while changes in theta band would relate rather to the learning of the new mapping. Altogether, this suggested that there is a tight relationship between modulation of the synchronization of low (theta) and higher (essentially beta) frequency oscillations in prefrontal and sensorimotor regions, respectively, and adaptive learning.

Testing silicone digit extensions as a way to suppress natural sensation to evaluate supplementary tactile feedback

Dexterous use of the hands depends critically on sensory feedback, so it is generally agreed that functional supplementary feedback would greatly improve the use of hand prostheses. Much research still focuses on improving non-invasive feedback that could potentially become available to all prosthesis users. However, few studies on supplementary tactile feedback for hand prostheses demonstrated a functional benefit. We suggest that confounding factors impede accurate assessment of feedback, e.g., testing non-amputee participants that inevitably focus intently on learning EMG control, the EMG’s susceptibility to noise and delays, and the limited dexterity of hand prostheses. In an attempt to assess the effect of feedback free from these constraints, we used silicone digit extensions to suppress natural tactile feedback from the fingertips and thus used the tactile feedback-deprived human hand as an approximation of an ideal feed-forward tool. Our non-amputee participants wore the extensions and performed a simple pick-and-lift task with known weight, followed by a more difficult pick-and-lift task with changing weight. They then repeated these tasks with one of three kinds of audio feedback. The tests were repeated over three days. We also conducted a similar experiment on a person with severe sensory neuropathy to test the feedback without the extensions. Furthermore, we used a questionnaire based on the NASA Task Load Index to gauge the subjective experience. Unexpectedly, we did not find any meaningful differences between the feedback groups, neither in the objective nor the subjective measurements. It is possible that the digit extensions did not fully suppress sensation, but since the participant with impaired sensation also did not improve with the supplementary feedback, we conclude that the feedback failed to provide relevant grasping information in our experiments. The study highlights the complex interaction between task, feedback variable, feedback delivery, and control, which seemingly rendered even rich, high-bandwidth acoustic feedback redundant, despite substantial sensory impairment.

Sensitivity to error during visuomotor adaptation is similarly modulated by abrupt, gradual and random perturbation schedules

It has been suggested that sensorimotor adaptation involves at least two processes (i.e., fast and slow) that differ in retention and error sensitivity. Previous work has shown that repeated exposure to an abrupt force field perturbation results in greater error sensitivity for both the fast and slow processes. While this implies that the faster relearning is associated with increased error sensitivity, it remains unclear what aspects of prior experience modulate error sensitivity. In the present study, we manipulated the initial training using different perturbation schedules, thought to differentially affect fast and slow learning processes based on error magnitude, and then observed what effect prior learning had on subsequent adaptation. During initial training of a visuomotor rotation task, we exposed three groups of participants to either an abrupt, a gradual, or a random perturbation schedule. During a testing session, all three groups were subsequently exposed to an abrupt perturbation schedule. Comparing the two sessions of the control group who experienced repetition of the same perturbation, we found an increased error sensitivity for both processes. We found that the error sensitivity was increased for both the fast and slow processes, with no reliable changes in the retention, for both the gradual and structural learning groups when compared to the first session of the control group. We discuss the findings in the context of how fast and slow learning processes respond to a history of errors.

Learning to stand with unexpected sensorimotor delays

Human standing balance relies on self-motion estimates that are used by the nervous system to detect unexpected movements and enable corrective responses and adaptations in control. These estimates must accommodate for inherent delays in sensory and motor pathways. Here, we used a robotic system to simulate human standing about the ankles in the anteroposterior direction and impose sensorimotor delays into the control of balance. Imposed delays destabilized standing, but through training, participants adapted and re-learned to balance with the delays. Before training, imposed delays attenuated vestibular contributions to balance and triggered perceptions of unexpected standing motion, suggesting increased uncertainty in the internal self-motion estimates. After training, vestibular contributions partially returned to baseline levels and larger delays were needed to evoke perceptions of unexpected standing motion. Through learning, the nervous system accommodates balance sensorimotor delays by causally linking whole-body sensory feedback (initially interpreted as imposed motion) to self-generated balance motor commands.

When standing, neurons in the brain send signals to skeletal muscles so we can adjust our movements to stay upright based on the requirements from the surrounding environment. The long nerves needed to connect our brain, muscles and sensors lead to considerable time delays (up to 160 milliseconds) between sensing the environment and the generation of balance-correcting motor signals. Such delays must be accounted for by the brain so it can adjust how it regulates balance and compensates for unexpected movements.

Aging and neurological disorders can lead to lengthened neural delays, which may result in poorer balance. Computer modeling suggests that we cannot maintain upright balance if delays are longer than 300-340 milliseconds. Directly assessing the destabilizing effects of increased delays in human volunteers can reveal how capable the brain is at adapting to this neurological change.

Using a custom-designed robotic balance simulator, Rasman et al. tested whether healthy volunteers could learn to balance with delays longer than the predicted 300-340 millisecond limit. In a series of experiments, 46 healthy participants stood on the balance simulator which recreates the physical sensations and neural signals for balancing upright based on a computer-driven virtual reality. This unique device enabled Rasman et al. to artificially impose delays by increasing the time between the generation of motor signals and resulting whole-body motion.

The experiments showed that lengthening the delay between motor signals and whole-body motion destabilized upright standing, decreased sensory contributions to balance and led to perceptions of unexpected movements. Over five days of training on the robotic balance simulator, participants regained their ability to balance, which was accompanied by recovered sensory contributions and perceptions of expected standing, despite the imposed delays. When a subset of participants was tested three months later, they were still able to compensate for the increased delay.

The experiments show that the human brain can learn to overcome delays up to 560 milliseconds in the control of balance. This discovery may have important implications for people who develop balance problems because of older age or neurologic diseases like multiple sclerosis. It is possible that robot-assisted training therapies, like the one in this study, could help people overcome their balance impairments.

Presence, flow, and narrative absorption: an interdisciplinary theoretical exploration with a new spatiotemporal integrated model based on predictive processing

Presence, flow, narrative absorption, immersion, transportation, and similar subjective phenomena are studied in many different disciplines, mostly in relation to mediated experiences (books, film, VR, games). Moreover, since real, virtual, or fictional agents are often involved, concepts like identification and state empathy are often linked to engaging media use. Based on a scoping review that identified similarities in the wording of various questionnaire items conceived to measure different phenomena, we categorize items into the most relevant psychological aspects and use this categorization to propose an interdisciplinary systematization. Then, based on a framework of embodied predictive processing, we present a new cognitive model of presence-related phenomena for mediated and non-mediated experiences, integrating spatial and temporal aspects and also considering the role of fiction and media design. Key processes described within the model are: selective attention, enactment of intentions, and interoception. We claim that presence is the state of perceived successful agency of an embodied mind able to correctly enact its predictions. The difference between real-life and simulated experiences ("book problem," "paradox of fiction") lays in the different precision weighting of exteroceptive and interoceptive signals.

Task-relevant and task-irrelevant variability causally shape error-based motor learning

Recent studies of motor learning show dissociable roles of reward- and sensory-prediction errors in updating motor commands by using typical adaptation paradigms where force or visual perturbations are imposed on hand movements. Such classic adaptation paradigms ignore a problem of redundancy inherently embedded in the motor pathways where the central nervous system has to find a unique solution in the high-dimensional motor command space. Computationally, a possible way of solving such a redundancy problem is exploring and updating motor commands based on the learned knowledge of the structures of both the motor pathways and the tasks. However, the effects of task-irrelevant motor command exploration in structure learning and its effects on reward-based and error-based learning have yet to be examined. Here, we used a redundant motor task where participants manipulated a cursor on a monitor screen with their hand gesture movements and then analyzed single-trial motor learning by fitting models consisting of reward-based and error-based learning contributions. We found that the error-based learning rate positively correlated with both task-relevant and task-irrelevant variability, likely reflecting the effect of motor exploration in structure learning. Further modeling results show that the effects of both task-relevant and task-irrelevant variability are simultaneous, and not mediated by one another. In contrast, the reward-based learning rate correlated with neither task-relevant nor task-irrelevant variability. Thus, although not having a direct influence on the task outcome, exploration in the task-irrelevant space late in training has a significant effect on the learning of a task structure used for error-based learning. This suggests that motor exploration, in both task-relevant and task-irrelevant spaces, has an essential role in error-based motor learning in a redundant motor mechanism.

Learning the Abstract General Task Structure in a Rapidly Changing Task Content

The ability to learn abstract generalized structures of tasks is crucial for humans to adapt to changing environments and novel tasks. In a series of five experiments, we investigated this ability using a Rapid Instructed Task Learning paradigm (RITL) comprising short miniblocks, each involving two novel stimulus-response rules. Each miniblock included (a) instructions for the novel stimulus-response rules, (b) a NEXT phase involving a constant (familiar) intervening task (0–5 trials), (c) execution of the newly instructed rules (2 trials). The results show that including a NEXT phase (and hence, a prospective memory demand) led to relatively more robust abstract learning as indicated by increasingly faster responses with experiment progress. Multilevel modeling suggests that the prospective memory demand was just another aspect of the abstract task structure which has been learned.

Sensorimotor knowledge from task-irrelevant feedback contributes to motor learning

Exposure to task-irrelevant feedback leads to perceptual learning, but its effect on motor learning has been understudied. Here, we asked human participants to reach a visual target with a hand-controlled cursor while observing another cursor moving independently in a different direction. Although the task-irrelevant feedback did not change the main task's performance, it elicited robust savings in subsequent adaptation to classical visuomotor rotation perturbation. We demonstrated that the saving effect resulted from a faster formation of strategic learning through a series of experiments, not from gains in the implicit learning process. Furthermore, the saving effect was robust against drastic changes in stimulus features (i.e., rotation size or direction) or task types (i.e., for motor adaptation and skill learning). However, the effect was absent when the task-irrelevant feedback did not carry the visuomotor relationship embedded in visuomotor rotation. Thus, though previous research on perceptual learning has related task-irrelevant feedback to changes in early sensory processes, our findings support its role in acquiring abstract sensorimotor knowledge during motor learning. Motor learning studies have traditionally focused on task-relevant feedback, but our study extends the scope of feedback processes and sheds new light on the dichotomy of explicit and implicit learning in motor adaptation and motor structure learning.NEW & NOTEWORTHY When the motor system faces perturbations, such as fatigue or new environmental changes, it adapts to these changes by voluntarily selecting new action plans or implicitly fine-tuning the control. We show that the action selection part can be enhanced without practice or explicit instruction. We further demonstrate that this enhancement is probably linked to the acquisition of abstract knowledge about the to-be-adapted novel visual feedback. Our findings draw an interesting parallel between motor and perceptual learning by showing that top-down information affects both types of procedural learning.

Multitask learning over shared subspaces

This paper uses constructs from machine learning to define pairs of learning tasks that either shared or did not share a common subspace. Human subjects then learnt these tasks using a feedback-based approach and we hypothesised that learning would be boosted for shared subspaces. Our findings broadly supported this hypothesis with either better performance on the second task if it shared the same subspace as the first, or positive correlations over task performance for shared subspaces. These empirical findings were compared to the behaviour of a Neural Network model trained using sequential Bayesian learning and human performance was found to be consistent with a minimal capacity variant of this model. Networks with an increased representational capacity, and networks without Bayesian learning, did not show these transfer effects. We propose that the concept of shared subspaces provides a useful framework for the experimental study of human multitask and transfer learning.

How does knowledge gained from previous experience affect learning of new tasks? This question of “Transfer Learning” has been addressed by teachers, psychologists, and more recently by researchers in the fields of neural networks and machine learning. Leveraging constructs from machine learning, we designed pairs of learning tasks that either shared or did not share a common subspace. We compared the dynamics of transfer learning in humans with those of a multitask neural network model, finding that human performance was consistent with a minimal capacity variant of the model. Learning was boosted in the second task if the same subspace was shared between tasks. Additionally, accuracy between tasks was positively correlated but only when they shared the same subspace. Our results highlight the roles of subspaces, showing how they could act as a learning boost if shared, and be detrimental if not.

Locomotor illusions are generated by perceptual body-environment organization

While one is walking, the stimulation by one's body forms a structure with the stimulation by the environment. This locomotor array of stimulation corresponds to the human-environment relation that one's body forms with the environment it is moving through. Thus, the perceptual experience of walking may arise from such a locomotor array of stimulation. Humans can also experience walking while they are sitting. In this case, there is no stimulation by one's walking body. Hence, one can experience walking although a basic component of a locomotor array of stimulation is missing. This may be facilitated by perception organizing the sensory input about one's body and environment into a perceptual structure that corresponds to a locomotor array of stimulation. We examined whether locomotor illusions are generated by this perceptual formation of a locomotor structure. We exposed sixteen seated individuals to environmental stimuli that elicited either the perceptual formation of a locomotor structure or that of a control structure. The study participants experienced distinct locomotor illusions when they were presented with environmental stimuli that elicited the perceptual formation of a locomotor structure. They did not experience distinct locomotor illusions when the stimuli instead elicited the perceptual formation of the control structure. These findings suggest that locomotor illusions are generated by the perceptual organization of sensory input about one's body and environment into a locomotor structure. This perceptual body-environment organization elucidates why seated human individuals experience the sensation of walking without any proprioceptive or kinaesthetic stimulation.

An Exploratory Meta-Analytic Review on the Empirical Evidence of Differential Learning as an Enhanced Motor Learning Method

Background: Differential learning (DL) is a motor learning method characterized by high amounts of variability during practice and is claimed to provide the learner with a higher learning rate than other methods. However, some controversy surrounds DL theory, and to date, no overview exists that compares the effects of DL to other motor learning methods.

Objective: To evaluate the effectiveness of DL in comparison to other motor learning methods in the acquisition and retention phase.

Design: Systematic review and exploratory meta-analysis.

Methods: PubMed (MEDLINE), Web of Science, and Google Scholar were searched until February 3, 2020. To be included, (1) studies had to be experiments where the DL group was compared to a control group engaged in a different motor learning method (lack of practice was not eligible), (2) studies had to describe the effects on one or more measures of performance in a skill or movement task, and (3) the study report had to be published as a full paper in a journal or as a book chapter.

Results: Twenty-seven studies encompassing 31 experiments were included. Overall heterogeneity for the acquisition phase (post-pre; I² = 77%) as well as for the retention phase (retention-pre; I² = 79%) was large, and risk of bias was high. The meta-analysis showed an overall small effect size of 0.26 [0.10, 0.42] in the acquisition phase for participants in the DL group compared to other motor learning methods. In the retention phase, an overall medium effect size of 0.61 [0.30, 0.91] was observed for participants in the DL group compared to other motor learning methods.

Discussion/Conclusion: Given the large amount of heterogeneity, limited number of studies, low sample sizes, low statistical power, possible publication bias, and high risk of bias in general, inferences about the effectiveness of DL would be premature. Even though DL shows potential to result in greater average improvements between pre- and post/retention test compared to non-variability-based motor learning methods, more high-quality research is needed before issuing such a statement. For robust comparisons on the relative effectiveness of DL to different variability-based motor learning methods, scarce and inconclusive evidence was found.

Exploring the Presence of Core Skills for Surgical Practice Through Simulation

OBJECTIVE

The ability to simulate procedures in silico has transformed surgical training and practice. Today's simulators, designed for the training of a highly specialized set of procedures, also present a powerful scientific tool for understanding the neural control processes that underpin the learning and application of surgical skills. Here, we examined whether 2 simulators designed for training in 2 different surgical domains could be used to examine the extent to which fundamental sensorimotor skills transcend surgical specialty.

DESIGN, SETTING & PARTICIPANTS

We used a high-fidelity virtual reality dental simulator and a laparoscopic box simulator to record the performance of 3 different groups. The groups comprised dentists, laparoscopic surgeons, and psychologists (each group n = 19).

RESULTS

The results revealed a specialization of performance, with laparoscopic surgeons showing the highest performance on the laparoscopic box simulator, while dentists demonstrated the highest skill levels on the virtual reality dental simulator. Importantly, we also found that a transfer learning effect, with laparoscopic surgeons and dentists showing superior performance to the psychologists on both tasks.

CONCLUSIONS

There are core sensorimotor skills that cut across surgical specialty. We propose that the identification of such fundamental skills could lead to improved training provision prior to specialization.

Examining the Trainability and Transferability of Working-Memory Gating Policies

Internal working memory (WM) gating control policies have been suggested to constitute a critical component of task-sets that can be learned and transferred to very similar task contexts (Bhandari and Badre (Cognition, 172, 33–43, 2018). Here, we attempt to expand these findings, examining whether such control policies can be also trained and transferred to other untrained cognitive control tasks, namely to task switching and AX-CPT. To this end, a context-processing WM task was used for training, allowing to manipulate either input (i.e., top-down selective entry of information into WM) or output (i.e., bottom-up selective retrieval of WM) gating control policies by employing either a context-first (CF) or context-last (CL) task structure, respectively. In this task, two contextual cues were each associated with two different stimuli. In CF condition, each trial began with a contextual cue, determining which of the two subsequent stimuli is target relevant. In contrast, in the CL condition the contextual cue appeared last, preceded by a target and non-target stimulus successively. Participants completed a task switching baseline assessment, followed by one practice and six training blocks with the WM context-processing training task. After completing training, task-switching and AX-CPT transfer blocks were administrated, respectively. As hypothesized, compared to CL training condition, CF training led to improved task-switching performance. However, contrary to our predictions, training type did not influence AX-CPT performance. Taken together, the current results provide further evidence that internal control policies are (1) inherent element of task-sets, also in task switching and (2) independent of S-R mappings. However, these results need to be cautiously interpreted due to baseline differences in task-switching performance between the conditions (overall slower RTs in the CF condition). Importantly though, our results open a new venue for the realm of cognitive enhancement, pointing here for the first time to the potential of control policies training in promoting wider transfer effects.

Reinforcement learning approaches to hippocampus-dependent flexible spatial navigation

Humans and non-human animals show great flexibility in spatial navigation, including the ability to return to specific locations based on as few as one single experience. To study spatial navigation in the laboratory, watermaze tasks, in which rats have to find a hidden platform in a pool of cloudy water surrounded by spatial cues, have long been used. Analogous tasks have been developed for human participants using virtual environments. Spatial learning in the watermaze is facilitated by the hippocampus. In particular, rapid, one-trial, allocentric place learning, as measured in the delayed-matching-to-place variant of the watermaze task, which requires rodents to learn repeatedly new locations in a familiar environment, is hippocampal dependent. In this article, we review some computational principles, embedded within a reinforcement learning framework, that utilise hippocampal spatial representations for navigation in watermaze tasks. We consider which key elements underlie their efficacy, and discuss their limitations in accounting for hippocampus-dependent navigation, both in terms of behavioural performance (i.e. how well do they reproduce behavioural measures of rapid place learning) and neurobiological realism (i.e. how well do they map to neurobiological substrates involved in rapid place learning). We discuss how an actor-critic architecture, enabling simultaneous assessment of the value of the current location and of the optimal direction to follow, can reproduce one-trial place learning performance as shown on watermaze and virtual delayed-matching-to-place tasks by rats and humans, respectively, if complemented with map-like place representations. The contribution of actor-critic mechanisms to delayed-matching-to-place performance is consistent with neurobiological findings implicating the striatum and hippocampo-striatal interaction in delayed-matching-to-place performance, given that the striatum has been associated with actor-critic mechanisms. Moreover, we illustrate that hierarchical computations embedded within an actor-critic architecture may help to account for aspects of flexible spatial navigation. The hierarchical reinforcement learning approach separates trajectory control via a temporal-difference error from goal selection via a goal prediction error and may account for flexible, trial-specific, navigation to familiar goal locations, as required in some arm-maze place memory tasks, although it does not capture one-trial learning of new goal locations, as observed in open field, including watermaze and virtual, delayed-matching-to-place tasks. Future models of one-shot learning of new goal locations, as observed on delayed-matching-to-place tasks, should incorporate hippocampal plasticity mechanisms that integrate new goal information with allocentric place representation, as such mechanisms are supported by substantial empirical evidence.

Single versus dual-rate learning when exposed to Coriolis forces during reaching movements

When we reach for an object during a passive whole body rotation, a tangential Coriolis force is generated on the arm. Yet, within a few trials, the brain adapts to this force so it does not disrupt the reach. Is this adaptation governed by a single-rate or dual-rate learning process? Here, guided by state-space modeling, we studied human reach adaptation in a fully-enclosed rotating room. After 90 pre-rotation reaches (baseline), participants were trained to make 240 to-and-fro reaches while the room rotated at 10 rpm (block A), then performed 6 reaches under opposite room rotation (block B), and subsequently made 100 post-rotation reaches (washout). A control group performed the same paradigm, but without the reaches during rotation block B. Single-rate and dual-rate models can be best dissociated if there would be full un-learning of compensation A during block B, but minimal learning of B. From the perspective of a dual-rate model, the un-learning observed in block B would mainly be caused by the faster state, such that the washout reaches would show retention effects of the slower state, called spontaneous recovery. Alternatively, following a single-rate model, the same state would govern the learning in block A and un-learning in block B, such that the washout reaches mimic the baseline reaches. Our results do not provide clear signs of spontaneous recovery in the washout reaches. Model fits further show that a single-rate process outperformed a dual-rate process. We suggest that a single-rate process underlies Coriolis force reach adaptation, perhaps because these forces relate to familiar body dynamics and are assigned to an internal cause.

Specialization in Hierarchical Learning Systems

Joining multiple decision-makers together is a powerful way to obtain more sophisticated decision-making systems, but requires to address the questions of division of labor and specialization. We investigate in how far information constraints in hierarchies of experts not only provide a principled method for regularization but also to enforce specialization. In particular, we devise an information-theoretically motivated on-line learning rule that allows partitioning of the problem space into multiple sub-problems that can be solved by the individual experts. We demonstrate two different ways to apply our method: (i) partitioning problems based on individual data samples and (ii) based on sets of data samples representing tasks. Approach (i) equips the system with the ability to solve complex decision-making problems by finding an optimal combination of local expert decision-makers. Approach (ii) leads to decision-makers specialized in solving families of tasks, which equips the system with the ability to solve meta-learning problems. We show the broad applicability of our approach on a range of problems including classification, regression, density estimation, and reinforcement learning problems, both in the standard machine learning setup and in a meta-learning setting.

Learning arbitrary stimulus-reward associations for naturalistic stimuli involves transition from learning about features to learning about objects

Most cognitive processes are studied using abstract or synthetic stimuli with specific features to fully control what is presented to subjects. However, recent studies have revealed enhancements of cognitive capacities (such as working memory) when processing naturalistic versus abstract stimuli. Using abstract stimuli constructed from distinct visual features (e.g., color and shape), we have recently shown that human subjects can learn multidimensional stimulus-reward associations via initially estimating reward value of individual features (feature-based learning) before gradually switching to learning about reward value of individual stimuli (object-based learning). Here, we examined whether similar strategies are adopted during learning about naturalistic stimuli that are clearly perceived as objects (instead of a combination of features) and contain both task-relevant and irrelevant features. We found that similar to learning about abstract stimuli, subjects initially adopted feature-based learning more strongly before transitioning to object-based learning. However, there were three key differences between learning about naturalistic and abstract stimuli. First, compared with abstract stimuli, the initial learning strategy was less feature-based for naturalistic stimuli. Second, subjects transitioned to object-based learning faster for naturalistic stimuli. Third, unexpectedly, subjects were more likely to adopt feature-based learning for naturalistic stimuli, both at the steady state and overall. These results suggest that despite the stronger tendency to perceive naturalistic stimuli as objects, which leads to greater likelihood of using object-based learning as the initial strategy and a faster transition to object-based learning, the influence of individual features on learning is stronger for these stimuli such that ultimately the object-based strategy is adopted less. Overall, our findings suggest that feature-based learning is a general initial strategy for learning about reward value of all types of multi-dimensional stimuli.

Dynamical Learning of Dynamics

The ability of humans and animals to quickly adapt to novel tasks is difficult to reconcile with the standard paradigm of learning by slow synaptic weight modification. Here, we show that fixed-weight neural networks can learn to generate required dynamics by imitation. After appropriate weight pretraining, the networks quickly and dynamically adapt to learn new tasks and thereafter continue to achieve them without further teacher feedback. We explain this ability and illustrate it with a variety of target dynamics, ranging from oscillatory trajectories to driven and chaotic dynamical systems.

A Review of Sensory Feedback in Upper-Limb Prostheses From the Perspective of Human Motor Control

This manuscript reviews historical and recent studies that focus on supplementary sensory feedback for use in upper limb prostheses. It shows that the inability of many studies to speak to the issue of meaningful performance improvements in real-life scenarios is caused by the complexity of the interactions of supplementary sensory feedback with other types of feedback along with other portions of the motor control process. To do this, the present manuscript frames the question of supplementary feedback from the perspective of computational motor control, providing a brief review of the main advances in that field over the last 20 years. It then separates the studies on the closed-loop prosthesis control into distinct categories, which are defined by relating the impact of feedback to the relevant components of the motor control framework, and reviews the work that has been done over the last 50+ years in each of those categories. It ends with a discussion of the studies, along with suggestions for experimental construction and connections with other areas of research, such as machine learning.

Exploring disturbance as a force for good in motor learning

Disturbance forces facilitate motor learning, but theoretical explanations for this counterintuitive phenomenon are lacking. Smooth arm movements require predictions (inference) about the force-field associated with a workspace. The Free Energy Principle (FEP) suggests that such 'active inference' is driven by 'surprise'. We used these insights to create a formal model that explains why disturbance might help learning. In two experiments, participants undertook a continuous tracking task where they learned how to move their arm in different directions through a novel 3D force field. We compared baseline performance before and after exposure to the novel field to quantify learning. In Experiment 1, the exposure phases (but not the baseline measures) were delivered under three different conditions: (i) robot haptic assistance; (ii) no guidance; (iii) robot haptic disturbance. The disturbance group showed the best learning as our model predicted. Experiment 2 further tested our FEP inspired model. Assistive and/or disturbance forces were applied as a function of performance (low surprise), and compared to a random error manipulation (high surprise). The random group showed the most improvement as predicted by the model. Thus, motor learning can be conceptualised as a process of entropy reduction. Short term motor strategies (e.g. global impedance) can mitigate unexpected perturbations, but continuous movements require active inference about external force-fields in order to create accurate internal models of the external world (motor learning). Our findings reconcile research on the relationship between noise, variability, and motor learning, and show that information is the currency of motor learning.

Guiding functional reorganization of motor redundancy using a body-machine interface

Background

Body-machine interfaces map movements onto commands to external devices. Redundant motion signals derived from inertial sensors are mapped onto lower-dimensional device commands. Then, the device users face two problems, a) the structural problem of understanding the operation of the interface and b) the performance problem of controlling the external device with high efficiency. We hypothesize that these problems, while being distinct are connected in that aligning the space of body movements with the space encoded by the interface, i.e. solving the structural problem, facilitates redundancy resolution towards increasing efficiency, i.e. solving the performance problem.

Methods

Twenty unimpaired volunteers practiced controlling the movement of a computer cursor by moving their arms. Eight signals from four inertial sensors were mapped onto the two cursor’s coordinates on a screen. The mapping matrix was initialized by asking each user to perform free-form spontaneous upper-limb motions and deriving the two main principal components of the motion signals. Participants engaged in a reaching task for 18 min, followed by a tracking task. One group of 10 participants practiced with the same mapping throughout the experiment, while the other 10 with an adaptive mapping that was iteratively updated by recalculating the principal components based on ongoing movements.

Results

Participants quickly reduced reaching time while also learning to distribute most movement variance over two dimensions. Participants with the fixed mapping distributed movement variance over a subspace that did not match the potent subspace defined by the interface map. In contrast, participant with the adaptive map reduced the difference between the two subspaces, resulting in a smaller amount of arm motions distributed over the null space of the interface map. This, in turn, enhanced movement efficiency without impairing generalization from reaching to tracking.

Conclusions

Aligning the potent subspace encoded by the interface map to the user’s movement subspace guides redundancy resolution towards increasing movement efficiency, with implications for controlling assistive devices. In contrast, in the pursuit of rehabilitative goals, results would suggest that the interface must change to drive the statistics of user’s motions away from the established pattern and toward the engagement of movements to be recovered.

Trial registration

ClinicalTrials.gov, NCT01608438, Registered 16 April 2012.

Self-controlled practice and nudging during structural learning of a novel control interface

Self-controlled practice schedules have been shown to enhance motor learning in several contexts, but their effectiveness in structural learning tasks, where the goal is to eventually learn an underlying structure or rule, is not well known. Here we examined the use of self-controlled practice in a novel control interface requiring structural learning. In addition, we examined the effect of ‘nudging’–i.e., whether altering task difficulty could influence self-selected strategies, and hence facilitate learning. Participants wore four inertial measurement units (IMUs) on their upper body and the goal was to use motions of the upper body to move a screen cursor to different targets presented on the screen. The structure in this task that had to be learned was based on the fact that the signals from the IMUs were linearly mapped to the x- and y- position of the cursor. Participants (N = 62) were split into 3 groups (random, self-selected, nudge) based on whether they had control over the sequence in which they could practice the targets. To test whether participants learned the underlying structure, participants were tested both on the trained targets, as well as novel targets that were not practiced during training. Results showed that during training, the self-selected group showed shorter movement times relative to the random group, and both self-selected and nudge groups adopted a strategy of tending to repeat targets. However, in the test phase, we found no significant differences in task performance between groups, indicating that structural learning was not reliably affected by the type of practice. In addition, nudging participants by adjusting task difficulty did not show any significant benefits to overall learning. These results suggest that although self-controlled practice influenced practice structure and facilitated learning, it did not provide any additional benefits relative to practicing on a random schedule in this task.