Structure learning in action

The computational and neural basis of cognitive control: charted territory and new frontiers

Cognitive control has long been one of the most active areas of computational modeling work in cognitive science. The focus on computational models as a medium for specifying and developing theory predates the PDP books, and cognitive control was not one of the areas on which they focused. However, the framework they provided has injected work on cognitive control with new energy and new ideas. On the occasion of the books' anniversary, we review computational modeling in the study of cognitive control, with a focus on the influence that the PDP approach has brought to bear in this area. Rather than providing a comprehensive review, we offer a framework for thinking about past and future modeling efforts in this domain. We define control in terms of the optimal parameterization of task processing. From this vantage point, the development of control systems in the brain can be seen as responding to the structure of naturalistic tasks, through the filter of the brain systems with which control directly interfaces. This perspective lays open a set of fascinating but difficult research questions, which together define an important frontier for future computational research.

Model averaging, optimal inference, and habit formation

Postulating that the brain performs approximate Bayesian inference generates principled and empirically testable models of neuronal function-the subject of much current interest in neuroscience and related disciplines. Current formulations address inference and learning under some assumed and particular model. In reality, organisms are often faced with an additional challenge-that of determining which model or models of their environment are the best for guiding behavior. Bayesian model averaging-which says that an agent should weight the predictions of different models according to their evidence-provides a principled way to solve this problem. Importantly, because model evidence is determined by both the accuracy and complexity of the model, optimal inference requires that these be traded off against one another. This means an agent's behavior should show an equivalent balance. We hypothesize that Bayesian model averaging plays an important role in cognition, given that it is both optimal and realizable within a plausible neuronal architecture. We outline model averaging and how it might be implemented, and then explore a number of implications for brain and behavior. In particular, we propose that model averaging can explain a number of apparently suboptimal phenomena within the framework of approximate (bounded) Bayesian inference, focusing particularly upon the relationship between goal-directed and habitual behavior.

Prior experience but not size of error improves motor learning on the split-belt treadmill in young children

Children can modify learned motor skills, such as walking, to adapt to new environments. Movement errors in these new situations drive the learning. We used split-belt walking to determine whether size of the error affects the degree of learning. Twenty-two children (aged 2–5 y) walked on the split-belt treadmill on two separate days spaced 1 week apart. Twenty-eight adults served as controls. On Day 1, children experienced an abrupt change in belt speeds (from 1∶1 to 2∶1 differential) resulting in large errors, or a gradual change (same change in speed over 12–15 min), resulting in small errors. Learning was measured by the size of the aftereffect upon return to a 1∶1 differential. On Day 2 (1 week later), the leg on the fast belt was reversed, as was the method of introducing the speed differential. We found that the error size did not affect learning. Unexpectedly, learning was greater on Day 2 compared to Day 1, especially for children under 4 y of age, despite the fact that the task was opposite to that of Day 1, and did not influence learning in adults. Hence, 11 additional children under 4 y of age were tested with belts running at the same speed on Day 1, and with a 2∶1 speed differential (abrupt introduction) on Day 2. Surprisingly, learning was again greater on Day 2. We conclude that size of error during split-belt walking does not affect learning, but experience on a treadmill does, especially for younger children.

Neural model for learning-to-learn of novel task sets in the motor domain

During development, infants learn to differentiate their motor behaviors relative to various contexts by exploring and identifying the correct structures of causes and effects that they can perform; these structures of actions are called task sets or internal models. The ability to detect the structure of new actions, to learn them and to select on the fly the proper one given the current task set is one great leap in infants cognition. This behavior is an important component of the child's ability of learning-to-learn, a mechanism akin to the one of intrinsic motivation that is argued to drive cognitive development. Accordingly, we propose to model a dual system based on (1) the learning of new task sets and on (2) their evaluation relative to their uncertainty and prediction error. The architecture is designed as a two-level-based neural system for context-dependent behavior (the first system) and task exploration and exploitation (the second system). In our model, the task sets are learned separately by reinforcement learning in the first network after their evaluation and selection in the second one. We perform two different experimental setups to show the sensorimotor mapping and switching between tasks, a first one in a neural simulation for modeling cognitive tasks and a second one with an arm-robot for motor task learning and switching. We show that the interplay of several intrinsic mechanisms drive the rapid formation of the neural populations with respect to novel task sets.

Probabilistic brains: knowns and unknowns

There is strong behavioral and physiological evidence that the brain both represents probability distributions and performs probabilistic inference. Computational neuroscientists have started to shed light on how these probabilistic representations and computations might be implemented in neural circuits. One particularly appealing aspect of these theories is their generality: they can be used to model a wide range of tasks, from sensory processing to high-level cognition. To date, however, these theories have only been applied to very simple tasks. Here we discuss the challenges that will emerge as researchers start focusing their efforts on real-life computations, with a focus on probabilistic learning, structural learning and approximate inference.

Extraction of basic movement from whole-body movement, based on gait variability

The aim of this study was to quantify the step-to-step variability (SSV) in speed-variant and speed-invariant movement components of the whole-body gait pattern during running. These separate aspects of variability can be used to gain insight into the neuromuscular control strategies that are engaged during running. Ten healthy, physically active, male recreational athletes performed five treadmill running trials at five different speeds (range: 1.3–4.9 m/sec). The whole-body movement was separated into principal movements (PM) using a principal component analysis. The PMs were split into two groups: a speed-variant group, where the range of motion (amplitude of PMs) changed with running speed; and a speed-invariant group, where the range of motion was constant across various speeds. The step-to-step variability (SSV) of the two groups was then quantified. The absolute SSV was the summed variability across all gait cycles, whereas the relative SSV was the summed variability divided by the magnitude of the movement. The absolute SSV of the speed-variant movements increased with running speed. By contrast, the relative SSV of the speed-variant group (as normalized to the PM amplitude) decreased asymptotically toward a minimal level as running speed increased. Both the absolute and relative SSV of the speed-invariant movements revealed a minimum at 3.1 m/sec. The whole-body gait pattern during running can be subdivided into speed-variant and speed-invariant movements. An interpretation of the SSV based on minimal intervention theory suggests that speed-variant movements are more tightly controlled, as evidenced by a lower degree of variability compared to the speed-invariant movements.

Augmented visual, auditory, haptic, and multimodal feedback in motor learning: a review

It is generally accepted that augmented feedback, provided by a human expert or a technical display, effectively enhances motor learning. However, discussion of the way to most effectively provide augmented feedback has been controversial. Related studies have focused primarily on simple or artificial tasks enhanced by visual feedback. Recently, technical advances have made it possible also to investigate more complex, realistic motor tasks and to implement not only visual, but also auditory, haptic, or multimodal augmented feedback. The aim of this review is to address the potential of augmented unimodal and multimodal feedback in the framework of motor learning theories. The review addresses the reasons for the different impacts of feedback strategies within or between the visual, auditory, and haptic modalities and the challenges that need to be overcome to provide appropriate feedback in these modalities, either in isolation or in combination. Accordingly, the design criteria for successful visual, auditory, haptic, and multimodal feedback are elaborated.

Adaptation paths to novel motor tasks are shaped by prior structure learning

After extensive practice with motor tasks sharing structural similarities (e.g., different dancing movements, or different sword techniques), new tasks of the same type can be learned faster. According to the recent "structure learning" hypothesis (Braun et al., 2009a), such rapid generalization of related motor skills relies on learning the dynamic and kinematic relationships shared by this set of skills. As a consequence, motor adaptation becomes constrained, effectively leading to a dimensionality reduction of the learning problem; at the same time, adaptation to tasks lying outside the structure becomes biased toward the structure. We tested these predictions by investigating how previously learned structures influence subsequent motor adaptation. Human subjects were making reaching movements in 3D virtual reality, experiencing perturbations either in the vertical or in the horizontal plane. Perturbations were either visuomotor rotations of varying angle or velocity-dependent forces of varying strength. We found that, after extensive training with both kinematic or dynamic perturbations, adaptation to unpracticed, diagonal, perturbations happened along the previously learned structure (vertical or horizontal), and resulting adaptation trajectories were curved. This effect is robust, can be observed on the single-subject level, and occurs during adaptation both within and across trials. Additionally, we demonstrate that structure learning changes involuntary visuomotor reflexes and therefore is not exclusively a high-level cognitive phenomenon.

Attention, learning, and the value of information

Despite many studies on selective attention, fundamental questions remain about its nature and neural mechanisms. Here I draw from the animal and machine learning fields that describe attention as a mechanism for active learning and uncertainty reduction and explore the implications of this view for understanding visual attention and eye movement control. I propose that a closer integration of these different views has the potential greatly to expand our understanding of oculomotor control and our ability to use this system as a window into high level but poorly understood cognitive functions, including the capacity for curiosity and exploration and for inferring internal models of the external world.

Self-organizing spiking neural model for learning fault-tolerant spatio-motor transformations

In this paper, we present a spiking neural model that learns spatio-motor transformations. The model is in the form of a multilayered architecture consisting of integrate and fire neurons and synapses that employ spike-timing-dependent plasticity learning rule to enable the learning of such transformations. We developed a simple 2-degree-of-freedom robot-based reaching task which involves the learning of a nonlinear function. Computer simulations demonstrate the capability of such a model for learning the forward and inverse kinematics for such a task and hence to learn spatio-motor transformations. The interesting aspect of the model is its capacity to be tolerant to partial absence of sensory or motor inputs at various stages of learning. We believe that such a model lays the foundation for learning other complex functions and transformations in real-world scenarios.

Structural learning in feedforward and feedback control

For smooth and efficient motor control, the brain needs to make fast corrections during the movement to resist possible perturbations. It also needs to adapt subsequent movements to improve future performance. It is important that both feedback corrections and feedforward adaptation need to be made based on noisy and often ambiguous sensory data. Therefore, the initial response of the motor system, both for online corrections and adaptive responses, is guided by prior assumptions about the likely structure of perturbations. In the context of correcting and adapting movements perturbed by a force field, we asked whether these priors are hard wired or whether they can be modified through repeated exposure to differently shaped force fields. We found that both feedback corrections to unexpected perturbations and feedforward adaptation to a new force field changed, such that they were appropriate to counteract the type of force field that participants had experienced previously. We then investigated whether these changes were driven by a common mechanism or by two separate mechanisms. Participants experienced force fields that were either temporally consistent, causing sustained adaptation, or temporally inconsistent, causing little overall adaptation. We found that the consistent force fields modified both feedback and feedforward responses. In contrast, the inconsistent force field modified the temporal shape of feedback corrections but not of the feedforward adaptive response. These results indicate that responses to force perturbations can be modified in a structural manner and that these modifications are at least partly dissociable for feedback and feedforward control.

Inferring relevance in a changing world

Reinforcement learning models of human and animal learning usually concentrate on how we learn the relationship between different stimuli or actions and rewards. However, in real-world situations “stimuli” are ill-defined. On the one hand, our immediate environment is extremely multidimensional. On the other hand, in every decision making scenario only a few aspects of the environment are relevant for obtaining reward, while most are irrelevant. Thus a key question is how do we learn these relevant dimensions, that is, how do we learn what to learn about? We investigated this process of “representation learning” experimentally, using a task in which one stimulus dimension was relevant for determining reward at each point in time. As in real life situations, in our task the relevant dimension can change without warning, adding ever-present uncertainty engendered by a constantly changing environment. We show that human performance on this task is better described by a suboptimal strategy based on selective attention and serial-hypothesis-testing rather than a normative strategy based on probabilistic inference. From this, we conjecture that the problem of inferring relevance in general scenarios is too computationally demanding for the brain to solve optimally. As a result the brain utilizes approximations, employing these even in simplified scenarios in which optimal representation learning is tractable, such as the one in our experiment.

Motor adaptation training for faster relearning

Adaptation is an error-driven motor learning process that can account for predictable changes in the environment (e.g., walking on ice) or in ourselves (e.g., injury). Our ability to recall and build upon adapted motor patterns across days is essential to this learning process. We investigated how different training paradigms affect the day-to-day memory of an adapted walking pattern. Healthy human adults walked on a split-belt treadmill, and returned the following day to assess recall, relearning rate, and performance. In the first experiment, one group adapted and de-adapted (i.e., washed-out the learning) several times on day 1 to practice the initial stage of learning where errors are large; another group adapted only one time and then practiced in the adapted ("learned") state where errors were small. On day 2, they performed washout trials before readapting. The group that repeatedly practiced the initial portion of adaptation where errors are large showed the fastest relearning on the second day. In fact, the memory was nearly as strong as that of a third group that was left overnight in the adapted state and was not washed-out before reexposure on the second day. This demonstrates that alternating exposures to early adaptation and washout can enhance readaptation. In the second experiment, we tested whether the opposite split-belt pattern interferes with day 2 relearning. Surprisingly, it did not, and instead was similar to practicing in the adapted state. These results show that the structure of the initial phase of learning influences the ease of motor relearning.

Facilitation of learning induced by both random and gradual visuomotor task variation

Motor task variation has been shown to be a key ingredient in skill transfer, retention, and structural learning. However, many studies only compare training of randomly varying tasks to either blocked or null training, and it is not clear how experiencing different nonrandom temporal orderings of tasks might affect the learning process. Here we study learning in human subjects who experience the same set of visuomotor rotations, evenly spaced between −60° and +60°, either in a random order or in an order in which the rotation angle changed gradually. We compared subsequent learning of three test blocks of +30°→−30°→+30° rotations. The groups that underwent either random or gradual training showed significant (P < 0.01) facilitation of learning in the test blocks compared with a control group who had not experienced any visuomotor rotations before. We also found that movement initiation times in the random group during the test blocks were significantly (P < 0.05) lower than for the gradual or the control group. When we fit a state-space model with fast and slow learning processes to our data, we found that the differences in performance in the test block were consistent with the gradual or random task variation changing the learning and retention rates of only the fast learning process. Such adaptation of learning rates may be a key feature of ongoing meta-learning processes. Our results therefore suggest that both gradual and random task variation can induce meta-learning and that random learning has an advantage in terms of shorter initiation times, suggesting less reliance on cognitive processes.

Sensory motor remapping of space in human-machine interfaces

Studies of adaptation to patterns of deterministic forces have revealed the ability of the motor control system to form and use predictive representations of the environment. These studies have also pointed out that adaptation to novel dynamics is aimed at preserving the trajectories of a controlled endpoint, either the hand of a subject or a transported object. We review some of these experiments and present more recent studies aimed at understanding how the motor system forms representations of the physical space in which actions take place. An extensive line of investigations in visual information processing has dealt with the issue of how the Euclidean properties of space are recovered from visual signals that do not appear to possess these properties. The same question is addressed here in the context of motor behavior and motor learning by observing how people remap hand gestures and body motions that control the state of an external device. We present some theoretical considerations and experimental evidence about the ability of the nervous system to create novel patterns of coordination that are consistent with the representation of extrapersonal space. We also discuss the perspective of endowing human-machine interfaces with learning algorithms that, combined with human learning, may facilitate the control of powered wheelchairs and other assistive devices.

Principles of sensorimotor learning

The exploits of Martina Navratilova and Roger Federer represent the pinnacle of motor learning. However, when considering the range and complexity of the processes that are involved in motor learning, even the mere mortals among us exhibit abilities that are impressive. We exercise these abilities when taking up new activities - whether it is snowboarding or ballroom dancing - but also engage in substantial motor learning on a daily basis as we adapt to changes in our environment, manipulate new objects and refine existing skills. Here we review recent research in human motor learning with an emphasis on the computational mechanisms that are involved.

A single-rate context-dependent learning process underlies rapid adaptation to familiar object dynamics

Motor learning has been extensively studied using dynamic (force-field) perturbations. These induce movement errors that result in adaptive changes to the motor commands. Several state-space models have been developed to explain how trial-by-trial errors drive the progressive adaptation observed in such studies. These models have been applied to adaptation involving novel dynamics, which typically occurs over tens to hundreds of trials, and which appears to be mediated by a dual-rate adaptation process. In contrast, when manipulating objects with familiar dynamics, subjects adapt rapidly within a few trials. Here, we apply state-space models to familiar dynamics, asking whether adaptation is mediated by a single-rate or dual-rate process. Previously, we reported a task in which subjects rotate an object with known dynamics. By presenting the object at different visual orientations, adaptation was shown to be context-specific, with limited generalization to novel orientations. Here we show that a multiple-context state-space model, with a generalization function tuned to visual object orientation, can reproduce the time-course of adaptation and de-adaptation as well as the observed context-dependent behavior. In contrast to the dual-rate process associated with novel dynamics, we show that a single-rate process mediates adaptation to familiar object dynamics. The model predicts that during exposure to the object across multiple orientations, there will be a degree of independence for adaptation and de-adaptation within each context, and that the states associated with all contexts will slowly de-adapt during exposure in one particular context. We confirm these predictions in two new experiments. Results of the current study thus highlight similarities and differences in the processes engaged during exposure to novel versus familiar dynamics. In both cases, adaptation is mediated by multiple context-specific representations. In the case of familiar object dynamics, however, the representations can be engaged based on visual context, and are updated by a single-rate process.

Skillful object manipulation is an essential feature of human behavior. How humans process and represent information associated with objects is thus a fundamental question in neuroscience. Here, we examine the representation of the mechanical properties of objects which define the mapping between the forces applied to an object and the motion that results. Knowledge of this mapping, which can change depending on the orientation with which an object is grasped, is essential for skillful manipulation. Subjects performed a virtual object manipulation task by grasping the handle of a novel robotic interface which simulated the dynamics of a familiar object which could be presented at different orientations. Using this task, we show that adaptation to the properties of a particular object is extremely rapid, and that such adaptation is confined to the specific orientation at which the object is experienced. Moreover, the pattern of adaptation observed when the orientation of the object and its mechanical properties were changed from trial-to-trial was reproduced by a model which included multiple representations and a generalization function tuned for object orientation. These results suggest that the skillful manipulation of objects with familiar dynamics is mediated by multiple context-specific representations.

A comprehensive model of audiovisual perception: both percept and temporal dynamics

The sparse information captured by the sensory systems is used by the brain to apprehend the environment, for example, to spatially locate the source of audiovisual stimuli. This is an ill-posed inverse problem whose inherent uncertainty can be solved by jointly processing the information, as well as introducing constraints during this process, on the way this multisensory information is handled. This process and its result - the percept - depend on the contextual conditions perception takes place in. To date, perception has been investigated and modeled on the basis of either one of two of its dimensions: the percept or the temporal dynamics of the process. Here, we extend our previously proposed audiovisual perception model to predict both these dimensions to capture the phenomenon as a whole. Starting from a behavioral analysis, we use a data-driven approach to elicit a Bayesian network which infers the different percepts and dynamics of the process. Context-specific independence analyses enable us to use the model's structure to directly explore how different contexts affect the way subjects handle the same available information. Hence, we establish that, while the percepts yielded by a unisensory stimulus or by the non-fusion of multisensory stimuli may be similar, they result from different processes, as shown by their differing temporal dynamics. Moreover, our model predicts the impact of bottom-up (stimulus driven) factors as well as of top-down factors (induced by instruction manipulation) on both the perception process and the percept itself.

Human sensorimotor learning: adaptation, skill, and beyond

Recent studies of upper limb movements have provided insights into the computations, mechanisms, and taxonomy of human sensorimotor learning. Motor tasks differ with respect to how they weight different learning processes. These include adaptation, an internal-model based process that reduces sensory-prediction errors in order to return performance to pre-perturbation levels, use-dependent plasticity, and operant reinforcement. Visuomotor rotation and force-field tasks impose systematic errors and thereby emphasize adaptation. In skill learning tasks, which for the most part do not involve a perturbation, improved performance is manifest as reduced motor variability and probably depends less on adaptation and more on success-based exploration. Explicit awareness and declarative memory contribute, to varying degrees, to motor learning. The modularity of motor learning processes maps, at least to some extent, onto distinct brain structures.

Multiplicity of control in the basal ganglia: computational roles of striatal subregions

The basal ganglia, in particular the striatum, are central to theories of behavioral control, and often identified as a seat of action selection. Reinforcement learning (RL) models--which have driven much recent experimental work on this region--cast striatum as a dynamic controller, integrating sensory and motivational information to construct efficient and enriching behavioral policies. Befitting this informationally central role, the BG sit at the nexus of multiple anatomical 'loops' of synaptic projections, connecting a wide range of cortical and subcortical structures. Numerous pioneering anatomical studies conducted over the past several decades have meticulously catalogued these loops, and labeled them according to the inferred functions of the connected regions. The specific cotermina of the projections are highly localized to several different subregions of the striatum, leading to the suggestion that these subregions perform complementary but distinct functions. However, until recently, the dominant computational framework outlined only a bipartite, dorsal/ventral, division of striatum. We review recent computational and experimental advances that argue for a more finely fractionated delineation. In particular, experimental data provide extensive insight into unique functions subserved by the dorsomedial striatum (DMS). These functions appear to correspond well with theories of a 'model-based' RL subunit, and may also shed light on the suborganization of ventral striatum. Finally, we discuss the limitations of these ideas and how they point the way toward future refinements of neurocomputational theories of striatal function, bringing them into contact with other areas of computational theory and other regions of the brain.

Dopamine in motivational control: rewarding, aversive, and alerting

Midbrain dopamine neurons are well known for their strong responses to rewards and their critical role in positive motivation. It has become increasingly clear, however, that dopamine neurons also transmit signals related to salient but nonrewarding experiences such as aversive and alerting events. Here we review recent advances in understanding the reward and nonreward functions of dopamine. Based on this data, we propose that dopamine neurons come in multiple types that are connected with distinct brain networks and have distinct roles in motivational control. Some dopamine neurons encode motivational value, supporting brain networks for seeking, evaluation, and value learning. Others encode motivational salience, supporting brain networks for orienting, cognition, and general motivation. Both types of dopamine neurons are augmented by an alerting signal involved in rapid detection of potentially important sensory cues. We hypothesize that these dopaminergic pathways for value, salience, and alerting cooperate to support adaptive behavior.

Structure learning in human sequential decision-making

Studies of sequential decision-making in humans frequently find suboptimal performance relative to an ideal actor that has perfect knowledge of the model of how rewards and events are generated in the environment. Rather than being suboptimal, we argue that the learning problem humans face is more complex, in that it also involves learning the structure of reward generation in the environment. We formulate the problem of structure learning in sequential decision tasks using Bayesian reinforcement learning, and show that learning the generative model for rewards qualitatively changes the behavior of an optimal learning agent. To test whether people exhibit structure learning, we performed experiments involving a mixture of one-armed and two-armed bandit reward models, where structure learning produces many of the qualitative behaviors deemed suboptimal in previous studies. Our results demonstrate humans can perform structure learning in a near-optimal manner.

Every decision-making experiment has a structure that specifies how rewards are obtained, which is usually explained to the subject at the beginning of the experiment. Participants frequently fail to act as if they understand the experimental structure, even in tasks as simple as determining which of two biased coins they should choose to maximize the number of trials that produce “heads”. We hypothesize that participants' behavior is not driven by top-down instructions—rather, participants must learn through experience how the rewards are generated. We formalize this hypothesis using a fully rational optimal Bayesian reinforcement learning approach that models optimal structure learning in sequential decision making. In an experimental test of structure learning in humans, we show that humans learn reward structure from experience in a near optimal manner. Our results demonstrate that behavior purported to show that humans are error-prone and suboptimal decision makers can result from an optimal learning approach. Our findings provide a compelling new family of rational hypotheses for behavior previously deemed irrational, including under- and over-exploration.

Reorganization of finger coordination patterns during adaptation to rotation and scaling of a newly learned sensorimotor transformation

We examined how people organize redundant kinematic control variables (finger joint configurations) while learning to make goal-directed movements of a virtual object (a cursor) within a low-dimensional task space (a computer screen). Subjects participated in three experiments performed on separate days. Learning progressed rapidly on day 1, resulting in reduced target capture error and increased cursor trajectory linearity. On days 2 and 3, one group of subjects adapted to a rotation of the nominal map, imposed either stepwise or randomly over trials. Another group experienced a scaling distortion. We report two findings. First, adaptation rates and memory-dependent motor command updating depended on distortion type. Stepwise application and removal of the rotation induced a marked increase in finger motion variability but scaling did not, suggesting that the rotation initiated a more exhaustive search through the space of viable finger motions to resolve the target capture task than did scaling. Indeed, subjects formed new coordination patterns in compensating the rotation but relied on patterns established during baseline practice to compensate the scaling. These findings support the idea that the brain compensates direction and extent errors separately and in computationally distinct ways, but are inconsistent with the idea that once a task is learned, command updating is limited to those degrees of freedom contributing to performance (thereby minimizing energetic or similar costs of control). Second, we report that subjects who learned a scaling while moving to just one target generalized more narrowly across directions than those who learned a rotation. This contrasts with results from whole-arm reaching studies, where a learned scaling generalizes more broadly across direction than rotation. Based on inverse- and forward-dynamics analyses of reaching with the arm, we propose the difference in results derives from extensive exposure in reaching with familiar arm dynamics versus the novelty of the manual task.

People efficiently explore the solution space of the computationally intractable traveling salesman problem to find near-optimal tours

Humans need to solve computationally intractable problems such as visual search, categorization, and simultaneous learning and acting, yet an increasing body of evidence suggests that their solutions to instantiations of these problems are near optimal. Computational complexity advances an explanation to this apparent paradox: (1) only a small portion of instances of such problems are actually hard, and (2) successful heuristics exploit structural properties of the typical instance to selectively improve parts that are likely to be sub-optimal. We hypothesize that these two ideas largely account for the good performance of humans on computationally hard problems. We tested part of this hypothesis by studying the solutions of 28 participants to 28 instances of the Euclidean Traveling Salesman Problem (TSP). Participants were provided feedback on the cost of their solutions and were allowed unlimited solution attempts (trials). We found a significant improvement between the first and last trials and that solutions are significantly different from random tours that follow the convex hull and do not have self-crossings. More importantly, we found that participants modified their current better solutions in such a way that edges belonging to the optimal solution ("good" edges) were significantly more likely to stay than other edges ("bad" edges), a hallmark of structural exploitation. We found, however, that more trials harmed the participants' ability to tell good from bad edges, suggesting that after too many trials the participants "ran out of ideas." In sum, we provide the first demonstration of significant performance improvement on the TSP under repetition and feedback and evidence that human problem-solving may exploit the structure of hard problems paralleling behavior of state-of-the-art heuristics.

Learning latent structure: carving nature at its joints

Reinforcement learning (RL) algorithms provide powerful explanations for simple learning and decision-making behaviors and the functions of their underlying neural substrates. Unfortunately, in real-world situations that involve many stimuli and actions, these algorithms learn pitifully slowly, exposing their inferiority in comparison to animal and human learning. Here we suggest that one reason for this discrepancy is that humans and animals take advantage of structure that is inherent in real-world tasks to simplify the learning problem. We survey an emerging literature on 'structure learning'--using experience to infer the structure of a task--and how this can be of service to RL, with an emphasis on structure in perception and action.

Structure learning in a sensorimotor association task

Learning is often understood as an organism's gradual acquisition of the association between a given sensory stimulus and the correct motor response. Mathematically, this corresponds to regressing a mapping between the set of observations and the set of actions. Recently, however, it has been shown both in cognitive and motor neuroscience that humans are not only able to learn particular stimulus-response mappings, but are also able to extract abstract structural invariants that facilitate generalization to novel tasks. Here we show how such structure learning can enhance facilitation in a sensorimotor association task performed by human subjects. Using regression and reinforcement learning models we show that the observed facilitation cannot be explained by these basic models of learning stimulus-response associations. We show, however, that the observed data can be explained by a hierarchical Bayesian model that performs structure learning. In line with previous results from cognitive tasks, this suggests that hierarchical Bayesian inference might provide a common framework to explain both the learning of specific stimulus-response associations and the learning of abstract structures that are shared by different task environments.