Credit assignment in multiple goal embodied visuomotor behavior

Feedback processing in cognitive and motor tasks: A meta-analysis on the feedback-related negativity

For motor learning, the processing of behavioral outcomes is of high significance. The feedback-related negativity (FRN) is an event-related potential, which is often described as a correlate of the reward prediction error in reinforcement learning. The number of studies examining the FRN in motor tasks is increasing. This meta-analysis summarizes the component in the motor domain and compares it to the cognitive domain. Therefore, a data set of a previous meta-analysis in the cognitive domain that comprised 47 studies  was reanalyzed and compared to additional 25 studies of the motor domain. Further, a moderator analysis for the studies in the motor domain was conducted. The FRN amplitude was higher in the motor domain than in the cognitive domain. This might be related to a higher task complexity and a higher feedback ambiguity of motor tasks. The FRN latency was shorter in the motor domain than in the cognitive domain. Given that sensory information can be used as an external feedback predictor prior to the presentation of the final feedback, reward processing in the motor domain may have been faster and reduced the FRN latency. The moderator variable analysis revealed that the feedback modality influenced the FRN latency, with shorter FRN latencies after bimodal than after visual feedback. Processing of outcome feedback seems to share basic principles in both domains; however, differences exist and should be considered in FRN studies. Future research is motivated to scrutinize the effects of bimodal feedback and other moderators within the motor domain.

Foraging for the self: Environment selection for agency inference

Sometimes agents choose to occupy environments that are neither traditionally rewarding nor worth exploring, but which rather promise to help minimise uncertainty related to what they can control. Selecting environments that afford inferences about agency seems a foundational aspect of environment selection dynamics - if an agent can't form reliable beliefs about what they can and can't control, then they can't act efficiently to achieve rewards. This relatively neglected aspect of environment selection is important to study so that we can better understand why agents occupy certain environments over others - something that may also be relevant for mental and developmental conditions, such as autism. This online experiment investigates the impact of uncertainty about agency on the way participants choose to freely move between two environments, one that has greater irreducible variability and one that is more complex to model. We hypothesise that increasingly erroneous predictions about the expected outcome of agency-exploring actions can be a driver of switching environments, and we explore which type of environment agents prefer. Results show that participants actively switch between the two environments following increases in prediction error, and that the tolerance for prediction error before switching is modulated by individuals' autism traits. Further, we find that participants more frequently occupy the variable environment, which is predicted by greater accuracy and higher confidence than the complex environment. This is the first online study to investigate relatively unconstrained ongoing foraging dynamics in support of judgements of agency, and in doing so represents a significant methodological advance.

Norepinephrine potentiates and serotonin depresses visual cortical responses by transforming eligibility traces

Reinforcement allows organisms to learn which stimuli predict subsequent biological relevance. Hebbian mechanisms of synaptic plasticity are insufficient to account for reinforced learning because neuromodulators signaling biological relevance are delayed with respect to the neural activity associated with the stimulus. A theoretical solution is the concept of eligibility traces (eTraces), silent synaptic processes elicited by activity which upon arrival of a neuromodulator are converted into a lasting change in synaptic strength. Previously we demonstrated in visual cortical slices the Hebbian induction of eTraces and their conversion into LTP and LTD by the retroactive action of norepinephrine and serotonin Here we show in vivo in mouse V1 that the induction of eTraces and their conversion to LTP/D by norepinephrine and serotonin respectively potentiates and depresses visual responses. We also show that the integrity of this process is crucial for ocular dominance plasticity, a canonical model of experience-dependent plasticity.

A survey of inverse reinforcement learning

Learning from demonstration, or imitation learning, is the process of learning to act in an environment from examples provided by a teacher. Inverse reinforcement learning (IRL) is a specific form of learning from demonstration that attempts to estimate the reward function of a Markov decision process from examples provided by the teacher. The reward function is often considered the most succinct description of a task. In simple applications, the reward function may be known or easily derived from properties of the system and hard coded into the learning process. However, in complex applications, this may not be possible, and it may be easier to learn the reward function by observing the actions of the teacher. This paper provides a comprehensive survey of the literature on IRL. This survey outlines the differences between IRL and two similar methods - apprenticeship learning and inverse optimal control. Further, this survey organizes the IRL literature based on the principal method, describes applications of IRL algorithms, and provides areas of future research.

The Hierarchical Evolution in Human Vision Modeling

Computational models of primate vision took a significant advance with David Marr's tripartite separation of the vision enterprise into the problem formulation, algorithm, and neural implementation; however, many subsequent parallel developments in robotics and modeling greatly refined the algorithm descriptions into very distinct levels that complement each other. This review traces the time course of these developments and shows how the current perspective evolved to have its alternative internal hierarchical organization.

Solving the Credit Assignment Problem With the Prefrontal Cortex

In naturalistic multi-cue and multi-step learning tasks, where outcomes of behavior are delayed in time, discovering which choices are responsible for rewards can present a challenge, known as the credit assignment problem. In this review, I summarize recent work that highlighted a critical role for the prefrontal cortex (PFC) in assigning credit where it is due in tasks where only a few of the multitude of cues or choices are relevant to the final outcome of behavior. Collectively, these investigations have provided compelling support for specialized roles of the orbitofrontal (OFC), anterior cingulate (ACC), and dorsolateral prefrontal (dlPFC) cortices in contingent learning. However, recent work has similarly revealed shared contributions and emphasized rich and heterogeneous response properties of neurons in these brain regions. Such functional overlap is not surprising given the complexity of reciprocal projections spanning the PFC. In the concluding section, I overview the evidence suggesting that the OFC, ACC and dlPFC communicate extensively, sharing the information about presented options, executed decisions and received rewards, which enables them to assign credit for outcomes to choices on which they are contingent. This account suggests that lesion or inactivation/inhibition experiments targeting a localized PFC subregion will be insufficient to gain a fine-grained understanding of credit assignment during learning and instead poses refined questions for future research, shifting the focus from focal manipulations to experimental techniques targeting cortico-cortical projections.

Communicating online information via streaming video: the role of user goal

Purpose

The purpose of this paper, building on the media richness theory (MRT), is to propose that while communicating product information via streaming video should enhance outcome measures, such an enhancement will be evident mainly for users with equivocal, latent goals (i.e. recreational browsing) rather than for those with less equivocal, concrete goals (i.e. the search of a specific product).

Design/methodology/approach

The experiment involved 337 potential online consumers in Canada, and had full factorial design with four conditions (two methods to communicate product information: textual vs streaming video, and two goals: product searching vs recreational browsing). Analysis of covariance was used to test the hypotheses.

Findings

The results lent support to the hypotheses. The perceived information quality, trusting competence, and arousal for participants with recreational browsing goals were significantly affected when product information where communicated using streaming video. For participants with concrete goals (product searchers), the traditional textual method was as effective as the streaming video method.

Practical implications

The findings entice practitioners to use rich media such as the streaming video method to communicate online information predominantly for users with experiential browsing goals, and to use lean media for users with less equivocal, concrete goals.

Originality/value

The results contribute to the sparse literature that underscores the key role of user goal in shaping the effectiveness of online information. The results provide empirical support to the prediction of MRT that the use of rich media to communicate information is advantageous for users with latent, equivocal goals.

Control of gaze while walking: Task structure, reward, and uncertainty

While it is universally acknowledged that both bottom up and top down factors contribute to allocation of gaze, we currently have limited understanding of how top-down factors determine gaze choices in the context of ongoing natural behavior. One purely top-down model by Sprague, Ballard, and Robinson (2007) suggests that natural behaviors can be understood in terms of simple component behaviors, or modules, that are executed according to their reward value, with gaze targets chosen in order to reduce uncertainty about the particular world state needed to execute those behaviors. We explore the plausibility of the central claims of this approach in the context of a task where subjects walk through a virtual environment performing interceptions, avoidance, and path following. Many aspects of both walking direction choices and gaze allocation are consistent with this approach. Subjects use gaze to reduce uncertainty for task-relevant information that is used to inform action choices. Notably the addition of motion to peripheral objects did not affect fixations when the objects were irrelevant to the task, suggesting that stimulus saliency was not a major factor in gaze allocation. The modular approach of independent component behaviors is consistent with the main aspects of performance, but there were a number of deviations suggesting that modules interact. Thus the model forms a useful, but incomplete, starting point for understanding top-down factors in active behavior.

The role of motor memory in action selection and procedural learning: insights from children with typical and atypical development

Motor memory is the process by which humans can adopt both persistent and flexible motor behaviours. Persistence and flexibility can be assessed through the examination of the cooperation/competition between new and old motor routines in the motor memory repertoire. Two paradigms seem to be particularly relevant to examine this competition/cooperation. First, a manual search task for hidden objects, namely the C-not-B task, which allows examining how a motor routine may influence the selection of action in toddlers. The second paradigm is procedural learning, and more precisely the consolidation stage, which allows assessing how a previously learnt motor routine becomes resistant to subsequent programming or learning of a new – competitive – motor routine. The present article defends the idea that results of both paradigms give precious information to understand the evolution of motor routines in healthy children. Moreover, these findings echo some clinical observations in developmental neuropsychology, particularly in children with Developmental Coordination Disorder. Such studies suggest that the level of equilibrium between persistence and flexibility of motor routines is an index of the maturity of the motor system.

Saccadic foraging: reduced reaction time to informative targets

The study of saccadic reaction times has revealed a great deal about the neural mechanisms underlying neural decision, in terms of Bayesian factors such as prior probability and information supply. In addition, recent work has shown that saccades are faster to visual targets associated with conventional monetary or food rewards. However, because the purpose of saccades is to acquire information, it could be argued that this is an unnatural situation: the most natural and fundamental reward is the amount of information supplied by a target. Here, we report the results of a study investigating the hypothesis that a saccade to a target whose colour provides information about the location of a subsequent target is faster than to one that does not. We show that the latencies of saccades to a location that provides reliable information about the location of a future target are indeed shorter, their distributions being shifted in a way that implies that the rate of rise of the underlying decision signal is increased. In a race between alternative targets, this means that expected information will be an important factor in deciding where to look, so that 'foraging' saccades are more likely to be made to useful targets.

Modeling task control of eye movements

In natural behavior, visual information is actively sampled from the environment by a sequence of gaze changes. The timing and choice of gaze targets, and the accompanying attentional shifts, are intimately linked with ongoing behavior. Nonetheless, modeling of the deployment of these fixations has been very difficult because they depend on characterizing the underlying task structure. Recently, advances in eye tracking during natural vision, together with the development of probabilistic modeling techniques, have provided insight into how the cognitive agenda might be included in the specification of fixations. These techniques take advantage of the decomposition of complex behaviors into modular components. A particular subset of these models casts the role of fixation as that of providing task-relevant information that is rewarding to the agent, with fixation being selected on the basis of expected reward and uncertainty about environmental state. We review this work here and describe how specific examples can reveal general principles in gaze control.

Neural correlates of temporal credit assignment in the parietal lobe

Empirical studies of decision making have typically assumed that value learning is governed by time, such that a reward prediction error arising at a specific time triggers temporally-discounted learning for all preceding actions. However, in natural behavior, goals must be acquired through multiple actions, and each action can have different significance for the final outcome. As is recognized in computational research, carrying out multi-step actions requires the use of credit assignment mechanisms that focus learning on specific steps, but little is known about the neural correlates of these mechanisms. To investigate this question we recorded neurons in the monkey lateral intraparietal area (LIP) during a serial decision task where two consecutive eye movement decisions led to a final reward. The underlying decision trees were structured such that the two decisions had different relationships with the final reward, and the optimal strategy was to learn based on the final reward at one of the steps (the “F” step) but ignore changes in this reward at the remaining step (the “I” step). In two distinct contexts, the F step was either the first or the second in the sequence, controlling for effects of temporal discounting. We show that LIP neurons had the strongest value learning and strongest post-decision responses during the transition after the F step regardless of the serial position of this step. Thus, the neurons encode correlates of temporal credit assignment mechanisms that allocate learning to specific steps independently of temporal discounting.

Information-seeking, curiosity, and attention: computational and neural mechanisms

Intelligent animals devote much time and energy to exploring and obtaining information, but the underlying mechanisms are poorly understood. We review recent developments on this topic that have emerged from the traditionally separate fields of machine learning, eye movements in natural behavior, and studies of curiosity in psychology and neuroscience. These studies show that exploration may be guided by a family of mechanisms that range from automatic biases toward novelty or surprise to systematic searches for learning progress and information gain in curiosity-driven behavior. In addition, eye movements reflect visual information searching in multiple conditions and are amenable for cellular-level investigations. This suggests that the oculomotor system is an excellent model system for understanding information-sampling mechanisms.

Priorities for selection and representation in natural tasks

Selecting and remembering visual information is an active and competitive process. In natural environments, representations are tightly coupled to task. Objects that are task-relevant are remembered better due to a combination of increased selection for fixation and strategic control of encoding and/or retaining viewed information. However, it is not understood how physically manipulating objects when performing a natural task influences priorities for selection and memory. In this study, we compare priorities for selection and memory when actively engaged in a natural task with first-person observation of the same object manipulations. Results suggest that active manipulation of a task-relevant object results in a specific prioritization for object position information compared with other properties and compared with action observation of the same manipulations. Experiment 2 confirms that this spatial prioritization is likely to arise from manipulation rather than differences in spatial representation in real environments and the movies used for action observation. Thus, our findings imply that physical manipulation of task relevant objects results in a specific prioritization of spatial information about task-relevant objects, possibly coupled with strategic de-prioritization of colour memory for irrelevant objects.

Modular inverse reinforcement learning for visuomotor behavior

In a large variety of situations one would like to have an expressive and accurate model of observed animal or human behavior. While general purpose mathematical models may capture successfully properties of observed behavior, it is desirable to root models in biological facts. Because of ample empirical evidence for reward-based learning in visuomotor tasks, we use a computational model based on the assumption that the observed agent is balancing the costs and benefits of its behavior to meet its goals. This leads to using the framework of reinforcement learning, which additionally provides well-established algorithms for learning of visuomotor task solutions. To quantify the agent's goals as rewards implicit in the observed behavior, we propose to use inverse reinforcement learning, which quantifies the agent's goals as rewards implicit in the observed behavior. Based on the assumption of a modular cognitive architecture, we introduce a modular inverse reinforcement learning algorithm that estimates the relative reward contributions of the component tasks in navigation, consisting of following a path while avoiding obstacles and approaching targets. It is shown how to recover the component reward weights for individual tasks and that variability in observed trajectories can be explained succinctly through behavioral goals. It is demonstrated through simulations that good estimates can be obtained already with modest amounts of observation data, which in turn allows the prediction of behavior in novel configurations.

The role of uncertainty and reward on eye movements in a virtual driving task

Eye movements during natural tasks are well coordinated with ongoing task demands and many variables could influence gaze strategies. Sprague and Ballard (2003) proposed a gaze-scheduling model that uses a utility-weighted uncertainty metric to prioritize fixations on task-relevant objects and predicted that human gaze should be influenced by both reward structure and task-relevant uncertainties. To test this conjecture, we tracked the eye movements of participants in a simulated driving task where uncertainty and implicit reward (via task priority) were varied. Participants were instructed to simultaneously perform a Follow Task where they followed a lead car at a specific distance and a Speed Task where they drove at an exact speed. We varied implicit reward by instructing the participants to emphasize one task over the other and varied uncertainty in the Speed Task with the presence or absence of uniform noise added to the car's velocity. Subjects' gaze data were classified for the image content near fixation and segmented into looks. Gaze measures, including look proportion, duration and interlook interval, showed that drivers more closely monitor the speedometer if it had a high level of uncertainty, but only if it was also associated with high task priority or implicit reward. The interaction observed appears to be an example of a simple mechanism whereby the reduction of visual uncertainty is gated by behavioral relevance. This lends qualitative support for the primary variables controlling gaze allocation proposed in the Sprague and Ballard model.

Attention, learning, and the value of information

Despite many studies on selective attention, fundamental questions remain about its nature and neural mechanisms. Here I draw from the animal and machine learning fields that describe attention as a mechanism for active learning and uncertainty reduction and explore the implications of this view for understanding visual attention and eye movement control. I propose that a closer integration of these different views has the potential greatly to expand our understanding of oculomotor control and our ability to use this system as a window into high level but poorly understood cognitive functions, including the capacity for curiosity and exploration and for inferring internal models of the external world.

Useful theories make predictions

Stephen and Van Orden (this issue) propose that there is a complex system approach to cognitive science, and collectively the authors of the papers presented in this issue believe that this approach provides the means to drive a revolution in the science of the mind. Unfortunately, however illuminating, this explanation is absent and hyperbole is all too extensive. In contrast, I argue (1) that dynamic systems theory is not new to cognitive science and does not provide a basis for a revolution, (2) it is not necessary to reject cognitive science in order to explain the constraints imposed by the body and the environment, (3) it is not necessary, as Silberstein and Chemero (this issue) appear to do, to reject cognitive science in order to explain consciousness, and (4) our understanding of pragmatics is not advanced by Gibbs and Van Orden's (this issue) "self-organized criticality".? Any debate about the future of cognitive science could usefully focus on predictive adequacy. Unfortunately, this is not the approach taken by the authors of this issue.

Eye guidance in natural vision: reinterpreting salience

Models of gaze allocation in complex scenes are derived mainly from studies of static picture viewing. The dominant framework to emerge has been image salience, where properties of the stimulus play a crucial role in guiding the eyes. However, salience-based schemes are poor at accounting for many aspects of picture viewing and can fail dramatically in the context of natural task performance. These failures have led to the development of new models of gaze allocation in scene viewing that address a number of these issues. However, models based on the picture-viewing paradigm are unlikely to generalize to a broader range of experimental contexts, because the stimulus context is limited, and the dynamic, task-driven nature of vision is not represented. We argue that there is a need to move away from this class of model and find the principles that govern gaze allocation in a broader range of settings. We outline the major limitations of salience-based selection schemes and highlight what we have learned from studies of gaze allocation in natural vision. Clear principles of selection are found across many instances of natural vision and these are not the principles that might be expected from picture-viewing studies. We discuss the emerging theoretical framework for gaze allocation on the basis of reward maximization and uncertainty reduction.