Insights into Seeing and Grasping: Distinguishing the Neural Correlates of Perception and Action

Separating cognitive and motor processes in the behaving mouse

The cognitive processes supporting complex animal behavior are closely associated with ubiquitous movements responsible for our posture, facial expressions, ability to actively sample our sensory environments, and other critical processes. These movements are strongly related to neural activity across much of the brain and are often highly correlated with ongoing cognitive processes, making it challenging to dissociate the neural dynamics that support cognitive processes from those supporting related movements. In such cases, a critical issue is whether cognitive processes are separable from related movements, or if they are driven by common neural mechanisms. Here, we demonstrate how the separability of cognitive and motor processes can be assessed, and, when separable, how the neural dynamics associated with each component can be isolated. We establish a novel two-context behavioral task in mice that involves multiple cognitive processes and show that commonly observed dynamics taken to support cognitive processes are strongly contaminated by movements. When cognitive and motor components are isolated using a novel approach for subspace decomposition, we find that they exhibit distinct dynamical trajectories. Further, properly accounting for movement revealed that largely separate populations of cells encode cognitive and motor variables, in contrast to the 'mixed selectivity' often reported. Accurately isolating the dynamics associated with particular cognitive and motor processes will be essential for developing conceptual and computational models of neural circuit function and evaluating the function of the cell types of which neural circuits are composed.

Two Distinct Systems Represent Contralateral and Ipsilateral Sensorimotor Processes in the Human Premotor Cortex: A Dense TMS Mapping Study

Animal brains contain behaviorally committed representations of the surrounding world, which integrate sensory and motor information. In primates, sensorimotor mechanisms reside in part in the premotor cortex (PM), where sensorimotor neurons are topographically clustered according to functional specialization. Detailed functional cartography of the human PM is still under investigation. We explored the topographic distribution of spatially dependent sensorimotor functions in healthy volunteers performing left or right, hand or foot, responses to visual cues presented in the left or right hemispace, thus combining independently stimulus side, effector side, and effector type. Event-related transcranial magnetic stimulation was applied to single spots of a dense grid of 10 points on the participants' left hemiscalp, covering the whole PM. Results showed: (1) spatially segregated hand and foot representations, (2) focal representations of contralateral cues and movements in the dorsal PM, and (3) distributed representations of ipsilateral cues and movements in the ventral and dorso-medial PM. The present novel causal information indicates that (1) the human PM is somatotopically organized and (2) the left PM contains sensory-motor representations of both hemispaces and of both hemibodies, but the hemispace and hemibody contralateral to the PM are mapped on a distinct, nonoverlapping cortical region compared to the ipsilateral ones.

The State of Lifelong Learning in Service Robots:

Service robots are appearing more and more in our daily life. The development of service robots combines multiple fields of research, from object perception to object manipulation. The state-of-the-art continues to improve to make a proper coupling between object perception and manipulation. This coupling is necessary for service robots not only to perform various tasks in a reasonable amount of time but also to continually adapt to new environments and safely interact with non-expert human users. Nowadays, robots are able to recognize various objects, and quickly plan a collision-free trajectory to grasp a target object in predefined settings. Besides, in most of the cases, there is a reliance on large amounts of training data. Therefore, the knowledge of such robots is fixed after the training phase, and any changes in the environment require complicated, time-consuming, and expensive robot re-programming by human experts. Therefore, these approaches are still too rigid for real-life applications in unstructured environments, where a significant portion of the environment is unknown and cannot be directly sensed or controlled. In such environments, no matter how extensive the training data used for batch learning, a robot will always face new objects. Therefore, apart from batch learning, the robot should be able to continually learn about new object categories and grasp affordances from very few training examples on-site. Moreover, apart from robot self-learning, non-expert users could interactively guide the process of experience acquisition by teaching new concepts, or by correcting insufficient or erroneous concepts. In this way, the robot will constantly learn how to help humans in everyday tasks by gaining more and more experiences without the need for re-programming. In this paper, we review a set of previously published works and discuss advances in service robots from object perception to complex object manipulation and shed light on the current challenges and bottlenecks.

Neural Code of Motor Planning and Execution during Goal-Directed Movements in Crows

The planning and execution of head-beak movements are vital components of bird behavior. They require integration of sensory input and internal processes with goal-directed motor output. Despite its relevance, the neurophysiological mechanisms underlying action planning and execution outside of the song system are largely unknown. We recorded single-neuron activity from the associative endbrain area nidopallium caudolaterale (NCL) of two male carrion crows (Corvus corone) trained to plan and execute head-beak movements in a spatial delayed response task. The crows were instructed to plan an impending movement toward one of eight possible targets on the left or right side of a touchscreen. In a fraction of trials, the crows were prompted to plan a movement toward a self-chosen target. NCL neurons signaled the impending motion direction in instructed trials. Tuned neuronal activity during motor planning categorically represented the target side, but also specific target locations. As a marker of intentional movement preparation, neuronal activity reliably predicted both target side and specific target location when the crows were free to select a target. In addition, NCL neurons were tuned to specific target locations during movement execution. A subset of neurons was tuned during both planning and execution period; these neurons experienced a sharpening of spatial tuning with the transition from planning to execution. These results show that the avian NCL not only represents high-level sensory and cognitive task components, but also transforms behaviorally-relevant information into dynamic action plans and motor execution during the volitional perception-action cycle of birds.SIGNIFICANCE STATEMENT Corvid songbirds have become exciting new models for understanding complex cognitive behavior. As a key neural underpinning, the endbrain area nidopallium caudolaterale (NCL) represents sensory and memory-related task components. How such representations are converted into goal-directed motor output remained unknown. In crows, we report that NCL neurons are involved in the planning and execution of goal-directed movements. NCL neurons prospectively signaled motion directions in instructed trials, but also when the crows were free to choose a target. NCL neurons showed a target-specific sharpening of tuning with the transition from the planning to the execution period. Thus, the avian NCL not only represents high-level sensory and cognitive task components, but also transforms relevant information into action plans and motor execution.

Identifying neural substrates of competitive interactions and sequence transitions during mechanosensory responses in Drosophila

Nervous systems have the ability to select appropriate actions and action sequences in response to sensory cues. The circuit mechanisms by which nervous systems achieve choice, stability and transitions between behaviors are still incompletely understood. To identify neurons and brain areas involved in controlling these processes, we combined a large-scale neuronal inactivation screen with automated action detection in response to a mechanosensory cue in Drosophila larva. We analyzed behaviors from 2.9x105 larvae and identified 66 candidate lines for mechanosensory responses out of which 25 for competitive interactions between actions. We further characterize in detail the neurons in these lines and analyzed their connectivity using electron microscopy. We found the neurons in the mechanosensory network are located in different regions of the nervous system consistent with a distributed model of sensorimotor decision-making. These findings provide the basis for understanding how selection and transition between behaviors are controlled by the nervous system.

Resynthesizing behavior through phylogenetic refinement

This article proposes that biologically plausible theories of behavior can be constructed by following a method of "phylogenetic refinement," whereby they are progressively elaborated from simple to complex according to phylogenetic data on the sequence of changes that occurred over the course of evolution. It is argued that sufficient data exist to make this approach possible, and that the result can more effectively delineate the true biological categories of neurophysiological mechanisms than do approaches based on definitions of putative functions inherited from psychological traditions. As an example, the approach is used to sketch a theoretical framework of how basic feedback control of interaction with the world was elaborated during vertebrate evolution, to give rise to the functional architecture of the mammalian brain. The results provide a conceptual taxonomy of mechanisms that naturally map to neurophysiological and neuroanatomical data and that offer a context for defining putative functions that, it is argued, are better grounded in biology than are some of the traditional concepts of cognitive science.

Brain-Machine Interfaces: From Basic Science to Neuroprostheses and Neurorehabilitation

Brain-machine interfaces (BMIs) combine methods, approaches, and concepts derived from neurophysiology, computer science, and engineering in an effort to establish real-time bidirectional links between living brains and artificial actuators. Although theoretical propositions and some proof of concept experiments on directly linking the brains with machines date back to the early 1960s, BMI research only took off in earnest at the end of the 1990s, when this approach became intimately linked to new neurophysiological methods for sampling large-scale brain activity. The classic goals of BMIs are 1) to unveil and utilize principles of operation and plastic properties of the distributed and dynamic circuits of the brain and 2) to create new therapies to restore mobility and sensations to severely disabled patients. Over the past decade, a wide range of BMI applications have emerged, which considerably expanded these original goals. BMI studies have shown neural control over the movements of robotic and virtual actuators that enact both upper and lower limb functions. Furthermore, BMIs have also incorporated ways to deliver sensory feedback, generated from external actuators, back to the brain. BMI research has been at the forefront of many neurophysiological discoveries, including the demonstration that, through continuous use, artificial tools can be assimilated by the primate brain's body schema. Work on BMIs has also led to the introduction of novel neurorehabilitation strategies. As a result of these efforts, long-term continuous BMI use has been recently implicated with the induction of partial neurological recovery in spinal cord injury patients.

Interfacing to the brain's motor decisions

It has been long known that neural activity, recorded with electrophysiological methods, contains rich information about a subject's motor intentions, sensory experiences, allocation of attention, action planning, and even abstract thoughts. All these functions have been the subject of neurophysiological investigations, with the goal of understanding how neuronal activity represents behavioral parameters, sensory inputs, and cognitive functions. The field of brain-machine interfaces (BMIs) strives for a somewhat different goal: it endeavors to extract information from neural modulations to create a communication link between the brain and external devices. Although many remarkable successes have been already achieved in the BMI field, questions remain regarding the possibility of decoding high-order neural representations, such as decision making. Could BMIs be employed to decode the neural representations of decisions underlying goal-directed actions? In this review we lay out a framework that describes the computations underlying goal-directed actions as a multistep process performed by multiple cortical and subcortical areas. We then discuss how BMIs could connect to different decision-making steps and decode the neural processing ongoing before movements are initiated. Such decision-making BMIs could operate as a system with prediction that offers many advantages, such as shorter reaction time, better error processing, and improved unsupervised learning. To present the current state of the art, we review several recent BMIs incorporating decision-making components.

Through the forest of motor representations

Following neuroscience, and using different labels, several philosophers have addressed the idea of the presence of a single representational mechanism lying in between (visual) perceptual processes and motor processes involved in different functions and useful for shaping suitable action performances: a motor representation (MR). MRs are the naturalized mental antecedents of action. This paper presents a new, non-monolithic view of MRs, according to which, contrarily to the received view, when looking at in between (visual) perceptual processes and motor processes, we find not only a single representational mechanism with different functions, but an ensemble of different sub-representational phenomena, each of which with a different function. This new view is able to avoid several issues emerging from the literature and to address something the literature is silent about, which however turns out to be crucial for a theory of MRs.

Reprogramming movements: extraction of motor intentions from cortical ensemble activity when movement goals change

The ability to inhibit unwanted movements and change motor plans is essential for behaviors of advanced organisms. The neural mechanisms by which the primate motor system rejects undesired actions have received much attention during the last decade, but it is not well understood how this neural function could be utilized to improve the efficiency of brain-machine interfaces (BMIs). Here we employed linear discriminant analysis (LDA) and a Wiener filter to extract motor plan transitions from the activity of ensembles of sensorimotor cortex neurons. Two rhesus monkeys, chronically implanted with multielectrode arrays in primary motor (M1) and primary sensory (S1) cortices, were overtrained to produce reaching movements with a joystick toward visual targets upon their presentation. Then, the behavioral task was modified to include a distracting target that flashed for 50, 150, or 250 ms (25% of trials each) followed by the true target that appeared at a different screen location. In the remaining 25% of trials, the initial target stayed on the screen and was the target to be approached. M1 and S1 neuronal activity represented both the true and distracting targets, even for the shortest duration of the distracting event. This dual representation persisted both when the monkey initiated movements toward the distracting target and then made corrections and when they moved directly toward the second, true target. The Wiener filter effectively decoded the location of the true target, whereas the LDA classifier extracted the location of both targets from ensembles of 50–250 neurons. Based on these results, we suggest developing real-time BMIs that inhibit unwanted movements represented by brain activity while enacting the desired motor outcome concomitantly.

Cortical correlates of fitts' law

Fitts’ law describes the fundamental trade-off between movement accuracy and speed: it states that the duration of reaching movements is a function of target size (TS) and distance. While Fitts’ law has been extensively studied in ergonomics and has guided the design of human–computer interfaces, there have been few studies on its neuronal correlates. To elucidate sensorimotor cortical activity underlying Fitts’ law, we implanted two monkeys with multielectrode arrays in the primary motor (M1) and primary somatosensory (S1) cortices. The monkeys performed reaches with a joystick-controlled cursor toward targets of different size. The reaction time (RT), movement time, and movement velocity changed with TS, and M1 and S1 activity reflected these changes. Moreover, modifications of cortical activity could not be explained by changes of movement parameters alone, but required TS as an additional parameter. Neuronal representation of TS was especially prominent during the early RT period where it influenced the slope of the firing rate rise preceding movement initiation. During the movement period, cortical activity was correlated with movement velocity. Neural decoders were applied to simultaneously decode TS and motor parameters from cortical modulations. We suggest that sensorimotor cortex activity reflects the characteristics of both the movement and the target. Classifiers that extract these parameters from cortical ensembles could improve neuroprosthetic control.

The What and How of prefrontal cortical organization

How is the prefrontal cortex (PFC) organized such that it is capable of making people more flexible and in control of their behavior? Is there any systematic organization across the many diverse areas that comprise the PFC, or is it uniquely adaptive such that no fixed representational structure can develop? Going against the current tide, this paper argues that there is indeed a systematic organization across PFC areas, with an important functional distinction between ventral and dorsal regions characterized as processing What versus How information, respectively. This distinction has implications for the rostro-caudal and medial-lateral axes of organization as well. The resulting large-scale functional map of PFC could prove useful in integrating diverse data, and in generating novel predictions.

Neural mechanisms for interacting with a world full of action choices

The neural bases of behavior are often discussed in terms of perceptual, cognitive, and motor stages, defined within an information processing framework that was originally inspired by models of human abstract problem solving. Here, we review a growing body of neurophysiological data that is difficult to reconcile with this influential theoretical perspective. As an alternative foundation for interpreting neural data, we consider frameworks borrowed from ethology, which emphasize the kinds of real-time interactive behaviors that animals have engaged in for millions of years. In particular, we discuss an ethologically-inspired view of interactive behavior as simultaneous processes that specify potential motor actions and select between them. We review how recent neurophysiological data from diverse cortical and subcortical regions appear more compatible with this parallel view than with the classical view of serial information processing stages.

Principles of neural ensemble physiology underlying the operation of brain-machine interfaces

Research on brain-machine interfaces has been ongoing for at least a decade. During this period, simultaneous recordings of the extracellular electrical activity of hundreds of individual neurons have been used for direct, real-time control of various artificial devices. Brain-machine interfaces have also added greatly to our knowledge of the fundamental physiological principles governing the operation of large neural ensembles. Further understanding of these principles is likely to have a key role in the future development of neuroprosthetics for restoring mobility in severely paralysed patients.

Cortical ensemble adaptation to represent velocity of an artificial actuator controlled by a brain-machine interface

Monkeys can learn to directly control the movements of an artificial actuator by using a brain-machine interface (BMI) driven by the activity of a sample of cortical neurons. Eventually, they can do so without moving their limbs. Neuronal adaptations underlying the transition from control of the limb to control of the actuator are poorly understood. Here, we show that rapid modifications in neuronal representation of velocity of the hand and actuator occur in multiple cortical areas during the operation of a BMI. Initially, monkeys controlled the actuator by moving a hand-held pole. During this period, the BMI was trained to predict the actuator velocity. As the monkeys started using their cortical activity to control the actuator, the activity of individual neurons and neuronal populations became less representative of the animal's hand movements while representing the movements of the actuator. As a result of this adaptation, the animals could eventually stop moving their hands yet continue to control the actuator. These results show that, during BMI control, cortical ensembles represent behaviorally significant motor parameters, even if these are not associated with movements of the animal's own limb.

Differential Representation of Perception and Action in the Frontal Cortex

A motor illusion was created to separate human subjects' perception of arm movement from their actual movement during figure drawing. Trajectories constructed from cortical activity recorded in monkeys performing the same task showed that the actual movement was represented in the primary motor cortex, whereas the visualized, presumably perceived, trajectories were found in the ventral premotor cortex. Perception and action representations can be differentially recognized in the brain and may be contained in separate structures.