Decoding hand kinematics from population responses in sensorimotor cortex during grasping

Accurate neural control of a hand prosthesis by posture-related activity in the primate grasping circuit

Brain-computer interfaces (BCIs) have the potential to restore hand movement for people with paralysis, but current devices still lack the fine control required to interact with objects of daily living. Following our understanding of cortical activity during arm reaches, hand BCI studies have focused primarily on velocity control. However, mounting evidence suggests that posture, and not velocity, dominates in hand-related areas. To explore whether this signal can causally control a prosthesis, we developed a BCI training paradigm centered on the reproduction of posture transitions. Monkeys trained with this protocol were able to control a multidimensional hand prosthesis with high accuracy, including execution of the very intricate precision grip. Analysis revealed that the posture signal in the target grasping areas was the main contributor to control. We present, for the first time, neural posture control of a multidimensional hand prosthesis, opening the door for future interfaces to leverage this additional information channel.

A dynamic subset of network interactions underlies tuning to natural movements in marmoset sensorimotor cortex

Mechanisms of computation in sensorimotor cortex must be flexible and robust to support skilled motor behavior. Patterns of neuronal coactivity emerge as a result of computational processes. Pairwise spike-time statistical relationships, across the population, can be summarized as a functional network (FN) which retains single-unit properties. We record populations of single-unit neural activity in forelimb sensorimotor cortex during prey-capture and spontaneous behavior and use an encoding model incorporating kinematic trajectories and network features to predict single-unit activity during forelimb movements. The contribution of network features depends on structured connectivity within strongly connected functional groups. We identify a context-specific functional group that is highly tuned to kinematics and reorganizes its connectivity between spontaneous and prey-capture movements. In the remaining context-invariant group, interactions are comparatively stable across behaviors and units are less tuned to kinematics. This suggests different roles in producing natural forelimb movements and contextualizes single-unit tuning properties within population dynamics.

Decoding hand kinetics and kinematics using somatosensory cortex activity in active and passive movement

Area 2 of the primary somatosensory cortex (S1), encodes proprioceptive information of limbs. Several studies investigated the encoding of movement parameters in this area. However, the single-trial decoding of these parameters, which can provide additional knowledge about the amount of information available in sub-regions of this area about instantaneous limb movement, has not been well investigated. We decoded kinematic and kinetic parameters of active and passive hand movement during center-out task using conventional and state-based decoders. Our results show that this area can be used to accurately decode position, velocity, force, moment, and joint angles of hand. Kinematics had higher accuracies compared to kinetics and active trials were decoded more accurately than passive trials. Although the state-based decoder outperformed the conventional decoder in the active task, it was the opposite in the passive task. These results can be used in intracortical micro-stimulation procedures to provide proprioceptive feedback to BCI subjects.

Sensory weighting of position and force feedback during pinching

Human hands are complex biomechanical systems that allow for dexterous tasks with many degrees of freedom. Coordination of the fingers is essential for many activities of daily living and involves integrating sensory signals. During this sensory integration, the central nervous system deals with the uncertainty of sensory signals. When handling compliant objects, force and position are related. Interactions with stiff objects result in reduced position changes and increased force changes compared to compliant objects. Literature has shown sensory integration of force and position at the shoulder. Nevertheless, differences in sensory requirements between proximal and distal joints may lead to different proprioceptive representations, hence findings at proximal joints cannot be directly transferred to distal joints, such as the digits. Here, we investigate the sensory integration of force and position during pinching. A haptic manipulator rendered a virtual spring with adjustable stiffness between the index finger and the thumb. Participants had to blindly reproduce a force against the spring. In both visual reference trials and blind reproduction trials, the relation between pinch force and spring compression was constant. However, by covertly changing the spring characteristics in catch trials into an adjusted force-position relation, the participants' weighting of force and position could be revealed. In agreement with previous studies on the shoulder, participants relied more on force sense in trials with higher stiffness. This study demonstrated stiffness-dependent sensory integration of force and position feedback during pinching.

Robust cortical encoding of 3D tongue shape during feeding in macaques

Dexterous tongue deformation underlies eating, drinking, and speaking. The orofacial sensorimotor cortex has been implicated in the control of coordinated tongue kinematics, but little is known about how the brain encodes-and ultimately drives-the tongue's 3D, soft-body deformation. Here we combine a biplanar x-ray video technology, multi-electrode cortical recordings, and machine-learning-based decoding to explore the cortical representation of lingual deformation. We trained long short-term memory (LSTM) neural networks to decode various aspects of intraoral tongue deformation from cortical activity during feeding in male Rhesus monkeys. We show that both lingual movements and complex lingual shapes across a range of feeding behaviors could be decoded with high accuracy, and that the distribution of deformation-related information across cortical regions was consistent with previous studies of the arm and hand.

Learnable latent embeddings for joint behavioural and neural analysis

Mapping behavioural actions to neural activity is a fundamental goal of neuroscience. As our ability to record large neural and behavioural data increases, there is growing interest in modelling neural dynamics during adaptive behaviours to probe neural representations1-3. In particular, although neural latent embeddings can reveal underlying correlates of behaviour, we lack nonlinear techniques that can explicitly and flexibly leverage joint behaviour and neural data to uncover neural dynamics3-5. Here, we fill this gap with a new encoding method, CEBRA, that jointly uses behavioural and neural data in a (supervised) hypothesis- or (self-supervised) discovery-driven manner to produce both consistent and high-performance latent spaces. We show that consistency can be used as a metric for uncovering meaningful differences, and the inferred latents can be used for decoding. We validate its accuracy and demonstrate our tool's utility for both calcium and electrophysiology datasets, across sensory and motor tasks and in simple or complex behaviours across species. It allows leverage of single- and multi-session datasets for hypothesis testing or can be used label free. Lastly, we show that CEBRA can be used for the mapping of space, uncovering complex kinematic features, for the production of consistent latent spaces across two-photon and Neuropixels data, and can provide rapid, high-accuracy decoding of natural videos from visual cortex.

Clinical neuroscience and neurotechnology: An amazing symbiosis

In the last decades, clinical neuroscience found a novel ally in neurotechnologies, devices able to record and stimulate electrical activity in the nervous system. These technologies improved the ability to diagnose and treat neural disorders. Neurotechnologies are concurrently enabling a deeper understanding of healthy and pathological dynamics of the nervous system through stimulation and recordings during brain implants. On the other hand, clinical neurosciences are not only driving neuroengineering toward the most relevant clinical issues, but are also shaping the neurotechnologies thanks to clinical advancements. For instance, understanding the etiology of a disease informs the location of a therapeutic stimulation, but also the way stimulation patterns should be designed to be more effective/naturalistic. Here, we describe cases of fruitful integration such as Deep Brain Stimulation and cortical interfaces to highlight how this symbiosis between clinical neuroscience and neurotechnology is closer to a novel integrated framework than to a simple interdisciplinary interaction.

Prehension kinematics in humans and macaques

Non-human primates, especially rhesus macaques, have been a dominant model to study sensorimotor control of the upper limbs. Indeed, human and macaques have similar hands and homologous neural circuits to mediate manual behavior. However, few studies have systematically and quantitatively compared the manual behaviors of the two species. Such comparison is critical for assessing the validity of using the macaque sensorimotor system as a model of its human counterpart. In this study, we systematically compared the prehensile behaviors of humans and rhesus macaques using an identical experimental setup. We found human and macaque prehension kinematics to be generally similar with a few subtle differences. While the structure of the pre-shaping hand postures is similar in humans and macaques, human postures are more object-specific and human joints are less intercorrelated. Conversely, monkeys demonstrate more stereotypical pre-shaping behaviors that are common across all objects and more variability in their postures across repeated presentations of the same object. Despite these subtle differences in manual behavior between humans and monkeys, our results bolster the use of the macaque model to understand the neural mechanisms of manual dexterity in humans.

The science and engineering behind sensitized brain-controlled bionic hands

Advances in our understanding of brain function, along with the development of neural interfaces that allow for the monitoring and activation of neurons, have paved the way for brain-machine interfaces (BMIs), which harness neural signals to reanimate the limbs via electrical activation of the muscles or to control extracorporeal devices, thereby bypassing the muscles and senses altogether. BMIs consist of reading out motor intent from the neuronal responses monitored in motor regions of the brain and executing intended movements with bionic limbs, reanimated limbs, or exoskeletons. BMIs also allow for the restoration of the sense of touch by electrically activating neurons in somatosensory regions of the brain, thereby evoking vivid tactile sensations and conveying feedback about object interactions. In this review, we discuss the neural mechanisms of motor control and somatosensation in able-bodied individuals and describe approaches to use neuronal responses as control signals for movement restoration and to activate residual sensory pathways to restore touch. Although the focus of the review is on intracortical approaches, we also describe alternative signal sources for control and noninvasive strategies for sensory restoration.

Decoding grasp and speech signals from the cortical grasp circuit in a tetraplegic human

The cortical grasp network encodes planning and execution of grasps and processes spoken and written aspects of language. High-level cortical areas within this network are attractive implant sites for brain-machine interfaces (BMIs). While a tetraplegic patient performed grasp motor imagery and vocalized speech, neural activity was recorded from the supramarginal gyrus (SMG), ventral premotor cortex (PMv), and somatosensory cortex (S1). In SMG and PMv, five imagined grasps were well represented by firing rates of neuronal populations during visual cue presentation. During motor imagery, these grasps were significantly decodable from all brain areas. During speech production, SMG encoded both spoken grasp types and the names of five colors. Whereas PMv neurons significantly modulated their activity during grasping, SMG's neural population broadly encoded features of both motor imagery and speech. Together, these results indicate that brain signals from high-level areas of the human cortex could be used for grasping and speech BMI applications.

The neural mechanisms of manual dexterity

The hand endows us with unparalleled precision and versatility in our interactions with objects, from mundane activities such as grasping to extraordinary ones such as virtuoso pianism. The complex anatomy of the human hand combined with expansive and specialized neuronal control circuits allows a wide range of precise manual behaviours. To support these behaviours, an exquisite sensory apparatus, spanning the modalities of touch and proprioception, conveys detailed and timely information about our interactions with objects and about the objects themselves. The study of manual dexterity provides a unique lens into the sensorimotor mechanisms that endow the nervous system with the ability to flexibly generate complex behaviour.

Decoding Different Reach-and-Grasp Movements Using Noninvasive Electroencephalogram

Grasping is one of the most indispensable functions of humans. Decoding reach-and-grasp actions from electroencephalograms (EEGs) is of great significance for the realization of intuitive and natural neuroprosthesis control, and the recovery or reconstruction of hand functions of patients with motor disorders. In this paper, we investigated decoding five different reach-and-grasp movements closely related to daily life using movement-related cortical potentials (MRCPs). In the experiment, nine healthy subjects were asked to naturally execute five different reach-and-grasp movements on the designed experimental platform, namely palmar, pinch, push, twist, and plug grasp. A total of 480 trials per subject (80 trials per condition) were recorded. The MRCPs amplitude from low-frequency (0.3-3 Hz) EEG signals were used as decoding features for further offline analysis. Average binary classification accuracy for grasping vs. the no-movement condition peaked at 75.06 ± 6.8%. Peak average accuracy for grasping vs. grasping conditions of 64.95 ± 7.4% could be reached. Grand average peak accuracy of multiclassification for five grasping conditions reached 36.7 ± 6.8% at 1.45 s after the movement onset. The analysis of MRCPs indicated that all the grasping conditions are more pronounced than the no-movement condition, and there are also significant differences between the grasping conditions. These findings clearly proved the feasibility of decoding multiple reach-and-grasp actions from noninvasive EEG signals. This work is significant for the natural and intuitive BCI application, particularly for neuroprosthesis control or developing an active human-machine interaction system, such as rehabilitation robot.

Real-time linear prediction of simultaneous and independent movements of two finger groups using an intracortical brain-machine interface

Modern brain-machine interfaces can return function to people with paralysis, but current upper extremity brain-machine interfaces are unable to reproduce control of individuated finger movements. Here, for the first time, we present a real-time, high-speed, linear brain-machine interface in nonhuman primates that utilizes intracortical neural signals to bridge this gap. We created a non-prehensile task that systematically individuates two finger groups, the index finger and the middle-ring-small fingers combined. During online brain control, the ReFIT Kalman filter could predict individuated finger group movements with high performance. Next, training ridge regression decoders with individual movements was sufficient to predict untrained combined movements and vice versa. Finally, we compared the postural and movement tuning of finger-related cortical activity to find that individual cortical units simultaneously encode multiple behavioral dimensions. Our results suggest that linear decoders may be sufficient for brain-machine interfaces to execute high-dimensional tasks with the performance levels required for naturalistic neural prostheses.

A markerless platform for ambulatory systems neuroscience

Motor systems neuroscience seeks to understand how the brain controls movement. To minimize confounding variables, large-animal studies typically constrain body movement from areas not under observation, ensuring consistent, repeatable behaviors. Such studies have fueled decades of research, but they may be artificially limiting the richness of neural data observed, preventing generalization to more natural movements and settings. Neuroscience studies of unconstrained movement would capture a greater range of behavior and a more complete view of neuronal activity, but instrumenting an experimental rig suitable for large animals presents substantial engineering challenges. Here, we present a markerless, full-body motion tracking and synchronized wireless neural electrophysiology platform for large, ambulatory animals. Composed of four depth (RGB-D) cameras that provide a 360° view of a 4.5-square-meters enclosed area, this system is designed to record a diverse range of neuroethologically relevant behaviors. This platform also allows for the simultaneous acquisition of hundreds of wireless neural recording channels in multiple brain regions. As behavioral and neuronal data are generated at rates below 200 megabytes per second, a single desktop can facilitate hours of continuous recording. This setup is designed for systems neuroscience and neuroengineering research, where synchronized kinematic behavior and neural data are the foundation for investigation. By enabling the study of previously unexplored movement tasks, this system can generate insights into the functioning of the mammalian motor system and provide a platform to develop brain-machine interfaces for unconstrained applications.

Visually and Tactually Guided Grasps Lead to Different Neuronal Activity in Non-human Primates

Movements are defining characteristics of all behaviors. Animals walk around, move their eyes to explore the world or touch structures to learn more about them. So far we only have some basic understanding of how the brain generates movements, especially when we want to understand how different areas of the brain interact with each other. In this study we investigated the influence of sensory object information on grasp planning in four different brain areas involved in vision, touch, movement planning, and movement generation in the parietal, somatosensory, premotor and motor cortex. We trained one monkey to grasp objects that he either saw or touched beforehand while continuously recording neural spiking activity with chronically implanted floating multi-electrode arrays. The animal was instructed to sit in the dark and either look at a shortly illuminated object or reach out and explore the object with his hand in the dark before lifting it up. In a first analysis we confirmed that the animal not only memorizes the object in both tasks, but also applies an object-specific grip type, independent of the sensory modality. In the neuronal population, we found a significant difference in the number of tuned units for sensory modalities during grasp planning that persisted into grasp execution. These differences were sufficient to enable a classifier to decode the object and sensory modality in a single trial exclusively from neural population activity. These results give valuable insights in how different brain areas contribute to the preparation of grasp movement and how different sensory streams can lead to distinct neural activity while still resulting in the same action execution.

Of mice and monkeys: Somatosensory processing in two prominent animal models

Our understanding of the neural basis of somatosensation is based largely on studies of the whisker system of mice and rats and the hands of macaque monkeys. Results across these animal models are often interpreted as providing direct insight into human somatosensation. Work on these systems has proceeded in parallel, capitalizing on the strengths of each model, but has rarely been considered as a whole. This lack of integration promotes a piecemeal understanding of somatosensation. Here, we examine the functions and morphologies of whiskers of mice and rats, the hands of macaque monkeys, and the somatosensory neuraxes of these three species. We then discuss how somatosensory information is encoded in their respective nervous systems, highlighting similarities and differences. We reflect on the limitations of these models of human somatosensation and consider key gaps in our understanding of the neural basis of somatosensation.

Somatosensory deafferentation reveals lateralized roles of proprioception in feedback and adaptive feedforward control of movement and posture

Proprioception provides crucial information necessary for determining limb position and movement, and plausibly also for updating internal models that might underlie the control of movement and posture. Seminal studies of upper-limb movements in individuals living with chronic, large fiber deafferentation have provided evidence for the role of proprioceptive information in the hypothetical formation and maintenance of internal models to produce accurate motor commands. Vision also contributes to sensorimotor functions but cannot fully compensate for proprioceptive deficits. More recent work has shown that posture and movement control processes are lateralized in the brain, and that proprioception plays a fundamental role in coordinating the contributions of these processes to the control of goal-directed actions. In fact, the behavior of each limb in a deafferented individual resembles the action of a controller in isolation. Proprioception, thus, provides state estimates necessary for the nervous system to efficiently coordinate multiple motor control processes.

Neural population dynamics in motor cortex are different for reach and grasp

Low-dimensional linear dynamics are observed in neuronal population activity in primary motor cortex (M1) when monkeys make reaching movements. This population-level behavior is consistent with a role for M1 as an autonomous pattern generator that drives muscles to give rise to movement. In the present study, we examine whether similar dynamics are also observed during grasping movements, which involve fundamentally different patterns of kinematics and muscle activations. Using a variety of analytical approaches, we show that M1 does not exhibit such dynamics during grasping movements. Rather, the grasp-related neuronal dynamics in M1 are similar to their counterparts in somatosensory cortex, whose activity is driven primarily by afferent inputs rather than by intrinsic dynamics. The basic structure of the neuronal activity underlying hand control is thus fundamentally different from that underlying arm control.