Decoding of human hand actions to handle missing limbs in neuroprosthetics

The spatio-temporal architecture of everyday manual behavior

In everyday activities, humans move alike to manipulate objects. Prior works suggest that hand movements are built by a limited set of basic building blocks consisting of a set of common postures. However, how the low dimensionality of hand movements supports the adaptability and flexibility of natural behavior is unknown. Through a sensorized glove, we collected kinematics data from thirty-six participants preparing and having breakfast in naturalistic conditions. By means of an unbiased analysis, we identified a repertoire of hand states. Then, we tracked their transitions over time. We found that manual behavior can be described in space through a complex organization of basic configurations. These, even in an unconstrained experiment, recurred across subjects. A specific temporal structure, highly consistent within the sample, seems to integrate such identified hand shapes to realize skilled movements. These findings suggest that the simplification of the motor commands unravels in the temporal dimension more than in the spatial one.

Playing the piano with a robotic third thumb: assessing constraints of human augmentation

Contemporary robotics gives us mechatronic capabilities for augmenting human bodies with extra limbs. However, how our motor control capabilities pose limits on such augmentation is an open question. We developed a Supernumerary Robotic 3rd Thumbs (SR3T) with two degrees-of-freedom controlled by the user’s body to endow them with an extra contralateral thumb on the hand. We demonstrate that a pianist can learn to play the piano with 11 fingers within an hour. We then evaluate 6 naïve and 6 experienced piano players in their prior motor coordination and their capability in piano playing with the robotic augmentation. We show that individuals’ augmented performance with the SR3T could be explained by our new custom motor coordination assessment, the Human Augmentation Motor Coordination Assessment (HAMCA) performed pre-augmentation. Our work demonstrates how supernumerary robotics can augment humans in skilled tasks and that individual differences in their augmentation capability are explainable by their individual motor coordination abilities.

The measurement, evolution, and neural representation of action grammars of human behavior

Human behaviors from toolmaking to language are thought to rely on a uniquely evolved capacity for hierarchical action sequencing. Testing this idea will require objective, generalizable methods for measuring the structural complexity of real-world behavior. Here we present a data-driven approach for extracting action grammars from basic ethograms, exemplified with respect to the evolutionarily relevant behavior of stone toolmaking. We analyzed sequences from the experimental replication of ~ 2.5 Mya Oldowan vs. ~ 0.5 Mya Acheulean tools, finding that, while using the same "alphabet" of elementary actions, Acheulean sequences are quantifiably more complex and Oldowan grammars are a subset of Acheulean grammars. We illustrate the utility of our complexity measures by re-analyzing data from an fMRI study of stone toolmaking to identify brain responses to structural complexity. Beyond specific implications regarding the co-evolution of language and technology, this exercise illustrates the general applicability of our method to investigate naturalistic human behavior and cognition.

Gaussian Process Autoregression for Joint Angle Prediction Based on sEMG Signals

There is uncertainty in the neuromusculoskeletal system, and deterministic models cannot describe this significant presence of uncertainty, affecting the accuracy of model predictions. In this paper, a knee joint angle prediction model based on surface electromyography (sEMG) signals is proposed. To address the instability of EMG signals and the uncertainty of the neuromusculoskeletal system, a non-parametric probabilistic model is developed using a Gaussian process model combined with the physiological properties of muscle activation. Since the neuromusculoskeletal system is a dynamic system, the Gaussian process model is further combined with a non-linear autoregressive with eXogenous inputs (NARX) model to create a Gaussian process autoregression model. In this paper, the normalized root mean square error (NRMSE) and the correlation coefficient (CC) are compared between the joint angle prediction results of the Gaussian process autoregressive model prediction and the actual joint angle under three test scenarios: speed-dependent, multi-speed and speed-independent. The mean of NRMSE and the mean of CC for all test scenarios in the healthy subjects dataset and the hemiplegic patients dataset outperform the results of the Gaussian process model, with significant differences (p < 0.05 and p < 0.05, p < 0.05 and p < 0.05). From the perspective of uncertainty, a non-parametric probabilistic model for joint angle prediction is established by using Gaussian process autoregressive model to achieve accurate prediction of human movement.

The organizational principles of de-differentiated topographic maps in somatosensory cortex

Topographic maps are a fundamental feature of cortex architecture in the mammalian brain. One common theory is that the de-differentiation of topographic maps links to impairments in everyday behavior due to less precise functional map readouts. Here, we tested this theory by characterizing de-differentiated topographic maps in primary somatosensory cortex (SI) of younger and older adults by means of ultra-high resolution functional magnetic resonance imaging together with perceptual finger individuation and hand motor performance. Older adults' SI maps showed similar amplitude and size to younger adults' maps, but presented with less representational similarity between distant fingers. Larger population receptive field sizes in older adults' maps did not correlate with behavior, whereas reduced cortical distances between D2 and D3 related to worse finger individuation but better motor performance. Our data uncover the drawbacks of a simple de-differentiation model of topographic map function, and motivate the introduction of feature-based models of cortical reorganization.

Embodied virtual reality for the study of real-world motor learning

Motor-learning literature focuses on simple laboratory-tasks due to their controlled manner and the ease to apply manipulations to induce learning and adaptation. Recently, we introduced a billiards paradigm and demonstrated the feasibility of real-world-neuroscience using wearables for naturalistic full-body motion-tracking and mobile-brain-imaging. Here we developed an embodied virtual-reality (VR) environment to our real-world billiards paradigm, which allows to control the visual feedback for this complex real-world task, while maintaining sense of embodiment. The setup was validated by comparing real-world ball trajectories with the trajectories of the virtual balls, calculated by the physics engine. We then ran our short-term motor learning protocol in the embodied VR. Subjects played billiard shots when they held the physical cue and hit a physical ball on the table while seeing it all in VR. We found comparable short-term motor learning trends in the embodied VR to those we previously reported in the physical real-world task. Embodied VR can be used for learning real-world tasks in a highly controlled environment which enables applying visual manipulations, common in laboratory-tasks and rehabilitation, to a real-world full-body task. Embodied VR enables to manipulate feedback and apply perturbations to isolate and assess interactions between specific motor-learning components, thus enabling addressing the current questions of motor-learning in real-world tasks. Such a setup can potentially be used for rehabilitation, where VR is gaining popularity but the transfer to the real-world is currently limited, presumably, due to the lack of embodiment.

Motor learning in real-world pool billiards

The neurobehavioral mechanisms of human motor-control and learning evolved in free behaving, real-life settings, yet this is studied mostly in reductionistic lab-based experiments. Here we take a step towards a more real-world motor neuroscience using wearables for naturalistic full-body motion-tracking and the sports of pool billiards to frame a real-world skill learning experiment. First, we asked if well-known features of motor learning in lab-based experiments generalize to a real-world task. We found similarities in many features such as multiple learning rates, and the relationship between task-related variability and motor learning. Our data-driven approach reveals the structure and complexity of movement, variability, and motor learning, enabling an in-depth understanding of the structure of motor learning in three ways: First, while expecting most of the movement learning is done by the cue-wielding arm, we find that motor learning affects the whole body, changing motor-control from head to toe. Second, during learning, all subjects decreased their movement variability and their variability in the outcome. Subjects who were initially more variable were also more variable after learning. Lastly, when screening the link across subjects between initial variability in individual joints and learning, we found that only the initial variability in the right forearm supination shows a significant correlation to the subjects' learning rates. This is in-line with the relationship between learning and variability: while learning leads to an overall reduction in movement variability, only initial variability in specific task-relevant dimensions can facilitate faster learning.

Sensing form - finger gaiting as key to tactile object exploration - a data glove analysis of a prototypical daily task

Background

Motor hand skill and associated dexterity is important for meeting the challenges of daily activity and an important resource post-stroke. In this context, the present study investigated the finger movements of right-handed subjects during tactile manipulation of a cuboid, a prototypical task underlying tactile exploration. During one motor act, the thumb and fingers of one hand surround the cuboid in a continuous and regular manner. While the object is moved by the guiding thumb, the opposed supporting fingers are replaced once they reach their joint limits by free fingers, a mechanism termed finger gaiting.

Methods

For both hands of 22 subjects, we acquired the time series of consecutive manipulations of a cuboid at a frequency of 1 Hz, using a digital data glove consisting of 29 sensors. Using principle component analysis, we decomposed the short action into motor patterns related to successive manipulations of the cuboid. The components purport to represent changing grasp configurations involving the stabilizing fingers and guiding thumb. The temporal features of the components permits testing whether the distinct configurations occur at the frequency of 1 Hz, i.e. within the time window of 1 s, and, thus, taxonomic classification of the manipulation as finger gaiting.

Results

The fraction of variance described by the principal components indicated that three components described the salient features of the single motor acts for each hand. Striking in the finger patterns was the prominent and varying roles of the MCP and PIP joints of the fingers, and the CMC joint of the thumb. An important aspect of the three components was their representation of distinct finger configurations within the same motor act. Principal component and graph theory analysis confirmed modular, functionally synchronous action of the involved joints. The computation of finger trajectories in one subject illustrated the workspace of the task, which differed for the right and left hands.

Conclusion

In this task one complex motor act of 1 s duration could be described by three elementary and hierarchically ordered grasp configurations occurring at the prescribed frequency of 1 Hz. Therefore, these configurations represent finger gaiting, described until now only in artificial systems, as the principal mechanism underlying this prototypical task.

Trial registration

clinicaltrials.gov, NCT02865642, registered 12 August 2016.

Brain Activity Reveals Multiple Motor-Learning Mechanisms in a Real-World Task

Many recent studies found signatures of motor learning in neural beta oscillations (13-30 Hz), and specifically in the post-movement beta rebound (PMBR). All these studies were in controlled laboratory-tasks in which the task designed to induce the studied learning mechanism. Interestingly, these studies reported opposing dynamics of the PMBR magnitude over learning for the error-based and reward-based tasks (increase vs. decrease, respectively). Here, we explored the PMBR dynamics during real-world motor-skill-learning in a billiards task using mobile-brain-imaging. Our EEG recordings highlight the opposing dynamics of PMBR magnitudes (increase vs. decrease) between different subjects performing the same task. The groups of subjects, defined by their neural dynamics, also showed behavioral differences expected for different learning mechanisms. Our results suggest that when faced with the complexity of the real-world different subjects might use different learning mechanisms for the same complex task. We speculate that all subjects combine multi-modal mechanisms of learning, but different subjects have different predominant learning mechanisms.

Decoding hand kinematics from population responses in sensorimotor cortex during grasping

OBJECTIVE

The hand-a complex effector comprising dozens of degrees of freedom of movement-endows us with the ability to flexibly, precisely, and effortlessly interact with objects. The neural signals associated with dexterous hand movements in primary motor cortex (M1) and somatosensory cortex (SC) have received comparatively less attention than have those associated with proximal upper limb control.

APPROACH

To fill this gap, we trained two monkeys to grasp objects varying in size and shape while tracking their hand postures and recording single-unit activity from M1 and SC. We then decoded their hand kinematics across tens of joints from population activity in these areas.

MAIN RESULTS

We found that we could accurately decode kinematics with a small number of neural signals and that different cortical fields carry different amounts of information about hand kinematics. In particular, neural signals in rostral M1 led to better performance than did signals in caudal M1, whereas Brodmann's area 3a outperformed areas 1 and 2 in SC. Moreover, decoding performance was higher for joint angles than joint angular velocities, in contrast to what has been found with proximal limb decoders.

SIGNIFICANCE

We conclude that cortical signals can be used for dexterous hand control in brain machine interface applications and that postural representations in SC may be exploited via intracortical stimulation to close the sensorimotor loop.

Neurobehavioural signatures in race car driving: a case study

Recent technological developments in mobile brain and body imaging are enabling new frontiers of real-world neuroscience. Simultaneous recordings of body movement and brain activity from highly skilled individuals as they demonstrate their exceptional skills in real-world settings, can shed new light on the neurobehavioural structure of human expertise. Driving is a real-world skill which many of us acquire to different levels of expertise. Here we ran a case-study on a subject with the highest level of driving expertise-a Formula E Champion. We studied the driver's neural and motor patterns while he drove a sports car on the "Top Gear" race track under extreme conditions (high speed, low visibility, low temperature, wet track). His brain activity, eye movements and hand/foot movements were recorded. Brain activity in the delta, alpha, and beta frequency bands showed causal relation to hand movements. We herein demonstrate the feasibility of using mobile brain and body imaging even in very extreme conditions (race car driving) to study the sensory inputs, motor outputs, and brain states which characterise complex human skills.

Gaussian Process-Integrated State Space Model for Continuous Joint Angle Prediction from EMG and Interactive Force in a Human-Exoskeleton System

As one of the most direct indicators of the transparency between a human and an exoskeleton, interactive force has rarely been fused with electromyography (EMG) in the control of human-exoskeleton systems, the performances of which are largely determined by the accuracy of the continuous joint angle prediction. To achieve intuitive and naturalistic human intent learning, a state space model (SSM) for continuous angle prediction of knee joint is developed. When the influence of the interactive force is often ignored in the existing models of human-exoskeleton systems, interactive force is applied as the measurement model output of the proposed SSM, and the EMG signal is used as the state model input signal to indicate muscle activation. The forward dynamics of joint motion and the human-machine interaction mechanism, i.e., the biomechanical interpretations of the interactive force generation mechanism, are derived as the bases for the state model and measurement model based on Hill’s muscle model and semiphenomenological (SP) muscular model, respectively. Gaussian process (GP)-based nonlinear autoregressive with the exogenous inputs (NARX) model and back-propagation neural network (BPNN) are applied to provide better adaptivity for the SSM in practical applications. Corresponding experimental results demonstrate the validity and superiority of the method.

A cryptography-based approach for movement decoding

Brain decoders use neural recordings to infer the activity or intent of a user. To train a decoder, one generally needs to infer the measured variables of interest (covariates) from simultaneously measured neural activity. However, there are cases for which obtaining supervised data is difficult or impossible. Here, we describe an approach for movement decoding that does not require access to simultaneously measured neural activity and motor outputs. We use the statistics of movement-much like cryptographers use the statistics of language-to find a mapping between neural activity and motor variables, and then align the distribution of decoder outputs with the typical distribution of motor outputs by minimizing their Kullback-Leibler divergence. By using datasets collected from the motor cortex of three non-human primates performing either a reaching task or an isometric force-production task, we show that the performance of such a distribution-alignment decoding algorithm is comparable to the performance of supervised approaches. Distribution-alignment decoding promises to broaden the set of potential applications of brain decoding.

Gaussian Process Autoregression for Simultaneous Proportional Multi-Modal Prosthetic Control With Natural Hand Kinematics

Matching the dexterity, versatility, and robustness of the human hand is still an unachieved goal in bionics, robotics, and neural engineering. A major limitation for hand prosthetics lies in the challenges of reliably decoding user intention from muscle signals when controlling complex robotic hands. Most of the commercially available prosthetic hands use muscle-related signals to decode a finite number of predefined motions and some offer proportional control of open/close movements of the whole hand. Here, in contrast, we aim to offer users flexible control of individual joints of their artificial hand. We propose a novel framework for decoding neural information that enables a user to independently control 11 joints of the hand in a continuous manner-much like we control our natural hands. Toward this end, we instructed six able-bodied subjects to perform everyday object manipulation tasks combining both dynamic, free movements (e.g., grasping) and isometric force tasks (e.g., squeezing). We recorded the electromyographic and mechanomyographic activities of five extrinsic muscles of the hand in the forearm, while simultaneously monitoring 11 joints of hand and fingers using a sensorized data glove that tracked the joints of the hand. Instead of learning just a direct mapping from current muscle activity to intended hand movement, we formulated a novel autoregressive approach that combines the context of previous hand movements with instantaneous muscle activity to predict future hand movements. Specifically, we evaluated a linear vector autoregressive moving average model with exogenous inputs and a novel Gaussian process ( ) autoregressive framework to learn the continuous mapping from hand joint dynamics and muscle activity to decode intended hand movement. Our approach achieves high levels of performance (RMSE of 8°/s and ). Crucially, we use a small set of sensors that allows us to control a larger set of independently actuated degrees of freedom of a hand. This novel undersensored control is enabled through the combination of nonlinear autoregressive continuous mapping between muscle activity and joint angles. The system evaluates the muscle signals in the context of previous natural hand movements. This enables us to resolve ambiguities in situations, where muscle signals alone cannot determine the correct action as we evaluate the muscle signals in their context of natural hand movements. autoregression is a particularly powerful approach which makes not only a prediction based on the context but also represents the associated uncertainty of its predictions, thus enabling the novel notion of risk-based control in neuroprosthetics. Our results suggest that autoregressive approaches with exogenous inputs lend themselves for natural, intuitive, and continuous control in neurotechnology, with the particular focus on prosthetic restoration of natural limb function, where high dexterity is required for complex movements.

Muscle moment arms and sensitivity analysis of a mouse hindlimb musculoskeletal model

Musculoskeletal modelling has become a valuable tool with which to understand how neural, muscular, skeletal and other tissues are integrated to produce movement. Most musculoskeletal modelling work has to date focused on humans or their close relatives, with few examples of quadrupedal animal limb models. A musculoskeletal model of the mouse hindlimb could have broad utility for questions in medicine, genetics, locomotion and neuroscience. This is due to this species’ position as a premier model of human disease, having an array of genetic tools for manipulation of the animal in vivo, and being a small quadruped, a category for which few models exist. Here, the methods used to develop the first three‐dimensional (3D) model of a mouse hindlimb and pelvis are described. The model, which represents bones, joints and 39 musculotendon units, was created through a combination of previously gathered muscle architecture data from microdissections, contrast‐enhanced micro‐computed tomography (CT) scanning and digital segmentation. The model allowed muscle moment arms as well as muscle forces to be estimated for each musculotendon unit throughout a range of joint rotations. Moment arm analysis supported the reliability of musculotendon unit placement within the model, and comparison to a previously published rat hindlimb model further supported the model's reliability. A sensitivity analysis performed on both the force‐generating parameters and muscle's attachment points of the model indicated that the maximal isometric muscle moment is generally most sensitive to changes in either tendon slack length or the coordinates of insertion, although the degree to which the moment is affected depends on several factors. This model represents the first step in the creation of a fully dynamic 3D computer model of the mouse hindlimb and pelvis that has application to neuromuscular disease, comparative biomechanics and the neuromechanical basis of movement. Capturing the morphology and dynamics of the limb, it enables future dissection of the complex interactions between the nervous and musculoskeletal systems as well as the environment.

Transfer of tactile perceptual learning to untrained neighboring fingers reflects natural use relationships

Tactile learning transfers from trained to untrained fingers in a pattern that reflects overlap between the representations of fingers in the somatosensory system (e.g., neurons with multifinger receptive fields). While physical proximity on the body is known to determine the topography of somatosensory representations, tactile coactivation is also an established organizing principle of somatosensory topography. In this study we investigated whether tactile coactivation, induced by habitual inter-finger cooperative use (use pattern), shapes inter-finger overlap. To this end, we used psychophysics to compare the transfer of tactile learning from the middle finger to its adjacent fingers. This allowed us to compare transfer to two fingers that are both physically and cortically adjacent to the middle finger but have differing use patterns. Specifically, the middle finger is used more frequently with the ring than with the index finger. We predicted this should lead to greater representational overlap between the former than the latter pair. Furthermore, this difference in overlap should be reflected in differential learning transfer from the middle to index vs. ring fingers. Subsequently, we predicted temporary learning-related changes in the middle finger's representation (e.g., cortical magnification) would cause transient interference in perceptual thresholds of the ring, but not the index, finger. Supporting this, longitudinal analysis revealed a divergence where learning transfer was fast to the index finger but relatively delayed to the ring finger. Our results support the theory that tactile coactivation patterns between digits affect their topographic relationships. Our findings emphasize how action shapes perception and somatosensory organization.