Performance limitations from delay in human and mechanical motor control

A portable, self-contained neuroprosthetic hand with deep learning-based finger control

Objective.Deep learning-based neural decoders have emerged as the prominent approach to enable dexterous and intuitive control of neuroprosthetic hands. Yet few studies have materialized the use of deep learning in clinical settings due to its high computational requirements.Approach.Recent advancements of edge computing devices bring the potential to alleviate this problem. Here we present the implementation of a neuroprosthetic hand with embedded deep learning-based control. The neural decoder is designed based on the recurrent neural network architecture and deployed on the NVIDIA Jetson Nano-a compacted yet powerful edge computing platform for deep learning inference. This enables the implementation of the neuroprosthetic hand as a portable and self-contained unit with real-time control of individual finger movements.Main results.A pilot study with a transradial amputee is conducted to evaluate the proposed system using peripheral nerve signals acquired from implanted intrafascicular microelectrodes. The preliminary experiment results show the system's capabilities of providing robust, high-accuracy (95%-99%) and low-latency (50-120 ms) control of individual finger movements in various laboratory and real-world environments.Conclusion.This work is a technological demonstration of modern edge computing platforms to enable the effective use of deep learning-based neural decoders for neuroprosthesis control as an autonomous system.Significance.The proposed system helps pioneer the deployment of deep neural networks in clinical applications underlying a new class of wearable biomedical devices with embedded artificial intelligence.Clinical trial registration: DExterous Hand Control Through Fascicular Targeting (DEFT). Identifier: NCT02994160.

Short-delay neurofeedback facilitates training of the parietal alpha rhythm

OBJECTIVE

Feedback latency was shown to be a critical parameter in a range of applications that imply learning. The therapeutic effects of neurofeedback (NFB) remain controversial. We hypothesized that often encountered unreliable results of NFB intervention could be associated with large feedback latency values that are often uncontrolled and may preclude the efficient learning.

APPROACH

We engaged our subjects into a parietal alpha power unpregulating paradigm faciliated by visual neurofeedback based on the invidually extracted envelope of the alpha-rhythm at P4 electrode. NFB was displayed either as soon as EEG envelope was processed, or with an extra 250 or 500-ms delay. The feedback training consisted of 15 two-minute long blocks interleaved with 15s pauses. We have also recorded two minute long baselines immediately before and after the training.

MAIN RESULTS

The time course of NFB-induced changes in the alpha rhythm power clearly depended on NFB latency, as shown with the adaptive Neyman test. NFB had a strong effect on the alpha-spindle incidence rate, but not on their duration or amplitude. The sustained changes in alpha activity measured after the completion of NFB training were negatively correlated to latency, with the maximum change for the shortest tested latency and no change for the longest.

SIGNIFICANCE

Here we for the first time show that visual NFB of parietal electroencephalographic (EEG) alpha-activity is efficient only when delivered to human subjects at short latency, which guarantees that NFB arrives when an alpha spindle is still ongoing. Such a considerable effect of NFB latency on the alpha-activity temporal structure could explain some of the previous inconsistent results, where latency was neither controlled nor documented. Clinical practitioners and manufacturers of NFB equipment should add latency to their specifications while enabling latency monitoring and supporting short-latency operations.

The Effects of Low Latency on Pointing and Steering Tasks

Latency is detrimental to interactive systems, especially pseudo-physical systems that emulate real-world behaviour. It prevents users from making quick corrections to their movement, and causes their experience to deviate from their expectations. Latency is a result of the processing and transport delays inherent in current computer systems. As such, while a number of studies have hypothesized that any latency will have a degrading effect, few have been able to test this for latencies less than ∼ 50 ms. In this study we investigate the effects of latency on pointing and steering tasks. We design an apparatus with a latency lower than typical interactive systems, using it to perform interaction tasks based on Fitts's law and the Steering law. We find evidence that latency begins to affect performance at ∼ 16 ms, and that the effect is non-linear. Further, we find latency does not affect the various components of an aiming motion equally. We propose a three stage characterisation of pointing movements with each stage affected independently by latency. We suggest that understanding how users execute movement is essential for studying latency at low levels, as high level metrics such as total movement time may be misleading.

Human strategies in balancing an inverted pendulum with time delay

The strategy of human manual control is investigated in balancing an inverted pendulum under time-delay constraints. Experiments show that as the task becomes more difficult due to the increase in time delay, the human operator adopts a more discrete-like strategy. Interpretation of the discrete-control mechanism is provided by relating the observed human responses with a human-performance model.

Estimation of psychomotor delay from the Fitts' law coefficients

An intrinsic property of human motor behavior is a trade-off between speed and accuracy. This is classically described by Fitts' law, a model derived by assuming that the human body has a limited capacity to transmit information in organizing motor behavior. However, Fitts' law can also be realized as an emergent property of movements generated by delayed feedback. In this article, we describe the relationship between the Fitts' law coefficients and the physiological parameters of the underlying delayed feedback circuit: the relaxation rate or time constant, and the psychomotor delay of the feedback process. This relationship is then used to estimate the motor circuit delay of several tasks for which Fitts' law data are available in the literature. We consistently estimate the delay to be between 0 and 112 ms. A further consequence of this model is that not all combinations of slope and Y-intercept in Fitts' law are possible when movements are generated by delayed feedback. In fact, it is only possible for an observed speed-accuracy trade-off to be generated by delayed feedback if the Fitts' law coefficients satisfy -0.482 < or = a/b < or = 3.343 [bits] where b represents the slope in bits per second and a represents the Y-intercept in seconds. If we assume human movement is generated by delayed feedback, then the Fitts' law coefficients should always be restricted to this range of values.