Instance-based learning with prototype reduction for real-time proportional myocontrol: a randomized user study demonstrating accuracy-preserving data reduction for prosthetic embedded systems

This work presents the design, implementation and validation of learning techniques based on the kNN scheme for gesture detection in prosthetic control. To cope with high computational demands in instance-based prediction, methods of dataset reduction are evaluated considering real-time determinism to allow for the reliable integration into battery-powered portable devices. The influence of parameterization and varying proportionality schemes is analyzed, utilizing an eight-channel-sEMG armband. Besides offline cross-validation accuracy, success rates in real-time pilot experiments (online target achievement tests) are determined. Based on the assessment of specific dataset reduction techniques' adequacy for embedded control applications regarding accuracy and timing behaviour, decision surface mapping (DSM) proves itself promising when applying kNN on the reduced set. A randomized, double-blind user study was conducted to evaluate the respective methods (kNN and kNN with DSM-reduction) against ridge regression (RR) and RR with random Fourier features (RR-RFF). The kNN-based methods performed significantly better ([Formula: see text]) than the regression techniques. Between DSM-kNN and kNN, there was no statistically significant difference (significance level 0.05). This is remarkable in consideration of only one sample per class in the reduced set, thus yielding a reduction rate of over 99% while preserving success rate. The same behaviour could be confirmed in an extended user study. With [Formula: see text], which turned out to be an excellent choice, the runtime complexity of both kNN (in every prediction step) as well as DSM-kNN (in the training phase) becomes linear concerning the number of original samples, favouring dependable wearable prosthesis applications.

Competitive motivation increased home use and improved prosthesis self-perception after Cybathlon 2020 for neuromusculoskeletal prosthesis user

Background

Assistive technologies, such as arm prostheses, are intended to improve the quality of life of individuals with physical disabilities. However, certain training and learning is usually required from the user to make these technologies more effective. Moreover, some people can be encouraged to train more through competitive motivation.

Methods

In this study, we investigated if the training for and participation in a competitive event (Cybathlon 2020) could promote behavioral changes in an individual with upper limb amputation (the pilot). We defined behavioral changes as the active time while his prosthesis was actuated, ratio of opposing and simultaneous movements, and the pilot’s ability to finely modulate his movement speeds. The investigation was based on extensive home-use data from the period before, during and after the Cybathlon 2020 competition.

Results

Relevant behavioral changes were found from both quantitative and qualitative analyses. The pilot’s home use of his prosthesis nearly doubled in the period before the Cybathlon, and remained 66% higher than baseline after the competition. Moreover, he improved his speed modulation when controlling his prosthesis, and he learned and routinely operated new movements in the prosthesis (wrist rotation) at home. Additionally, as confirmed by semi-structured interviews, his self-perception of the prosthetic arm and its functionality also improved.

Conclusions

An event like the Cybathlon may indeed promote behavioral changes in how competitive individuals with amputation use their prostheses. Provided that the prosthesis is suitable in terms of form and function for both competition and at-home daily use, daily activities can become opportunities for training, which in turn can improve prosthesis function and create further opportunities for daily use. Moreover, these changes appeared to remain even well after the event, albeit relevant only for individuals who continue using the technology employed in the competition.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12984-022-01024-4.

Regenerative Peripheral Nerve Interfaces for Advanced Control of Upper Extremity Prosthetic Devices

The quest to find the ideal prosthetic device interface that enables intuitive control has motivated several recent innovations. Although current prosthetic device control strategies have advanced the field of neuroprosthetic control, they are limited in their ability to generate reliable, stable, and specific signals to replicate the complex movements of the upper extremity. The regenerative peripheral nerve interface (RPNI) is a promising solution to enhance prosthetic device control. This article describes the development of RPNIs and summarizes its successful use in the control of advanced prosthetic devices in patients with upper extremity amputations.

Regenerative Peripheral Nerve Interfaces for the Treatment and Prevention of Neuromas and Neuroma Pain

A neuroma occurs when a regenerating transected peripheral nerve has no distal target to reinnervate. This situation can result in a hypersensitive free nerve ending that causes debilitating pain to affected patients. No techniques to treat symptomatic neuromas have shown consistent results. One novel physiologic solution is the regenerative peripheral nerve interface (RPNI). RPNI consists of a transected peripheral nerve that is implanted into an autologous free skeletal muscle graft. Early clinical studies have shown promising results in the use of RPNIs to treat and prevent symptomatic neuromas. This review article describes the rationale behind the success of RPNIs and its clinical applications.

Fascicle-Specific Targeting of Longitudinal Intrafascicular Electrodes for Motor and Sensory Restoration in Upper-Limb Amputees

Multichannel longitudinal intrafascicular electrode (LIFE) interfaces provide optimized balance of invasiveness and stability for chronic sensory stimulation and motor recording/decoding of peripheral nerve signals. Using a fascicle-specific targeting (FAST)-LIFE approach, where electrodes are individually placed within discrete sensory- and motor-related fascicular subdivisions of the residual ulnar and/or median nerves in an amputated upper limb, FAST-LIFE interfacing can provide discernment of motor intent for individual digit control of a robotic hand, and restoration of touch- and movement-related sensory feedback. The authors describe their findings from clinical studies performed with 6 human amputee trials using FAST-LIFE interfacing of the residual upper limb.

Recommendations for the Successful Implementation of Upper Limb Prosthetic Technology

Despite the numerous prosthetic hand designs that are commercially available, people with upper limb loss still frequently report dissatisfaction and abandonment. Over the past decade there have been numerous advances in prosthetic design, control, sensation, and device attachment. Each offers the potential to enhance function and satisfaction, but most come at high costs and involve surgical risks. Here, we discuss potential barriers and solutions to promote the widespread use of novel prosthetic technology. With appropriate reimbursement, multidisciplinary care teams, device-specific rehabilitation, and patient and clinician education, such technology has the potential to revolutionize the field and improve patient outcomes.

In-silico development and assessment of a Kalman filter motor decoder for prosthetic hand control

Up to 50% of amputees abandon their prostheses, partly due to rapid degradation of the control systems, which require frequent recalibration. The goal of this study was to develop a Kalman filter-based approach to decoding motoneuron activity to identify movement kinematics and thereby provide stable, long-term, accurate, real-time decoding. The Kalman filter-based decoder was examined via biologically varied datasets generated from a high-fidelity computational model of the spinal motoneuron pool. The estimated movement kinematics controlled a simulated MuJoCo prosthetic hand. This clear-box approach showed successful estimation of hand movements under eight varied physiological conditions with no retraining. The mean correlation coefficient of 0.98 and mean normalized root mean square error of 0.06 over these eight datasets provide proof of concept that this decoder would improve long-term integrity of performance while performing new, untrained movements. Additionally, the decoder operated in real-time (~0.3 ms). Further results include robust performance of the Kalman filter when re-trained to more severe post-amputation limitations in the type and number of motoneurons remaining. An additional analysis shows that the decoder achieves better accuracy when using the firing of individual motoneurons as input, compared to using aggregate pool firing. Moreover, the decoder demonstrated robustness to noise affecting both the trained decoder parameters and the decoded motoneuron activity. These results demonstrate the utility of a proof of concept Kalman filter decoder that can support prosthetics' control systems to maintain accurate and stable real-time movement performance.

Stable, simultaneous and proportional 4-DoF prosthetic hand control via synergy-inspired linear interpolation: a case series

BACKGROUND

Current commercial prosthetic hand controllers limit patients' ability to fully engage high Degree-of-Freedom (DoF) prosthetic hands. Available feedforward controllers rely on large training data sets for controller setup and a need for recalibration upon prosthesis donning. Recently, an intuitive, proportional, simultaneous, regression-based 3-DoF controller remained stable for several months without retraining by combining chronically implanted electromyography (ciEMG) electrodes with a K-Nearest-Neighbor (KNN) mapping technique. The training dataset requirements for simultaneous KNN controllers increase exponentially with DoF, limiting the realistic development of KNN controllers in more than three DoF. We hypothesize that a controller combining linear interpolation, the muscle synergy framework, and a sufficient number of ciEMG channels (at least two per DoF), can allow stable, high-DoF control.

METHODS

Two trans-radial amputee subjects, S6 and S8, were implanted with percutaneously interfaced bipolar intramuscular electrodes. At the time of the study, S6 and S8 had 6 and 8 bipolar EMG electrodes, respectively. A Virtual Reality (VR) system guided users through single and paired training movements in one 3-DoF and four different 4-DoF cases. A linear model of user activity was built by partitioning EMG feature space into regions bounded by vectors of steady state movement EMG patterns. The controller evaluated online EMG signals by linearly interpolating the movement class labels for surrounding trained EMG movements. This yields a simultaneous, continuous, intuitive, and proportional controller. Controllers were evaluated in 3-DoF and 4-DoF through a target-matching task in which subjects controlled a virtual hand to match 80 targets spanning the available movement space. Match Percentage, Time-To-Target, and Path Efficiency were evaluated over a 10-month period based on subject availability.

RESULTS AND CONCLUSIONS

In 3-DoF, S6 and S8 matched most targets and demonstrated stable control after 8 and 10 months, respectively. In 4-DoF, both subjects initially found two of four 4-DoF controllers usable, matching most targets. S8 4-DoF controllers were stable, and showed improving trends over 7-9 months without retraining or at-home practice. S6 4-DoF controllers were unstable after 7 months without retraining. These results indicate that the performance of the controller proposed in this study may remain stable, or even improve, provided initial viability and a sufficient number of EMG channels. Overall, this study demonstrates a controller capable of stable, simultaneous, proportional, intuitive, and continuous control in 3-DoF for up to ten months and in 4-DoF for up to nine months without retraining or at-home use with minimal training times.

Activities of daily living with bionic arm improved by combination training and latching filter in prosthesis control comparison

BACKGROUND

Advanced prostheses can restore function and improve quality of life for individuals with amputations. Unfortunately, most commercial control strategies do not fully utilize the rich control information from residual nerves and musculature. Continuous decoders can provide more intuitive prosthesis control using multi-channel neural or electromyographic recordings. Three components influence continuous decoder performance: the data used to train the algorithm, the algorithm, and smoothing filters on the algorithm's output. Individual groups often focus on a single decoder, so very few studies compare different decoders using otherwise similar experimental conditions.

METHODS

We completed a two-phase, head-to-head comparison of 12 continuous decoders using activities of daily living. In phase one, we compared two training types and a smoothing filter with three algorithms (modified Kalman filter, multi-layer perceptron, and convolutional neural network) in a clothespin relocation task. We compared training types that included only individual digit and wrist movements vs. combination movements (e.g., simultaneous grasp and wrist flexion). We also compared raw vs. nonlinearly smoothed algorithm outputs. In phase two, we compared the three algorithms in fragile egg, zipping, pouring, and folding tasks using the combination training and smoothing found beneficial in phase one. In both phases, we collected objective, performance-based (e.g., success rate), and subjective, user-focused (e.g., preference) measures.

RESULTS

Phase one showed that combination training improved prosthesis control accuracy and speed, and that the nonlinear smoothing improved accuracy but generally reduced speed. Phase one importantly showed simultaneous movements were used in the task, and that the modified Kalman filter and multi-layer perceptron predicted more simultaneous movements than the convolutional neural network. In phase two, user-focused metrics favored the convolutional neural network and modified Kalman filter, whereas performance-based metrics were generally similar among all algorithms.

CONCLUSIONS

These results confirm that state-of-the-art algorithms, whether linear or nonlinear in nature, functionally benefit from training on more complex data and from output smoothing. These studies will be used to select a decoder for a long-term take-home trial with implanted neuromyoelectric devices. Overall, clinical considerations may favor the mKF as it is similar in performance, faster to train, and computationally less expensive than neural networks.

Learning regularized representations of categorically labelled surface EMG enables simultaneous and proportional myoelectric control

Background

Processing the surface electromyogram (sEMG) to decode movement intent is a promising approach for natural control of upper extremity prostheses. To this end, this paper introduces and evaluates a new framework which allows for simultaneous and proportional myoelectric control over multiple degrees of freedom (DoFs) in real-time. The framework uses multitask neural networks and domain-informed regularization in order to automatically find nonlinear mappings from the forearm sEMG envelope to multivariate and continuous encodings of concurrent hand- and wrist kinematics, despite only requiring categorical movement instruction stimuli signals for calibration.

Methods

Forearm sEMG with 8 channels was collected from healthy human subjects (N = 20) and used to calibrate two myoelectric control interfaces, each with two output DoFs. The interfaces were built from (I) the proposed framework, termed Myoelectric Representation Learning (MRL), and, to allow for comparisons, from (II) a standard pattern recognition framework based on Linear Discriminant Analysis (LDA). The online performances of both interfaces were assessed with a Fitts’s law type test generating 5 quantitative performance metrics. The temporal stabilities of the interfaces were evaluated by conducting identical tests without recalibration 7 days after the initial experiment session.

Results

Metric-wise two-way repeated measures ANOVA with factors method (MRL vs LDA) and session (day 1 vs day 7) revealed a significant (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p<0.05$$\end{document}p<0.05) advantage for MRL over LDA in 5 out of 5 performance metrics, with metric-wise effect sizes (Cohen’s \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d$$\end{document}d) separating MRL from LDA ranging from \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left|d\right|=0.62$$\end{document}d=0.62 to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left|d\right|=1.13$$\end{document}d=1.13. No significant effect on any metric was detected for neither session nor interaction between method and session, indicating that none of the methods deteriorated significantly in control efficacy during one week of intermission.

Conclusions

The results suggest that MRL is able to successfully generate stable mappings from EMG to kinematics, thereby enabling myoelectric control with real-time performance superior to that of the current commercial standard for pattern recognition (as represented by LDA). It is thus postulated that the presented MRL approach can be of practical utility for muscle-computer interfaces.

Virtual reality-based action observation facilitates the acquisition of body-powered prosthetic control skills

Background

Regular body-powered (BP) prosthesis training facilitates the acquisition of skills through repeated practice but requires adequate time and motivation. Therefore, auxiliary tools such as indirect training may improve the training experience and speed of skill acquisition. In this study, we examined the effects of action observation (AO) using virtual reality (VR) as an auxiliary tool. We used two modalities during AO: three-dimensional (3D) VR and two-dimensional (2D) computer tablet devices (Tablet). Each modality was tested from first- and third-person perspectives.

Methods

We studied 40 healthy right-handed participants wearing a BP prosthesis simulator on their non-dominant hands. The participants were divided into five groups based on combinations of the different modalities and perspectives: first-person perspective on VR (VR1), third-person perspective on VR (VR3), first-person perspective on a tablet (Tablet1), third-person perspective on a tablet (Tablet3), and a control group (Control). The intervention groups observed and imitated the video image of prosthesis operation for 10 min in each of two sessions. We evaluated the level of immersion during AO using the visual analogue scale. Prosthetic control skills were evaluated using the Box and Block Test (BBT) and a bowknot task (BKT).

Results

In the BBT, there were no significant differences in the amount of change in the skills between the five groups. In contrast, the relative changes in the BKT prosthetic control skills in VR1 (p < 0.001, d = 3.09) and VR3 (p < 0.001, d = 2.16) were significantly higher than those in the control group. Additionally, the immersion scores of VR1 (p < 0.05, d = 1.45) and VR3 (p < 0.05, d = 1.18) were higher than those of Tablet3. There was a significant negative correlation between the immersion scores and the relative change in the BKT scores (Spearman’s r_s = − 0.47, p < 0.01).

Conclusions

Using the BKT of bilateral manual dexterity, VR-based AO significantly improved short-term prosthetic control acquisition. Additionally, it appeared that the higher the immersion score was, the shorter the execution time of the BKT task. Our findings suggest that VR-based AO training may be effective in acquiring bilateral BP prosthetic control, which requires more 3D-based operation.

Methodology for creating a chronic osseointegrated neural interface for prosthetic control in rabbits

BACKGROUND

Chronic stability and high degrees of selectivity are both essential but somewhat juxtaposed components for creating an implantable bi-directional PNI capable of controlling of a prosthetic limb. While the more invasive implantable electrode arrays provide greater specificity, they are less stable over time due to compliance mismatch with the dynamic soft tissue environment in which the interface is created.

NEW METHOD

This paper takes the surgical approach of transposing nerves into bone to create neural interface within the medullary canal of long bones, an osseointegrated neural interface, to provide greater stability for implantable electrodes. In this context, we describe the surgical model for transfemoral amputation with transposition of the sciatic nerve into the medullary canal in rabbits. We investigate the capacity to create a neural interface within the medullary canal histolomorphologically. In a separate proof of concept experiment, we quantify the chronic physiological capacity of transposed nerves to conduct compound nerve action potentials evoked via an Osseointegrated Neural Interface.

COMPARISON WITH EXISTING METHOD(S)

The rabbit serves as an important animal model for both amputation neuroma and osseointegration research, but is underutilized for the exploration neural interfacing in an amputation setting.

RESULTS

Our findings demonstrate that transposed nerves remain stable over 12 weeks. Creating a neural interface within the medullary canal is possible and does not impede nerve regeneration or physiological capacity.

CONCLUSIONS

This article represents the first evidence that an Osseointegrated Neural Interface can be surgically created, capable of chronic stimulation/recording from amputated nerves required for future prosthetic control.

Grip control and motor coordination with implanted and surface electrodes while grasping with an osseointegrated prosthetic hand

Background

Replacement of a lost limb by an artificial substitute is not yet ideal. Resolution and coordination of motor control approximating that of a biological limb could dramatically improve the functionality of prosthetic devices, and thus reduce the gap towards a suitable limb replacement.

Methods

In this study, we investigated the control resolution and coordination exhibited by subjects with transhumeral amputation who were implanted with epimysial electrodes and an osseointegrated interface that provides bidirectional communication in addition to skeletal attachment (e-OPRA Implant System). We assessed control resolution and coordination in the context of routine and delicate grasping using the Pick and Lift and the Virtual Eggs Tests. Performance when utilizing implanted electrodes was compared with the standard-of-care technology for myoelectric prostheses, namely surface electrodes.

Results

Results showed that implanted electrodes provide superior controllability over the prosthetic terminal device compared to conventional surface electrodes. Significant improvements were found in the control of the grip force and its reliability during object transfer. However, these improvements failed to increase motor coordination, and surprisingly decreased the temporal correlation between grip and load forces observed with surface electrodes. We found that despite being more functional and reliable, prosthetic control via implanted electrodes still depended highly on visual feedback.

Conclusions

Our findings indicate that incidental sensory feedback (visual, auditory, and osseoperceptive in this case) is insufficient for restoring natural grasp behavior in amputees, and support the idea that supplemental tactile sensory feedback is needed to learn and maintain the motor tasks internal model, which could ultimately restore natural grasp behavior in subjects using prosthetic hands.

Online mapping of EMG signals into kinematics by autoencoding

Background

In this paper, we propose a nonlinear minimally supervised method based on autoencoding (AEN) of EMG for myocontrol. The proposed method was tested against the state-of-the-art (SOA) control scheme using a Fitts’ law approach.

Methods

Seven able-bodied subjects performed a series of target acquisition myoelectric control tasks using the AEN and SOA algorithms for controlling two degrees-of-freedom (radial/ulnar deviation and flexion/extension of the wrist), and their online performance was characterized by six metrics.

Results

Both methods allowed a completion rate close to 100%, however AEN outperformed SOA for all other performance metrics, e.g. it allowed to perform the tasks on average in half the time with respect to SOA. Moreover, the amount of information transferred by the proposed method in bit/s was nearly twice the throughput of SOA.

Conclusions

These results show that autoencoders can map EMG signals into kinematics with the potential of providing intuitive and dexterous control of artificial limbs for amputees.

An Alternative Myoelectric Pattern Recognition Approach for the Control of Hand Prostheses: A Case Study of Use in Daily Life by a Dysmelia Subject

The functionality of upper limb prostheses can be improved by intuitive control strategies that use bioelectric signals measured at the stump level. One such strategy is the decoding of motor volition via myoelectric pattern recognition (MPR), which has shown promising results in controlled environments and more recently in clinical practice. Moreover, not much has been reported about daily life implementation and real-time accuracy of these decoding algorithms. This paper introduces an alternative approach in which MPR allows intuitive control of four different grips and open/close in a multifunctional prosthetic hand. We conducted a clinical proof-of-concept in activities of daily life by constructing a self-contained, MPR-controlled, transradial prosthetic system provided with a novel user interface meant to log errors during real-time operation. The system was used for five days by a unilateral dysmelia subject whose hand had never developed, and who nevertheless learned to generate patterns of myoelectric activity, reported as intuitive, for multi-functional prosthetic control. The subject was instructed to manually log errors when they occurred via the user interface mounted on the prosthesis. This allowed the collection of information about prosthesis usage and real-time classification accuracy. The assessment of capacity for myoelectric control test was used to compare the proposed approach to the conventional prosthetic control approach, direct control. Regarding the MPR approach, the subject reported a more intuitive control when selecting the different grips, but also a higher uncertainty during proportional continuous movements. This paper represents an alternative to the conventional use of MPR, and this alternative may be particularly suitable for a certain type of amputee patients. Moreover, it represents a further validation of MPR with dysmelia cases.