Encoding contact size using static and dynamic electrotactile finger stimulation: natural decoding vs. trained cues

Electrotactile stimulation through matrix electrodes is a promising technology to restore high-resolution tactile feedback in extended reality applications. One of the fundamental tactile effects that should be simulated is the change in the size of the contact between the finger and a virtual object. The present study investigated how participants perceive the increase of stimulation area when stimulating the index finger using static or dynamic (moving) stimuli produced by activating 1 to 6 electrode pads. To assess the ability to interpret the stimulation from the natural cues (natural decoding), without any prior training, the participants were instructed to draw the size of the stimulated area and identify the size difference when comparing two consecutive stimulations. To investigate if other "non-natural" cues can improve the size estimation, the participants were asked to enumerate the number of active pads following a training protocol. The results demonstrated that participants could perceive the change in size without prior training (e.g., the estimated area correlated with the stimulated area, p < 0.001; ≥ two-pad difference recognized with > 80% success rate). However, natural decoding was also challenging, as the response area changed gradually and sometimes in complex patterns when increasing the number of active pads (e.g., four extra pads needed for the statistically significant difference). Nevertheless, by training the participants to utilize additional cues the limitations of natural perception could be compensated. After the training, the mismatch in the activated and estimated number of pads was less than one pad regardless of the stimulus size. Finally, introducing the movement of the stimulus substantially improved discrimination (e.g., 100% median success rate to recognize ≥ one-pad difference). The present study, therefore, provides insights into stimulation size perception, and practical guidelines on how to modulate pad activation to change the perceived size in static and dynamic scenarios.

Using mobile eye tracking to measure cognitive load through gaze behavior during walking in lower limb prosthesis users: A preliminary assessment

BACKGROUND

Lower limb amputation does not affect only physical and psychological functioning but the use of a prosthetic device can also lead to increased cognitive demands. Measuring cognitive load objectively is challenging, and therefore, most studies use questionnaires that are easy to apply but can suffer from subjective bias. Motivated by this, the present study investigated whether a mobile eye tracker can be used to objectively measure cognitive load by monitoring gaze behavior during a set of motor tasks.

METHODS

Five prosthetic users and eight able-bodied controls participated in this study. Eye tracking data and kinematics were recorded during a set of motor tasks (level ground walking, walking on uneven terrain, obstacle avoidance, stairs up and ramp down, as well as ramp up and stairs down) while the participants were asked to focus their gaze on a visual target for as long as possible. Target fixation times and increase in pupil diameters were determined and correlated to subjective ratings of cognitive load.

FINDINGS

Overall, target fixation time and pupil diameter showed strong negative and positive correlations, respectively, to the subjective rating of cognitive load in the able-bodied controls (-0.75 and 0.80, respectively). However, the individual correlation strength, and in some cases, even the sign, was different across participants. A similar trend could be observed in prosthetic users.

INTERPRETATION

The results of this study showed that a mobile eye tracker may be used to estimate cognitive load in prosthesis users during locomotor tasks. This paves the way to establish a new approach to assessing cognitive load, which is objective and yet practical and simple to administer. Nevertheless, future studies should corroborate these results by comparing them to other objective measures as well as focus on translating the proposed approach outside of a laboratory.

Measuring and monitoring skill learning in closed-loop myoelectric hand prostheses using speed-accuracy tradeoffs

Objective.Closed-loop myoelectric prostheses, which combine supplementary sensory feedback and electromyography (EMG) based control, hold the potential to narrow the divide between natural and bionic hands. The use of these devices, however, requires dedicated training. Therefore, it is crucial to develop methods that quantify how users acquire skilled control over their prostheses to effectively monitor skill progression and inform the development of interfaces that optimize this process.Approach.Building on theories of skill learning in human motor control, we measured speed-accuracy tradeoff functions (SAFs) to comprehensively characterize learning-induced changes in skill-as opposed to merely tracking changes in task success across training-facilitated by a closed-loop interface that combined proportional control and EMG feedback. Sixteen healthy participants and one individual with a transradial limb loss participated in a three-day experiment where they were instructed to perform the box-and-blocks task using a timed force-matching paradigm at four specified speeds to reach two target force levels, such that the SAF could be determined.Main results.We found that the participants' accuracy increased in a similar way across all speeds we tested. Consequently, the shape of the SAF remained similar across days, at both force levels. Further, we observed that EMG feedback enabled participants to improve their motor execution in terms of reduced trial-by-trial variability, a hallmark of skilled behavior. We then fit a power law model of the SAF, and demonstrated how the model parameters could be used to identify and monitor changes in skill.Significance.We comprehensively characterized how an EMG feedback interface enabled skill acquisition, both at the level of task performance and movement execution. More generally, we believe that the proposed methods are effective for measuring and monitoring user skill progression in closed-loop prosthesis control.

Brain computer interface training with motor imagery and functional electrical stimulation for patients with severe upper limb paresis after stroke: a randomized controlled pilot trial

BACKGROUND

Restorative Brain-Computer Interfaces (BCI) that combine motor imagery with visual feedback and functional electrical stimulation (FES) may offer much-needed treatment alternatives for patients with severely impaired upper limb (UL) function after a stroke.

OBJECTIVES

This study aimed to examine if BCI-based training, combining motor imagery with FES targeting finger/wrist extensors, is more effective in improving severely impaired UL motor function than conventional therapy in the subacute phase after stroke, and if patients with preserved cortical-spinal tract (CST) integrity benefit more from BCI training.

METHODS

Forty patients with severe UL paresis (< 13 on Action Research Arm Test (ARAT) were randomized to either a 12-session BCI training as part of their rehabilitation or conventional UL rehabilitation. BCI sessions were conducted 3-4 times weekly for 3-4 weeks. At baseline, Transcranial Magnetic Stimulation (TMS) was performed to examine CST integrity. The main endpoint was the ARAT at 3 months post-stroke. A binominal logistic regression was conducted to examine the effect of treatment group and CST integrity on achieving meaningful improvement. In the BCI group, electroencephalographic (EEG) data were analyzed to investigate changes in event-related desynchronization (ERD) during the course of therapy.

RESULTS

Data from 35 patients (15 in the BCI group and 20 in the control group) were analyzed at 3-month follow-up. Few patients (10/35) improved above the minimally clinically important difference of 6 points on ARAT, 5/15 in the BCI group, 5/20 in control. An independent-samples Mann-Whitney U test revealed no differences between the two groups, p = 0.382. In the logistic regression only CST integrity was a significant predictor for improving UL motor function, p = 0.007. The EEG analysis showed significant changes in ERD of the affected hemisphere and its lateralization only during unaffected UL motor imagery at the end of the therapy.

CONCLUSION

This is the first RCT examining BCI training in the subacute phase where only patients with severe UL paresis were included. Though more patients in the BCI group improved relative to the group size, the difference between the groups was not significant. In the present study, preserved CTS integrity was much more vital for UL improvement than which type of intervention the patients received. Larger studies including only patients with some preserved CST integrity should be attempted.

A Novel Method for Vibrotactile Proprioceptive Feedback Using Spatial Encoding and Gaussian Interpolation

OBJECTIVE

The bidirectional communication between the user and the prosthesis is an important requirement when developing prosthetic hands. Proprioceptive feedback is fundamental to perceiving prosthesis movement without the need for constant visual attention. We propose a novel solution to encode wrist rotation using a vibromotor array and Gaussian interpolation of vibration intensity. The approach generates tactile sensation that smoothly rotates around the forearm congruently with prosthetic wrist rotation. The performance of this scheme was systematically assessed for a range of parameter values (number of motors and Gaussian standard deviation).

METHODS

Fifteen non-disabled subjects and one individual with congenital limb deficiency used vibrational feedback to control a virtual hand in the target-achievement control test. Performance was assessed by end-point error and efficiency as well as subjective impressions.

RESULTS

The results showed a preference for smooth feedback and a higher number of motors (8 and 6 versus 4). With 8 and 6 motors, the standard deviation, determining the sensation spread and continuity, could be modulated through a broad range of values (0.1 - 2) without a significant performance loss. The overall average error and efficiency across these feedback configurations were ∼ 10% and ∼ 30%, respectively. For low values of standard deviation (0.1-0.5), the number of motors could be reduced to 4 without a significant performance decrease.

CONCLUSION

The study demonstrated that the developed strategy provided meaningful rotation feedback. Moreover, the results indicate that the Gaussian standard deviation could be used as an independent parameter to encode an additional feedback variable.

SIGNIFICANCE

The proposed method is a flexible and effective approach to provide proprioceptive feedback while adjusting the trade-off between sensation quality and the number of vibromotors.

A Novel Sensory Feedback Approach to Facilitate Both Predictive and Corrective Control of Grasping Force in Myoelectric Prostheses

Reliable force control is especially important when using myoelectric upper-limb prostheses as the force defines whether an object will be firmly grasped, damaged, or dropped. It is known from human motor control that the grasping of non-disabled subjects is based on a combination of anticipation and feedback correction. Inspired by this insight, the present study proposes a novel approach to provide artificial sensory feedback to the user of a myoelectric prosthesis using vibrotactile stimulation to facilitate both predictive and corrective processes characteristic of grasping in non-disabled people. Specifically, the level of EMG was conveyed to the subjects while closing the prosthesis (predictive strategy), whereas the actual grasping force was transmitted when the prosthesis closed (corrective strategy). To investigate if this combined EMG and force feedback is indeed an effective method to explicitly close the control loop, 16 non-disabled and 3 transradial amputee subjects performed a set of functional tasks, inspired by the "Box and Block" test, with six target force levels, in three conditions: no feedback, only EMG feedback, and combined feedback. The highest overall performance in non-disabled subjects was obtained with combined feedback (79.6±9.9%), whereas the lowest was achieved with no feedback (53±11.5%). The combined feedback, however, increased the task completion time compared to the other two conditions. A similar trend was obtained also in three amputee subjects. The results, therefore, indicate that the feedback inspired by human motor control is indeed an effective approach to improve prosthesis grasping in realistic conditions when other sources of feedback (vision and audition) are not blocked.

Toward self-contained bidirectional bionic limbs with high information throughput

Surgical neural engineering and human-machine interfacing enable bionic limbs with dexterous control and sensory feedback.

Novel Electrode Designs for Electrotactile Stimulation of the Finger: A Comparative Assessment

Electrotactile stimulation can be an attractive technology to restore tactile feedback in different application scenarios (e.g., virtual and augmented reality, tele-manipulation). This technology allows designing compact solutions with no mechanical elements that can integrate a high-density matrix of stimulation points. The present study introduced four novel multi-pad finger-electrode designs with different arrangements (two matrix and two circular) and shapes of active pads (producing sensation) and reference pads (ideally, no sensation produced below the pad). The electrodes were used to investigate the subjects' ability to spatially discriminate active pads within phalanges individually (6-9 pads) as well as across the full finger (18-19 pads). The tests were conducted in 12 subjects and the results showed that all designs led to high success rates when applied to the fingertip (70-81%). When tested on the full finger, the matrix and circular designs were characterized with similar performance (54-57%), and when the phalanges were analyzed individually, the spatial discrimination was best at the fingertip. Additionally, new approaches for faster amplitude calibration were proposed and tested, demonstrating that calibration duration can be reduced by approximately 40% compared to the standard approach of calibrating single pads individually. Finally, discrimination tests of dynamic tactile patterns were conducted using circular and matrix designs on the fingertip and full finger, respectively. The tests showed that the different patterns generated by the two arrangements could be clearly discriminated, especially in the case of full-finger matrix-style patterns. The present study, therefore, provides several important insights that are relevant when delivering tactile feedback to the finger using an electrotactile interface.

EMG feedback improves grasping of compliant objects using a myoelectric prosthesis

BACKGROUND

Closing the control loop in myoelectric prostheses by providing artificial somatosensory feedback is recognized as an important goal. However, designing a feedback interface that is effective in realistic conditions is still a challenge. Namely, in some situations, feedback can be redundant, as the information it provides can be readily obtained through hearing or vision (e.g., grasping force estimated from the deformation of a compliant object). EMG feedback is a non-invasive method wherein the tactile stimulation conveys to the user the level of their own myoelectric signal, hence a measurement intrinsic to the interface, which cannot be accessed incidentally.

METHODS

The present study investigated the efficacy of EMG feedback in prosthesis force control when 10 able-bodied participants and a person with transradial amputation used a myoelectric prosthesis to grasp compliant objects of different stiffness values. The performance with feedback was compared to that achieved when the participants relied solely on incidental cues.

RESULTS

The main outcome measures were the task success rate and completion time. EMG feedback resulted in significantly higher success rates regardless of pin stiffness, indicating that the feedback enhanced the accuracy of force application despite the abundance of incidental cues. Contrary to expectations, there was no difference in the completion time between the two feedback conditions. Additionally, the data revealed that the participants could produce smoother control signals when they received EMG feedback as well as more consistent commands across trials, signifying better control of the system by the participants.

CONCLUSIONS

The results presented in this study further support the efficacy of EMG feedback when closing the prosthesis control loop by demonstrating its benefits in particularly challenging conditions which maximized the utility of intrinsic feedback sources.

The impact of walking on the perception of multichannel electrotactile stimulation in individuals with lower-limb amputation and able-bodied participants

BACKGROUND

One of the drawbacks of lower-limb prostheses is that they do not provide explicit somatosensory feedback to their users. Electrotactile stimulation is an attractive technology to restore such feedback because it enables compact solutions with multiple stimulation points. This allows stimulating a larger skin area to provide more information concurrently and modulate parameters spatially as well as in amplitude. However, for effective use, electrotactile stimulation needs to be calibrated and it would be convenient to perform this procedure while the subject is seated. However, amplitude and spatial perception can be affected by motion and/or physical coupling between the residual limb and the socket. In the present study, we therefore evaluated and compared the psychometric properties of multichannel electrotactile stimulation applied to the thigh/residual limb during sitting versus walking.

METHODS

The comprehensive assessment included the measurement of the sensation and discomfort thresholds (ST & DT), just noticeable difference (JND), number of distinct intervals (NDI), two-point discrimination threshold (2PD), and spatial discrimination performance (SD). The experiment involved 11 able-bodied participants (4 females and 7 males; 29.2 ± 3.8 years), 3 participants with transtibial amputation, and 3 participants with transfemoral amputation.

RESULTS

In able-bodied participants, the results were consistent for all the measured parameters, and they indicated that both amplitude and spatial perception became worse during walking. More specifically, ST and DT increased significantly during walking vs. sitting (2.90 ± 0.82 mA vs. 2.00 ± 0.52 mA; p < 0.001 for ST and 7.74 ± 0.84 mA vs. 7.21 ± 1.30 mA; p < 0.05 for DT) and likewise for the JND (22.47 ± 12.21% vs. 11.82 ± 5.07%; p < 0.01), while the NDI became lower (6.46 ± 3.47 vs. 11.27 ± 5.18 intervals; p < 0.01). Regarding spatial perception, 2PD was higher during walking (69.78 ± 17.66 mm vs. 57.85 ± 14.87 mm; p < 0.001), while the performance of SD was significantly lower (56.70 ± 10.02% vs. 64.55 ± 9.44%; p < 0.01). For participants with lower-limb amputation, the ST, DT, and performance in the SD assessment followed the trends observed in the able-bodied population. The results for 2PD and JND were however different and subject-specific.

CONCLUSION

The conducted evaluation demonstrates that electrotactile feedback should be calibrated in the conditions in which it will be used (e.g., during walking). The calibration during sitting, while more convenient, might lead to an overly optimistic (or in some cases pessimistic) estimate of sensitivity. In addition, the results underline that calibration is particularly important in people affected by lower-limb loss to capture the substantial variability in the conditions of the residual limb and prosthesis setup. These insights are important for the implementation of artificial sensory feedback in lower-limb prosthetics applications.

Encoding force modulation in two electrotactile feedback parameters strengthens sensory integration according to maximum likelihood estimation

Bidirectional human-machine interfaces involve commands from the central nervous system to an external device and feedback characterizing device state. Such feedback may be elicited by electrical stimulation of somatosensory nerves, where a task-relevant variable is encoded in stimulation amplitude or frequency. Recently, concurrent modulation in amplitude and frequency (multimodal encoding) was proposed. We hypothesized that feedback with multimodal encoding may effectively be processed by the central nervous system as two independent inputs encoded in amplitude and frequency, respectively, thereby increasing state estimate quality in accordance with maximum-likelihood estimation. Using an adaptation paradigm, we tested this hypothesis during a grasp force matching task where subjects received electrotactile feedback encoding instantaneous force in amplitude, frequency, or both, in addition to their natural force feedback. The results showed that adaptations in grasp force with multimodal encoding could be accurately predicted as the integration of three independent inputs according to maximum-likelihood estimation: amplitude modulated electrotactile feedback, frequency modulated electrotactile feedback, and natural force feedback (r² = 0.73). These findings show that multimodal electrotactile feedback carries an intrinsic advantage for state estimation accuracy with respect to single-variable modulation and suggest that this scheme should be the preferred strategy for bidirectional human-machine interfaces with electrotactile feedback.

Nonlinear Mapping From EMG to Prosthesis Closing Velocity Improves Force Control With EMG Biofeedback

When using EMG biofeedback to control the grasping force of a myoelectric prosthesis, subjects need to activate their muscles and maintain the myoelectric signal within an appropriate interval. However, their performance decreases for higher forces, because the myoelectric signal is more variable for stronger contractions. Therefore, the present study proposes to implement EMG biofeedback using nonlinear mapping, in which EMG intervals of increasing size are mapped to equal-sized intervals of the prosthesis velocity. To validate this approach, 20 non-disabled subjects performed force-matching tasks using Michelangelo prosthesis with and without EMG biofeedback with linear and nonlinear mapping. Additionally, four transradial amputees performed a functional task in the same feedback and mapping conditions. The success rate in producing desired force was significantly higher with feedback (65.4±15.9%) compared to no feedback (46.2±14.9%) as well as when using nonlinear (62.4±16.8%) versus linear mapping (49.2±17.2%). Overall, in non-disabled subjects, the highest success rate was obtained when EMG biofeedback was combined with nonlinear mapping (72%), and the opposite for linear mapping with no feedback (39.6%). The same trend was registered also in four amputee subjects. Therefore, EMG biofeedback improved prosthesis force control, especially when combined with nonlinear mapping, which showed to be an effective approach to counteract increasing variability of myoelectric signal for stronger contractions.

A hierarchical sensorimotor control framework for human-in-the-loop robotic hands

Human manual dexterity relies critically on touch. Robotic and prosthetic hands are much less dexterous and make little use of the many tactile sensors available. We propose a framework modeled on the hierarchical sensorimotor controllers of the nervous system to link sensing to action in human-in-the-loop, haptically enabled, artificial hands.

Using embedded prosthesis sensors for clinical gait analyses in people with lower limb amputation: A feasibility study

BACKGROUND

Biomechanical gait analyses are typically performed in laboratory settings, and are associated with limitations due to space, marker placement, and tasks that are not representative of the real-world usage of lower limb prostheses. Therefore, the purpose of this study was to investigate the possibility of accurately measuring gait parameters using embedded sensors in a microprocessor-controlled knee joint.

METHODS

Ten participants were recruited for this study and equipped with a Genium X3 prosthetic knee joint. They performed level walking, stair/ramp descent, and ascent. During these tasks, kinematics and kinetics (sagittal knee and thigh segment angle, and knee moment) were recorded using an optical motion capture system and force plates (gold standard), as well as the prosthesis-embedded sensors. Root mean square errors, relative errors, correlation coefficients, and discrete outcome variables of clinical relevance were calculated and compared between the gold standard and the embedded sensors.

FINDINGS

The average root mean square errors were found to be 0.6°, 5.3°, and 0.08 Nm/kg, for the knee angle, thigh angle, and knee moment, respectively. The average relative errors were 0.75% for the knee angle, 11.67% for the thigh angle, and 9.66%, for the knee moment. The discrete outcome variables showed small but significant differences between the two measurement systems for a number of tasks (higher differences only at the thigh).

INTERPRETATION

The findings highlight the potential of prosthesis-embedded sensors to accurately measure gait parameters across a wide range of tasks. This paves the way for assessing prosthesis performance in realistic environments outside the lab.

Closed-loop control of a multifunctional myoelectric prosthesis with full-state anatomically congruent electrotactile feedback

State-of-the-art myoelectric hand prostheses provide multi-functional control but lack somatosensory feedback. To accommodate the full functionality of a dexterous prosthesis, the artificial sensory feedback needs to convey several degrees of freedom (DoF) simultaneously. However, this is a challenge with current methods as they are characterized by a low information bandwidth. In this study, we leverage the flexibility of a recently developed system for simultaneous electrotactile stimulation and electromyography (EMG) recording to present the first solution for closed-loop myoelectric control of a multifunctional prosthesis with full-state anatomically congruent electrotactile feedback. The novel feedback scheme (coupled encoding) conveyed proprioceptive (hand aperture, wrist rotation) and exteroceptive information (grasping force). The coupled encoding was compared to the conventional approach (sectorized encoding) and incidental feedback in 10 able-bodied and one amputee participant who used the system to perform a functional task. The results showed that both feedback approaches increased the accuracy of position control compared to incidental feedback. However, the feedback increased completion time, and it did not significantly improve grasping force control. Importantly, the performance of the coupled feedback was not significantly different compared to the conventional scheme, despite the latter being easier to learn during training. Overall, the results indicate that the developed feedback can improve prosthesis control across multiple DoFs but they also highlight the subjects' ability to exploit minimal incidental information. Importantly, the current setup is the first to convey three feedback variables simultaneously using electrotactile stimulation while providing multi-DoF myoelectric control with all hardware components mounted on the same forearm.

Biomimetic Tendon-Based Mechanism for Finger Flexion and Extension in a Soft Hand Exoskeleton: Design and Experimental Assessment

This study proposes a bioinspired exotendon routing configuration for a tendon-based mechanism to provide finger flexion and extension that utilizes a single motor to reduce the complexity of the system. The configuration was primarily inspired by the extrinsic muscle-tendon units of the human musculoskeletal system. The function of the intrinsic muscle-tendon units was partially compensated by adding a minor modification to the configuration of the extrinsic units. The finger kinematics produced by this solution during flexion and extension were experimentally evaluated on an artificial finger and compared to that obtained using the traditional mechanism, where one exotendon was inserted at the distal phalanx. The experiments were conducted on nine healthy subjects who wore a soft exoskeleton glove equipped with the novel tendon mechanism. Contrary to the traditional approach, the proposed mechanism successfully prevented the hyperextension of the distal interphalangeal (DIP) and the metacarpophalangeal (MCP) joints. During flexion, the DIP joint angles produced by the novel mechanism were smaller than the angles generated by the traditional approach for the same proximal interphalangeal (PIP) joint angles. This provided a flexion trajectory closer to the voluntary flexion motion and avoided straining the interphalangeal coupling between the DIP and PIP joints. Finally, the proposed solution generated similar trajectories when applied to a stiff artificial finger (simulating spasticity). The results, therefore, demonstrate that the proposed approach is indeed an effective solution for the envisioned soft hand exoskeleton system.

A Novel Screen-Printed Textile Interface for High-Density Electromyography Recording

Recording electrical muscle activity using a dense matrix of detection points (high-density electromyography, EMG) is of interest in a range of different applications, from human-machine interfacing to rehabilitation and clinical assessment. The wider application of high-density EMG is, however, limited as the clinical interfaces are not convenient for practical use (e.g., require conductive gel/cream). In the present study, we describe a novel dry electrode (TEX) in which the matrix of sensing pads is screen printed on textile and then coated with a soft polymer to ensure good skin-electrode contact. To benchmark the novel solution, an identical electrode was produced using state-of-the-art technology (polyethylene terephthalate with hydrogel, PET) and a process that ensured a high-quality sample. The two electrodes were then compared in terms of signal quality as well as functional application. The tests showed that the signals collected using PET and TEX were characterised by similar spectra, magnitude, spatial distribution and signal-to-noise ratio. The electrodes were used by seven healthy subjects and an amputee participant to recognise seven hand gestures, leading to similar performance during offline analysis and online control. The comprehensive assessment, therefore, demonstrated that the proposed textile interface is an attractive solution for practical applications.

Continuous Transition Impairs Discrimination of Electrotactile Frequencies

Just-noticeable difference (JND), indicating the ability to accurately identify small differences in stimulation parameters, can be used to choose more sensitive stimulation methods as well as to calibrate tactile feedback in closed-loop human-machine interfacing. The JND is typically estimated using a forced-choice-discrimination task, in which two stimuli with different intensities are delivered separated by a brief pause. In the applications of tactile feedback, however, the stimulation parameters are typically modulated continuously. It is unclear if the discriminability of stimuli separated in time characterizes the ability to distinguish continuous changes in stimulation intensity. The present study compared the JND when pairs of frequency-modulated electrotactile stimuli were separated in time and presented continuously at two different baseline frequencies (20 and 60 Hz). The results showed that the JND was significantly smaller with time-separation between stimuli, but that the JND obtained with different types of transitions were in most cases linearly associated. In conclusion, the discriminability of time-separated stimuli is systematically better compared to that of the stimuli presented continuously. This can have an impact when calibrating the tactile feedback where the conventional method of the JND assessment might lead to an overly optimistic estimate of detectable changes.

EMG feedback outperforms force feedback in the presence of prosthesis control disturbance

Closing the prosthesis control loop by providing artificial somatosensory feedback can improve utility and user experience. Additionally, closed-loop control should be more robust with respect to disturbance, but this might depend on the type of feedback provided. Thus, the present study investigates and compares the performance of EMG and force feedback in the presence of control disturbances. Twenty able-bodied subjects and one transradial amputee performed delicate and power grasps with a prosthesis in a functional task, while the control signal gain was temporarily increased (high-gain disturbance) or decreased (low-gain disturbance) without their knowledge. Three outcome measures were considered: the percentage of trials successful in the first attempt (reaction to disturbance), the average number of attempts in trials where the wrong force was initially applied (adaptation to disturbance), and the average completion time of the last attempt in every trial. EMG feedback was shown to offer significantly better performance compared to force feedback during power grasping in terms of reaction to disturbance and completion time. During power grasping with high-gain disturbance, the median first-attempt success rate was significantly higher with EMG feedback (73.3%) compared to that achieved with force feedback (60%). Moreover, the median completion time for power grasps with low-gain disturbance was significantly longer with force feedback than with EMG feedback (3.64 against 2.48 s, an increase of 32%). Contrary to our expectations, there was no significant difference between feedback types with regards to adaptation to disturbances and the two feedback types performed similarly in delicate grasps. The results indicated that EMG feedback displayed better performance than force feedback in the presence of control disturbances, further demonstrating the potential of this approach to provide a reliable prosthesis-user interaction.

Estimating speed-accuracy trade-offs to evaluate and understand closed-loop prosthesis interfaces

OBJECTIVE

Closed-loop prosthesis interfaces, which combine electromyography (EMG)-based control with supplementary feedback, represent a promising direction for developing the next generation of bionic limbs. However, we still lack an understanding of how users utilize these interfaces and how to evaluate competing solutions. In this study, we used the framework of speed-accuracy trade-off functions (SAF) to understand, evaluate, and compare the performance of two closed-loop user-prosthesis interfaces.

APPROACH

Ten able-bodied participants and an amputee performed a force-matching task in a functional box-and-block setup at three different speeds. All participants were subjected to both interfaces in a crossover study design with a one-week washout period. Importantly, both interfaces used (identical) direct proportional control but differed in the feedback provided to the participant (EMG feedback vs. Force feedback). Therefore, we estimated the SAFs afforded by the two interfaces and sought to understand how the participants planned and executed the task under the various conditions.

MAIN RESULTS

We found that execution speed significantly influenced performance, and that EMG feedback afforded better overall performance, especially at medium speeds. Notably, we found that there was a difference in the SAF between the two interfaces, with EMG feedback enabling participants to attain higher accuracies faster than Force feedback. Furthermore, both interfaces enabled participants to develop flexible control policies, while EMG feedback also afforded participants the ability to generate smoother, more repeatable EMG commands.

SIGNIFICANCE

Overall, the results indicate that the performance of closed-loop prosthesis interfaces depends critically on the feedback approach and execution speed. This study showed that the SAF framework could be used to reveal the differences between feedback approaches, which might not have been detected if the assessment was performed at a single speed. Therefore, we argue that it is important to consider the speed-accuracy trade-offs to rigorously evaluate and compare user-prosthesis interfaces.

Combined spatial and frequency encoding for electrotactile feedback of myoelectric signals

Electrotactile stimulation has been commonly used in human–machine interfaces to provide feedback to the user, thereby closing the control loop and improving performance. The encoding approach, which defines the mapping of the feedback information into stimulation profiles, is a critical component of an electrotactile interface. Ideally, the encoding will provide a high-fidelity representation of the feedback variable while being easy to perceive and interpret by the subject. In the present study, we performed a closed-loop experiment wherein discrete and continuous coding schemes are combined to exploit the benefits of both techniques. Subjects performed a muscle activation-matching task relying solely on electrotactile feedback representing the generated myoelectric signal (EMG). In particular, we investigated the performance of two different coding schemes (spatial and spatial combined with frequency) at two feedback resolutions (low: 3 and high: 5 intervals). In both schemes, the stimulation electrodes were placed circumferentially around the upper arm. The magnitude of the normalized EMG was divided into intervals, and each electrode was associated with one interval. When the generated EMG entered one of the intervals, the associated electrode started stimulating. In the combined encoding, the additional frequency modulation of the active electrode also indicated the momentary magnitude of the signal within the interval. The results showed that combined coding decreased the undershooting rate, variability and absolute deviation when the resolution was low but not when the resolution was high, where it actually worsened the performance. This demonstrates that combined coding can improve the effectiveness of EMG feedback, but that this effect is limited by the intrinsic variability of myoelectric control. Our findings, therefore, provide important insights as well as elucidate limitations of the information encoding methods when using electrotactile stimulation to convey a feedback signal characterized by high variability (EMG biofeedback).

Full-hand electrotactile feedback using electronic skin and matrix electrodes for high-bandwidth human-machine interfacing

Tactile feedback is relevant in a broad range of human-machine interaction systems (e.g. teleoperation, virtual reality and prosthetics). The available tactile feedback interfaces comprise few sensing and stimulation units, which limits the amount of information conveyed to the user. The present study describes a novel technology that relies on distributed sensing and stimulation to convey comprehensive tactile feedback to the user of a robotic end effector. The system comprises six flexible sensing arrays (57 sensors) integrated on the fingers and palm of a robotic hand, embedded electronics (64 recording channels), a multichannel stimulator and seven flexible electrodes (64 stimulation pads) placed on the volar side of the subject's hand. The system was tested in seven subjects asked to recognize contact positions and identify contact sliding on the electronic skin, using distributed anode configuration (DAC) and single dedicated anode configuration. The experiments demonstrated that DAC resulted in substantially better performance. Using DAC, the system successfully translated the contact patterns into electrotactile profiles that the subjects could recognize with satisfactory accuracy ([Formula: see text] for static and [Formula: see text] for dynamic patterns). The proposed system is an important step towards the development of a high-density human-machine interfacing between the user and a robotic hand. This article is part of the theme issue 'Advanced neurotechnologies: translating innovation for health and well-being'.

Reducing the number of EMG electrodes during online hand gesture classification with changing wrist positions

Background

Myoelectric control based on hand gesture classification can be used for effective, contactless human–machine interfacing in general applications (e.g., consumer market) as well as in the clinical context. However, the accuracy of hand gesture classification can be impacted by several factors including changing wrist position. The present study aimed at investigating how channel configuration (number and placement of electrode pads) affects performance in hand gesture recognition across wrist positions, with the overall goal of reducing the number of channels without the loss of performance with respect to the benchmark (all channels).

Methods

Matrix electrodes (256 channels) were used to record high-density EMG from the forearm of 13 healthy subjects performing a set of 8 gestures in 3 wrist positions and 2 force levels (low and moderate). A reduced set of channels was chosen by applying sequential forward selection (SFS) and simple circumferential placement (CIRC) and used for gesture classification with linear discriminant analysis. The classification success rate and task completion rate were the main outcome measures for offline analysis across the different number of channels and online control using 8 selected channels, respectively.

Results

The offline analysis demonstrated that good accuracy (> 90%) can be achieved with only a few channels. However, using data from all wrist positions required more channels to reach the same performance. Despite the targeted placement (SFS) performing similarly to CIRC in the offline analysis, the task completion rate [median (lower–upper quartile)] in the online control was significantly higher for SFS [71.4% (64.8–76.2%)] compared to CIRC [57.1% (51.8–64.8%), p < 0.01], especially for low contraction levels [76.2% (66.7–84.5%) for SFS vs. 57.1% (47.6–60.7%) for CIRC, p < 0.01]. For the reduced number of electrodes, the performance with SFS was comparable to that obtained when using the full matrix, while the selected electrodes were highly subject-specific.

Conclusions

The present study demonstrated that the number of channels required for gesture classification with changing wrist positions could be decreased substantially without loss of performance, if those channels are placed strategically along the forearm and individually for each subject. The results also emphasize the importance of online assessment and motivate the development of configurable matrix electrodes with integrated channel selection.

Increase and Decrease in Velocity and Force During Exercise with a Hybrid Robotic-FES Rehabilitation System

Early rehabilitation is beneficial for stroke patients, but it is often delayed since the patients are often bedbound due to their general condition. New robotic rehabilitation devices such as ROBERT® enable patients to exercise even while bedbound. During pilot testing of an automated FES-delivery system combined with ROBERT®, we observed both increased and decreased exerted velocity and interaction force during repetitive exercising with the system. The goal of the current study was thus to investigate the extent of both potentiation and fatigue, as assessed in the velocity and interaction force produced in response to repetitive robotic-FES exercising. Eight healthy subjects completed 50 repetitions of leg-press exercises using the hybrid robotic-FES system. For individual subjects, significant changes were found for both mean and maximal velocities and interaction forces exerted during the exercise. Roughly half of the subjects had an increase in maximal velocity and interaction force during the exercise, and half exhibited an increase in mean velocity, whereas three subjects had an increase in mean interaction force during the exercise. The changes in mean velocity were in the range of -40.6 to 30.9% and for the maximal velocity they were in the range of -21.9 to 22.0%. The changes for mean interaction force were in the range of -5.8 to 11.0%, while for the maximal interaction force, they were in the range of -7.8 to 14.4%. These changes might pose significant challenges for future developments of hybrid robotic-FES rehabilitation systems, as the system must be able to comply with the observed changes, and appropriately adapt to them in order to maintain efficacy and safety.

The Impact of Size and Position of Reference Electrode on the Localization of Biphasic Electrotactile Stimulation on the Fingertips

Development of haptic interfaces to enrich augmented and virtual reality with the sense of touch is the next frontier for technological advancement of these systems. Among available technologies, electrotactile stimulation enables design of high-density interfaces that can provide natural-like sensation of touch in interaction with virtual objects. The present study investigates the human perception of electrotactile sensations on fingertips, focusing on the sensation localization in function of the size and position of reference electrode. Ten healthy subjects participated in the study, with the task to mark the sensations elicited by stimulating the index fingertip using an 8-pad electrode. The test systematically explored several configurations of the active (position) and reference (position and size) electrode pads. The results indicated that there was a spreading of perceived sensations across the fingertip, but that they were mostly localized below the active pad. The position and size of the reference electrode were shown to affect the location of the perceived sensations, which can potentially be exploited as an additional parameter to modulate the feedback. The present study demonstrates that the fingertip is a promising target for the delivery of high-resolution feedback.

Continuous Semi-autonomous Prosthesis Control Using a Depth Sensor on the Hand

Modern myoelectric prostheses can perform multiple functions (e.g., several grasp types and wrist rotation) but their intuitive control by the user is still an open challenge. It has been recently demonstrated that semi-autonomous control can allow the subjects to operate complex prostheses effectively; however, this approach often requires placing sensors on the user. The present study proposes a system for semi-autonomous control of a myoelectric prosthesis that requires a single depth sensor placed on the dorsal side of the hand. The system automatically pre-shapes the hand (grasp type, size, and wrist rotation) and allows the user to grasp objects of different shapes, sizes and orientations, placed individually or within cluttered scenes. The system “reacts” to the side from which the object is approached, and enables the user to target not only the whole object but also an object part. Another unique aspect of the system is that it relies on online interaction between the user and the prosthesis; the system reacts continuously on the targets that are in its focus, while the user interprets the movement of the prosthesis to adjust aiming. Experimental assessment was conducted in ten able-bodied participants to evaluate the feasibility and the impact of training on prosthesis-user interaction. The subjects used the system to grasp a set of objects individually (Phase I) and in cluttered scenarios (Phase II), while the time to accomplish the task (TAT) was used as the performance metric. In both phases, the TAT improved significantly across blocks. Some targets (objects and/or their parts) were more challenging, requiring thus significantly more time to handle, but all objects and scenes were successfully accomplished by all subjects. The assessment therefore demonstrated that the system is indeed robust and effective, and that the subjects could successfully learn how to aim with the system after a brief training. This is an important step toward the development of a self-contained semi-autonomous system convenient for clinical applications.

Electrotactile and Vibrotactile Feedback Enable Similar Performance in Psychometric Tests and Closed-Loop Control

Electro- and vibro-tactile stimulation are commonly employed for feedback in closed-loop human-machine interfacing. Although these feedback systems have been extensively investigated individually, they are rarely objectively compared. In this study, two state-of-the-art stimulation units (concentric electrode and C2-tactor) similar in shape and size were compared in psychometric and online control tests. The just noticeable difference and number of discriminable levels for intensity and frequency modulation were determined across values of carrier frequency and intensity, respectively. Next, subjects performed a compensatory tracking task, in which the feedback encoded the momentary tracking error. In the psychometric tests, intensity modulation outperformed frequency modulation and electrotactile stimulation enabled significantly higher resolution than vibrotactile stimulation, for the same carrier frequency. However, for the best-case settings (eletro-tactile: 100 Hz; vibro-tactile: 200 Hz), the two stimulation modalities were equivalent in the psychometric tests and in the online control tests, where the two stimulation methods resulted in similar correlation and deviation between the target and the generated trajectory. Time delay was slightly but significantly lower for the vibrotactile modality. Overall, the present assessment shows that despite psychometric differences between the two stimulation methods, they enable similar online control performance when parameters are optimally selected for each modality.

Impact of Shared Control Modalities on Performance and Usability of Semi-autonomous Prostheses

Semi-autonomous (SA) control of upper-limb prostheses can improve the performance and decrease the cognitive burden of a user. In this approach, a prosthesis is equipped with additional sensors (e.g., computer vision) that provide contextual information and enable the system to accomplish some tasks automatically. Autonomous control is fused with a volitional input of a user to compute the commands that are sent to the prosthesis. Although several promising prototypes demonstrating the potential of this approach have been presented, methods to integrate the two control streams (i.e., autonomous and volitional) have not been systematically investigated. In the present study, we implemented three shared control modalities (i.e., sequential, simultaneous, and continuous) and compared their performance, as well as the cognitive and physical burdens imposed on the user. In the sequential approach, the volitional input disabled the autonomous control. In the simultaneous approach, the volitional input to a specific degree of freedom (DoF) activated autonomous control of other DoFs, whereas in the continuous approach, autonomous control was always active except for the DoFs controlled by the user. The experiment was conducted in ten able-bodied subjects, and these subjects used an SA prosthesis to perform reach-and-grasp tasks while reacting to audio cues (dual tasking). The results demonstrated that, compared to the manual baseline (volitional control only), all three SA modalities accomplished the task in a shorter time and resulted in less volitional control input. The simultaneous SA modality performed worse than the sequential and continuous SA approaches. When systematic errors were introduced in the autonomous controller to generate a mismatch between the goals of the user and controller, the performance of SA modalities substantially decreased, even below the manual baseline. The sequential SA scheme was the least impacted one in terms of errors. The present study demonstrates that a specific approach for integrating volitional and autonomous control is indeed an important factor that significantly affects the performance and physical and cognitive load, and therefore these should be considered when designing SA prostheses.

Embedded Electrotactile Feedback System for Hand Prostheses Using Matrix Electrode and Electronic Skin

As the technology moves towards more human-like bionic limbs, it is necessary to develop a feedback system that provides active touch feedback to a user of a prosthetic hand. Most of the contemporary sensory substitution methods comprise simple position and force sensors combined with few discrete stimulation units, and hence they are characterized with a limited amount of information that can be transmitted by the feedback. The present study describes a novel system for tactile feedback integrating advanced multipoint sensing (electronic skin) and stimulation (matrix electrodes). The system comprises a flexible sensing array (16 sensors) integrated on the index finger of a Michelangelo prosthetic hand mockup, embedded interface electronics and multichannel stimulator connected to a flexible matrix electrode (24 pads). The developed system conveys contact information (binary detections) to the user. To demonstrate the feasibility, the system was tested in six able-bodied subjects who were asked to recognize static patterns (contact position) with two different spatial resolutions and dynamic movement patterns (i.e., sliding along and/or across the finger) presented on the electronic skin. The experiments demonstrated that the system successfully translated the mechanical interaction into electrotactile profiles, which the subjects could recognize with good performance. The success rates (mean ± standard deviation) for the static patterns were 91 ± 4% and 58 ± 10% for low and high spatial resolution, respectively, while the success rate for sliding touch was 94 ± 4%. These results demonstrate that the developed system is an important step towards a new generation of tactile feedback interfaces that can provide high-bandwidth connection between the user and his/her bionic limb. Such systems would allow mimicking spatially distributed natural feedback, thereby facilitating the control and embodiment of the artificial device into the user body scheme.

The impact of objective functions on control policies in closed-loop control of grasping force with a myoelectric prosthesis

Objective.Supplemental sensory feedback for myoelectric prostheses can provide both psychosocial and functional benefits during prosthesis control. However, the impact of feedback depends on multiple factors and there is insufficient understanding about the fundamental role of such feedback in prosthesis use. The framework of human motor control enables us to systematically investigate the user-prosthesis control loop. In this study, we explore how different task objectives such as speed and accuracy shape the control policy developed by participants in a prosthesis force-matching task.Approach.Participants were randomly assigned to two groups that both used identical electromyography control interface and prosthesis force feedback, through vibrotactile stimulation, to perform a prosthesis force-matching task. However, the groups received different task objectives specifying speed and accuracy demands. We then investigated the control policies developed by the participants. To this end, we not only evaluated how successful or fast participants were but also analyzed the behavioral strategies adopted by the participants to obtain such performance gains.Main results.First, we observed that participants successfully integrated supplemental prosthesis force feedback to develop both feedforward and feedback control policies, as demanded by the task objectives. We then observed that participants who first developed a (slow) feedback policy were quickly able to adapt their policy to more stringent speed demands, by switching to a combined feedforward-feedback control strategy. However, the participants who first developed a (fast) feedforward policy were not able to change their control policy and adjust to greater accuracy demands.Significance.Overall, the results signify how the framework of human motor control can be applied to study the role of feedback in user-prosthesis interaction. The results also reveal the utility of training prosthesis users to integrate supplemental feedback into their state estimation by designing training protocols that encourage the development of combined feedforward and feedback policy.

Electrotactile feedback outweighs natural feedback in sensory integration during control of grasp force

Objective.The nervous system subconsciously estimates the state of the body as a weighted average of the information from various sensory sources, where the weights reflect the perceived reliability of each source. Loss of motor functions can be partially compensated using assistive systems (e.g. prostheses), which may also restore somatosensory feedback through tactile stimulation. Whether such artificial feedback is integrated in the neural state estimation process is not known.Approach.In this study, able-bodied subjects performed a grasp force matching task with supplementary non-invasive electrotactile stimulation with a frequency proportional to grasp force magnitude. Before the task, a brief training session taught the subjects to associate the sensation of electrotactile stimulation with the generated grasp force. In some trials, the force-frequency mapping was biased to introduce an unnoticeable mismatch between natural and electrotactile force feedback, thereby provoking the subject to subconsciously estimate the force as a compromise between the two sources of information.Main results.The outcome of this compromise revealed the weights assigned to each feedback type. The grasp forces were significantly affected by the biased mappings, as indicated by the average estimated relative weights (electrotactile: 0.69 ± 0.29; natural: 0.31 ± 0.29). Across subjects, this weight was correlatedr2=0.75) with the improvement in force matching precision when adding the unbiased electrotactile feedback to the natural force feedback, as predicted by maximum likelihood estimation. This shows that even after minimal training the nervous system adopts electrotactile stimulation as a highly reliable source of information that can improve the precision in the estimation of the grip force.Significance.This result has important implications for the restoration of sensory feedback in upper limb prostheses as it indicates that even non-invasive stimulation can be integrated naturally (i.e. subconsciously and effectively) in the motor controlloop.

Participant-specific classifier tuning increases the performance of hand movement detection from EEG in patients with amyotrophic lateral sclerosis

Objective.Brain-computer interface (BCI) systems can be employed to provide motor and communication assistance to patients suffering from neuromuscular diseases, such as amyotrophic lateral sclerosis (ALS). Movement related cortical potentials (MRCPs), which are naturally generated during movement execution, can be used to implement a BCI triggered by motor attempts. Such BCI could assist impaired motor functions of ALS patients during disease progression, and facilitate the training for the generation of reliable MRCPs. The training aspect is relevant to establish a communication channel in the late stage of the disease. Therefore, the aim of this study was to investigate the possibility of detecting MRCPs associated to movement intention in ALS patients with different levels of disease progression from slight to complete paralysis.Approach.Electroencephalography signals were recorded from nine channels in 30 ALS patients at various stages of the disease while they performed or attempted to perform hand movements timed to a visual cue. The movement detection was implemented using offline classification between movement and rest phase. Temporal and spectral features were extracted using 500 ms sliding windows with 50% overlap. The detection was tested for each individual channel and two surrogate channels by performing feature selection followed by classification using linear and non-linear support vector machine and linear discriminant analysis.Main results.The results demonstrated that the detection performance was high in all patients (accuracy 80.5 ± 5.6%) but that the classification parameters (channel, features and classifier) leading to the best performance varied greatly across patients. When the same channel and classifier were used for all patients (participant-generic analysis), the performance significantly decreased (accuracy 74 ± 8.3%).Significance.The present study demonstrates that to maximize the detection of brain waves across ALS patients at different stages of the disease, the classification pipeline should be tuned to each patient individually.

Generative Adversarial Networks-Based Data Augmentation for Brain-Computer Interface

The performance of a classifier in a brain-computer interface (BCI) system is highly dependent on the quality and quantity of training data. Typically, the training data are collected in a laboratory where the users perform tasks in a controlled environment. However, users' attention may be diverted in real-life BCI applications and this may decrease the performance of the classifier. To improve the robustness of the classifier, additional data can be acquired in such conditions, but it is not practical to record electroencephalogram (EEG) data over several long calibration sessions. A potentially time- and cost-efficient solution is artificial data generation. Hence, in this study, we proposed a framework based on the deep convolutional generative adversarial networks (DCGANs) for generating artificial EEG to augment the training set in order to improve the performance of a BCI classifier. To make a comparative investigation, we designed a motor task experiment with diverted and focused attention conditions. We used an end-to-end deep convolutional neural network for classification between movement intention and rest using the data from 14 subjects. The results from the leave-one subject-out (LOO) classification yielded baseline accuracies of 73.04% for diverted attention and 80.09% for focused attention without data augmentation. Using the proposed DCGANs-based framework for augmentation, the results yielded a significant improvement of 7.32% for diverted attention ( ) and 5.45% for focused attention ( ). In addition, we implemented the method on the data set IVa from BCI competition III to distinguish different motor imagery tasks. The proposed method increased the accuracy by 3.57% ( ). This study shows that using GANs for EEG augmentation can significantly improve BCI performance, especially in real-life applications, whereby users' attention may be diverted.

The effect of calibration parameters on the control of a myoelectric hand prosthesis using EMG feedback

Objective.The implementation of somatosensory feedback in upper limb myoelectric prostheses is an important step towards the restoration of lost sensory-motor functions. EMG feedback is a recently proposed method for closing the control loop wherein the myoelectric signal that drives the prosthesis is also used to generate the feedback provided to the user. Therefore, the characteristics of the myoelectric signal (variability and sensitivity) are likely to significantly affect the ability of the subject to utilize this feedback for online control of the prosthesis.Approach.In the present study, we investigated how the cutoff frequency of the low-pass filter (0.5, 1 and 1.5 Hz) and normalization value (20%, 40% and 60% of the maximum voluntary contraction (MVC)), that are used for the generation of the myoelectric signal, affect the quality of closed-loop control with EMG feedback. Lower cutoff and normalization decrease the intrinsic variability of the EMG but also increase the time lag between the contraction and the feedback (cutoff) as well as the sensitivity of the myoelectric signal (normalization). Ten participants were asked to generate three grasp force levels with a myoelectric prosthetic hand, while receiving five-level vibrotactile EMG feedback, over nine experimental runs (all parameter combinations).Main results.The outcome measure was the success rate (SR) in achieving the appropriate level of myoelectric signal (primary outcome) and grasping force (secondary outcome). Overall, the experiments demonstrated that EMG feedback provided robust control across conditions. Nevertheless, the performance was significantly better for the lowest cutoff (0.5 Hz) and higher normalization (40% and 60%). The highest SR for the EMG was 71.9%, achieved in the condition (40% MVC and 0.5 Hz), and this was 24.1% higher than that in the condition (20% MVC and 1.5 Hz), which resulted in the lowest performance. The SR for the force followed a similar trend.Significance.This is the first study that systematically explored the parameter space for the calibration of EMG feedback, which is a critical step for the future clinical application of this approach.

A compact system for simultaneous stimulation and recording for closed-loop myoelectric control

Background

Despite important advancements in control and mechatronics of myoelectric prostheses, the communication between the user and his/her bionic limb is still unidirectional, as these systems do not provide somatosensory feedback. Electrotactile stimulation is an attractive technology to close the control loop since it allows flexible modulation of multiple parameters and compact interface design via multi-pad electrodes. However, the stimulation interferes with the recording of myoelectric signals and this can be detrimental to control.

Methods

We present a novel compact solution for simultaneous recording and stimulation through dynamic blanking of stimulation artefacts. To test the system, a feedback coding scheme communicating wrist rotation and hand aperture was developed specifically to stress the myoelectric control while still providing meaningful information to the subjects. Ten subjects participated in an experiment, where the quality of closed-loop myoelectric control was assessed by controlling a cursor in a two degrees of freedom target-reaching task. The benchmark performance with visual feedback was compared to that achieved by combining visual feedback and electrotactile stimulation as well as by using electrotactile feedback only.

Results

There was no significant difference in performance between visual and combined feedback condition with regards to successfully reached targets, time to reach a target, path efficiency and the number of overshoots. Therefore, the quality of myoelectric control was preserved in spite of the stimulation. As expected, the tactile condition was significantly poorer in completion rate (100/4% and 78/25% for combined and tactile condition, respectively) and time to reach a target (9/2 s and 13/4 s for combined and tactile condition, respectively). However, the performance in the tactile condition was still good, with no significant difference in path efficiency (38/8%) and the number of overshoots (0.5/0.4 overshoots), indicating that the stimulation was meaningful for the subjects and useful for closed-loop control.

Conclusions

Overall, the results demonstrated that the developed system can provide robust closed-loop control using electrotactile stimulation. The system supports different encoding schemes and allows placing the recording and stimulation electrodes next to each other. This is an important step towards an integrated solution where the developed unit will be embedded into a prosthetic socket.

Hybrid Tongue - Myoelectric Control Improves Functional Use of a Robotic Hand Prosthesis

OBJECTIVE

This study aims at investigating the functional performance of a novel prosthesis control scheme integrating an inductive tongue interface and myoelectric control. The tongue interface allowed direct selection of the desired grasp while myoelectric signals were used to open and close the robotic hand.

METHODS

The novel method was compared to a conventional sequential on/off myoelectric control scheme using functional tasks defined by Assistive Hand Assessment protocol. Ten able-bodied participants were fitted with the SmartHand on their left forearm. They used both the conventional myoelectric control and the Tongue and Myoelectric Hybrid interface (TMH) to accomplish two activities of daily living (i.e., preparing a sandwich and gift wrapping). Sessions were video recorded and the outcome measure was the completion time for the subtasks as well as the full tasks.

RESULTS

The sandwich task was completed significantly faster, with 19% decrease in the completion time, using the TMH when compared to the conventional sequential on/off myoelectric control scheme (p < 0.05).

CONCLUSION

The results indicate that the TMH control scheme facilitates the active use of the prosthetic device by simplifying grasp selection, leading thereby to faster completion of challenging and relevant tasks involving bimanual activities.

Online Closed-Loop Control Using Tactile Feedback Delivered Through Surface and Subdermal Electrotactile Stimulation

Aim

Limb loss is a dramatic event with a devastating impact on a person’s quality of life. Prostheses have been used to restore lost motor abilities and cosmetic appearance. Closing the loop between the prosthesis and the amputee by providing somatosensory feedback to the user might improve the performance, confidence of the amputee, and embodiment of the prosthesis. Recently, a minimally invasive method, in which the electrodes are placed subdermally, was presented and psychometrically evaluated. The present study aimed to assess the quality of online control with subdermal stimulation and compare it to that achieved using surface stimulation (common benchmark) as well as to investigate the impact of training on the two modalities.

Methods

Ten able-bodied subjects performed a PC-based compensatory tracking task. The subjects employed a joystick to track a predefined pseudorandom trajectory using feedback on the momentary tracking error, which was conveyed via surface and subdermal electrotactile stimulation. The tracking performance was evaluated using the correlation coefficient (CORR), root mean square error (RMSE), and time delay between reference and generated trajectories.

Results

Both stimulation modalities resulted in good closed-loop control, and surface stimulation outperformed the subdermal approach. There was significant difference in CORR (86 vs 77%) and RMSE (0.23 vs 0.31) between surface and subdermal stimulation (all p < 0.05). The RMSE of the subdermal stimulation decreased significantly in the first few trials.

Conclusion

Subdermal stimulation is a viable method to provide tactile feedback. The quality of online control is, however, somewhat worse compared to that achieved using surface stimulation. Nevertheless, due to minimal invasiveness, compactness, and power efficiency, the subdermal interface could be an attractive solution for the functional application in sensate prostheses.

Immersive augmented reality system for the training of pattern classification control with a myoelectric prosthesis

BACKGROUND

Hand amputation can have a truly debilitating impact on the life of the affected person. A multifunctional myoelectric prosthesis controlled using pattern classification can be used to restore some of the lost motor abilities. However, learning to control an advanced prosthesis can be a challenging task, but virtual and augmented reality (AR) provide means to create an engaging and motivating training.

METHODS

In this study, we present a novel training framework that integrates virtual elements within a real scene (AR) while allowing the view from the first-person perspective. The framework was evaluated in 13 able-bodied subjects and a limb-deficient person divided into intervention (IG) and control (CG) groups. The IG received training by performing simulated clothespin task and both groups conducted a pre- and posttest with a real prosthesis. When training with the AR, the subjects received visual feedback on the generated grasping force. The main outcome measure was the number of pins that were successfully transferred within 20 min (task duration), while the number of dropped and broken pins were also registered. The participants were asked to score the difficulty of the real task (posttest), fun-factor and motivation, as well as the utility of the feedback.

RESULTS

The performance (median/interquartile range) consistently increased during the training sessions (4/3 to 22/4). While the results were similar for the two groups in the pretest, the performance improved in the posttest only in IG. In addition, the subjects in IG transferred significantly more pins (28/10.5 versus 14.5/11), and dropped (1/2.5 versus 3.5/2) and broke (5/3.8 versus 14.5/9) significantly fewer pins in the posttest compared to CG. The participants in IG assigned (mean ± std) significantly lower scores to the difficulty compared to CG (5.2 ± 1.9 versus 7.1 ± 0.9), and they highly rated the fun factor (8.7 ± 1.3) and usefulness of feedback (8.5 ± 1.7).

CONCLUSION

The results demonstrated that the proposed AR system allows for the transfer of skills from the simulated to the real task while providing a positive user experience. The present study demonstrates the effectiveness and flexibility of the proposed AR framework. Importantly, the developed system is open source and available for download and further development.

Closed-loop Control using Electrotactile Feedback Encoded in Frequency and Pulse Width

Sensory substitution by electrotactile stimulation has been widely investigated for improving the functionality of human-machine interfaces. Few studies, however, have objectively compared different ways in which such systems can be implemented. In this study, we compare encoding of a feedback variable in stimulation pulse width or stimulation frequency during a closed-loop control task. Specifically, participants were asked to track a predefined pseudorandom trajectory using a joystick with electrotactile feedback as the only indication of the tracking error. Each participant performed eight 90 s trials per encoding scheme. Tracking performance using frequency modulation enabled lower tracking error (RMSE: Frequency modulation: 0.27 ± 0.03; Pulse width modulation: 0.31 ± 0.05; p < 0.05) and a higher correlation with the target trajectory (Frequency modulation: 83.4 ± 4.1%; Pulse width modulation: 79.8 ± 5.2%; p < 0.05). There was no significant improvement in performance over the eight trials. Furthermore, frequency-domain analysis revealed that frequency modulation was characterized with a higher gain at lower error frequencies. In summary, the results indicate that encoding of feedback variables in the frequency of pulses enables better control than pulse width modulation in closed-loop dynamic tasks.

Amplitude versus spatially modulated electrotactile feedback for myoelectric control of two degrees of freedom

OBJECTIVE

Artificial proprioceptive feedback from a myoelectric prosthesis is an important aspect in enhancing embodiment and user satisfaction, possibly lowering the demand for visual attention while controlling a prosthesis in everyday tasks. Contemporary myoelectric prostheses are advanced mechatronic systems with multiple degrees of freedom, and therefore, to communicate the prosthesis state, the feedback interface needs to transmit several variables simultaneously. In the present study, two different configurations for conveying proprioceptive information of wrist rotation and hand aperture through multichannel electrotactile stimulation were developed and evaluated during online myoelectric control.

APPROACH

Myoelectric recordings were acquired from the dominant forearm and electrotactile stimulation was delivered on the non-dominant forearm using a compact interface. The first feedback configuration, which was based on spatial coding, transmitted the information using a moving tactile stimulus, whereas the second, amplitude-based configuration conveyed the position via sensation intensity. Thirteen able-bodied subjects used pattern classification-based myoelectric control with both feedback configurations to accomplish a target-reaching task.

MAIN RESULTS

High task performance (completion rate > 90%) was observed for both configurations, with no significant difference in completion rate, time to reach the target, distance error and path efficiency, respectively.

SIGNIFICANCE

Overall, the results demonstrated that both feedback configurations allowed subjects to perceive and interpret two feedback variables delivered simultaneously, despite using a compact stimulation interface. This is an encouraging result for the prospect of communicating the full state of a multifunctional hand prosthesis.

Temporal Asynchrony but Not Total Energy Nor Duration Improves the Judgment of Numerosity in Electrotactile Stimulation

Stroke patients suffer from impairments of both motor and somatosensory functions. The functional recovery of upper extremities is one of the primary goals of rehabilitation programs. Additional somatosensory deficits limit sensorimotor function and significantly affect its recovery after the neuromotor injury. Sensory substitution systems, providing tactile feedback, might facilitate manipulation capability, and improve patient's dexterity during grasping movements. As a first step toward this aim, we evaluated the ability of healthy subjects in exploiting electrotactile feedback on the shoulder to determine the number of perceived stimuli in numerosity judgment tasks. During the experiment, we compared four different stimulation patterns (two simultaneous: short and long, intermittent and sequential) differing in total duration, total energy, or temporal synchrony. The experiment confirmed that the subject ability to enumerate electrotactile stimuli decreased with increasing the number of active electrodes. Furthermore, we found that, in electrotactile stimulation, the temporal coding schemes, and not total energy or duration modulated the accuracy in numerosity judgment. More precisely, the sequential condition resulted in significantly better numerosity discrimination than intermittent and simultaneous stimulation. These findings, together with the fact that the shoulder appeared to be a feasible stimulation site to communicate tactile information via electrotactile feedback, can serve as a guide to deliver tactile feedback to proximal areas in stroke survivors who lack sensory integrity in distal areas of their affected arm, but retain motor skills.

The Interaction Between Feedback Type and Learning in Routine Grasping With Myoelectric Prostheses

While prosthetic fitting after upper-limb loss allows for restoration of motor functions, it deprives the amputee of tactile sensations that are essential for grasp control in able-bodied subjects. Therefore, it is commonly assumed that restoring the force feedback would improve the control of prosthesis grasping force. However, the literature regarding the benefit of feedback is controversial. Here, we investigated how the type of feedback affects learning and steady-state performance of routine grasping with a prosthesis. The experimental task was to grasp an object using a prosthesis and generate a low or high target-force range (TFR), both initially unknown, in three feedback conditions: basic auditory feedback on task outcome, and additional visual or vibratory feedback on the force magnitude. The results demonstrated that the performance was rather good and stable for the low TFR, whereas it was substantially worse for the high TFR with a pronounced training effect. Surprisingly, learning curve and steady-state performance did not depend on the feedback condition. Hence, in the specific context of routing grasping with a prosthesis controlled via surface EMG, the basic feedback on task outcome was not outperformed by force-related end-of-trial feedback and hence seemed to be sufficient for accomplishing the task.This conclusion applies to the context of routine grasping using a myoelectric prosthesis with surface EMG electrodes, which means that the control signals are variable and the feedback is perceived and processed at the end of the trial (motor adaption).

A Review of Sensory Feedback in Upper-Limb Prostheses From the Perspective of Human Motor Control

This manuscript reviews historical and recent studies that focus on supplementary sensory feedback for use in upper limb prostheses. It shows that the inability of many studies to speak to the issue of meaningful performance improvements in real-life scenarios is caused by the complexity of the interactions of supplementary sensory feedback with other types of feedback along with other portions of the motor control process. To do this, the present manuscript frames the question of supplementary feedback from the perspective of computational motor control, providing a brief review of the main advances in that field over the last 20 years. It then separates the studies on the closed-loop prosthesis control into distinct categories, which are defined by relating the impact of feedback to the relevant components of the motor control framework, and reviews the work that has been done over the last 50+ years in each of those categories. It ends with a discussion of the studies, along with suggestions for experimental construction and connections with other areas of research, such as machine learning.

Dual-Parameter Modulation Improves Stimulus Localization in Multichannel Electrotactile Stimulation

Among most challenging open issues in prosthetic research is the development of a robust bidirectional interface between a prosthesis and its user. Commercially available prosthetic systems are mechanically advanced, but they do not provide somatosensory feedback. Here, we present a novel non-invasive interface for multichannel electrotactile feedback, comprising a matrix of 24 pads, and we investigate the ability of able-bodied human subjects to localize the electrotactile stimulus delivered through the matrix. For this purpose, we tested conventional stimulation (same frequency for all pads) and a novel dual-parameter modulation scheme (interleaved frequency and intensity) designed to facilitate the spatial localization over the electrode. Electrotactile stimulation was also compared to mechanical stimulation of the same locations on the skin. Experimental results on eight able-bodied subjects demonstrated that the proposed interleaved coding substantially improved the spatial localization compared to same-frequency stimulation. The results also showed that same-frequency stimulation was equivalent to mechanical stimulation, whereas the performance with dual-parameter modulation was significantly better. These are encouraging outcomes for the application of a multichannel interface for the restoration of feedback in prosthetics. The high-resolution augmented interfaces might be used to explore novel scenarios for effective communication with the prosthesis user enabled by maximizing information transmission.

Closed-Loop Multi-Amplitude Control for Robust and Dexterous Performance of Myoelectric Prosthesis

In the case of a hand amputation, the affected person can use a myoelectric prosthesis to substitute the missing limb and regain motor functions. Unfortunately, commercial methods for myoelectric control, although robust and simple, are unintuitive and cognitively taxing when applied to an advanced multi-functional prosthesis. The state-of-the-art methods developed in academia are based on machine learning and therefore require long training and suffer from a lack of robustness. This work presents a novel closed-loop multi-level amplitude controller (CMAC), which aims at overcoming these drawbacks. The CMAC implements three degrees-of-freedom (DoF) control by thresholding the muscle contraction intensity during wrist flexion and extension movements. Unique features of the controller are the vibrotactile feedback that communicates the state of the controller to the user and a scheme for proportional control. These components allow exploiting the full dexterity of the prosthesis using a simple two-channel myoelectric interface. The CMAC was compared to a commonly implemented pattern-recognition method (linear discriminant analysis - LDA) using clinically relevant tests in 12 able-bodied and 2 amputee subjects. The experimental assessment demonstrated that CMAC was similarly fast as LDA in dexterous tests (clothespin and cube manipulation), while it was somewhat slower than LDA during a simple, single DoF task (box and blocks). In addition, in all the tasks, LDA and CMAC resulted in a similarly low error rate. On the other hand, to an amputee that could not generate six distinguishable classes using LDA, the CMAC still enabled the control of all the prosthesis DoFs. Importantly, the overall setup and training time in CMAC were significantly lower compared to LDA. In conclusion, the novel method is convenient for clinical applications, and allows substantially higher control dexterity compared to what can be normally achieved using conventional two channel EMG. Therefore, CMAC provides performance comparable to advanced machine-learning algorithms and the robustness and ease of use that is characteristic for the simple two-channel myoelectric interface.

Psychometric characterization of incidental feedback sources during grasping with a hand prosthesis

Background

A prosthetic system should ideally reinstate the bidirectional communication between the user’s brain and its end effector by restoring both motor and sensory functions lost after an amputation. However, current commercial prostheses generally do not incorporate somatosensory feedback. Even without explicit feedback, grasping using a prosthesis partly relies on sensory information. Indeed, the prosthesis operation is characterized by visual and sound cues that could be exploited by the user to estimate the prosthesis state. However, the quality of this incidental feedback has not been objectively evaluated.

Methods

In this study, the psychometric properties of the auditory and visual feedback of prosthesis motion were assessed and compared to that of a vibro-tactile interface. Twelve able-bodied subjects passively observed prosthesis closing and grasping an object, and they were asked to discriminate (experiment I) or estimate (experiment II) the closing velocity of the prosthesis using visual (VIS), acoustic (SND), or combined (VIS + SND) feedback. In experiment II, the subjects performed the task also with a vibrotactile stimulus (VIB) delivered using a single tactor. The outcome measures for the discrimination and estimation experiments were just noticeable difference (JND) and median absolute estimation error (MAE), respectively.

Results

The results demonstrated that the incidental sources provided a remarkably good discrimination and estimation of the closing velocity, significantly outperforming the vibrotactile feedback. Using incidental sources, the subjects could discriminate almost the minimum possible increment/decrement in velocity that could be commanded to the prosthesis (median JND < 2% for SND and VIS + SND). Similarly, the median MAE in estimating the prosthesis velocity randomly commanded from the full working range was also low, i.e., approximately 5% in SND and VIS + SND.

Conclusions

Since the closing velocity is proportional to grasping force in state-of-the-art myoelectric prostheses, the results of the present study imply that the incidental feedback, when available, could be usefully exploited for grasping force control. Therefore, the impact of incidental feedback needs to be considered when designing a feedback interface in prosthetics, especially since the quality of estimation using supplemental sources (e.g., vibration) can be worse compared to that of the intrinsic cues.

The Variability of Psychophysical Parameters Following Surface and Subdermal Stimulation: A Multiday Study in Amputees

Electrotactile stimulation has been suggested as a modality for providing sensory feedback in upper limb prostheses. This study investigates the multiday variability of subdermal and surface stimulation. Electrical stimulation was delivered using either surface or fine wire electrodes placed right under the skin in eight amputees for seven consecutive days. The variability of psychophysical measurements, including detection threshold (DT), pain threshold (PT), dynamic range (DR), just noticeable difference (JND), Weber fraction (WF) and quality of evoked sensations, was evaluated using the coefficient of variation (CoV). In addition, the systematic change in the mean of the parameters across days was assessed in both stimulation modalities. In the case of DT, PT, DR, and perceived intensity at 100 Hz, the CoV of surface stimulation was significantly smaller than that of subdermal stimulation. Only PT showed a significant systematic change in the mean value across days for both modalities. The outcome of this study has implications for the choice of modality in delivering sensory feedback, though the significance of the quantified variability needs to be evaluated using usability tests with user feedback.

Improving bimanual interaction with a prosthesis using semi-autonomous control

BACKGROUND

The loss of a hand is a traumatic experience that substantially compromises an individual's capability to interact with his environment. The myoelectric prostheses are state-of-the-art (SoA) functional replacements for the lost limbs. Their overall mechanical design and dexterity have improved over the last few decades, but the users have not been able to fully exploit these advances because of the lack of effective and intuitive control. Bimanual tasks are particularly challenging for an amputee since prosthesis control needs to be coordinated with the movement of the sound limb. So far, the bimanual activities have been often neglected by the prosthetic research community.

METHODS

We present a novel method to prosthesis control, which uses a semi-autonomous approach in order to simplify bimanual interactions. The approach supplements the commercial SoA two-channel myoelectric control with two additional sensors. Two inertial measurement units were attached to the prosthesis and the sound hand to detect the movement of both limbs. Once a bimanual interaction is detected, the system mimics the coordination strategies of able-bodied subjects to automatically adjust the prosthesis wrist rotation (pronation, supination) and grip type (lateral, palmar) to assist the sound hand during a bimanual task. The system has been evaluated in eight able-bodied subjects performing functional uni- and bi-manual tasks using the novel method and SoA two-channel myocontrol. The outcome measures were time to accomplish the task, semi-autonomous system misclassification rate, subjective rating of intuitiveness, and perceived workload (NASA TLX).

RESULTS

The results demonstrated that the novel control interface substantially outperformed the SoA myoelectric control. While using the semi-autonomous control the time to accomplish the task and the perceived workload decreased for 25 and 27%, respectively, while the subjects rated the system as more intuitive then SoA myocontrol.

CONCLUSIONS

The novel system uses minimal additional hardware (two inertial sensors) and simple processing and it is therefore convenient for practical implementation. By using the proposed control scheme, the prosthesis assists the user's sound hand in performing bimanual interactions while decreasing cognitive burden.