Object discrimination using electrotactile feedback

Integration of proprioception in upper limb prostheses through non-invasive strategies: a review

Proprioception plays a key role in moving our body dexterously and effortlessly. Nevertheless, the majority of investigations evaluating the benefits of providing supplemental feedback to prosthetics users focus on delivering touch restitution. These studies evaluate the influence of touch sensation in an attempt to improve the controllability of current robotic devices. Contrarily, investigations evaluating the capabilities of proprioceptive supplemental feedback have yet to be comprehensively analyzed to the same extent, marking a major gap in knowledge within the current research climate. The non-invasive strategies employed so far to restitute proprioception are reviewed in this work. In the absence of a clearly superior strategy, approaches employing vibrotactile, electrotactile and skin-stretch stimulation achieved better and more consistent results, considering both kinesthetic and grip force information, compared with other strategies or any incidental feedback. Although emulating the richness of the physiological sensory return through artificial feedback is the primary hurdle, measuring its effects to eventually support the integration of cumbersome and energy intensive hardware into commercial prosthetic devices could represent an even greater challenge. Thus, we analyze the strengths and limitations of previous studies and discuss the possible benefits of coupling objective measures, like neurophysiological parameters, as well as measures of prosthesis embodiment and cognitive load with behavioral measures of performance. Such insights aim to provide additional and collateral outcomes to be considered in the experimental design of future investigations of proprioception restitution that could, in the end, allow researchers to gain a more detailed understanding of possibly similar behavioral results and, thus, support one strategy over another.

Restoration of sensory information via bionic hands

Individuals who have lost the use of their hands because of amputation or spinal cord injury can use prosthetic hands to restore their independence. A dexterous prosthesis requires the acquisition of control signals that drive the movements of the robotic hand, and the transmission of sensory signals to convey information to the user about the consequences of these movements. In this Review, we describe non-invasive and invasive technologies for conveying artificial sensory feedback through bionic hands, and evaluate the technologies' long-term prospects.

EEG guided electrical stimulation parameters generation from texture force profiles

Objective.Our aim is to enhance sensory perception and spatial presence in artificial interfaces guided by EEG. This is done by developing a closed-loop electro-tactile system guided by EEG that adaptively update the electrical stimulation parameters to achieve EEG responses similar to the EEG responses generated from touching textured surface.Approach.In this work, we introduce a model that defines the relationship between the contact force profiles and the electrical stimulation parameters. This is done by using the EEG and force data collected from two experiments. The first was conducted by moving a set of textured surfaces against the subjects' fingertip, while collecting both EEG and force data. Whereas the second was carried out by applying a set of different pulse and amplitude modulated electrical stimuli to the subjects' index finger while recording EEG.Main results.We were able to develop a model which could generate electrical stimulation parameters corresponding to different textured surfaces. We showed by offline testing and validation analysis that the average error between the EEG generated from the estimated electrical stimulation parameters and the actual EEG generated from touching textured surfaces is around 7%.Significance.Haptic feedback plays a vital role in our daily life, as it allows us to become aware of our environment. Even though a number of methods have been developed to measure perception of spatial presence and provide sensory feedback in virtual reality environments, there is currently no closed-loop control of sensory stimulation. The proposed model provides an initial step towards developing a closed loop electro-tactile haptic feedback model that delivers more realistic touch sensation through electrical stimulation.

Electrotactile Perception Properties and Its Applications: A Review

With the increased demands of human-machine interaction, haptic feedback is becoming increasingly critical. However, the high cost, large size and low efficiency of current haptic systems severely hinder further development. As a portable and efficient technology, cutaneous electrotactile stimulation has shown promising potential for these issues. This paper presents a review on and insight into cutaneous electrotactile perception and its applications. Research results on perceptual properties and evaluation methods have been summarized and discussed to understand the effects of electrotactile stimulation on humans. Electrotactile applications are presented in categories to understand the methods and progress in various fields such as prostheses control, sensory substitution, sensory restoration and sensorimotor restoration. State of the art has demonstrated the superiority of electrotactile feedback, its efficiency and its flexibility. However, the complex factors and the limitations of evaluation methods made it challenging for precise electrotactile control. Groundbreaking innovation in electrotactile theory is expected to overcome challenges such as precise perception control, information capacity increasing, comprehension burden reducing and implementation costs.

Electrotactile feedback improves grip force control and enables object stiffness recognition while using a myoelectric hand

Current myoelectric hands are limited in their ability to provide effective sensory feedback to the users, which highly affects their functionality and utility. Although the sensory information of a myoelectric hand can be acquired with equipped sensors, transforming the sensory signals into effective tactile sensations on users for functional tasks is a largely unsolved challenge. The purpose of this study aims to demonstrate that electrotactile feedback of the grip force improves the sensorimotor control of a myoelectric hand and enables object stiffness recognition. The grip force of a sensorized myoelectric hand was delivered to its users via electrotactile stimulation based on four kinds of typical encoding strategies, including graded (G), linear amplitude (LA), linear frequency (LF), and biomimetic (B) modulation. Object stiffness was encoded with the change of electrotactile sensations triggered by final grip force, as the prosthesis grasped the objects. Ten able-bodied subjects and two transradial amputees were recruited to participate in a dual-task virtual eggs test (VET) and an object stiffness discrimination test (OSDT) to quantify the prosthesis users' ability to handle fragile objects and recognize object stiffnesses, respectively. The quantified results showed that with electrotactile feedback enabled, the four kinds of encoding strategies allowed subjects to better able to handle fragile objects with similar performance, and the subjects were able to differentiate four levels of object stiffness at favorable accuracies (>86%) and high manual efficiency. Strategy LA presented the best stiffness discrimination performance, while strategy B was able to reduce the discrimination time but the discrimination accuracy was not better than the other three strategies. Electrotactile feedback also enhanced prosthesis embodiment and improved the users' confidence in prosthetic control. Outcomes indicate electrotactile feedback can be effectively exploited by the prosthesis users for grip force control and object stiffness recognition, proving the feasibility of functional sensory restoration of myoelectric prostheses equipped with electrotactile feedback.

The science and engineering behind sensitized brain-controlled bionic hands

Advances in our understanding of brain function, along with the development of neural interfaces that allow for the monitoring and activation of neurons, have paved the way for brain-machine interfaces (BMIs), which harness neural signals to reanimate the limbs via electrical activation of the muscles or to control extracorporeal devices, thereby bypassing the muscles and senses altogether. BMIs consist of reading out motor intent from the neuronal responses monitored in motor regions of the brain and executing intended movements with bionic limbs, reanimated limbs, or exoskeletons. BMIs also allow for the restoration of the sense of touch by electrically activating neurons in somatosensory regions of the brain, thereby evoking vivid tactile sensations and conveying feedback about object interactions. In this review, we discuss the neural mechanisms of motor control and somatosensation in able-bodied individuals and describe approaches to use neuronal responses as control signals for movement restoration and to activate residual sensory pathways to restore touch. Although the focus of the review is on intracortical approaches, we also describe alternative signal sources for control and noninvasive strategies for sensory restoration.

Object Recognition via Evoked Sensory Feedback during Control of a Prosthetic Hand

Haptic and proprioceptive feedback is critical for sensorimotor integration when we use our hand to perform daily tasks. Here, we evaluated how externally evoked haptic and proprioceptive feedback and myoelectric control strategies affected the recognition of object properties when participants controlled a prosthetic hand. Fingertip haptic sensation was elicited using a transcutaneous nerve stimulation grid to encode the prosthetic's fingertip forces. An array of tactors elicited patterned vibratory stimuli to encode tactile-proprioceptive kinematic information of the prosthetic finger joint. Myoelectric signals of the finger flexor and extensor were used to control the position or velocity of joint angles of the prosthesis. Participants were asked to perform object property (stiffness and size) recognition, by controlling the prosthetic hand with concurrent haptic and tactile-proprioceptive feedback. With the evoked feedback, intact and amputee participants recognized the object stiffness and size at success rates ranging from 50% to 100% in both position and velocity control with no significant difference across control schemes. Our findings show that evoked somatosensory feedback in a non-invasive manner can facilitate closed-loop control of the prosthetic hand and allowed for simultaneous recognition of different object properties. The outcomes can facilitate our understanding on the role of sensory feedback during bidirectional human-machine interactions, which can potentially promote user experience in object interactions using prosthetic hands.

Closed-loop control of a prosthetic finger via evoked proprioceptive information

Objective.Proprioceptive information plays an important role for recognizing and coordinating our limb's static and dynamic states relative to our body or the environment. In this study, we determined how artificially evoked proprioceptive feedback affected the continuous control of a prosthetic finger.Approach.We elicited proprioceptive information regarding the joint static position and dynamic movement of a prosthetic finger via a vibrotactor array placed around the subject's upper arm. Myoelectric signals of the finger flexor and extensor muscles were used to control the prosthesis, with or without the evoked proprioceptive feedback. Two control modes were evaluated: the myoelectric signal amplitudes were continuously mapped to either the position or the velocity of the prosthetic joint.Main results.Our results showed that the evoked proprioceptive information improved the control accuracy of the joint angle, with comparable performance in the position- and velocity-control conditions. However, greater angle variability was prominent during position-control than velocity-control. Without the proprioceptive feedback, the position-control tended to show a smaller angle error than the velocity-control condition.Significance.Our findings suggest that closed-loop control of a prosthetic device can potentially be achieved using non-invasive evoked proprioceptive feedback delivered to intact participants. Moreover, the evoked sensory information was integrated during myoelectric control effectively for both control strategies. The outcomes can facilitate our understanding of the sensorimotor integration process during human-machine interactions, which can potentially promote fine control of prosthetic hands.

Reduction of Phantom Limb Pain and Improved Proprioception through a TSR-Based Surgical Technique: A Case Series of Four Patients with Lower Limb Amputation

Four patients underwent targeted sensory reinnervation (TSR), a surgical technique in which a defined skin area is first selectively denervated and then surgically reinnervated by another sensory nerve. In our case, either the area of the lateral femoral cutaneous nerve or the saphenous nerve was reinnervated by the sural nerve. Patients were then fitted with a special prosthetic device capable of transferring the sense of pressure from the sole of the prosthesis to the newly wired skin area. Pain reduction after TSR was highly significant in all patients. In three patients, permanent pain medication could even be discontinued, in one patient the pain medication has been significantly reduced. Two of the four patients were completely pain-free after the surgical intervention. Surgical rewiring of existing sensory nerves by TSR can provide the brain with new afferent signals seeming to originate from the missing limb. These signals help to reduce phantom limb pain and to restore a more normal body image. In combination with special prosthetic devices, the amputee can be provided with sensory feedback from the prosthesis, thus improving gait and balance.

A Hybrid Microstructure Piezoresistive Sensor with Machine Learning Approach for Gesture Recognition

Developments in flexible electronics have adopted various approaches which have enhanced the applicability of human–machine interface fields. Recently, microstructural integration and hybrid functional materials were designed for realizing human somatosensory. Nonetheless, designing tactile sensors with smart structures using facile and low-cost fabrication processes remains challenging. Furthermore, using the sensors for recognizing stimuli and feedback applications remains poorly validated. In this study, a highly flexible piezoresistive tactile sensor was developed by homogeneously dispersing carbon black (CB) in a microstructure porous sugar/PDMS-based sponge. Owning to its high flexibility and softness, the sensor can be mounted on human or robotic systems for different clinical applications. We validated the applicability of the proposed sensor by applying it to recognizing grasp and release forces in an open setting and to classifying hand motions that surgeons apply on the master interface of a robotic system during intravascular catheterization. For this purpose, we implemented the long short-term memory (LSTM)-dense classification model and five traditional machine learning methods, namely, support vector machine, multilayer perceptron, decision tree, and k-nearest neighbor. The models were used to classify the different hand gestures obtained in an open-setting experiment. Amongst all, the LSTM-dense method yielded the highest overall recognition accuracy (87.38%). Nevertheless, the performance of the other models was in a similar range, showing that our sensor structure can be applied in intelligence sensing or tactile feedback systems. Secondly, the sensor prototype was applied to analyze the motions made while manipulating an interventional robot. We analyzed the displacement and velocity of the master interface during typical axial (push/pull) and radial operations with the robot. The results obtained show that the sensor is capable of recording unique patterns during different operations. Thus, a combination of the flexible wearable sensors and machine learning could yield a future generation of flexible materials and artificial intelligence of things (AIoT) devices.

Static and dynamic proprioceptive recognition through vibrotactile stimulation

Objective.Proprioceptive information provides individuals with a sense of our limb's static position and dynamic movement. Impaired or a lack of such feedback can diminish our ability to perform dexterous motions with our biological limbs or assistive devices. Here we seek to determine whether both static and dynamic components of proprioception can be recognized using variation of the spatial and temporal components of vibrotactile feedback.Approach.An array of five vibrotactors was placed on the forearm of each subject. Each tactor was encoded to represent one of the five forearm postures. Vibratory stimulus was elicited to convey the static position and movement of the forearm. Four experimental blocks were performed to test each subject's recognition of a forearm's simulated static position, rotational amplitude, rotational amplitude and direction, and rotational speed.Main results.Our results showed that the subjects were able to perform proprioceptive recognition based on the delivered vibrotactile information. Specifically, rotational amplitude recognition resulted in the highest level of accuracy (99.0%), while the recognition accuracy of the static position and the rotational amplitude-direction was the lowest (91.7% and 90.8%, respectively). Nevertheless, all proprioceptive properties were perceived with >90% accuracy, indicating that the implemented vibrotactile encoding scheme could effectively provide proprioceptive information to the users.Significance.The outcomes suggest that information pertaining to static and dynamic aspects of proprioception can be accurately delivered using an array of vibrotactors. This feedback approach could be used to potentially evaluate the sensorimotor integration processes during human-machine interactions, and to improve sensory feedback in clinical populations with somatosensory impairments.

Perception of Static Position and Kinesthesia of the Finger using Vibratory Stimulation

Proprioception provides information regarding the state of an individual's limb in terms of static position and kinesthesia (dynamic movement). When such feedback is lost or impaired, the performance of dexterous control of our biological limbs or assistive devices tends to deteriorate. In this study, we determined if external vibratory stimulation patterns could allow for the perception of a finger's static position and kinesthesia. Using four tactors and two stimulus levels, eight vibratory settings corresponded to eight discrete finger positions. The transition patterns between these eight settings corresponded to kinesthesia. Three experimental blocks assessed the perception of a finger's static position, speed, and movement (amplitude and direction). Our results demonstrated that both position and kinesthesia could be recognized with over 93% accuracy. The outcomes suggest that vibratory stimulus can inform subjects of static and dynamic aspects of finger proprioception. This sensory stimulation approach can be implemented to improve outcomes in clinical populations with sensory deficits, and to enhance user experience when users interact with assistive devices.

Experiment Study for Wrist-Wearable Electro-Tactile Display

Tactile sensation is a promising information display channel for human beings that involves supplementing or replacing degraded visual or auditory channels. In this paper, a wrist-wearable tactile rendering system based on electro-tactile stimulation is designed for information expression, where a square array with 8 × 8 spherical electrodes is used as the touch panel. To verify and improve this touch-based information display method, the optimal mode for stimulus signals was firstly investigated through comparison experiments, which show that sequential stimuli with consecutive-electrode-in-active mode have a better performance than those with single-electrode-in-active mode. Then, simple Chinese and English characters and 26 English characters’ recognition experiments were carried out and the proposed method was verified with an average recognition rate of 95% and 82%, respectively. This wrist-wearable tactile display system would be a new and promising medium for communication and could be of great value for visually impaired people.

Recent Developments in Prosthesis Sensors, Texture Recognition, and Sensory Stimulation for Upper Limb Prostheses

Current developments being made in upper limb prostheses are focused on replacing lost sensory information to the amputees. Providing sensory stimulation from the prosthesis can directly improve control over the prosthetic and provide a sense of body ownership. The focus of this review article is on recent developments while including foundational knowledge for some of the critical concepts in neural prostheses. Reported concepts follow the flow of information from sensors to signal processing, with emphasis on texture recognition, and then to sensory stimulation strategies that reestablish the lost sensory feedback loop. Prosthetic sensors are used to detect the physical environment, converting pressure, force, and position into electrical signals. The electrical signals can then be processed in an effort to identify the surrounding environment using distinctive characteristics such as stiffness and texture. In order for the amputee to use this information in a natural manner, there must be real-time sensory stimulation, perception, and motor control of the prosthesis. Although truly complete sensory replacement has not yet been realized, some basic percepts can be partially restored, allowing progress towards a more realistic prosthesis with natural sensations.

Amplitude versus spatially modulated electrotactile feedback for myoelectric control of two degrees of freedom

OBJECTIVE

Artificial proprioceptive feedback from a myoelectric prosthesis is an important aspect in enhancing embodiment and user satisfaction, possibly lowering the demand for visual attention while controlling a prosthesis in everyday tasks. Contemporary myoelectric prostheses are advanced mechatronic systems with multiple degrees of freedom, and therefore, to communicate the prosthesis state, the feedback interface needs to transmit several variables simultaneously. In the present study, two different configurations for conveying proprioceptive information of wrist rotation and hand aperture through multichannel electrotactile stimulation were developed and evaluated during online myoelectric control.

APPROACH

Myoelectric recordings were acquired from the dominant forearm and electrotactile stimulation was delivered on the non-dominant forearm using a compact interface. The first feedback configuration, which was based on spatial coding, transmitted the information using a moving tactile stimulus, whereas the second, amplitude-based configuration conveyed the position via sensation intensity. Thirteen able-bodied subjects used pattern classification-based myoelectric control with both feedback configurations to accomplish a target-reaching task.

MAIN RESULTS

High task performance (completion rate > 90%) was observed for both configurations, with no significant difference in completion rate, time to reach the target, distance error and path efficiency, respectively.

SIGNIFICANCE

Overall, the results demonstrated that both feedback configurations allowed subjects to perceive and interpret two feedback variables delivered simultaneously, despite using a compact stimulation interface. This is an encouraging result for the prospect of communicating the full state of a multifunctional hand prosthesis.

Object Shape and Surface Topology Recognition Using Tactile Feedback Evoked through Transcutaneous Nerve Stimulation

Tactile feedback is critical for distinguishing different object properties. In this article, we determined if tactile feedback evoked by transcutaneous nerve stimulation can be used to detect objects of different shape and surface topology. To evoke tactile sensation at different fingers, a 2x8 electrode grid was placed along the subject's upper arm, and two concurrent electrical stimulation trains targeted the median and ulnar nerve bundles, which evoked individually modulated sensations at different fingers. Fingertip forces of the prosthetic hand were transformed to stimulation current amplitude. Object shape was encoded based on finger-object contact timing. Surface topology represented by ridge height and spacing was encoded through current amplitude and stimulation time interval, respectively. The elicited sensation allowed subjects to determine object shape with success rates >84%. Surface topology recognition resulted in success rates >81%. Our findings suggest that tactile feedback evoked from transcutaneous nerve stimulation allows the recognition of object shape and surface topology. The ability to recognize these properties may help improve object manipulation and promote fine control of a prosthetic hand.

Object stiffness recognition using haptic feedback delivered through transcutaneous proximal nerve stimulation

OBJECTIVE

Haptic feedback is crucial when we manipulate objects. Information pertaining to an object's stiffness in particular can help facilitate fine motor control. In this study, we seek to determine whether objects of different stiffness levels can be recognized using haptic feedback provided by transcutaneous electrical stimulation of peripheral nerves.

APPROACH

Using a stimulation electrode grid placed along the medial side of the upper arm, the median and ulnar nerve bundles were targeted to evoke haptic sensation on the palmar side of the hand. Stimulation current amplitude was modulated in real-time with the fingertip force recorded from a sensorized prosthetic hand. In order to evaluate which stimulation pattern was more critical, object stiffness was encoded either by the rate of change of the stimulus amplitude or the level of peak stimulus amplitude, as the prosthesis grasped the objects.

MAIN RESULTS

Both encoding methods allowed the subjects to differentiate objects of different stiffness levels with  >90% accuracy. No significant difference was observed between the two encoding methods, which indicated that both the rate of change of the stimulation amplitude and the peak stimulation amplitude could effectively provide stiffness information of the objects.

SIGNIFICANCE

The outcomes suggest that it is possible to elicit haptic sensations describing various object stiffness levels using transcutaneous nerve stimulation. The haptic feedback associated with object stiffness can facilitate object manipulation/interactions. It may also improve user experience during human-machine interactions, when object stiffness information is incorporated.