Tactile Feedback in Upper Limb Prosthetic Devices Using Flexible Textile Force Sensors

Neuromorphic applications in medicine

In recent years, there has been a growing demand for miniaturization, low power consumption, quick treatments, and non-invasive clinical strategies in the healthcare industry. To meet these demands, healthcare professionals are seeking new technological paradigms that can improve diagnostic accuracy while ensuring patient compliance. Neuromorphic engineering, which uses neural models in hardware and software to replicate brain-like behaviors, can help usher in a new era of medicine by delivering low power, low latency, small footprint, and high bandwidth solutions. This paper provides an overview of recent neuromorphic advancements in medicine, including medical imaging and cancer diagnosis, processing of biosignals for diagnosis, and biomedical interfaces, such as motor, cognitive, and perception prostheses. For each section, we provide examples of how brain-inspired models can successfully compete with conventional artificial intelligence algorithms, demonstrating the potential of neuromorphic engineering to meet demands and improve patient outcomes. Lastly, we discuss current struggles in fitting neuromorphic hardware with non-neuromorphic technologies and propose potential solutions for future bottlenecks in hardware compatibility.

An Improved Approach for Grasp Force Sensing and Control of Upper Limb Soft Robotic Prosthetics

The following research proposes a closed loop force control system, which is implemented on a soft robotic prosthetic hand. The proposed system uses a force sensing approach that does not require any sensing elements to be embedded in the prosthetic's fingers, therefore maintaining their monolithic structural integrity, and subsequently decreasing the cost and manufacturing complexity. This is achieved by embedding an aluminum test specimen with a full bridge strain gauge circuit directly inside the actuator's housing rather than in the finger. The location of the test specimen is precisely at the location of the critical section of the bending moment on the actuator housing due to the tension in the driving tendon. Therefore, the resulting loadcell can acquire a signal proportional to the prosthetic's grasping force. A PI controller is implemented and tested using this force sensing approach. The experiment design includes a flexible test object, which serves to visually demonstrate the force controller's performance through the deformation that the test object experiences. Setpoints corresponding to "light", "medium", and "hard" grasps were tested with pinch, tripod, and full grasps and the results of these tests are documented in this manuscript. The developed controller was found to have an accuracy of ±2%. Additionally, the deformation of the test object increased proportionally with the given grasp force setpoint, with almost no deformation during the light grasp test, slight deformation during the medium grasp test, and relatively large deformation of the test object during the hard grasp test.

An Adaptive Socket Attaches onto Residual Limb Using Smart Polymers for Upper Limb Prosthesis

A major challenge for upper limb amputees is discomfort due to improper socket fit on the residual limb during daily use of their prosthesis. Our work introduces the implementation of soft robotic actuators into a prosthetic socket. The soft actuators are a type of electrically-powered actuator. The actuator is driven through changes in internal temperature causing actuation due to vapor pressure, which results in high and reliable force outputs. A regression fit was generated to model how the smart polymer's temperature relates to force output, and the model was cross-validated based on training data collected from each actuator. A proportional integral (PI) controller regulated the force exerted by the actuators based off of tactile and temperature feedback. Results showed that a socket system can be integrated with smart polymers and sensors, and demonstrated the ability to control two actuators and reach desired forces from set temperatures.

Texture Discrimination using a Soft Biomimetic Finger for Prosthetic Applications

Soft robotic fingers have shown great potential for use in prostheses due to their inherent compliant, light, and dexterous nature. Recent advancements in sensor technology for soft robotic systems showcase their ability to perceive and respond to static cues. However, most of the soft fingers for use in prosthetic applications are not equipped with sensors which have the ability to perceive texture like humans can. In this work, we present a dexterous, soft, biomimetic solution which is capable of discrimination of textures. We fabricated a soft finger with two individually controllable degrees of freedom with a tactile sensor embedded at the fingertip. The output of the tac- tile sensor, as texture plates were palpated, was converted into spikes, mimicking the behavior of a biological mechanoreceptor. We explored the spatial properties of the textures captured in the form of spiking patterns by generating spatial event plots and analyzing the similarity between spike trains generated for each texture. Unique features representative of the different textures were then extracted from the spikes and input to a classifier. The textures were successfully classified with an accuracy of 94% when palpating at a rate of 42 mm/s. This work demonstrates the potential of providing amputees with a soft finger with sensing capabilities, which could potentially help discriminate between different objects and surfaces during activities of daily living (ADL) through palpation.

Neuromorphic vision and tactile fusion for upper limb prosthesis control

A major issue with upper limb prostheses is the disconnect between sensory information perceived by the user and the information perceived by the prosthesis. Advances in prosthetic technology introduced tactile information for monitoring grasping activity, but visual information, a vital component in the human sensory system, is still not fully utilized as a form of feedback to the prosthesis. For able-bodied individuals, many of the decisions for grasping or manipulating an object, such as hand orientation and aperture, are made based on visual information before contact with the object. We show that inclusion of neuromorphic visual information, combined with tactile feedback, improves the ability and efficiency of both able-bodied and amputee subjects to pick up and manipulate everyday objects. We discovered that combining both visual and tactile information in a real-time closed loop feedback strategy generally decreased the completion time of a task involving picking up and manipulating objects compared to using a single modality for feedback. While the full benefit of the combined feedback was partially obscured by experimental inaccuracies of the visual classification system, we demonstrate that this fusion of neuromorphic signals from visual and tactile sensors can provide valuable feedback to a prosthetic arm for enhancing real-time function and usability.

Dynamically Mapping Socket Loading Conditions During Real Time Operation of an Upper Limb Prosthesis

A continuing problem faced by amputees is that extended use of a prosthesis leads to discomfort along the residual limb. In this work, we use a novel pressure sensor array and an inertial measuring unit to monitor the changes in the pressure distribution within an upper limb socket in response to its position and the real time performance of a grasping task. These experiments illustrate that the operation of a prosthetic hand produces distinct features in the time derivative and spatial component of the sensor outputs, which correspond to the orientation and task-dependent changes in loading conditions within the socket. The significance of this study is that it highlights the use of a combined pressure sensor array and inertial measuring unit as a way to characterize the loading conditions within a prosthesis based on both temporal and spatial information during movement. This method of real time pressure sensing in prosthetic sockets will be useful for adaptive socket technology aimed towards decreasing the discomfort caused by long-term use of a prosthesis.

Prosthesis with neuromorphic multilayered e-dermis perceives touch and pain

The human body is a template for many state-of-the-art prosthetic devices and sensors. Perceptions of touch and pain are fundamental components of our daily lives that convey valuable information about our environment while also providing an element of protection from damage to our bodies. Advances in prosthesis designs and control mechanisms can aid an amputee's ability to regain lost function but often lack meaningful tactile feedback or perception. Through transcutaneous electrical nerve stimulation (TENS) with an amputee, we discovered and quantified stimulation parameters to elicit innocuous (non-painful) and noxious (painful) tactile perceptions in the phantom hand. Electroencephalography (EEG) activity in somatosensory regions confirms phantom hand activation during stimulation. We invented a multilayered electronic dermis (e-dermis) with properties based on the behavior of mechanoreceptors and nociceptors to provide neuromorphic tactile information to an amputee. Our biologically inspired e-dermis enables a prosthesis and its user to perceive a continuous spectrum from innocuous to noxious touch through a neuromorphic interface that produces receptor-like spiking neural activity. In a Pain Detection Task (PDT), we show the ability of the prosthesis and amputee to differentiate non-painful or painful tactile stimuli using sensory feedback and a pain reflex feedback control system. In this work, an amputee can use perceptions of touch and pain to discriminate object curvature, including sharpness. This work demonstrates possibilities for creating a more natural sensation spanning a range of tactile stimuli for prosthetic hands.

An Extreme Learning Machine-Based Neuromorphic Tactile Sensing System for Texture Recognition

Despite significant advances in computational algorithms and development of tactile sensors, artificial tactile sensing is strikingly less efficient and capable than the human tactile perception. Inspired by efficiency of biological systems, we aim to develop a neuromorphic system for tactile pattern recognition. We particularly target texture recognition as it is one of the most necessary and challenging tasks for artificial sensory systems. Our system consists of a piezoresistive fabric material as the sensor to emulate skin, an interface that produces spike patterns to mimic neural signals from mechanoreceptors, and an extreme learning machine (ELM) chip to analyze spiking activity. Benefiting from intrinsic advantages of biologically inspired event-driven systems and massively parallel and energy-efficient processing capabilities of the ELM chip, the proposed architecture offers a fast and energy-efficient alternative for processing tactile information. Moreover, it provides the opportunity for the development of low-cost tactile modules for large-area applications by integration of sensors and processing circuits. We demonstrate the recognition capability of our system in a texture discrimination task, where it achieves a classification accuracy of 92% for categorization of ten graded textures. Our results confirm that there exists a tradeoff between response time and classification accuracy (and information transfer rate). A faster decision can be achieved at early time steps or by using a shorter time window. This, however, results in deterioration of the classification accuracy and information transfer rate. We further observe that there exists a tradeoff between the classification accuracy and the input spike rate (and thus energy consumption). Our work substantiates the importance of development of efficient sparse codes for encoding sensory data to improve the energy efficiency. These results have a significance for a wide range of wearable, robotic, prosthetic, and industrial applications.

Targeted Transcutaneous Electrical Nerve Stimulation for Phantom Limb Sensory Feedback

In this work, we investigated the use of noninvasive, targeted transcutaneous electrical nerve stimulation (TENS) of peripheral nerves to provide sensory feedback to two amputees, one with targeted sensory reinnervation (TSR) and one without TSR. A major step in developing a closed-loop prosthesis is providing the sense of touch back to the amputee user. We investigated the effect of targeted nerve stimulation amplitude, pulse width, and frequency on stimulation perception. We discovered that both subjects were able to reliably detect stimulation patterns with pulses less than 1 ms. We utilized the psychophysical results to produce a subject specific stimulation pattern using a leaky integrate and fire (LIF) neuron model from force sensors on a prosthetic hand during a grasping task. For the first time, we show that TENS is able to provide graded sensory feedback at multiple sites in both TSR and non-TSR amputees while using behavioral results to tune a neuromorphic stimulation pattern driven by a force sensor output from a prosthetic hand.

Biologically inspired multi-layered synthetic skin for tactile feedback in prosthetic limbs

The human body offers a template for many state-of-the-art prosthetic devices and sensors. In this work, we present a novel, sensorized synthetic skin that mimics the natural multi-layered nature of mechanoreceptors found in healthy glabrous skin to provide tactile information. The multi-layered sensor is made up of flexible piezoresistive textiles that act as force sensitive resistors (FSRs) to convey tactile information, which are embedded within a silicone rubber to resemble the compliant nature of human skin. The top layer of the synthetic skin is capable of detecting small loads less than 5 N whereas the bottom sensing layer responds reliably to loads over 7 N. Finite element analysis (FEA) of a simplified human fingertip and the synthetic skin was performed. Results suggest similarities in behavior during loading. A natural tactile event is simulated by loading the synthetic skin on a prosthetic limb. Results show the sensors' ability to detect applied loads as well as the ability to simulate neural spiking activity based on the derivative and temporal differences of the sensor response. During the tactile loading, the top sensing layer responded 0.24 s faster than the bottom sensing layer. A synthetic biologically-inspired skin such as this will be useful for enhancing the functionality of prosthetic limbs through tactile feedback.

Neuromimetic Event-Based Detection for Closed-Loop Tactile Feedback Control of Upper Limb Prostheses

Upper limb amputees lack the valuable tactile sensing that helps provide context about the surrounding environment. Here we utilize tactile information to provide active touch feedback to a prosthetic hand. First, we developed fingertip tactile sensors for producing biomimetic spiking responses for monitoring contact, release, and slip of an object grasped by a prosthetic hand. We convert the sensor output into pulses, mimicking the rapid and slowly adapting spiking responses of receptor afferents found in the human body. Second, we designed and implemented two neuromimetic event-based algorithms, Compliant Grasping and Slip Prevention, on a prosthesis to create a local closed-loop tactile feedback control system (i.e. tactile information is sent to the prosthesis). Grasping experiments were designed to assess the benefit of this biologically inspired neuromimetic tactile feedback to a prosthesis. Results from able-bodied and amputee subjects show the average number of objects that broke or slipped during grasping decreased by over 50% and the average time to complete a grasping task decreased by at least 10% for most trials when comparing neuromimetic tactile feedback with no feedback on a prosthesis. Our neuromimetic method of closed-loop tactile sensing is a novel approach to improving the function of upper limb prostheses.