Texture Discrimination with a Soft Biomimetic Finger Using a Flexible Neuromorphic Tactile Sensor Array That Provides Sensory Feedback

Three-dimensional micro strain gauges as flexible, modular tactile sensors for versatile integration with micro- and macroelectronics

Flexible tactile sensors play important roles in many areas, like human-machine interface, robotic manipulation, and biomedicine. However, their flexible form factor poses challenges in their integration with wafer-based devices, commercial chips, or circuit boards. Here, we introduce manufacturing approaches, device designs, integration strategies, and biomedical applications of a set of flexible, modular tactile sensors, which overcome the above challenges and achieve cooperation with commercial electronics. The sensors exploit lithographically defined thin wires of metal or alloy as the sensing elements. Arranging these elements across three-dimensional space enables accurate, hysteresis-free, and decoupled measurements of temperature, normal force, and shear force. Assembly of such sensors on flexible printed circuit boards together with commercial electronics forms various flexible electronic systems with capabilities in wireless measurements at the skin interface, continuous monitoring of biomechanical signals, and spatial mapping of tactile information. The flexible, modular tactile sensors expand the portfolio of functional components in both microelectronics and macroelectronics.

Restoration of sensory feedback from the foot and reduction of phantom limb pain via closed-loop spinal cord stimulation

Restoring somatosensory feedback in individuals with lower-limb amputations would reduce the risk of falls and alleviate phantom limb pain. Here we show, in three individuals with transtibial amputation (one traumatic and two owing to diabetic peripheral neuropathy), that sensations from the missing foot, with control over their location and intensity, can be evoked via lateral lumbosacral spinal cord stimulation with commercially available electrodes and by modulating the intensity of stimulation in real time on the basis of signals from a wireless pressure-sensitive shoe insole. The restored somatosensation via closed-loop stimulation improved balance control (with a 19-point improvement in the composite score of the Sensory Organization Test in one individual) and gait stability (with a 5-point improvement in the Functional Gait Assessment in one individual). And over the implantation period of the stimulation leads, the three individuals experienced a clinically meaningful decrease in phantom limb pain (with an average reduction of nearly 70% on a visual analogue scale). Our findings support the further clinical assessment of lower-limb neuroprostheses providing somatosensory feedback.

Cobalt MOF-Based Porous Carbonaceous Spheres for Multimodal Soft Actuator Exhibiting Intricate Biomimetic Motions

The advancement of active electrode materials is essential to meet the demand for multifaceted soft robotic interactions. In this study, a new type of porous carbonaceous sphere (PCS) for a multimodal soft actuator capable of both magnetoactive and electro-ionic responses is reported. The PCS, derived from the simultaneous oxidative and reductive breakdown of specially designed cobalt-based metal-organic frameworks (Co-MOFs) with varying metal-to-ligand ratios, exhibits a high specific surface area of 529 m2 g-1 and a saturated magnetization of 142.7 Am2 kg-1. The size of the PCS can be controlled through the Ostwald ripening mechanism, while the porous structure can be regulated by adjusting the metal-to-ligand mol ratio. Its exceptional compatibility with poly(3,4-ethylene-dioxythiophene)-poly(styrenesulfonate) enables the creation of uniform electrode, crucial for producing soft actuators that work in both magnetic and electrical fields. Operated at an ultralow voltage of 1 V, the PCS-based actuator generates a blocking force of 47.5 mN and exhibits significant bending deflection even at an oscillation frequency of 10 Hz. Employing this simultaneous multimodal actuation ensures the dynamic and complex motions of a balancing bird robot and a dynamic eagle robot. This advancement marks a significant step toward the realization of more dynamic and versatile soft robotic systems.

An Amphibious Fully-Soft Centimeter-Scale Miniature Crawling Robot Powered by Electrohydraulic Fluid Kinetic Energy

Miniature locomotion robots with the ability to navigate confined environments show great promise for a wide range of tasks, including search and rescue operations. Soft miniature locomotion robots, as a burgeoning field, have attracted significant research interest due to their exceptional terrain adaptability and safety features. Here, a fully-soft centimeter-scale miniature crawling robot directly powered by fluid kinetic energy generated by an electrohydraulic actuator is introduced. Through optimization of the operating voltage and design parameters, the average crawling velocity of the robot is dramatically enhanced, reaching 16 mm s-1. The optimized robot weighs 6.3 g and measures 5 cm in length, 5 cm in width, and 6 mm in height. By combining two robots in parallel, the robot can achieve a turning rate of ≈3° s-1. Additionally, by reconfiguring the distribution of electrodes in the electrohydraulic actuator, the robot can achieve 2 degrees-of-freedom translational motion, improving its maneuverability in narrow spaces. Finally, the use of a soft water-proof skin is demonstrated for underwater locomotion and actuation. In comparison with other soft miniature crawling robots, this robot with full softness can achieve relatively high crawling velocity as well as increased robustness and recovery.

Human-Inspired Tactile Perception System for Real-Time and Multimodal Detection of Tactile Stimuli

A human can intuitively perceive and comprehend complicated tactile information because the cutaneous receptors distributed in the fingertip skin receive different tactile stimuli simultaneously and the tactile signals are immediately transmitted to the brain. Although many research groups have attempted to mimic the structure and function of human skin, it remains a challenge to implement human-like tactile perception process inside one system. In this study, we developed a real-time and multimodal tactile system that mimics the function of cutaneous receptors and the transduction of tactile stimuli from receptors to the brain, by using multiple sensors, a signal processing and transmission circuit module, and a signal analysis module. The proposed system is capable of simultaneously acquiring four types of decoupled tactile information with a compact system, thereby enabling differentiation between various tactile stimuli, texture characteristics, and consecutive complex motions. This skin-like three-dimensional integrated design provides further opportunities in multimodal tactile sensing systems.

Active learning strategies for robotic tactile texture recognition tasks

Accurate texture classification empowers robots to improve their perception and comprehension of the environment, enabling informed decision-making and appropriate responses to diverse materials and surfaces. Still, there are challenges for texture classification regarding the vast amount of time series data generated from robots' sensors. For instance, robots are anticipated to leverage human feedback during interactions with the environment, particularly in cases of misclassification or uncertainty. With the diversity of objects and textures in daily activities, Active Learning (AL) can be employed to minimize the number of samples the robot needs to request from humans, streamlining the learning process. In the present work, we use AL to select the most informative samples for annotation, thus reducing the human labeling effort required to achieve high performance for classifying textures. We also use a sliding window strategy for extracting features from the sensor's time series used in our experiments. Our multi-class dataset (e.g., 12 textures) challenges traditional AL strategies since standard techniques cannot control the number of instances per class selected to be labeled. Therefore, we propose a novel class-balancing instance selection algorithm that we integrate with standard AL strategies. Moreover, we evaluate the effect of sliding windows of two-time intervals (3 and 6 s) on our AL Strategies. Finally, we analyze in our experiments the performance of AL strategies, with and without the balancing algorithm, regarding f1-score, and positive effects are observed in terms of performance when using our proposed data pipeline. Our results show that the training data can be reduced to 70% using an AL strategy regardless of the machine learning model and reach, and in many cases, surpass a baseline performance. Finally, exploring the textures with a 6-s window achieves the best performance, and using either Extra Trees produces an average f1-score of 90.21% in the texture classification data set.

Advanced Stiffness Sensing through the Pincer Grasping of Soft Pneumatic Grippers

In this study, a comprehensive approach for sensing object stiffness through the pincer grasping of soft pneumatic grippers (SPGs) is presented. This study was inspired by the haptic sensing of human hands that allows us to perceive object properties through grasping. Many researchers have tried to imitate this capability in robotic grippers. The association between gripper performance and object reaction must be determined for this purpose. However, soft pneumatic actuators (SPA), the main components of SPGs, are extremely compliant. SPA compliance makes the determination of the association challenging. Methodologically, the connection between the behaviors of grasped objects and those of SPAs was clarified. A new concept of SPA modeling was then introduced. A method for stiffness sensing through SPG pincer grasping was developed based on this connection, and demonstrated on four samples. This method was validated through compression testing on the same samples. The results indicate that the proposed method yielded similar stiffness trends with slight deviations in compression testing. A main limitation in this study was the occlusion effect, which leads to dramatic deviations when grasped objects greatly deform. This is the first study to enable stiffness sensing and SPG grasping to be carried out in the same attempt. This study makes a major contribution to research on soft robotics by progressing the role of sensing for SPG grasping and object classification by offering an efficient method for acquiring another effective class of classification input. Ultimately, the proposed framework shows promise for future applications in inspecting and classifying visually indistinguishable objects.

Texture recognition based on multi-sensory integration of proprioceptive and tactile signals

The sense of touch plays a fundamental role in enabling us to interact with our surrounding environment. Indeed, the presence of tactile feedback in prostheses greatly assists amputees in doing daily tasks. In this line, the present study proposes an integration of artificial tactile and proprioception receptors for texture discrimination under varying scanning speeds. Here, we fabricated a soft biomimetic fingertip including an 8 × 8 array tactile sensor and a piezoelectric sensor to mimic Merkel, Meissner, and Pacinian mechanoreceptors in glabrous skin, respectively. A hydro-elastomer sensor was fabricated as an artificial proprioception sensor (muscle spindles) to assess the instantaneous speed of the biomimetic fingertip. In this study, we investigated the concept of the complex receptive field of RA-I and SA-I afferents for naturalistic textures. Next, to evaluate the synergy between the mechanoreceptors and muscle spindle afferents, ten naturalistic textures were manipulated by a soft biomimetic fingertip at six different speeds. The sensors' outputs were converted into neuromorphic spike trains to mimic the firing pattern of biological mechanoreceptors. These spike responses are then analyzed using machine learning classifiers and neural coding paradigms to explore the multi-sensory integration in real experiments. This synergy between muscle spindle and mechanoreceptors in the proposed neuromorphic system represents a generalized texture discrimination scheme and interestingly irrespective of the scanning speed.

Nonlinear Tactile Estimation Model Based on Perceptibility of Mechanoreceptors Improves Quantitative Tactile Sensing

Tactile sensing has attracted significant attention as a tactile quantitative evaluation method because the tactile sensation is an important factor while evaluating consumer products. Although the human tactile perception mechanism has nonlinearity, previous studies have often developed linear regression models. In contrast, this study proposes a nonlinear tactile estimation model that can estimate sensory evaluation scores from physical measurements. We extracted features from the vibration data obtained by a tactile sensor based on the perceptibility of mechanoreceptors. In parallel, a sensory evaluation test was conducted using 10 evaluation words. Then, the relationship between the extracted features and the tactile evaluation results was modeled using linear/nonlinear regressions. The best model was concluded by comparing the mean squared error between the model predictions and the actual values. The results imply that there are multiple evaluation words suitable for adopting nonlinear regression models, and the average error was 43.8% smaller than that of building only linear regression models.

Touch, Texture and Haptic Feedback: A Review on How We Feel the World around Us

Touch is one most of the important aspects of human life. Nearly all interactions, when broken down, involve touch in one form or another. Recent advances in technology, particularly in the field of virtual reality, have led to increasing interest in the research of haptics. However, accurately capturing touch is still one of most difficult engineering challenges currently being faced. Recent advances in technology such as those found in microcontrollers which allow the creation of smaller sensors and feedback devices may provide the solution. Beyond capturing and measuring touch, replicating touch is also another unique challenge due to the complexity and sensitivity of the human skin. The development of flexible, soft-wearable devices, however, has allowed for the creating of feedback systems that conform to the human form factor with minimal loss of accuracy, thus presenting possible solutions and opportunities. Thus, in this review, the researchers aim to showcase the technologies currently being used in haptic feedback, and their strengths and limitations.

Fine Texture Detection Based on a Solid–Liquid Composite Flexible Tactile Sensor Array

Surface texture information plays an important role in the cognition and manipulation of an object. Vision and touch are the two main methods for extracting an object’s surface texture information. However, vision is often limited since the viewing angle is uncertain during manipulation. In this article, we propose a fine surface texture detection method based on a stochastic resonance algorithm through a novel solid–liquid composite flexible tactile sensor array. A thin flexible layer and solid–liquid composite conduction structure on the sensor effectively reduce the attenuation of the contact force and enhance the sensitivity of the sensor. A series of ridge texture samples with different heights (0.9, 4, 10 μm), different widths (0.3, 0.5, 0.7, 1 mm), but the same spatial period (2 mm) of ridges were used in the experiment. The experimental results prove that the stochastic resonance algorithm can significantly improve the signal characteristic of the output signal of the sensor. The sensor has the capability to detect fine ridge texture information. The mean relative error of the estimation for the spatial period was 1.085%, and the ridge width and ridge height, respectively, have a monotonic mapping relationship with the corresponding model output parameters. The sensing capability to sense a fine texture of tactile senor surpasses the limit of human fingers.

Texture Identification and Object Recognition Using a Soft Robotic Hand Innervated Bio-Inspired Proprioception

In this study, we innervated bio-inspired proprioception into a soft hand, facilitating a robust perception of textures and object shapes. The tendon-driven soft finger with three joints, inspired by the human finger, was detailed. With tension sensors embedded in the tendon that simulate the Golgi tendon organ of the human body, 17 types of textures can be identified under uncertain rotation angles and actuator displacements. Four classifiers were used and the highest identification accuracy was 98.3%. A three-fingered soft hand based on the bionic finger was developed. Its basic grasp capability was tested experimentally. The soft hand can distinguish 10 types of objects that vary in shape with top grasp and side grasp, with the highest accuracies of 96.33% and 96.00%, respectively. Additionally, for six objects with close shapes, the soft hand obtained an identification accuracy of 97.69% with a scan-grasp method. This study offers a novel bionic solution for the texture identification and object recognition of soft manipulators.

A stretchable and strain-unperturbed pressure sensor for motion interference-free tactile monitoring on skins

A stretchable pressure sensor is a necessary tool for perceiving physical interactions that take place on soft/deformable skins present in human bodies, prosthetic limbs, or soft robots. However, all existing types of stretchable pressure sensors have an inherent limitation, which is the interference of stretching with pressure sensing accuracy. Here, we present a design for a highly stretchable and highly sensitive pressure sensor that can provide unaltered sensing performance under stretching, which is realized through the synergistic creations of an ionic capacitive sensing mechanism and a mechanically hierarchical microstructure. Via this optimized structure, our sensor exhibits 98% strain insensitivity up to 50% strain and a low pressure detection limit of 0.2 Pa. With the capability to provide all the desired characteristics for quantitative pressure sensing on a deformable surface, this sensor has been used to realize the accurate sensation of physical interactions on human or soft robotic skin.

A Biomimetic Circuit for Electronic Skin With Application in Hand Prosthesis

One major challenge in upper limb prostheses is providing sensory feedback to amputees. Reproducing the spiking patterns of human primary tactile afferents can be considered as the first step for this challenging problem. In this study, a novel biomimetic circuit for SA-I and RA-I afferents is proposed to functionally replicate the spiking response of the biological tactile afferents to indentation stimuli. The circuit has been designed, laid out, and simulated in TSMC 180nm CMOS technology with a 1.8V supply voltage. A pair of SA-I and RA-I afferent circuits consume [Formula: see text] of power. The occupied silicon area is [Formula: see text] for 32 afferents. To provide the inputs for circuit testing, a patch of skin with a grid of mechanoreceptors is simulated and tested by an edge stimulus presented at different orientations. Experimental data are collected using indentation of 3D-printed edges at different orientations on a tactile sensor mounted on a robotic arm. Inspired by innervation patterns observed in biology, the artificial afferents are connected to several neighboring mechanoreceptors with different weights to form complex receptive fields which cover the entire mechanoreceptor grid. Machine learning algorithms are applied offline to classify the edge orientations based on the pattern of neural responses. Our results show that the complex receptive fields arising from the innervation pattern led to smaller circuit area and lower power consumption, while facilitating data encoding from high-resolution sensors. The proposed biomimetic circuit and tactile encoding example demonstrate potential applications in modern tactile sensing modules for developing novel bio-robotic and prosthetic technologies.

A bionic piezoelectric tactile sensor for features recognition of object surface based on machine learning

Based on the tactile mechanism of human fingertips, a bionic tactile sensor fabricated from polyvinylidene fluoride piezoelectric film is proposed, which can identify the surface softness, viscoelasticity, thermal conductivity, and texture roughness of the object. The tactile sensor is mounted on the fingertip of the bionic manipulator, which obtains the surface features by touching and sliding the object. The time-domain features of the output signal are used for preliminarily discriminating the softness, viscoelasticity, and heat conduction of the object. Finally, based on the Back Propagation and the Particle Swarm Optimization-Back Propagation neural network algorithm, the recognition experiment of texture roughness is carried out using the PSO algorithm to improve the BP neural network so that the optimized BP algorithm has a higher convergence accuracy. The results show that the PSO-BP algorithm achieved the highest accuracy of 98% for identifying samples with different roughnesses and the average recognition achieved an accuracy of 94%. The bionic piezoelectric tactile sensor proposed in this paper has a good application development prospect in recognizing the surface features of objects and intelligent robots.

Hierarchical Tactile Sensation Integration from Prosthetic Fingertips Enables Multi-Texture Surface Recognition

Multifunctional flexible tactile sensors could be useful to improve the control of prosthetic hands. To that end, highly stretchable liquid metal tactile sensors (LMS) were designed, manufactured via photolithography, and incorporated into the fingertips of a prosthetic hand. Three novel contributions were made with the LMS. First, individual fingertips were used to distinguish between different speeds of sliding contact with different surfaces. Second, differences in surface textures were reliably detected during sliding contact. Third, the capacity for hierarchical tactile sensor integration was demonstrated by using four LMS signals simultaneously to distinguish between ten complex multi-textured surfaces. Four different machine learning algorithms were compared for their successful classification capabilities: K-nearest neighbor (KNN), support vector machine (SVM), random forest (RF), and neural network (NN). The time-frequency features of the LMSs were extracted to train and test the machine learning algorithms. The NN generally performed the best at the speed and texture detection with a single finger and had a 99.2 ± 0.8% accuracy to distinguish between ten different multi-textured surfaces using four LMSs from four fingers simultaneously. The capability for hierarchical multi-finger tactile sensation integration could be useful to provide a higher level of intelligence for artificial hands.

Development and Grasp Stability Estimation of Sensorized Soft Robotic Hand

This paper introduces the development of an anthropomorphic soft robotic hand integrated with multiple flexible force sensors in the fingers. By leveraging on the integrated force sensing mechanism, grip state estimation networks have been developed. The robotic hand was tasked to hold the given object on the table for 1.5 s and lift it up within 1 s. The object manipulation experiment of grasping and lifting the given objects were conducted with various pneumatic pressure (50, 80, and 120 kPa). Learning networks were developed to estimate occurrence of object instability and slippage due to acceleration of the robot or insufficient grasp strength. Hence the grip state estimation network can potentially feedback object stability status to the pneumatic control system. This would allow the pneumatic system to use suitable pneumatic pressure to efficiently handle different objects, i.e., lower pneumatic pressure (50 kPa) for lightweight objects which do not require high grasping strength. The learning process of the soft hand is made challenging by curating a diverse selection of daily objects, some of which displays dynamic change in shape upon grasping. To address the cost of collecting extensive training datasets, we adopted one-shot learning (OSL) technique with a long short-term memory (LSTM) recurrent neural network. OSL aims to allow the networks to learn based on limited training data. It also promotes the scalability of the network to accommodate more grasping objects in the future. Three types of LSTM-based networks have been developed and their performance has been evaluated in this study. Among the three LSTM networks, triplet network achieved overall stability estimation accuracy at 89.96%, followed by LSTM network with 88.00% and Siamese LSTM network with 85.16%.

Cut wires: The Electrophysiology of Regenerated Tissue

When nerves are damaged by trauma or disease, they are still capable of firing off electrical command signals that originate from the brain. Furthermore, those damaged nerves have an innate ability to partially regenerate, so they can heal from trauma and even reinnervate new muscle targets. For an amputee who has his/her damaged nerves surgically reconstructed, the electrical signals that are generated by the reinnervated muscle tissue can be sensed and interpreted with bioelectronics to control assistive devices or robotic prostheses. No two amputees will have identical physiologies because there are many surgical options for reconstructing residual limbs, which may in turn impact how well someone can interface with a robotic prosthesis later on. In this review, we aim to investigate what the literature has to say about different pathways for peripheral nerve regeneration and how each pathway can impact the neuromuscular tissue’s final electrophysiology. This information is important because it can guide us in planning the development of future bioelectronic devices, such as prosthetic limbs or neurostimulators. Future devices will primarily have to interface with tissue that has undergone some natural regeneration process, and so we have explored and reported here what is known about the bioelectrical features of neuromuscular tissue regeneration.