Neuromorphic Artificial Touch for Categorization of Naturalistic Textures

Neuromorphic hardware for somatosensory neuroprostheses

In individuals with sensory-motor impairments, missing limb functions can be restored using neuroprosthetic devices that directly interface with the nervous system. However, restoring the natural tactile experience through electrical neural stimulation requires complex encoding strategies. Indeed, they are presently limited in effectively conveying or restoring tactile sensations by bandwidth constraints. Neuromorphic technology, which mimics the natural behavior of neurons and synapses, holds promise for replicating the encoding of natural touch, potentially informing neurostimulation design. In this perspective, we propose that incorporating neuromorphic technologies into neuroprostheses could be an effective approach for developing more natural human-machine interfaces, potentially leading to advancements in device performance, acceptability, and embeddability. We also highlight ongoing challenges and the required actions to facilitate the future integration of these advanced technologies.

Uneven Terrain Recognition Using Neuromorphic Haptic Feedback

Recent years have witnessed relevant advancements in the quality of life of persons with lower limb amputations thanks to the technological developments in prosthetics. However, prostheses that provide information about the foot-ground interaction, and in particular about terrain irregularities, are still missing on the market. The lack of tactile feedback from the foot sole might lead subjects to step on uneven terrains, causing an increase in the risk of falling. To address this issue, a biomimetic vibrotactile feedback system that conveys information about gait and terrain features sensed by a dedicated insole has been assessed with intact subjects. After having shortly experienced both even and uneven terrains, the recruited subjects discriminated them with an accuracy of 87.5%, solely relying on the replay of the vibrotactile feedback. With the objective of exploring the human decoding mechanism of the feedback startegy, a KNN classifier was trained to recognize the uneven terrains. The outcome suggested that the subjects achieved such performance with a temporal dynamics of 45 ms. This work is a leap forward to assist lower-limb amputees to appreciate the floor conditions while walking, adapt their gait and promote a more confident use of their artificial limb.

Restoration of sensory information via bionic hands

Individuals who have lost the use of their hands because of amputation or spinal cord injury can use prosthetic hands to restore their independence. A dexterous prosthesis requires the acquisition of control signals that drive the movements of the robotic hand, and the transmission of sensory signals to convey information to the user about the consequences of these movements. In this Review, we describe non-invasive and invasive technologies for conveying artificial sensory feedback through bionic hands, and evaluate the technologies' long-term prospects.

Texture recognition based on multi-sensory integration of proprioceptive and tactile signals

The sense of touch plays a fundamental role in enabling us to interact with our surrounding environment. Indeed, the presence of tactile feedback in prostheses greatly assists amputees in doing daily tasks. In this line, the present study proposes an integration of artificial tactile and proprioception receptors for texture discrimination under varying scanning speeds. Here, we fabricated a soft biomimetic fingertip including an 8 × 8 array tactile sensor and a piezoelectric sensor to mimic Merkel, Meissner, and Pacinian mechanoreceptors in glabrous skin, respectively. A hydro-elastomer sensor was fabricated as an artificial proprioception sensor (muscle spindles) to assess the instantaneous speed of the biomimetic fingertip. In this study, we investigated the concept of the complex receptive field of RA-I and SA-I afferents for naturalistic textures. Next, to evaluate the synergy between the mechanoreceptors and muscle spindle afferents, ten naturalistic textures were manipulated by a soft biomimetic fingertip at six different speeds. The sensors' outputs were converted into neuromorphic spike trains to mimic the firing pattern of biological mechanoreceptors. These spike responses are then analyzed using machine learning classifiers and neural coding paradigms to explore the multi-sensory integration in real experiments. This synergy between muscle spindle and mechanoreceptors in the proposed neuromorphic system represents a generalized texture discrimination scheme and interestingly irrespective of the scanning speed.

Braille letter reading: A benchmark for spatio-temporal pattern recognition on neuromorphic hardware

Spatio-temporal pattern recognition is a fundamental ability of the brain which is required for numerous real-world activities. Recent deep learning approaches have reached outstanding accuracies in such tasks, but their implementation on conventional embedded solutions is still very computationally and energy expensive. Tactile sensing in robotic applications is a representative example where real-time processing and energy efficiency are required. Following a brain-inspired computing approach, we propose a new benchmark for spatio-temporal tactile pattern recognition at the edge through Braille letter reading. We recorded a new Braille letters dataset based on the capacitive tactile sensors of the iCub robot's fingertip. We then investigated the importance of spatial and temporal information as well as the impact of event-based encoding on spike-based computation. Afterward, we trained and compared feedforward and recurrent Spiking Neural Networks (SNNs) offline using Backpropagation Through Time (BPTT) with surrogate gradients, then we deployed them on the Intel Loihi neuromorphic chip for fast and efficient inference. We compared our approach to standard classifiers, in particular to the Long Short-Term Memory (LSTM) deployed on the embedded NVIDIA Jetson GPU, in terms of classification accuracy, power, and energy consumption together with computational delay. Our results show that the LSTM reaches ~97% of accuracy, outperforming the recurrent SNN by ~17% when using continuous frame-based data instead of event-based inputs. However, the recurrent SNN on Loihi with event-based inputs is ~500 times more energy-efficient than the LSTM on Jetson, requiring a total power of only ~30 mW. This work proposes a new benchmark for tactile sensing and highlights the challenges and opportunities of event-based encoding, neuromorphic hardware, and spike-based computing for spatio-temporal pattern recognition at the edge.

Massive-Scale Aerial Photo Categorization by Cross-Resolution Visual Perception Enhancement

Categorizing aerial photographs with varied weather/lighting conditions and sophisticated geomorphic factors is a key module in autonomous navigation, environmental evaluation, and so on. Previous image recognizers cannot fulfill this task due to three challenges: 1) localizing visually/semantically salient regions within each aerial photograph in a weakly annotated context due to the unaffordable human resources required for pixel-level annotation; 2) aerial photographs are generally with multiple informative attributes (e.g., clarity and reflectivity), and we have to encode them for better aerial photograph modeling; and 3) designing a cross-domain knowledge transferal module to enhance aerial photograph perception since multiresolution aerial photographs are taken asynchronistically and are mutually complementary. To handle the above problems, we propose to optimize aerial photograph's feature learning by leveraging the low-resolution spatial composition to enhance the deep learning of perceptual features with a high resolution. More specifically, we first extract many BING-based object patches (Cheng et al., 2014) from each aerial photograph. A weakly supervised ranking algorithm selects a few semantically salient ones by seamlessly incorporating multiple aerial photograph attributes. Toward an interpretable aerial photograph recognizer indicative to human visual perception, we construct a gaze shifting path (GSP) by linking the top-ranking object patches and, subsequently, derive the deep GSP feature. Finally, a cross-domain multilabel SVM is formulated to categorize each aerial photograph. It leverages the global feature from low-resolution counterparts to optimize the deep GSP feature from a high-resolution aerial photograph. Comparative results on our compiled million-scale aerial photograph set have demonstrated the competitiveness of our approach. Besides, the eye-tracking experiment has shown that our ranking-based GSPs are over 92% consistent with the real human gaze shifting sequences.

Touch, Texture and Haptic Feedback: A Review on How We Feel the World around Us

Touch is one most of the important aspects of human life. Nearly all interactions, when broken down, involve touch in one form or another. Recent advances in technology, particularly in the field of virtual reality, have led to increasing interest in the research of haptics. However, accurately capturing touch is still one of most difficult engineering challenges currently being faced. Recent advances in technology such as those found in microcontrollers which allow the creation of smaller sensors and feedback devices may provide the solution. Beyond capturing and measuring touch, replicating touch is also another unique challenge due to the complexity and sensitivity of the human skin. The development of flexible, soft-wearable devices, however, has allowed for the creating of feedback systems that conform to the human form factor with minimal loss of accuracy, thus presenting possible solutions and opportunities. Thus, in this review, the researchers aim to showcase the technologies currently being used in haptic feedback, and their strengths and limitations.

Memristive Izhikevich Spiking Neuron Model and Its Application in Oscillatory Associative Memory

The Izhikevich (IZH) spiking neuron model can display spiking and bursting behaviors of neurons. Based on the switching property and bio-plausibility of the memristor, the memristive Izhikevich (MIZH) spiking neuron model is built. Firstly, the MIZH spiking model is introduced and used to generate 23 spiking patterns. We compare the 23 spiking patterns produced by the IZH and MIZH spiking models. Secondly, the MIZH spiking model actively reproduces various neuronal behaviors, including the excitatory cortical neurons, the inhibitory cortical neurons, and other cortical neurons. Finally, the collective dynamic activities of the MIZH neuronal network are performed, and the MIZH oscillatory network is constructed. Experimental results illustrate that the constructed MIZH spiking neuron model performs high firing frequency and good frequency adaptation. The model can easily simulate various spiking and bursting patterns of distinct neurons in the brain. The MIZH neuronal network realizes the synchronous and asynchronous collective behaviors. The MIZH oscillatory network can memorize and retrieve the information patterns correctly and efficiently with high retrieval accuracy.

Fine Texture Detection Based on a Solid–Liquid Composite Flexible Tactile Sensor Array

Surface texture information plays an important role in the cognition and manipulation of an object. Vision and touch are the two main methods for extracting an object’s surface texture information. However, vision is often limited since the viewing angle is uncertain during manipulation. In this article, we propose a fine surface texture detection method based on a stochastic resonance algorithm through a novel solid–liquid composite flexible tactile sensor array. A thin flexible layer and solid–liquid composite conduction structure on the sensor effectively reduce the attenuation of the contact force and enhance the sensitivity of the sensor. A series of ridge texture samples with different heights (0.9, 4, 10 μm), different widths (0.3, 0.5, 0.7, 1 mm), but the same spatial period (2 mm) of ridges were used in the experiment. The experimental results prove that the stochastic resonance algorithm can significantly improve the signal characteristic of the output signal of the sensor. The sensor has the capability to detect fine ridge texture information. The mean relative error of the estimation for the spatial period was 1.085%, and the ridge width and ridge height, respectively, have a monotonic mapping relationship with the corresponding model output parameters. The sensing capability to sense a fine texture of tactile senor surpasses the limit of human fingers.

A Biomimetic Circuit for Electronic Skin With Application in Hand Prosthesis

One major challenge in upper limb prostheses is providing sensory feedback to amputees. Reproducing the spiking patterns of human primary tactile afferents can be considered as the first step for this challenging problem. In this study, a novel biomimetic circuit for SA-I and RA-I afferents is proposed to functionally replicate the spiking response of the biological tactile afferents to indentation stimuli. The circuit has been designed, laid out, and simulated in TSMC 180nm CMOS technology with a 1.8V supply voltage. A pair of SA-I and RA-I afferent circuits consume [Formula: see text] of power. The occupied silicon area is [Formula: see text] for 32 afferents. To provide the inputs for circuit testing, a patch of skin with a grid of mechanoreceptors is simulated and tested by an edge stimulus presented at different orientations. Experimental data are collected using indentation of 3D-printed edges at different orientations on a tactile sensor mounted on a robotic arm. Inspired by innervation patterns observed in biology, the artificial afferents are connected to several neighboring mechanoreceptors with different weights to form complex receptive fields which cover the entire mechanoreceptor grid. Machine learning algorithms are applied offline to classify the edge orientations based on the pattern of neural responses. Our results show that the complex receptive fields arising from the innervation pattern led to smaller circuit area and lower power consumption, while facilitating data encoding from high-resolution sensors. The proposed biomimetic circuit and tactile encoding example demonstrate potential applications in modern tactile sensing modules for developing novel bio-robotic and prosthetic technologies.

Tactile Avatar: Tactile Sensing System Mimicking Human Tactile Cognition

As a surrogate for human tactile cognition, an artificial tactile perception and cognition system are proposed to produce smooth/soft and rough tactile sensations by its user's tactile feeling; and named this system as "tactile avatar". A piezoelectric tactile sensor is developed to record dynamically various physical information such as pressure, temperature, hardness, sliding velocity, and surface topography. For artificial tactile cognition, the tactile feeling of humans to various tactile materials ranging from smooth/soft to rough are assessed and found variation among participants. Because tactile responses vary among humans, a deep learning structure is designed to allow personalization through training based on individualized histograms of human tactile cognition and recording physical tactile information. The decision error in each avatar system is less than 2% when 42 materials are used to measure the tactile data with 100 trials for each material under 1.2N of contact force with 4cm s-1 of sliding velocity. As a tactile avatar, the machine categorizes newly experienced materials based on the tactile knowledge obtained from training data. The tactile sensation showed a high correlation with the specific user's tendency. This approach can be applied to electronic devices with tactile emotional exchange capabilities, as well as advanced digital experiences.

Sharpness recognition based on synergy between bio-inspired nociceptors and tactile mechanoreceptors

Touch and pain sensations are complementary aspects of daily life that convey crucial information about the environment while also providing protection to our body. Technological advancements in prosthesis design and control mechanisms assist amputees to regain lost function but often they have no meaningful tactile feedback or perception. In the present study, we propose a bio-inspired tactile system with a population of 23 digital afferents: 12 RA-I, 6 SA-I, and 5 nociceptors. Indeed, the functional concept of the nociceptor is implemented on the FPGA for the first time. One of the main features of biological tactile afferents is that their distal axon branches in the skin, creating complex receptive fields. Given these physiological observations, the bio-inspired afferents are randomly connected to the several neighboring mechanoreceptors with different weights to form their own receptive field. To test the performance of the proposed neuromorphic chip in sharpness detection, a robotic system with three-degree of freedom equipped with the tactile sensor indents the 3D-printed objects. Spike responses of the biomimetic afferents are then collected for analysis by rate and temporal coding algorithms. In this way, the impact of the innervation mechanism and collaboration of afferents and nociceptors on sharpness recognition are investigated. Our findings suggest that the synergy between sensory afferents and nociceptors conveys more information about tactile stimuli which in turn leads to the robustness of the proposed neuromorphic system against damage to the taxels or afferents. Moreover, it is illustrated that spiking activity of the biomimetic nociceptors is amplified as the sharpness increases which can be considered as a feedback mechanism for prosthesis protection. This neuromorphic approach advances the development of prosthesis to include the sensory feedback and to distinguish innocuous (non-painful) and noxious (painful) stimuli.

Run-off election-based decision method for the training and inference process in an artificial neural network

Generally, the decision rule for classifying unstructured data in an artificial neural network system depends on the sequence results of an activation function determined by vector-matrix multiplication between the input bias signal and the analog synaptic weight quantity of each node in a matrix array. Although a sequence-based decision rule can efficiently extract a common feature in a large data set in a short time, it can occasionally fail to classify similar species because it does not intrinsically consider other quantitative configurations of the activation function that affect the synaptic weight update. In this work, we implemented a simple run-off election-based decision rule via an additional filter evaluation to mitigate the confusion from proximity of output activation functions, enabling the improved training and inference performance of artificial neural network system. Using the filter evaluation selected via the difference among common features of classified images, the recognition accuracy achieved for three types of shoe image data sets reached ~ 82.03%, outperforming the maximum accuracy of ~ 79.23% obtained via the sequence-based decision rule in a fully connected single layer network. This training algorithm with an independent filter can precisely supply the output class in the decision step of the fully connected network.

Weakly Supervised Complets Ranking for Deep Image Quality Modeling

Despite the competitive prediction performance, recent deep image quality models suffer from the following limitations. First, it is deficiently effective to interpret and quantify the region-level quality, which contributes to global features during deep architecture training. Second, human visual perception is sensitive to compositional features (i.e., the sophisticated spatial configurations among regions), but explicitly incorporating them into a deep model is challenging. Third, the state-of-the-art deep quality models typically use rectangular image patches as inputs, but there is no evidence that these rectangles can reflect arbitrarily shaped objects, such as beaches and jungles. By defining the complet, which is a set of image segments collaboratively characterizing the spatial/geometric distribution of multiple visual elements, we propose a novel quality-modeling framework that involves two key modules: a complet ranking algorithm and a spatially-aware dual aggregation network (SDA-Net). Specifically, to describe the region-level quality features, we build complets to characterize the high-order spatial interactions among the arbitrarily shaped segments in each image. To obtain complets that are highly descriptive to image compositions, a weakly supervised complet ranking algorithm is designed by quantifying the quality of each complet. The algorithm seamlessly encodes three factors: the image-level quality discrimination, weakly supervised constraint, and complet geometry of each image. Based on the top-ranking complets, a novel multi-column convolutional neural network (CNN) called SDA-Net is designed, which supports input segments with arbitrary shapes. The key is a dual-aggregation mechanism that fuses the intracomplet deep features and the intercomplet deep features under a unified framework. Thorough experimental validations on a series of benchmark data sets demonstrated the superiority of our method.

Texture Discrimination with a Soft Biomimetic Finger Using a Flexible Neuromorphic Tactile Sensor Array That Provides Sensory Feedback

The compliant nature of soft fingers allows for safe and dexterous manipulation of objects by humans in an unstructured environment. A soft prosthetic finger design with tactile sensing capabilities for texture discrimination and subsequent sensory stimulation has the potential to create a more natural experience for an amputee. In this work, a pneumatically actuated soft biomimetic finger is integrated with a textile neuromorphic tactile sensor array for a texture discrimination task. The tactile sensor outputs were converted into neuromorphic spike trains, which emulate the firing pattern of biological mechanoreceptors. Spike-based features from each taxel compressed the information and were then used as inputs for the support vector machine classifier to differentiate the textures. Our soft biomimetic finger with neuromorphic encoding was able to achieve an average overall classification accuracy of 99.57% over 16 independent parameters when tested on 13 standardized textured surfaces. The 16 parameters were the combination of 4 angles of flexion of the soft finger and 4 speeds of palpation. To aid in the perception of more natural objects and their manipulation, subjects were provided with transcutaneous electrical nerve stimulation to convey a subset of four textures with varied textural information. Three able-bodied subjects successfully distinguished two or three textures with the applied stimuli. This work paves the way for a more human-like prosthesis through a soft biomimetic finger with texture discrimination capabilities using neuromorphic techniques that provide sensory feedback; furthermore, texture feedback has the potential to enhance user experience when interacting with their surroundings.

Embodied tactile perception and learning

Various living creatures exhibit embodiment intelligence, which is reflected by a collaborative interaction of the brain, body, and environment. The actual behavior of embodiment intelligence is generated by a continuous and dynamic interaction between a subject and the environment through information perception and physical manipulation. The physical interaction between a robot and the environment is the basis for realizing embodied perception and learning. Tactile information plays a critical role in this physical interaction process. It can be used to ensure safety, stability, and compliance, and can provide unique information that is difficult to capture using other perception modalities. However, due to the limitations of existing sensors and perception and learning methods, the development of robotic tactile research lags significantly behind other sensing modalities, such as vision and hearing, thereby seriously restricting the development of robotic embodiment intelligence. This paper presents the current challenges related to robotic tactile embodiment intelligence and reviews the theory and methods of robotic embodied tactile intelligence. Tactile perception and learning methods for embodiment intelligence can be designed based on the development of new large‐scale tactile array sensing devices, with the aim to make breakthroughs in the neuromorphic computing technology of tactile intelligence.

Cuneate spiking neural network learning to classify naturalistic texture stimuli under varying sensing conditions

We implemented a functional neuronal network that was able to learn and discriminate haptic features from biomimetic tactile sensor inputs using a two-layer spiking neuron model and homeostatic synaptic learning mechanism. The first order neuron model was used to emulate biological tactile afferents and the second order neuron model was used to emulate biological cuneate neurons. We have evaluated 10 naturalistic textures using a passive touch protocol, under varying sensing conditions. Tactile sensor data acquired with five textures under five sensing conditions were used for a synaptic learning process, to tune the synaptic weights between tactile afferents and cuneate neurons. Using post-learning synaptic weights, we evaluated the individual and population cuneate neuron responses by decoding across 10 stimuli, under varying sensing conditions. This resulted in a high decoding performance. We further validated the decoding performance across stimuli, irrespective of sensing velocities using a set of 25 cuneate neuron responses. This resulted in a median decoding performance of 96% across the set of cuneate neurons. Being able to learn and perform generalized discrimination across tactile stimuli, makes this functional spiking tactile system effective and suitable for further robotic applications.

Morphological Neural Computation Restores Discrimination of Naturalistic Textures in Trans-radial Amputees

Humans rely on their sense of touch to interact with the environment. Thus, restoring lost tactile sensory capabilities in amputees would advance their quality of life. In particular, texture discrimination is an important component for the interaction with the environment, but its restoration in amputees has been so far limited to simplified gratings. Here we show that naturalistic textures can be discriminated by trans-radial amputees using intraneural peripheral stimulation and tactile sensors located close to the outer layer of the artificial skin. These sensors exploit the morphological neural computation (MNC) approach, i.e., the embodiment of neural computational functions into the physical structure of the device, encoding normal and shear stress to guarantee a faithful neural temporal representation of stimulus spatial structure. Two trans-radial amputees successfully discriminated naturalistic textures via the MNC-based tactile feedback. The results also allowed to shed light on the relevance of spike temporal encoding in the mechanisms used to discriminate naturalistic textures. Our findings pave the way to the development of more natural bionic limbs.

A Digital Hardware System for Spiking Network of Tactile Afferents

In the present research, we explore the possibility of utilizing a hardware-based neuromorphic approach to develop a tactile sensory system at the level of first-order afferents, which are slowly adapting type 1 (SA-I) and fast adapting type 1 (FA-I) afferents. Four spiking models are used to mimic neural signals of both SA-I and FA-I primary afferents. Next, a digital circuit is designed for each spiking model for both afferents to be implemented on the field-programmable gate array (FPGA). The four different digital circuits are then compared from source utilization point of view to find the minimum cost circuit for creating a population of digital afferents. In this way, the firing responses of both SA-I and FA-I afferents are physically measured in hardware. Finally, a population of 243 afferents consisting of 90 SA-I and 153 FA-I digital neuromorphic circuits are implemented on the FPGA. The FPGA also receives nine inputs from the force sensors through an interfacing board. Therefore, the data of multiple inputs are processed by the spiking network of tactile afferents, simultaneously. Benefiting from parallel processing capabilities of FPGA, the proposed architecture offers a low-cost neuromorphic structure for tactile information processing. Applying machine learning algorithms on the artificial spiking patterns collected from FPGA, we successfully classified three different objects based on the firing rate paradigm. Consequently, the proposed neuromorphic system provides the opportunity for development of new tactile processing component for robotic and prosthetic applications.

Texture Discrimination using a Soft Biomimetic Finger for Prosthetic Applications

Soft robotic fingers have shown great potential for use in prostheses due to their inherent compliant, light, and dexterous nature. Recent advancements in sensor technology for soft robotic systems showcase their ability to perceive and respond to static cues. However, most of the soft fingers for use in prosthetic applications are not equipped with sensors which have the ability to perceive texture like humans can. In this work, we present a dexterous, soft, biomimetic solution which is capable of discrimination of textures. We fabricated a soft finger with two individually controllable degrees of freedom with a tactile sensor embedded at the fingertip. The output of the tac- tile sensor, as texture plates were palpated, was converted into spikes, mimicking the behavior of a biological mechanoreceptor. We explored the spatial properties of the textures captured in the form of spiking patterns by generating spatial event plots and analyzing the similarity between spike trains generated for each texture. Unique features representative of the different textures were then extracted from the spikes and input to a classifier. The textures were successfully classified with an accuracy of 94% when palpating at a rate of 42 mm/s. This work demonstrates the potential of providing amputees with a soft finger with sensing capabilities, which could potentially help discriminate between different objects and surfaces during activities of daily living (ADL) through palpation.

A neuro-inspired artificial peripheral nervous system for scalable electronic skins

The human sense of touch is essential for dexterous tool usage, spatial awareness, and social communication. Equipping intelligent human-like androids and prosthetics with electronic skins-a large array of sensors spatially distributed and capable of rapid somatosensory perception-will enable them to work collaboratively and naturally with humans to manipulate objects in unstructured living environments. Previously reported tactile-sensitive electronic skins largely transmit the tactile information from sensors serially, resulting in readout latency bottlenecks and complex wiring as the number of sensors increases. Here, we introduce the Asynchronously Coded Electronic Skin (ACES)-a neuromimetic architecture that enables simultaneous transmission of thermotactile information while maintaining exceptionally low readout latencies, even with array sizes beyond 10,000 sensors. We demonstrate prototype arrays of up to 240 artificial mechanoreceptors that transmitted events asynchronously at a constant latency of 1 ms while maintaining an ultra-high temporal precision of <60 ns, thus resolving fine spatiotemporal features necessary for rapid tactile perception. Our platform requires only a single electrical conductor for signal propagation, realizing sensor arrays that are dynamically reconfigurable and robust to damage. We anticipate that the ACES platform can be integrated with a wide range of skin-like sensors for artificial intelligence (AI)-enhanced autonomous robots, neuroprosthetics, and neuromorphic computing hardware for dexterous object manipulation and somatosensory perception.

Tactile Decoding of Edge Orientation With Artificial Cuneate Neurons in Dynamic Conditions

Generalization ability in tactile sensing for robotic manipulation is a prerequisite to effectively perform tasks in ever-changing environments. In particular, performing dynamic tactile perception is currently beyond the ability of robotic devices. A biomimetic approach to achieve this dexterity is to develop machines combining compliant robotic manipulators with neuroinspired architectures displaying computational adaptation. Here we demonstrate the feasibility of this approach for dynamic touch tasks experimented by integrating our sensing apparatus in a 6 degrees of freedom robotic arm via a soft wrist. We embodied in the system a model of spike-based neuromorphic encoding of tactile stimuli, emulating the discrimination properties of cuneate nucleus neurons based on pathways with differential delay lines. These strategies allowed the system to correctly perform a dynamic touch protocol of edge orientation recognition (ridges from 0 to 40°, with a step of 5°). Crucially, the task was robust to contact noise and was performed with high performance irrespectively of sensing conditions (sensing forces and velocities). These results are a step forward toward the development of robotic arms able to physically interact in real-world environments with tactile sensing.

Integrate-and-fire network model of activity propagation from thalamus to cortex

The thalamus plays a crucial role in modulating the cortical activity underlying sensory and cognitive processes. In particular, recent experimental findings highlighted that the thalamus does not merely act as a binary gate for sensory stimuli, but rather participates to the processing of sensory information. Clarifying such thalamic influence on cortical dynamics is also important as the thalamus is the target of therapies such as DBS for Tourette patients. In this perspective, various computational models have been proposed in the last decades. However, a detailed description of the propagation of thalamic activity to the cortex is missing. Here we present a simple computational model of thalamocortical connectivity accounting for the propagation of activity from the thalamus to the cortex. The model includes both the single-neuron scale and the mesoscopic level of Local Field Potential (LFP) signals. Numerical simulations at both levels reproduce typical thalamocortical dynamics which are consistent with experimental measurements and robust to parameters changes. In particular, our model correctly reproduces locally generated rhythms as spindle oscillations in the thalamus and gamma oscillations in the cortex. Our model paves the way to deeper investigations of the thalamic influence on cortical dynamics, with and without sensory inputs or therapeutic electrical stimulation.

A Python Code for Simulating Single Tactile Receptors and the Spiking Responses of Their Afferents

This work presents a pieces of Python code to rapidly simulate the spiking responses of large numbers of single cutaneous tactile afferents with millisecond precision. To simulate the spike responses of all the major types of cutaneous tactile afferents, we proposed an electromechanical circuit model, in which a two-channel filter was developed to characterize the mechanical selectivity of tactile receptors, and a spike synthesizer was designed to recreate the action potentials evoked in afferents. The parameters of this model were fitted using previous neurophysiological datasets. Several simulation examples were presented in this paper to reproduce action potentials, sensory adaptation, frequency characteristics and spiking timing for each afferent type. The results indicated that the simulated responses matched previous neurophysiological recordings well. The model allows for a real-time reproduction of the spiking responses of about 4,000 tactile units with a timing precision of <6 ms. The current work provides a valuable guidance to designing highly realistic tactile interfaces such as neuroprosthesis and haptic devices.

Haptic Glove and Platform with Gestural Control For Neuromorphic Tactile Sensory Feedback In Medical Telepresence †

Advancements in the study of the human sense of touch are fueling the field of haptics. This is paving the way for augmenting sensory perception during object palpation in tele-surgery and reproducing the sensed information through tactile feedback. Here, we present a novel tele-palpation apparatus that enables the user to detect nodules with various distinct stiffness buried in an ad-hoc polymeric phantom. The contact force measured by the platform was encoded using a neuromorphic model and reproduced on the index fingertip of a remote user through a haptic glove embedding a piezoelectric disk. We assessed the effectiveness of this feedback in allowing nodule identification under two experimental conditions of real-time telepresence: In Line of Sight (ILS), where the platform was placed in the visible range of a user; and the more demanding Not In Line of Sight (NILS), with the platform and the user being 50 km apart. We found that the entailed percentage of identification was higher for stiffer inclusions with respect to the softer ones (average of 74% within the duration of the task), in both telepresence conditions evaluated. These promising results call for further exploration of tactile augmentation technology for telepresence in medical interventions.

Intracellular Dynamics in Cuneate Nucleus Neurons Support Self-Stabilizing Learning of Generalizable Tactile Representations

How the brain represents the external world is an unresolved issue for neuroscience, which could provide fundamental insights into brain circuitry operation and solutions for artificial intelligence and robotics. The neurons of the cuneate nucleus form the first interface for the sense of touch in the brain. They were previously shown to have a highly skewed synaptic weight distribution for tactile primary afferent inputs, suggesting that their connectivity is strongly shaped by learning. Here we first characterized the intracellular dynamics and inhibitory synaptic inputs of cuneate neurons in vivo and modeled their integration of tactile sensory inputs. We then replaced the tactile inputs with input from a sensorized bionic fingertip and modeled the learning-induced representations that emerged from varied sensory experiences. The model reproduced both the intrinsic membrane dynamics and the synaptic weight distributions observed in cuneate neurons in vivo. In terms of higher level model properties, individual cuneate neurons learnt to identify specific sets of correlated sensors, which at the population level resulted in a decomposition of the sensor space into its recurring high-dimensional components. Such vector components could be applied to identify both past and novel sensory experiences and likely correspond to the fundamental haptic input features these neurons encode in vivo. In addition, we show that the cuneate learning architecture is robust to a wide range of intrinsic parameter settings due to the neuronal intrinsic dynamics. Therefore, the architecture is a potentially generic solution for forming versatile representations of the external world in different sensor systems.

Correlation of neural activity with behavioral kinematics reveals distinct sensory encoding and evidence accumulation processes during active tactile sensing

Many real-world decisions rely on active sensing, a dynamic process for directing our sensors (e.g. eyes or fingers) across a stimulus to maximize information gain. Though ecologically pervasive, limited work has focused on identifying neural correlates of the active sensing process. In tactile perception, we often make decisions about an object/surface by actively exploring its shape/texture. Here we investigate the neural correlates of active tactile decision-making by simultaneously measuring electroencephalography (EEG) and finger kinematics while subjects interrogated a haptic surface to make perceptual judgments. Since sensorimotor behavior underlies decision formation in active sensing tasks, we hypothesized that the neural correlates of decision-related processes would be detectable by relating active sensing to neural activity. Novel brain-behavior correlation analysis revealed that three distinct EEG components, localizing to right-lateralized occipital cortex (LOC), middle frontal gyrus (MFG), and supplementary motor area (SMA), respectively, were coupled with active sensing as their activity significantly correlated with finger kinematics. To probe the functional role of these components, we fit their single-trial-couplings to decision-making performance using a hierarchical-drift-diffusion-model (HDDM), revealing that the LOC modulated the encoding of the tactile stimulus whereas the MFG predicted the rate of information integration towards a choice. Interestingly, the MFG disappeared from components uncovered from control subjects performing active sensing but not required to make perceptual decisions. By uncovering the neural correlates of distinct stimulus encoding and evidence accumulation processes, this study delineated, for the first time, the functional role of cortical areas in active tactile decision-making.

A Digital Hardware Realization for Spiking Model of Cutaneous Mechanoreceptor

Inspired by the biology of human tactile perception, a hardware neuromorphic approach is proposed for spiking model of mechanoreceptors to encode the input force. In this way, a digital circuit is designed for a slowly adapting type I (SA-I) and fast adapting type I (FA-I) mechanoreceptors to be implemented on a low-cost digital hardware, such as field-programmable gate array (FPGA). This system computationally replicates the neural firing responses of both afferents. Then, comparative simulations are shown. The spiking models of mechanoreceptors are first simulated in MATLAB and next the digital neuromorphic circuits simulated in VIVADO are also compared to show that obtained results are in good agreement both quantitatively and qualitatively. Finally, we test the performance of the proposed digital mechanoreceptors in hardware using a prepared experimental set up. Hardware synthesis and physical realization on FPGA indicate that the digital mechanoreceptors are able to replicate essential characteristics of different firing patterns including bursting and spiking responses of the SA-I and FA-I mechanoreceptors. In addition to parallel computation, a main advantage of this method is that the mechanoreceptor digital circuits can be implemented in real-time through low-power neuromorphic hardware. This novel engineering framework is generally suitable for use in robotic and hand-prosthetic applications, so progressing the state of the art for tactile sensing.

An Extreme Learning Machine-Based Neuromorphic Tactile Sensing System for Texture Recognition

Despite significant advances in computational algorithms and development of tactile sensors, artificial tactile sensing is strikingly less efficient and capable than the human tactile perception. Inspired by efficiency of biological systems, we aim to develop a neuromorphic system for tactile pattern recognition. We particularly target texture recognition as it is one of the most necessary and challenging tasks for artificial sensory systems. Our system consists of a piezoresistive fabric material as the sensor to emulate skin, an interface that produces spike patterns to mimic neural signals from mechanoreceptors, and an extreme learning machine (ELM) chip to analyze spiking activity. Benefiting from intrinsic advantages of biologically inspired event-driven systems and massively parallel and energy-efficient processing capabilities of the ELM chip, the proposed architecture offers a fast and energy-efficient alternative for processing tactile information. Moreover, it provides the opportunity for the development of low-cost tactile modules for large-area applications by integration of sensors and processing circuits. We demonstrate the recognition capability of our system in a texture discrimination task, where it achieves a classification accuracy of 92% for categorization of ten graded textures. Our results confirm that there exists a tradeoff between response time and classification accuracy (and information transfer rate). A faster decision can be achieved at early time steps or by using a shorter time window. This, however, results in deterioration of the classification accuracy and information transfer rate. We further observe that there exists a tradeoff between the classification accuracy and the input spike rate (and thus energy consumption). Our work substantiates the importance of development of efficient sparse codes for encoding sensory data to improve the energy efficiency. These results have a significance for a wide range of wearable, robotic, prosthetic, and industrial applications.

Neuromorphic Vibrotactile Stimulation of Fingertips for Encoding Object Stiffness in Telepresence Sensory Substitution and Augmentation Applications

We present a tactile telepresence system for real-time transmission of information about object stiffness to the human fingertips. Experimental tests were performed across two laboratories (Italy and Ireland). In the Italian laboratory, a mechatronic sensing platform indented different rubber samples. Information about rubber stiffness was converted into on-off events using a neuronal spiking model and sent to a vibrotactile glove in the Irish laboratory. Participants discriminated the variation of the stiffness of stimuli according to a two-alternative forced choice protocol. Stiffness discrimination was based on the variation of the temporal pattern of spikes generated during the indentation of the rubber samples. The results suggest that vibrotactile stimulation can effectively simulate surface stiffness when using neuronal spiking models to trigger vibrations in the haptic interface. Specifically, fractional variations of stiffness down to 0.67 were significantly discriminated with the developed neuromorphic haptic interface. This is a performance comparable, though slightly worse, to the threshold obtained in a benchmark experiment evaluating the same set of stimuli naturally with the own hand. Our paper presents a bioinspired method for delivering sensory feedback about object properties to human skin based on contingency-mimetic neuronal models, and can be useful for the design of high performance haptic devices.

Discrimination of Dynamic Tactile Contact by Temporally Precise Event Sensing in Spiking Neuromorphic Networks

This paper presents a neuromorphic tactile encoding methodology that utilizes a temporally precise event-based representation of sensory signals. We introduce a novel concept where touch signals are characterized as patterns of millisecond precise binary events to denote pressure changes. This approach is amenable to a sparse signal representation and enables the extraction of relevant features from thousands of sensing elements with sub-millisecond temporal precision. We also proposed measures adopted from computational neuroscience to study the information content within the spiking representations of artificial tactile signals. Implemented on a state-of-the-art 4096 element tactile sensor array with 5.2 kHz sampling frequency, we demonstrate the classification of transient impact events while utilizing 20 times less communication bandwidth compared to frame based representations. Spiking sensor responses to a large library of contact conditions were also synthesized using finite element simulations, illustrating an 8-fold improvement in information content and a 4-fold reduction in classification latency when millisecond-precise temporal structures are available. Our research represents a significant advance, demonstrating that a neuromorphic spatiotemporal representation of touch is well suited to rapid identification of critical contact events, making it suitable for dynamic tactile sensing in robotic and prosthetic applications.

Frontiers of robotic endoscopic capsules: a review

Digestive diseases are a major burden for society and healthcare systems, and with an aging population, the importance of their effective management will become critical. Healthcare systems worldwide already struggle to insure quality and affordability of healthcare delivery and this will be a significant challenge in the midterm future. Wireless capsule endoscopy (WCE), introduced in 2000 by Given Imaging Ltd., is an example of disruptive technology and represents an attractive alternative to traditional diagnostic techniques. WCE overcomes conventional endoscopy enabling inspection of the digestive system without discomfort or the need for sedation. Thus, it has the advantage of encouraging patients to undergo gastrointestinal (GI) tract examinations and of facilitating mass screening programmes. With the integration of further capabilities based on microrobotics, e.g. active locomotion and embedded therapeutic modules, WCE could become the key-technology for GI diagnosis and treatment. This review presents a research update on WCE and describes the state-of-the-art of current endoscopic devices with a focus on research-oriented robotic capsule endoscopes enabled by microsystem technologies. The article also presents a visionary perspective on WCE potential for screening, diagnostic and therapeutic endoscopic procedures.

Intraneural stimulation elicits discrimination of textural features by artificial fingertip in intact and amputee humans

Restoration of touch after hand amputation is a desirable feature of ideal prostheses. Here, we show that texture discrimination can be artificially provided in human subjects by implementing a neuromorphic real-time mechano-neuro-transduction (MNT), which emulates to some extent the firing dynamics of SA1 cutaneous afferents. The MNT process was used to modulate the temporal pattern of electrical spikes delivered to the human median nerve via percutaneous microstimulation in four intact subjects and via implanted intrafascicular stimulation in one transradial amputee. Both approaches allowed the subjects to reliably discriminate spatial coarseness of surfaces as confirmed also by a hybrid neural model of the median nerve. Moreover, MNT-evoked EEG activity showed physiologically plausible responses that were superimposable in time and topography to the ones elicited by a natural mechanical tactile stimulation. These findings can open up novel opportunities for sensory restoration in the next generation of neuro-prosthetic hands.

DOI:http://dx.doi.org/10.7554/eLife.09148.001

Our hands provide us with a wide variety of information about our surroundings, enabling us to detect pain, temperature and pressure. Our sense of touch also allows us to interact with objects by feeling their texture and solidity. However, completely reproducing a sense of touch in artificial or prosthetic hands has proven challenging. While commercial prostheses can mimic the range of movements of natural limbs, even the latest experimental prostheses have only a limited ability to ‘feel’ the objects being manipulated. Oddo, Raspopovic et al. have now brought this ability a step closer by exploiting an artificial fingertip and appropriate neural interfaces through which different textures can be identified.

The initial experiments were performed in four healthy volunteers with intact limbs. Oddo, Raspopovic et al. connected the artificial fingertip to the volunteers via an electrode inserted into a nerve in the arm. When moved over a rough surface, sensors in the fingertip produced patterns of electrical pulses that stimulated the nerve, causing the volunteers to feel like they were touching the surface. The volunteers were even able to tell the difference between the different surface textures the artificial fingertip moved across.

The temporary electrodes used in this group of volunteers are unsuitable for use with prosthetic limbs because they can easily be knocked out of position. Therefore, in a further experiment involving a volunteer who had undergone an arm amputation a number of years previously, Oddo, Raspopovic et al. tested an implanted electrode array that could, in principle, remain in place long-term. This volunteer could also identify the different textures the artificial fingertip touched, with a slightly higher degree of accuracy than the previous group of intact volunteers. Further studies are now required to explore the potential of this approach in larger groups of volunteers.

DOI:http://dx.doi.org/10.7554/eLife.09148.002