Discrimination of Dynamic Tactile Contact by Temporally Precise Event Sensing in Spiking Neuromorphic Networks

Wavelet Transforms Significantly Sparsify and Compress Tactile Interactions

As higher spatiotemporal resolution tactile sensing systems are being developed for prosthetics, wearables, and other biomedical applications, they demand faster sampling rates and generate larger data streams. Sparsifying transformations can alleviate these requirements by enabling compressive sampling and efficient data storage through compression. However, research on the best sparsifying transforms for tactile interactions is lagging. In this work we construct a library of orthogonal and biorthogonal wavelet transforms as sparsifying transforms for tactile interactions and compare their tradeoffs in compression and sparsity. We tested the sparsifying transforms on a publicly available high-density tactile object grasping dataset (548 sensor tactile glove, grasping 26 objects). In addition, we investigated which dimension wavelet transform-1D, 2D, or 3D-would best compress these tactile interactions. Our results show that wavelet transforms are highly efficient at compressing tactile data and can lead to very sparse and compact tactile representations. Additionally, our results show that 1D transforms achieve the sparsest representations, followed by 3D, and lastly 2D. Overall, the best wavelet for coarse approximation is Symlets 4 evaluated temporally which can sparsify to 0.5% sparsity and compress 10-bit tactile data to an average of 0.04 bits per pixel. Future studies can leverage the results of this paper to assist in the compressive sampling of large tactile arrays and free up computational resources for real-time processing on computationally constrained mobile platforms like neuroprosthetics.

MEMS-Based Tactile Sensors: Materials, Processes and Applications in Robotics

Commonly encountered problems in the manipulation of objects with robotic hands are the contact force control and the setting of approaching motion. Microelectromechanical systems (MEMS) sensors on robots offer several solutions to these problems along with new capabilities. In this review, we analyze tactile, force and/or pressure sensors produced by MEMS technologies including off-the-shelf products such as MEMS barometric sensors. Alone or in conjunction with other sensors, MEMS platforms are considered very promising for robots to detect the contact forces, slippage and the distance to the objects for effective dexterous manipulation. We briefly reviewed several sensing mechanisms and principles, such as capacitive, resistive, piezoresistive and triboelectric, combined with new flexible materials technologies including polymers processing and MEMS-embedded textiles for flexible and snake robots. We demonstrated that without taking up extra space and at the same time remaining lightweight, several MEMS sensors can be integrated into robotic hands to simulate human fingers, gripping, hardness and stiffness sensations. MEMS have high potential of enabling new generation microactuators, microsensors, micro miniature motion-systems (e.g., microrobots) that will be indispensable for health, security, safety and environmental protection.

Neuromorphic Tactile Edge Orientation Classification in an Unsupervised Spiking Neural Network

Dexterous manipulation in robotic hands relies on an accurate sense of artificial touch. Here we investigate neuromorphic tactile sensation with an event-based optical tactile sensor combined with spiking neural networks for edge orientation detection. The sensor incorporates an event-based vision system (mini-eDVS) into a low-form factor artificial fingertip (the NeuroTac). The processing of tactile information is performed through a Spiking Neural Network with unsupervised Spike-Timing-Dependent Plasticity (STDP) learning, and the resultant output is classified with a 3-nearest neighbours classifier. Edge orientations were classified in 10-degree increments while tapping vertically downward and sliding horizontally across the edge. In both cases, we demonstrate that the sensor is able to reliably detect edge orientation, and could lead to accurate, bio-inspired, tactile processing in robotics and prosthetics applications.

Artificial SA-I and RA-I afferents for tactile sensing of ridges and gratings

For robot touch to reach the capabilities of human touch, artificial tactile sensors may require transduction principles like those of natural tactile afferents. Here we propose that a biomimetic tactile sensor (the TacTip) could provide suitable artificial analogues of the tactile skin dynamics, afferent responses and population encoding. Our three-dimensionally printed sensor skin is based on the physiology of the dermal-epidermal interface with an underlying mesh of biomimetic intermediate ridges and dermal papillae, comprising inner pins tipped with markers. Slowly adapting SA-I activity is modelled by marker displacements and rapidly adapting RA-I activity by marker speeds. We test the biological plausibility of these artificial population codes with three classic experiments used for natural touch: (1a) responses to normal pressure to test adaptation of single afferents and spatial modulation across the population; (1b) responses to bars, edges and gratings to compare with measurements from monkey primary afferents; and (2) discrimination of grating orientation to compare with human perceptual performance. Our results show a match between artificial and natural touch at single afferent, population and perceptual levels. As expected, natural skin is more sensitive, which raises a challenge to fabricate a biomimetic fingertip that demonstrates human sensitivity using the transduction principles of human touch.

Artificial SA-I, RA-I and RA-II/vibrotactile afferents for tactile sensing of texture

Robot touch can benefit from how humans perceive tactile textural information, from the stimulation mode to which tactile channels respond, then the tactile cues and encoding. Using a soft biomimetic tactile sensor (the TacTip) based on the physiology of the dermal-epidermal boundary, we construct two biomimetic tactile channels based on slowly adapting SA-I and rapidly adapting RA-I afferents, and introduce an additional sub-modality for vibrotactile information with an embedded microphone interpreted as an artificial RA-II channel. These artificial tactile channels are stimulated dynamically with a set of 13 artificial rigid textures comprising raised-bump patterns on a rotating drum that vary systematically in roughness. Methods employing spatial, spatio-temporal and temporal codes are assessed for texture classification insensitive to stimulation speed. We find: (i) spatially encoded frictional cues provide a salient representation of texture; (ii) a simple transformation of spatial tactile features to model natural afferent responses improves the temporal coding; and (iii) the harmonic structure of induced vibrations provides a pertinent code for speed-invariant texture classification. Just as human touch relies on an interplay between slowly adapting (SA-I), rapidly adapting (RA-I) and vibrotactile (RA-II) channels, this tripartite structure may be needed for future robot applications with human-like dexterity, from prosthetics to materials testing, handling and manipulation.

Event-Based Computation for Touch Localization Based on Precise Spike Timing

Precise spike timing and temporal coding are used extensively within the nervous system of insects and in the sensory periphery of higher order animals. However, conventional Artificial Neural Networks (ANNs) and machine learning algorithms cannot take advantage of this coding strategy, due to their rate-based representation of signals. Even in the case of artificial Spiking Neural Networks (SNNs), identifying applications where temporal coding outperforms the rate coding strategies of ANNs is still an open challenge. Neuromorphic sensory-processing systems provide an ideal context for exploring the potential advantages of temporal coding, as they are able to efficiently extract the information required to cluster or classify spatio-temporal activity patterns from relative spike timing. Here we propose a neuromorphic model inspired by the sand scorpion to explore the benefits of temporal coding, and validate it in an event-based sensory-processing task. The task consists in localizing a target using only the relative spike timing of eight spatially-separated vibration sensors. We propose two different approaches in which the SNNs learns to cluster spatio-temporal patterns in an unsupervised manner and we demonstrate how the task can be solved both analytically and through numerical simulation of multiple SNN models. We argue that the models presented are optimal for spatio-temporal pattern classification using precise spike timing in a task that could be used as a standard benchmark for evaluating event-based sensory processing models based on temporal coding.

Neuromorphic Processing of Pressure Signal Using Integrated Sensor-Synaptic Device Capable of Selective and Reversible Short- and Long-Term Plasticity Operation

To mimic the tactile sensing properties of the human skin, signals from tactile sensors need to be processed in an efficient manner. The integration of the tactile sensor with a neuromorphic device can potentially address this issue, as the neuromorphic device has both signal processing and memory capability through which parallel and efficient processing of information is possible. In this article, an intelligent haptic perception device (IHPD) is presented that combines pressure sensing with an organic electrochemical transistor-based synaptic device into a simple device architecture. More importantly, the IHPD is capable of rapid and reversible switching between short-term plasticity (STP) and long-term plasticity (LTP) operation through which accelerated learning, processing of new information, and distinctive operation of STP and LTP are possible. Various types of pressure information such as magnitude, rate, and duration were processed utilizing STP by which error-tolerant perception was demonstrated. Meanwhile, memorization and learning of pressure through a stepwise change in a conductive state was demonstrated using LTP. These demonstrations present unique approaches to process and learn tactile information, which can potentially be utilized in various electronic skin applications in the future.

Cuneate spiking neural network learning to classify naturalistic texture stimuli under varying sensing conditions

We implemented a functional neuronal network that was able to learn and discriminate haptic features from biomimetic tactile sensor inputs using a two-layer spiking neuron model and homeostatic synaptic learning mechanism. The first order neuron model was used to emulate biological tactile afferents and the second order neuron model was used to emulate biological cuneate neurons. We have evaluated 10 naturalistic textures using a passive touch protocol, under varying sensing conditions. Tactile sensor data acquired with five textures under five sensing conditions were used for a synaptic learning process, to tune the synaptic weights between tactile afferents and cuneate neurons. Using post-learning synaptic weights, we evaluated the individual and population cuneate neuron responses by decoding across 10 stimuli, under varying sensing conditions. This resulted in a high decoding performance. We further validated the decoding performance across stimuli, irrespective of sensing velocities using a set of 25 cuneate neuron responses. This resulted in a median decoding performance of 96% across the set of cuneate neurons. Being able to learn and perform generalized discrimination across tactile stimuli, makes this functional spiking tactile system effective and suitable for further robotic applications.

A neuro-inspired artificial peripheral nervous system for scalable electronic skins

The human sense of touch is essential for dexterous tool usage, spatial awareness, and social communication. Equipping intelligent human-like androids and prosthetics with electronic skins-a large array of sensors spatially distributed and capable of rapid somatosensory perception-will enable them to work collaboratively and naturally with humans to manipulate objects in unstructured living environments. Previously reported tactile-sensitive electronic skins largely transmit the tactile information from sensors serially, resulting in readout latency bottlenecks and complex wiring as the number of sensors increases. Here, we introduce the Asynchronously Coded Electronic Skin (ACES)-a neuromimetic architecture that enables simultaneous transmission of thermotactile information while maintaining exceptionally low readout latencies, even with array sizes beyond 10,000 sensors. We demonstrate prototype arrays of up to 240 artificial mechanoreceptors that transmitted events asynchronously at a constant latency of 1 ms while maintaining an ultra-high temporal precision of <60 ns, thus resolving fine spatiotemporal features necessary for rapid tactile perception. Our platform requires only a single electrical conductor for signal propagation, realizing sensor arrays that are dynamically reconfigurable and robust to damage. We anticipate that the ACES platform can be integrated with a wide range of skin-like sensors for artificial intelligence (AI)-enhanced autonomous robots, neuroprosthetics, and neuromorphic computing hardware for dexterous object manipulation and somatosensory perception.

Tactile Decoding of Edge Orientation With Artificial Cuneate Neurons in Dynamic Conditions

Generalization ability in tactile sensing for robotic manipulation is a prerequisite to effectively perform tasks in ever-changing environments. In particular, performing dynamic tactile perception is currently beyond the ability of robotic devices. A biomimetic approach to achieve this dexterity is to develop machines combining compliant robotic manipulators with neuroinspired architectures displaying computational adaptation. Here we demonstrate the feasibility of this approach for dynamic touch tasks experimented by integrating our sensing apparatus in a 6 degrees of freedom robotic arm via a soft wrist. We embodied in the system a model of spike-based neuromorphic encoding of tactile stimuli, emulating the discrimination properties of cuneate nucleus neurons based on pathways with differential delay lines. These strategies allowed the system to correctly perform a dynamic touch protocol of edge orientation recognition (ridges from 0 to 40°, with a step of 5°). Crucially, the task was robust to contact noise and was performed with high performance irrespectively of sensing conditions (sensing forces and velocities). These results are a step forward toward the development of robotic arms able to physically interact in real-world environments with tactile sensing.