A Machine-Learning-Based Approach to Solve Both Contact Location and Force in Soft Material Tactile Sensors

Compliant Grasping Control for a Tactile Self-Sensing Soft Gripper

Soft grippers with good passive compliance can effectively adapt to the shape of a target object and have better safe grasping performance than rigid grippers. However, for soft or fragile objects, passive compliance is insufficient to prevent grippers from crushing the target. Thus, to complete nondestructive grasping tasks, precision force sensing and control are immensely important for soft grippers. In this article, we proposed an online learning self-tuning nonlinearity impedance controller for a tactile self-sensing two-finger soft gripper so that its grasping force can be controlled accurately. For the soft gripper, its grasping force is sensed by a liquid lens-based optical tactile sensing unit that contains a self-sensing fingertip and a liquid lens module and has many advantages of a rapid response time (about 0.04 s), stable output, good sensitivity (>0.4985 V/N), resolution (0.03 N), linearity (R2 > 0.96), and low cost (power consumption: 5 mW, preparation cost Collapse

Key Words

• force control
• full online sequential extreme learning machine
• liquid lens-based optical tactile sensing unit
• soft gripper

Collapse

MESH Headings Collapse

Grants Collapse

Affiliation(s)

Hui Yang Dynamic Image Perception Lab, Institute of Semiconductors, Guangdong Academy of Sciences, Guangdong, China Biomechanics and Soft Robotics Lab, School of Mechanical Engineering and Automation, Beihang University, Beijing, China
Jiaqi Liu Biomechanics and Soft Robotics Lab, School of Mechanical Engineering and Automation, Beihang University, Beijing, China
Wenbo Liu Biomechanics and Soft Robotics Lab, School of Mechanical Engineering and Automation, Beihang University, Beijing, China
Weirui Liu Department of Mechanical and Electrical Engineering, School of Mechanical Engineering and Automation, Liaoning Petrochemical University, Fushun, China
Zilong Deng Department of Mechanical and Electrical Engineering, School of Mechanical Engineering and Automation, Liaoning Petrochemical University, Fushun, China
Yunzhi Ling Dynamic Image Perception Lab, Institute of Semiconductors, Guangdong Academy of Sciences, Guangdong, China
Changan Wang Dynamic Image Perception Lab, Institute of Semiconductors, Guangdong Academy of Sciences, Guangdong, China
Meixia Wu Dynamic Image Perception Lab, Institute of Semiconductors, Guangdong Academy of Sciences, Guangdong, China
Lihui Wang Dynamic Image Perception Lab, Institute of Semiconductors, Guangdong Academy of Sciences, Guangdong, China
Li Wen Biomechanics and Soft Robotics Lab, School of Mechanical Engineering and Automation, Beihang University, Beijing, China

Collapse

Machine Learning Applications in Optical Fiber Sensing: A Research Agenda

The constant monitoring and control of various health, infrastructure, and natural factors have led to the design and development of technological devices in a wide range of fields. This has resulted in the creation of different types of sensors that can be used to monitor and control different environments, such as fire, water, temperature, and movement, among others. These sensors detect anomalies in the input data to the system, allowing alerts to be generated for early risk detection. The advancement of artificial intelligence has led to improved sensor systems and networks, resulting in devices with better performance and more precise results by incorporating various features. The aim of this work is to conduct a bibliometric analysis using the PRISMA 2020 set to identify research trends in the development of machine learning applications in fiber optic sensors. This methodology facilitates the analysis of a dataset comprised of documents obtained from Scopus and Web of Science databases. It enables the evaluation of both the quantity and quality of publications in the study area based on specific criteria, such as trends, key concepts, and advances in concepts over time. The study found that deep learning techniques and fiber Bragg gratings have been extensively researched in infrastructure, with a focus on using fiber optic sensors for structural health monitoring in future research. One of the main limitations is the lack of research on the use of novel materials, such as graphite, for designing fiber optic sensors. One of the main limitations is the lack of research on the use of novel materials, such as graphite, for designing fiber optic sensors. This presents an opportunity for future studies.

A Deep Learning Framework for Soft Robots with Synthetic Data

Data-driven methods with deep neural networks demonstrate promising results for accurate modeling in soft robots. However, deep neural network models rely on voluminous data in discovering the complex and nonlinear representations inherent in soft robots. Consequently, while it is not always possible, a substantial amount of effort is required for data acquisition, labeling, and annotation. This article introduces a data-driven learning framework based on synthetic data to circumvent the exhaustive data collection process. More specifically, we propose a novel time series generative adversarial network with a self-attention mechanism, Transformer TimeGAN (TTGAN) to precisely learn the complex dynamics of a soft robot. On top of that, the TTGAN is incorporated with a conditioning network that enables it to produce synthetic data for specific soft robot behaviors. The proposed framework is verified on a widely used pneumatic-based soft gripper as an exemplary experimental setup. Experimental results demonstrate that the TTGAN generates synthetic time series data with realistic soft robot dynamics. Critically, a combination of the synthetic and only partially available original data produces a data-driven model with estimation accuracy comparable to models obtained from using complete original data.

Soft Biomimetic Fiber-Optic Tactile Sensors Capable of Discriminating Temperature and Pressure

Tactile sensors with high softness and multisensory functions are highly desirable for applications in humanoid robotics, smart prosthetics, and human-machine interfaces. Here, we report a soft biomimetic fiber-optic tactile (SBFT) sensor that offers skin-like tactile sensing abilities to perceive and discriminate temperature and pressure. The SBFT sensor is fabricated by encapsulating a macrobent fiber Bragg grating (FBG) in an elastomeric droplet-shaped structure that results in two optical resonances associated with the FBG and excited whispering gallery modes (WGMs) propagating along the bent region. Benefiting from the different thermo-optic and stress-optic effects of FBG and WGM resonances, the pressure and temperature can be fully decoupled with a high precision of 0.2 °C and 0.8 mN, respectively. To achieve a compact system for signal demodulation, a single-cavity dual-comb fiber laser is developed to interrogate the SBFT sensor based on dual-comb spectroscopy, which enables fast spectral sampling with a single photodiode. We show that the SBFT sensor is capable of perceiving pressure, temperature, and hardness in touching soft tissues and human skins, demonstrating great promise for soft tissue palpation and human-like robotic perception.

A Learning-Based Approach to Sensorize Soft Robots

Soft actuators and their sensors have always been separate entities with two distinct roles. The omnidirectional compliance of soft robots thus means that multiple sensors have to be used to sense different modalities in the respective planes of motion. With the recent emergence of self-sensing actuators, the two roles have gradually converged to simplify sensing requirements. Self-sensing typically involves embedding a conductive sensing element into the soft actuator and provides multiple state information along the continuum. However, most of these self-sensing actuators are fabricated through manual methods, which results in inconsistent sensing performance. Soft material compliance also imply that both actuator and sensor exhibit nonlinear behaviors during actuation, making sensing more complex. In this regard, machine learning has shown promise in characterizing the nonlinear behavior of soft sensors. Beyond characterization, we show that applying machine learning to soft actuators eliminates the need to implant a sensing element to achieve self-sensing. Fabrication is done using 3D printing, thus ensuring that sensing performance is consistent across the actuators. In addition, our proposed technique is able to estimate the bending curvature of a soft continuum actuator and the external forces applied to the tip of the actuator in real time. Our methodology is generalizable and aims to provide a novel way of multimodal sensing for soft robots across a variety of applications.

2D Temperature Field Reconstruction Using Optical Frequency Domain Reflectometry and Machine-Learning Algorithms

We present experimental results on the reconstruction of the 2D temperature field on the surface of a 250 × 250 mm sensor panel based on the distributed frequency shift measured by an optical backscatter reflectometer. A linear regression and a feed-forward neural network algorithm, trained by varying the temperature field and capturing thermal images of the panel, are used for the reconstruction. In this approach, we do not use any information about the exact trajectory of the fiber, material properties of the sensor panel, and a temperature sensitivity coefficient of the fiber. Mean absolute errors of 0.118 °C and 0.086 °C are achieved in the case of linear regression and feed-forward neural network, respectively.

Highly sensitive soft optical fiber tactile sensor

A soft highly sensitive tactile sensor based on an in-fiber interferometer embedded in polydimethylsiloxane (PDMS) structure is studied. Theoretical simulation obtains that the high order sensing modes and PDMS can improve the sensitivity. Experiments show that different order sensing modes, derived by fast Fourier transform (FFT) and inverse FFT methods, present different sensing performance. Corresponding to high order mode, 1.3593 nm/kPa sensitivity and 37 Pa (0.015 N) detection limit is obtained. Meanwhile, it also shows very good stability, reproducibility, and response time. This study not only demonstrates a tactile sensor with high sensitivity but also provides a novel sensing modes analysis method.

Functional mimicry of Ruffini receptors with fibre Bragg gratings and deep neural networks enables a bio-inspired large-area tactile-sensitive skin

Collaborative robots are expected to physically interact with humans in daily living and the workplace, including industrial and healthcare settings. A key related enabling technology is tactile sensing, which currently requires addressing the outstanding scientific challenge to simultaneously detect contact location and intensity by means of soft conformable artificial skins adapting over large areas to the complex curved geometries of robot embodiments. In this work, the development of a large-area sensitive soft skin with a curved geometry is presented, allowing for robot total-body coverage through modular patches. The biomimetic skin consists of a soft polymeric matrix, resembling a human forearm, embedded with photonic fibre Bragg grating transducers, which partially mimics Ruffini mechanoreceptor functionality with diffuse, overlapping receptive fields. A convolutional neural network deep learning algorithm and a multigrid neuron integration process were implemented to decode the fibre Bragg grating sensor outputs for inference of contact force magnitude and localization through the skin surface. Results of 35 mN (interquartile range 56 mN) and 3.2 mm (interquartile range 2.3 mm) median errors were achieved for force and localization predictions, respectively. Demonstrations with an anthropomorphic arm pave the way towards artificial intelligence based integrated skins enabling safe human–robot cooperation via machine intelligence.

Smart laparoscopic grasper integrated with fiber Bragg grating based tactile sensor for real-time force feedback

Minimally invasive surgery, such as laparoscopic surgery, has developed rapidly due to its small wound, less bleeding and quick recovery. However, a lack of force feedback, which leads to tissue damage, is still unsolved. Many sensors have been used to offer force feedback but still limited by their large size, low security and high complexity. Based on the advantages of small size, high sensitivity and immunity to electromagnetic interferences, we propose a tactile sensor integrated with fiber Bragg gratings (FBGs) at the tip of laparoscopic grasper to offer real-time force feedback in the laparoscopic surgery. The tactile sensor shows a force sensitivity of 0.076 nm/N with a repeatable accuracy of 0.118 N. A bench test is conducted in a laparoscopic training box to verify its feasibility. Test results illustrate that gripping force exerted on the laparoscopic grasper in terms of peak and standard deviation values reduce significantly for the novice subjects with force feedback compared to those without force feedback. The proposed sensor integrated at the tip of the laparoscopic grasper demonstrates a better control of the gripping force among the novice surgeons and indicates that the smart grasper can help surgeons achieve precise gripping force to reduce unnecessary tissue trauma.

Radio Frequency Resonator-Based Flexible Wireless Pressure Sensor with MWCNT-PDMS Bilayer Microstructure

Flexible pressure sensors have been widely applied in wearable devices, e-skin, and the new generation of robots. However, most of the current sensors use connecting wires for energy supply and signal transmission, which presents an obstacle for application scenarios requiring long endurance and large movement, especially. Flexible sensors combined with wireless technology is a promising research field for realizing efficient state sensing in an active state. Here, we designed and fabricated a soft wireless passive pressure sensor, with a fully flexible Ecoflex substrate and a multi-walled carbon nanotube/polydimethylsiloxane (MWCNT/PDMS) bilayer pyramid dielectric structure. Based on the principle of the radio-frequency resonator, the device achieved pressure sensing with a changeable capacitance. Subsequently, the effect of the pyramid density was simulated by the finite element method to improve the sensitivity. With one-step embossing and spin-coating methods, the fabricated sensor had an optimized sensitivity of 14.25 MHz/kPa in the low-pressure range. The sensor exhibited the potential for application in limb bending monitoring, thus demonstrating its value for long-term wireless clinical monitoring. Moreover, the radio frequency coupling field can be affected by approaching objects, which provides a possible route for realizing non-contact sensing in applications such as pre-collision warning.

A Model for Estimating Tactile Sensation by Machine Learning Based on Vibration Information Obtained while Touching an Object

The tactile sensation is an important indicator of the added value of a product, and it is thus important to be able to evaluate this sensation quantitatively. Sensory evaluation is generally used to quantitatively evaluate the tactile sensation of an object. However, statistical evaluation of the tactile sensation requires many participants and is, thus, time-consuming and costly. Therefore, tactile sensing technology, as opposed to sensory evaluation, is attracting attention. In establishing tactile sensing technology, it is necessary to estimate the tactile sensation of an object from information obtained by a tactile sensor. In this research, we developed a tactile sensor made of two-layer silicone rubber with two strain gauges in each layer and obtained vibration information as the sensor traced an object. We then extracted features from the vibration information using deep autoencoders, following the nature of feature extraction by neural firing due to vibrations perceived within human fingers. We also conducted sensory evaluation to obtain tactile scores for different words from participants. We finally developed a tactile sensation estimation model for each of the seven samples and evaluated the accuracy of estimating the tactile sensation of unknown samples. We demonstrated that the developed model can properly estimate the tactile sensation for at least four of the seven samples.

Soft System Based on Fiber Bragg Grating Sensor for Loss of Resistance Detection during Epidural Procedures: In Silico and In Vivo Assessment

Epidural analgesia represents a clinical common practice aiming at pain mitigation. This loco-regional technique is widely used in several applications such as labor, surgery and lower back pain. It involves the injections of anesthetics or analgesics into the epidural space (ES). The ES detection is still demanding and is usually performed by the techniques named loss of resistance (LOR). In this study, we propose a novel soft system (SS) based on one fiber Bragg grating sensor (FBG) embedded in a soft polymeric matrix for LOR detection during the epidural puncture. The SS was designed to allow instrumenting the syringe's plunger without relevant modifications of the anesthetist's sensations during the procedure. After the metrological characterization of the SS, we assessed the capability of this solution in detecting LOR by carrying it out in silico and in clinical settings. For both trials, results revealed the capability of the proposed solutions in detecting the LOR and then in recording the force exerted on the plunger.

FBG-Based Estimation of External Forces Along Flexible Instrument Bodies

A variety of medical treatment and diagnostic procedures rely on flexible instruments such as catheters and endoscopes to navigate through tortuous and soft anatomies like the vasculature. Knowledge of the interaction forces between these flexible instruments and patient anatomy is extremely valuable. This can aid interventionalists in having improved awareness and decision-making abilities, efficient navigation, and increased procedural safety. In many applications, force interactions are inherently distributed. While knowledge of their locations and magnitudes is highly important, retrieving this information from instruments with conventional dimensions is far from trivial. Robust and reliable methods have not yet been found for this purpose. In this work, we present two new approaches to estimate the location, magnitude, and number of external point and distributed forces applied to flexible and elastic instrument bodies. Both methods employ the knowledge of the instrument’s curvature profile. The former is based on piecewise polynomial-based curvature segmentation, whereas the latter on model-based parameter estimation. The proposed methods make use of Cosserat rod theory to model the instrument and provide force estimates at rates over 30 Hz. Experiments on a Nitinol rod embedded with a multi-core fiber, inscribed with fiber Bragg gratings, illustrate the feasibility of the proposed methods with mean force error reaching 7.3% of the maximum applied force, for the point load case. Furthermore, simulations of a rod subjected to two distributed loads with varying magnitudes and locations show a mean force estimation error of 1.6% of the maximum applied force.

User-Interactive Robot Skin with Large-Area Scalability for Safer and Natural Human-Robot Collaboration in Future Telehealthcare

With the fourth revolution of healthcare, i.e., Healthcare 4.0, collaborative robotics is spilling out from traditional manufacturing and will blend into human living or working environments to deliver care services, especially telehealthcare. Because of the frequent and seamless interaction between robots and care recipients, it poses several challenges that require careful consideration: 1) the ability of the human to collaborate with the robots in a natural manner; and 2) the safety of the human collaborating with the robot. In this regard, we have proposed a proximity sensing solution based on the self-capacitive technology to provide an extended sense of touch for collaborative robots, allowing approach and contact measurement to enhance safe and natural human-robot collaboration. The modular design of our solution enables it to scale up to form a large-area sensing system. The sensing solution is proposed to work in two operation modes: the interaction mode and the safety mode. In the interaction mode, utilizing the ability of the sensor to localize the point of action, gesture command is used for robot manipulation. In the safety mode, the sensor enables the robot to actively avoid obstacles.

Random forest assisted vector displacement sensor based on a multicore fiber

We proposed a two-dimensional vector displacement sensor with the capability of distinguishing the direction and amplitude of the displacement simultaneously, with improved performance assisted by random forest, a powerful machine learning algorithm. The sensor was designed based on a seven-core multi-core fiber inscribed with Bragg gratings, with a displacement direction range of 0-360° and the amplitude range related to the length of the sensor body. The displacement information was obtained under a random circumstance, where the performances with theoretical model and random forest model were studied. With the theoretical model, the sensor performed well over a shorter linear range (from 0 to 9 mm). Whereas the sensor assisted with random forest algorithm exhibits better performance in two aspects, a wider measurement range (from 0 to 45 mm) and a reduced measurement error of displacement. Mean absolute errors of direction and amplitude reconstruction were decreased by 60% and 98%, respectively. The proposed displacement sensor shows the possibility of machine learning methods to be applied in point-based optical systems for multi-parameter sensing.

Parasitic Capacitance-Free Flexible Tactile Sensor with a Real-Contact Trigger

In this study, a parasitic capacitance-free tactile sensor with a floating electrode that is capable of identifying actual physical contact pressure by distinguishing from parasitic effects and applicable to sensor arrays is presented. Although capacitive pressure sensors are known for their excellent pressure sensing capabilities in wide range with high sensitivity, they tend to suffer from a parasitic capacitance noise and unwanted proximity effects. Electromagnetic interference shielding was conventionally used to prevent this noise; however, it was not entirely successful in multicell array sensors. Parasitic capacitance-free method involves the use of a floating electrode, which functions as a contact trigger by causing sudden changes in capacitance only when the actual physical contact pressure has been applied or removed. The proposed method is robust, consistent, and precise. Experimental results show a wide range of pressure response up to 2.4 MPa with a sensitivity of 0.179 MPa^-1 (up to 0.74 MPa) and negligible hysteresis.

Seedless Hydrothermal Growth of ZnO Nanorods as a Promising Route for Flexible Tactile Sensors

Hydrothermal growth of ZnO nanorods has been widely used for the development of tactile sensors, with the aid of ZnO seed layers, favoring the growth of dense and vertically aligned nanorods. However, seed layers represent an additional fabrication step in the sensor design. In this study, a seedless hydrothermal growth of ZnO nanorods was carried out on Au-coated Si and polyimide substrates. The effects of both the Au morphology and the growth temperature on the characteristics of the nanorods were investigated, finding that smaller Au grains produced tilted rods, while larger grains provided vertical rods. Highly dense and high-aspect-ratio nanorods with hexagonal prismatic shape were obtained at 75 °C and 85 °C, while pyramid-like rods were grown when the temperature was set to 95 °C. Finite-element simulations demonstrated that prismatic rods produce higher voltage responses than the pyramid-shaped ones. A tactile sensor, with an active area of 1 cm², was fabricated on flexible polyimide substrate and embedding the nanorods forest in a polydimethylsiloxane matrix as a separation layer between the bottom and the top Au electrodes. The prototype showed clear responses upon applied loads of 2-4 N and vibrations over frequencies in the range of 20-800 Hz.