Vision-Based Tactile Sensor Mechanism for the Estimation of Contact Position and Force Distribution Using Deep Learning

Validations of various in-hand object manipulation strategies employing a novel tactile sensor developed for an under-actuated robot hand

Prisma Hand II is an under-actuated prosthetic hand developed at the University of Naples, Federico II to study in-hand manipulations during grasping activities. 3 motors equipped on the robotic hand drive 19 joints using elastic tendons. The operations of the hand are achieved by combining tactile hand sensing with under-actuation capabilities. The hand has the potential to be employed in both industrial and prosthetic applications due to its dexterous motion capabilities. However, currently there are no commercially available tactile sensors with compatible dimensions suitable for the prosthetic hand. Hence, in this work, we develop a novel tactile sensor designed based on an opto-electronic technology for the Prisma Hand II. The optimised dimensions of the proposed sensor made it possible to be integrated with the fingertips of the prosthetic hand. The output voltage obtained from the novel tactile sensor is used to determine optimum grasping forces and torques during in-hand manipulation tasks employing Neural Networks (NNs). The grasping force values obtained using a Convolutional Neural Network (CNN) and an Artificial Neural Network (ANN) are compared based on Mean Square Error (MSE) values to find out a better training network for the tasks. The tactile sensing capabilities of the proposed novel sensing method are presented and compared in simulation studies and experimental validations using various hand manipulation tasks. The developed tactile sensor is found to be showcasing a better performance compared to previous version of the sensor used in the hand.

Bio-inspired affordance learning for 6-DoF robotic grasping: A transformer-based global feature encoding approach

The 6-Degree-of-Freedom (6-DoF) robotic grasping is a fundamental task in robot manipulation, aimed at detecting graspable points and corresponding parameters in a 3D space, i.e affordance learning, and then a robot executes grasp actions with the detected affordances. Existing research works on affordance learning predominantly focus on learning local features directly for each grid in a voxel scene or each point in a point cloud scene, subsequently filtering the most promising candidate for execution. Contrarily, cognitive models of grasping highlight the significance of global descriptors, such as size, shape, and orientation, in grasping. These global descriptors indicate a grasp path closely tied to actions. Inspired by this, we propose a novel bio-inspired neural network that explicitly incorporates global feature encoding. In particular, our method utilizes a Truncated Signed Distance Function (TSDF) as input, and employs the recently proposed Transformer model to encode the global features of a scene directly. With the effective global representation, we then use deconvolution modules to decode multiple local features to generate graspable candidates. In addition, to integrate global and local features, we propose using a skip-connection module to merge lower-layer global features with higher-layer local features. Our approach, when tested on a recently proposed pile and packed grasping dataset for a decluttering task, surpassed state-of-the-art local feature learning methods by approximately 5% in terms of success and declutter rates. We also evaluated its running time and generalization ability, further demonstrating its superiority. We deployed our model on a Franka Panda robot arm, with real-world results aligning well with simulation data. This underscores our approach's effectiveness for generalization and real-world applications.

Toxicological assessment of divalent ion-modified ZnO nanomaterials through artificial intelligence and in vivo study

The nanotechnology-driven industrial revolution widely relies on metal oxide-based nanomaterial (NM). Zinc oxide (ZnO) production has rapidly increased globally due to its outstanding physical and chemical properties and versatile applications in industries including cement, rubber, paints, cosmetics, and more. Nevertheless, releasing Zn2+ ions into the environment can profoundly impact living systems and affect water-based ecosystems, including biological ones. In aquatic environments, Zn2+ ions can change water properties, directly influencing underwater ecosystems, especially fish populations. These ions can accumulate in fish tissues when fish are exposed to contaminated water and pose health risks to humans who consume them, leading to symptoms such as nausea, vomiting, and even organ damage. To address this issue, safety of ZnO NMs should be enhanced without altering their nanoscale properties, thus preventing toxic-related problems. In this study, an eco-friendly precipitation method was employed to prepare ZnO NMs. These NMs were found to reduce ZnO toxicity levels by incorporating elements such as Mg, Ca, Sr, and Ba. Structural, morphological, and optical properties of synthesized NMs were thoroughly investigated. In vitro tests demonstrated potential antioxidative properties of NMs with significant effects on free radical scavenging activities. In vivo, toxicity tests were conducted using Oreochromis mossambicus fish and male Swiss Albino mice to compare toxicities of different ZnO NMs. Fish and mice exposed to these NMs exhibited biochemical changes and histological abnormalities. Notably, ZnCaO NMs demonstrated lower toxicity to fish and mice than other ZnO NMs. This was attributed to its Ca2+ ions, which could enhance body growth metabolism compared to other metals, thus improving material safety. Furthermore, whether nanomaterials' surface roughness might contribute to their increased toxicity in biological systems was investigated utilizing computer vision (CV)-based AI tools to obtain SEM images of NMs, providing valuable image-based surface morphology data that could be correlated with relevant toxicology studies.

Dynamic Focusing (DF) Cone-Based Omnidirectional Fingertip Pressure Sensor with High Sensitivity in a Wide Pressure Range

It is essential to detect pressure from a robot's fingertip in every direction to ensure efficient and secure grasping of objects with diverse shapes. Nevertheless, creating a simple-designed sensor that offers cost-effective and omnidirectional pressure sensing poses substantial difficulties. This is because it often requires more intricate mechanical solutions than when designing non-omnidirectional pressure sensors of robot fingertips. This paper introduces an innovative pressure sensor for fingertips. It utilizes a uniquely designed dynamic focusing cone to visually detect pressure with omnidirectional sensitivity. This approach enables cost-effective measurement of pressure from all sides of the fingertip. The experimental findings demonstrate the great potential of the newly introduced sensor. Its implementation is both straightforward and uncomplicated, offering high sensitivity (0.07 mm/N) in all directions and a broad pressure sensing range (up to 40 N) for robot fingertips.

Research on Finger Pressure Tactile Sensor with Square Hole Structure Based on Fiber Bragg Grating

Aiming at the problems of lateral force interference and non-uniform strain of robot fingers in the process of pressure tactile sensing, a flexible tactile sensor with a square hole structure based on fiber Bragg grating (FBG) is proposed in this paper. Firstly, the optimal embedding depth of the FBG in the sensor matrix model was determined by finite element simulation. Secondly, according to the size of the finger knuckle and the simulation analysis based on the pressure tactile sensor element for the robot finger, the square hole structure was designed, and the overall dimensions of the sensing element and size of the square hole were determined. Thirdly, the FBG was embedded in the polydimethylsiloxane (PDMS) elastic matrix to make a sensor model, and the tactile sensor was fabricated. Finally, the FBG pressure tactile sensing system platform was built by using optical fiber sensing technology, and the experiment of the FBG tactile sensor was completed through the sensing system platform. Experimental results show that the tactile sensor designed in this paper has good repeatability and creep resistance. The sensitivity is 8.85 pm/N, and the resolution is 0.2 N. The loading sensitivity based on the robot finger is 27.3 pm/N, the goodness of fit is 0.996, and the average value of interference in the sensing process is 7.63%, which is lower than the solid structure sensor. These results verify that the sensor can effectively reduce the lateral force interference and solve the problem of non-uniform strain and has high fit with fingers, which has a certain application value for the research of robot pressure tactile intelligent perception.

Implementation of Field-Programmable Gate Array Platform for Object Classification Tasks Using Spike-Based Backpropagated Deep Convolutional Spiking Neural Networks

This paper investigates the performance of deep convolutional spiking neural networks (DCSNNs) trained using spike-based backpropagation techniques. Specifically, the study examined temporal spike sequence learning via backpropagation (TSSL-BP) and surrogate gradient descent via backpropagation (SGD-BP) as effective techniques for training DCSNNs on the field programmable gate array (FPGA) platform for object classification tasks. The primary objective of this experimental study was twofold: (i) to determine the most effective backpropagation technique, TSSL-BP or SGD-BP, for deeper spiking neural networks (SNNs) with convolution filters across various datasets; and (ii) to assess the feasibility of deploying DCSNNs trained using backpropagation techniques on low-power FPGA for inference, considering potential configuration adjustments and power requirements. The aforementioned objectives will assist in informing researchers and companies in this field regarding the limitations and unique perspectives of deploying DCSNNs on low-power FPGA devices. The study contributions have three main aspects: (i) the design of a low-power FPGA board featuring a deployable DCSNN chip suitable for object classification tasks; (ii) the inference of TSSL-BP and SGD-BP models with novel network architectures on the FPGA board for object classification tasks; and (iii) a comparative evaluation of the selected spike-based backpropagation techniques and the object classification performance of DCSNNs across multiple metrics using both public (MNIST, CIFAR10, KITTI) and private (INHA_ADAS, INHA_KLP) datasets.

Spatial Calibration of Humanoid Robot Flexible Tactile Skin for Human-Robot Interaction

Recent developments in robotics have enabled humanoid robots to be used in tasks where they have to physically interact with humans, including robot-supported caregiving. This interaction-referred to as physical human-robot interaction (pHRI)-requires physical contact between the robot and the human body; one way to improve this is to use efficient sensing methods for the physical contact. In this paper, we use a flexible tactile sensing array and integrate it as a tactile skin for the humanoid robot HRP-4C. As the sensor can take any shape due to its flexible property, a particular focus is given on its spatial calibration, i.e., the determination of the locations of the sensor cells and their normals when attached to the robot. For this purpose, a novel method of spatial calibration using B-spline surfaces has been developed. We demonstrate with two methods that this calibration method gives a good approximation of the sensor position and show that our flexible tactile sensor can be fully integrated on a robot and used as input for robot control tasks. These contributions are a first step toward the use of flexible tactile sensors in pHRI applications.

Speckle-based high-resolution multimodal soft sensing

Skin-like soft sensors are key components for human–machine interfaces; however, the simultaneous sensing of several types of stimuli remains challenging because large-scale sensor integration is required with numerous wire connections. We propose an optical high-resolution multimodal sensing approach, which does not require integrating multiple sensors. This approach is based on the combination of an optical scattering phenomenon, which can encode the information of various stimuli as a speckle pattern, and a decoding technique using deep learning. We demonstrate the simultaneous sensing of three different physical quantities—contact force, contact location, and temperature—with a single soft material. Another unique capability of the proposed approach is spatially continuous sensing with an ultrahigh resolution of few tens of micrometers, in contrast to previous multimodal sensing approaches. Furthermore, a haptic soft device is presented for a human–machine interface. Our approach encourages the development of high-performance smart skin-like sensors.

Optical Fiber Array Sensor for Force Estimation and Localization in TAVI Procedure: Design, Modeling, Analysis and Validation

Transcatheter aortic valve implantation has shown superior clinical outcomes compared to open aortic valve replacement surgery. The loss of the natural sense of touch, inherited from its minimally invasive nature, could lead to misplacement of the valve in the aortic annulus. In this study, a cylindrical optical fiber sensor is proposed to be integrated with valve delivery catheters. The proposed sensor works based on intensity modulation principle and is capable of measuring and localizing lateral force. The proposed sensor was constituted of an array of optical fibers embedded on a rigid substrate and covered by a flexible shell. The optical fibers were modeled as Euler–Bernoulli beams with both-end fixed boundary conditions. To study the sensing principle, a parametric finite element model of the sensor with lateral point loads was developed and the deflection of the optical fibers, as the determinant of light intensity modulation was analyzed. Moreover, the sensor was fabricated, and a set of experiments were performed to study the performance of the sensor in lateral force measurement and localization. The results showed that the transmitted light intensity decreased up to 24% for an external force of 1 N. Additionally, the results showed the same trend between the simulation predictions and experimental results. The proposed sensor was sensitive to the magnitude and position of the external force which shows its capability for lateral force measurement and localization.

Automated Inorganic Pigment Classification in Plastic Material Using Terahertz Spectroscopy

This paper presents an automatic classification of plastic material's inorganic pigment using terahertz spectroscopy and convolutional neural networks (CNN). The plastic materials were placed between the THz transmitter and receiver, and the acquired THz signals were classified using a supervised learning approach. A THz frequency band between 0.1-1.2 THz produced a one-dimensional (1D) vector that is almost impossible to classify directly using supervised learning. This paper proposes a novel pre-processing of 1D THz data that transforms 1D data into 2D data, which are processed efficiently using a convolutional neural network. The proposed pre-processing algorithm consists of four steps: peak detection, envelope extraction, and a down-sampling procedure. The last main step introduces the windowing with spectrum dilatation that reorders 1D data into 2D data that can be considered as an image. The spectrum dilation techniques ensure the classifier's robustness by suppressing measurement bias, reducing the complexity of the THz dataset with negligible loss of accuracy, and speeding up the network classification. The experimental results showed that the proposed approach achieved high accuracy using a CNN classifier, and outperforms 1D classification of THz data using support vector machine, naive Bayes, and other popular classification algorithms.