1
|
Mao Q, Liao Z, Yuan J, Zhu R. Multimodal tactile sensing fused with vision for dexterous robotic housekeeping. Nat Commun 2024; 15:6871. [PMID: 39127714 PMCID: PMC11316753 DOI: 10.1038/s41467-024-51261-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/17/2024] [Accepted: 08/02/2024] [Indexed: 08/12/2024] Open
Abstract
As robots are increasingly participating in our daily lives, the quests to mimic human abilities have driven the advancements of robotic multimodal senses. However, current perceptual technologies still unsatisfied robotic needs for home tasks/environments, particularly facing great challenges in multisensory integration and fusion, rapid response capability, and highly sensitive perception. Here, we report a flexible tactile sensor utilizing thin-film thermistors to implement multimodal perceptions of pressure, temperature, matter thermal property, texture, and slippage. Notably, the tactile sensor is endowed with an ultrasensitive (0.05 mm/s) and ultrafast (4 ms) slip sensing that is indispensable for dexterous and reliable grasping control to avoid crushing fragile objects or dropping slippery objects. We further propose and develop a robotic tactile-visual fusion architecture that seamlessly encompasses multimodal sensations from the bottom level to robotic decision-making at the top level. A series of intelligent grasping strategies with rapid slip feedback control and a tactile-visual fusion recognition strategy ensure dexterous robotic grasping and accurate recognition of daily objects, handling various challenging tasks, for instance grabbing a paper cup containing liquid. Furthermore, we showcase a robotic desktop-cleaning task, the robot autonomously accomplishes multi-item sorting and cleaning desktop, demonstrating its promising potential for smart housekeeping.
Collapse
Affiliation(s)
- Qian Mao
- State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University, Beijing, China
| | - Zijian Liao
- State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University, Beijing, China
| | - Jinfeng Yuan
- State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University, Beijing, China
| | - Rong Zhu
- State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University, Beijing, China.
| |
Collapse
|
2
|
Yu H, Liang H, Bai J, Li C. The controlled growth CuS nanosheets on the surface of functionalization carbon fibers with SiO 2. INORG NANO-MET CHEM 2022. [DOI: 10.1080/24701556.2021.2025106] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/18/2022]
Affiliation(s)
- Haiyan Yu
- Chemical Engineering College, Inner Mongolia University of Technology, Hohhot, People’s Republic of China
- Inner Mongolia Key Laboratory of Industrial Catalysis, Hohhot, People’s Republic of China
- Chemistry College, Baotou Teachers' College, Baotou, People’s Republic of China
| | - Haiou Liang
- Chemical Engineering College, Inner Mongolia University of Technology, Hohhot, People’s Republic of China
- Inner Mongolia Key Laboratory of Industrial Catalysis, Hohhot, People’s Republic of China
| | - Jie Bai
- Chemical Engineering College, Inner Mongolia University of Technology, Hohhot, People’s Republic of China
- Inner Mongolia Key Laboratory of Industrial Catalysis, Hohhot, People’s Republic of China
| | - Chunping Li
- Chemical Engineering College, Inner Mongolia University of Technology, Hohhot, People’s Republic of China
- Inner Mongolia Key Laboratory of Industrial Catalysis, Hohhot, People’s Republic of China
| |
Collapse
|
3
|
Li G, Liu S, Mao Q, Zhu R. Multifunctional Electronic Skins Enable Robots to Safely and Dexterously Interact with Human. ADVANCED SCIENCE (WEINHEIM, BADEN-WURTTEMBERG, GERMANY) 2022; 9:e2104969. [PMID: 35170258 PMCID: PMC9008439 DOI: 10.1002/advs.202104969] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/03/2021] [Revised: 12/12/2021] [Indexed: 06/14/2023]
Abstract
Human-robot collaboration is playing more and more important roles in current deployments of robotic systems in our lives. Haptic perception and intelligent control are essential to ensure safety and efficiency of human-robot interaction. However, existing robotic sensory and control systems are deficient in terms of performance issues, complexity, and cost. Here, the authors report a multifunctional electronic skin (e-skin) incorporating multiple perceptions with intelligent robotic control, by which robots can safely and dexterously interact with humans. The e-skin with a simple and cost-effective sensory structure has multimodal perceptions of proximity, temperature, contact force, and contact position with broad measuring range, high sensitivity, and fast response. The e-skin is applied onto robots to accomplish obstacle avoidance, safe and dexterous human-robot interaction, smart teaching, and playing Tai-Chi, which demonstrate a broad range of applications for intelligent robots equipped with e-skins.
Collapse
Affiliation(s)
- Guozhen Li
- State Key Laboratory of Precision Measurement Technology and InstrumentsDepartment of Precision InstrumentTsinghua UniversityBeijing100084China
| | - Shiqiang Liu
- State Key Laboratory of Precision Measurement Technology and InstrumentsDepartment of Precision InstrumentTsinghua UniversityBeijing100084China
| | - Qian Mao
- State Key Laboratory of Precision Measurement Technology and InstrumentsDepartment of Precision InstrumentTsinghua UniversityBeijing100084China
| | - Rong Zhu
- State Key Laboratory of Precision Measurement Technology and InstrumentsDepartment of Precision InstrumentTsinghua UniversityBeijing100084China
| |
Collapse
|
4
|
Interface sensors with skin piezo-thermic transduction enable motion artifact removal for wearable physiological monitoring. Biosens Bioelectron 2021; 188:113325. [PMID: 34030098 DOI: 10.1016/j.bios.2021.113325] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/28/2021] [Revised: 05/05/2021] [Accepted: 05/06/2021] [Indexed: 12/25/2022]
Abstract
Accurate measurements on physiological parameters using wearable monitoring devices during physical exercises are essential for personal healthcare and rehabilitation training, but still challenging owing to various motion artifacts (MA) caused by the interfacial dynamic change between wearable sensors and human skin. Here, we propose an interface sensor to detect noncontact proximity and contact pressure between wearable sensors and human skin. The interface sensor employs natural piezo-thermic transduction of human skin and enables direct interfacial proximity/pressure detection by using simple thin-film thermistors to detect the interfacial thermal field change. We develop a wearable watch-type heart rate (HR) monitor utilizing interface sensors to remove MA for a photoplethysmography (PPG) sensor through adaptive filtering. To validate the method, we conduct experiments for multiple subjects, who carry out HR monitoring using the wearable device while doing various physical exercises. The PPG-based HR estimations are corrected through MA removal using interface sensors and compared with that using conventional accelerometer-based MA removal. The experimental results verify that the interface sensors capture the interfacial dynamic change between the PPG sensor and skin better, and obtain more accurate HR estimations during irregular and muscle strength exercises. Utilizing natural transduction of human skin and simple thermometry, the interface sensor provides an advantageous way to overcome MA for wearable monitoring devices during physical activities and thus broadens wearable monitoring applications.
Collapse
|
5
|
Li G, Liu S, Wang L, Zhu R. Skin-inspired quadruple tactile sensors integrated on a robot hand enable object recognition. Sci Robot 2021; 5:5/49/eabc8134. [PMID: 33328298 DOI: 10.1126/scirobotics.abc8134] [Citation(s) in RCA: 118] [Impact Index Per Article: 39.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/16/2020] [Accepted: 11/17/2020] [Indexed: 12/19/2022]
Abstract
Robot hands with tactile perception can improve the safety of object manipulation and also improve the accuracy of object identification. Here, we report the integration of quadruple tactile sensors onto a robot hand to enable precise object recognition through grasping. Our quadruple tactile sensor consists of a skin-inspired multilayer microstructure. It works as thermoreceptor with the ability to perceive thermal conductivity of a material, measure contact pressure, as well as sense object temperature and environment temperature simultaneously and independently. By combining tactile sensing information and machine learning, our smart hand has the capability to precisely recognize different shapes, sizes, and materials in a diverse set of objects. We further apply our smart hand to the task of garbage sorting and demonstrate a classification accuracy of 94% in recognizing seven types of garbage.
Collapse
Affiliation(s)
- Guozhen Li
- State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University, Beijing 100084, China
| | - Shiqiang Liu
- State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University, Beijing 100084, China
| | - Liangqi Wang
- State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University, Beijing 100084, China
| | - Rong Zhu
- State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University, Beijing 100084, China.
| |
Collapse
|
6
|
Wang L, Zhu R, Li G. Temperature and Strain Compensation for Flexible Sensors Based on Thermosensation. ACS APPLIED MATERIALS & INTERFACES 2020; 12:1953-1961. [PMID: 31816229 DOI: 10.1021/acsami.9b21474] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/20/2023]
Abstract
Flexible sensors have wide applications in wearable electronics, health monitoring, humanoid robotics, and smart prosthesis. Problems of temperature drift and bending/stretching strain are challenging and should not be neglected in practical applications of flexible sensors. Here, we report a novel temperature and strain compensation method for thermosensation-based flexible sensors. Thermosensation is human-skin-inspired perception, which inspires diverse flexible sensors (pressure sensor, flow sensor, temperature sensor, material sensor, proximity sensor, etc.) and multisensory electronic skin. Thermosensation-based flexible sensors utilize thin-film sensing thermistors to detect external physical stimuli through perceptions of the conductive and convective heat transfers toward the surroundings, which enables high-density integration of multisensations while minimizing complexity due to the uniform sensing principle of thermistors that have simple structures and easy operations. To overcome the negative effects of temperature drift and bending/stretching strain in these flexible sensors, we propose to monolithically integrate a compensating thermistor that has a similar geometric shape and is of the same material with the sensing thermistor into a Wheatstone-bridge feedback circuit. When the sensing and compensating thermistors meet geometric similarity, the compensations of temperature and strain are self-sustained by a feedback control of a circuit. The effectiveness is validated through theoretical analysis and experiment measurements. As examples, flexible pressure sensor and flexible flow sensor with temperature and strain compensations are demonstrated. Results indicate that the temperature and strain effects can be tremendously eliminated using the proposed compensation method, which is fast, self-sustained, and expedient to realize. The compensation method enriches competences of flexible sensors and demonstrates competitive advantages for diverse flexible and stretching applications of wearable electronics.
Collapse
Affiliation(s)
- Liangqi Wang
- State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument , Tsinghua University , Beijing 100084 , China
| | - Rong Zhu
- State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument , Tsinghua University , Beijing 100084 , China
| | - Guozhen Li
- State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument , Tsinghua University , Beijing 100084 , China
| |
Collapse
|
7
|
Fu Y, Zhao S, Wang L, Zhu R. A Wearable Sensor Using Structured Silver-Particle Reinforced PDMS for Radial Arterial Pulse Wave Monitoring. Adv Healthc Mater 2019; 8:e1900633. [PMID: 31293071 DOI: 10.1002/adhm.201900633] [Citation(s) in RCA: 20] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/18/2019] [Revised: 06/27/2019] [Indexed: 01/31/2023]
Abstract
Human pulse signals contain important and useful physiological information for the auxiliary diagnosis of cardiovascular disease. Here, a wearable pulse sensor based on piezo-thermic transduction is reported using a structured silver-particle reinforced polydimethylsiloxane (PDMS) membrane, for monitoring radial arterial pulse waves. The structured silver-particle reinforced PDMS membrane is optimally designed to meet the specific requirements on sensitivity, linearity, and effective preload measuring range for pulse detection by adjusting the air gap volume fraction and silver particle volume fraction of the structured material. The sensor is endowed with high sensitivity, good linearity in preload measuring range, allowing to detect the subtle pulse waveforms of subjects at different ages under different contact pressures, such as superficial (Fu), medium (Zhong) and deep (Chen). The developed pulse device provides a promising approach for homecare pulse monitoring.
Collapse
Affiliation(s)
- Yu Fu
- State Key Laboratory of Precision Measurement Technology and InstrumentsDepartment of Precision InstrumentTsinghua University Beijing 100084 China
| | - Shuai Zhao
- State Key Laboratory of Precision Measurement Technology and InstrumentsDepartment of Precision InstrumentTsinghua University Beijing 100084 China
| | - Liangqi Wang
- State Key Laboratory of Precision Measurement Technology and InstrumentsDepartment of Precision InstrumentTsinghua University Beijing 100084 China
| | - Rong Zhu
- State Key Laboratory of Precision Measurement Technology and InstrumentsDepartment of Precision InstrumentTsinghua University Beijing 100084 China
| |
Collapse
|