1
|
Makarenko M, Burguete-Lopez A, Wang Q, Giancola S, Ghanem B, Passone L, Fratalocchi A. Hardware-accelerated integrated optoelectronic platform towards real-time high-resolution hyperspectral video understanding. Nat Commun 2024; 15:7051. [PMID: 39147787 PMCID: PMC11327253 DOI: 10.1038/s41467-024-51406-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/07/2024] [Accepted: 08/05/2024] [Indexed: 08/17/2024] Open
Abstract
Recent advancements in artificial intelligence have significantly expanded capabilities in processing language and images. However, the challenge of comprehensively understanding video content still needs to be solved. The main problem is the requirement to process real-time multidimensional video information at data rates exceeding 1 Tb/s, a demand that current hardware technologies cannot meet. This work introduces a hardware-accelerated integrated optoelectronic platform specifically designed for the real-time analysis of multidimensional video. By leveraging optical information processing within artificial intelligence hardware and combining it with advanced machine vision networks, the platform achieves data processing speeds of 1.2 Tb/s. This capability supports the analysis of hundreds of frequency bands with megapixel spatial resolution at video frame rates, significantly outperforming existing technologies in speed by three to four orders of magnitude. The platform demonstrates effectiveness for AI-driven tasks, such as video semantic segmentation and object understanding, across indoor and aerial scenarios. By overcoming the current data processing speed limitations, the platform shows promise in real-time AI video understanding, with potential implications for enhancing human-machine interactions and advancing cognitive processing technologies.
Collapse
Affiliation(s)
- Maksim Makarenko
- PRIMALIGHT, Faculty of Electrical Engineering; Applied Mathematics and Computational Science, King Abdullah University of Science and Technology, Thuwal, 23955-6900, Saudi Arabia
- AI & Advanced Computing Lab, EXPEC ARC, Saudi Aramco, 4143 Dhahran Blvd, Gharb Al Dhahran, Dhahran, 34466, Saudi Arabia
| | - Arturo Burguete-Lopez
- PRIMALIGHT, Faculty of Electrical Engineering; Applied Mathematics and Computational Science, King Abdullah University of Science and Technology, Thuwal, 23955-6900, Saudi Arabia
| | - Qizhou Wang
- PRIMALIGHT, Faculty of Electrical Engineering; Applied Mathematics and Computational Science, King Abdullah University of Science and Technology, Thuwal, 23955-6900, Saudi Arabia
| | - Silvio Giancola
- Image and Video Understanding Lab, Faculty of Electrical Engineering; Applied Mathematics and Computational Science, King Abdullah University of Science and Technology, Thuwal, 23955-6900, Saudi Arabia
| | - Bernard Ghanem
- Image and Video Understanding Lab, Faculty of Electrical Engineering; Applied Mathematics and Computational Science, King Abdullah University of Science and Technology, Thuwal, 23955-6900, Saudi Arabia
| | - Luca Passone
- Falconviz, King Abdullah University of Science and Technology Research Park Headquarters - Level 1 - Office 2225, Thuwal, 23955-6900, Saudi Arabia
| | - Andrea Fratalocchi
- PRIMALIGHT, Faculty of Electrical Engineering; Applied Mathematics and Computational Science, King Abdullah University of Science and Technology, Thuwal, 23955-6900, Saudi Arabia.
| |
Collapse
|
2
|
Pan X, Pu W, Liu Y, Xiao Y, Pu J, Shi Y, Wu H, Wang H. Self-Perceptional Soft Robotics by a Dielectric Elastomer. ACS APPLIED MATERIALS & INTERFACES 2024; 16:26797-26807. [PMID: 38722638 DOI: 10.1021/acsami.4c04700] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/24/2024]
Abstract
Soft robotics has been a rapidly growing field in recent decades due to its advantages of softness, deformability, and adaptability to various environments. However, the separation of perception and actuation in soft robot research hinders its progress toward compactness and flexibility. To address this limitation, we propose the use of a dielectric elastomer actuator (DEA), which exhibits both an actuation capability and perception stability. Specifically, we developed a DEA array to localize the 3D spatial position of objects. Subsequently, we integrate the actuation and sensing properties of DEA into soft robots to achieve self-perception. We have developed a system that integrates actuation and sensing and have proposed two modes to achieve this integration. Furthermore, we demonstrated the feasibility of this system for soft robots. When the robots detect an obstacle or an approaching object, they can swiftly respond by avoiding or escaping the obstacle. By eliminating the need for separate perception and motion considerations, self-perceptional soft robots can achieve an enhanced response performance and enable applications in a more compact and flexible field.
Collapse
Affiliation(s)
- Xinghai Pan
- School of Aeronautics and Astronautics, Sichuan University, Chengdu 610065, China
| | - Wei Pu
- School of Aeronautics and Astronautics, Sichuan University, Chengdu 610065, China
| | - Yanling Liu
- School of Aeronautics and Astronautics, Sichuan University, Chengdu 610065, China
| | - Yuhang Xiao
- School of Aeronautics and Astronautics, Sichuan University, Chengdu 610065, China
| | - Junhong Pu
- School of Fashion and Textiles, The Hong Kong Polytechnic University, Hong Kong 999077, China
| | - Ye Shi
- ZJU-UIUC Institute, Zhejiang University, Zhejiang 314400, China
| | - Hui Wu
- State Key Laboratory of New Ceramics and Fine Processing, School of Materials Science and Engineering, Tsinghua University, Beijing 100084, China
| | - Haolun Wang
- School of Aeronautics and Astronautics, Sichuan University, Chengdu 610065, China
| |
Collapse
|
3
|
Müller R. Bioinspiration from bats and new paradigms for autonomy in natural environments. BIOINSPIRATION & BIOMIMETICS 2024; 19:033001. [PMID: 38452384 DOI: 10.1088/1748-3190/ad311e] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/02/2023] [Accepted: 03/07/2024] [Indexed: 03/09/2024]
Abstract
Achieving autonomous operation in complex natural environment remains an unsolved challenge. Conventional engineering approaches to this problem have focused on collecting large amounts of sensory data that are used to create detailed digital models of the environment. However, this only postpones solving the challenge of identifying the relevant sensory information and linking it to action control to the domain of the digital world model. Furthermore, it imposes high demands in terms of computing power and introduces large processing latencies that hamper autonomous real-time performance. Certain species of bats that are able to navigate and hunt their prey in dense vegetation could be a biological model system for an alternative approach to addressing the fundamental issues associated with autonomy in complex natural environments. Bats navigating in dense vegetation rely on clutter echoes, i.e. signals that consist of unresolved contributions from many scatters. Yet, the animals are able to extract the relevant information from these input signals with brains that are often less than 1 g in mass. Pilot results indicate that information relevant to location identification and passageway finding can be directly obtained from clutter echoes, opening up the possibility that the bats' skill can be replicated in man-made autonomous systems.
Collapse
Affiliation(s)
- Rolf Müller
- Department of Mechanical Engineering, Virginia Tech, Blacksburg, VA 24061, United States of America
| |
Collapse
|
4
|
Chen Y, Li T, Wang Z, Yan Z, De Vita R, Tan T. A Metamaterial Computational Multi-Sensor of Grip-Strength Properties with Point-of-Care Human-Computer Interaction. ADVANCED SCIENCE (WEINHEIM, BADEN-WURTTEMBERG, GERMANY) 2023; 10:e2304091. [PMID: 37818760 PMCID: PMC10700692 DOI: 10.1002/advs.202304091] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/20/2023] [Revised: 08/11/2023] [Indexed: 10/13/2023]
Abstract
Grip strength is a biomarker of frailty and an evaluation indicator of brain health, cardiovascular morbidity, and psychological health. Yet, the development of a reliable, interactive, and point-of-care device for comprehensive multi-sensing of hand grip status is challenging. Here, a relation between soft buckling metamaterial deformations and built piezoelectric voltage signals is uncovered to achieve multiple sensing of maximal grip force, grip speed, grip impulse, and endurance indicators. A metamaterial computational sensor design is established by hyperelastic model that governs the mechanical characterization, machine learning models for computational sensing, and graphical user interface to provide visual cues. A exemplify grip measurement for left and right hands of seven elderly campus workers is conducted. By taking indicators of grip status as input parameters, human-computer interactive games are incorporated into the computational sensor to improve the user compliance with measurement protocols. Two elderly female schizophrenic patients are participated in the real-time interactive point-of-care grip assessment and training for potentially sarcopenia screening. The attractive features of this advanced intelligent metamaterial computational sensing system are crucial to establish a point-of-care biomechanical platform and advancing the human-computer interactive healthcare, ultimately contributing to a global health ecosystem.
Collapse
Affiliation(s)
- Yinghua Chen
- State Key Laboratory of Mechanical System and VibrationSchool of Mechanical EngineeringShanghai Jiao Tong UniversityShanghai200240P. R. China
| | - Tianrun Li
- State Key Laboratory of Mechanical System and VibrationSchool of Mechanical EngineeringShanghai Jiao Tong UniversityShanghai200240P. R. China
| | - Zhemin Wang
- State Key Laboratory of Mechanical System and VibrationSchool of Mechanical EngineeringShanghai Jiao Tong UniversityShanghai200240P. R. China
| | - Zhimiao Yan
- State Key Laboratory of Ocean EngineeringDepartment of MechanicsSchool of Naval ArchitectureOcean & Civil EngineeringShanghai Jiao Tong UniversityShanghai200240P. R. China
| | - Raffaella De Vita
- Department of Biomedical Engineering and MechanicsVirginia TechBlacksburgVA24061USA
| | - Ting Tan
- State Key Laboratory of Mechanical System and VibrationSchool of Mechanical EngineeringShanghai Jiao Tong UniversityShanghai200240P. R. China
| |
Collapse
|
5
|
Capshaw G, Brown AD, Peña JL, Carr CE, Christensen-Dalsgaard J, Tollin DJ, Womack MC, McCullagh EA. The continued importance of comparative auditory research to modern scientific discovery. Hear Res 2023; 433:108766. [PMID: 37084504 PMCID: PMC10321136 DOI: 10.1016/j.heares.2023.108766] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/01/2022] [Revised: 03/23/2023] [Accepted: 04/05/2023] [Indexed: 04/23/2023]
Abstract
A rich history of comparative research in the auditory field has afforded a synthetic view of sound information processing by ears and brains. Some organisms have proven to be powerful models for human hearing due to fundamental similarities (e.g., well-matched hearing ranges), while others feature intriguing differences (e.g., atympanic ears) that invite further study. Work across diverse "non-traditional" organisms, from small mammals to avians to amphibians and beyond, continues to propel auditory science forward, netting a variety of biomedical and technological advances along the way. In this brief review, limited primarily to tetrapod vertebrates, we discuss the continued importance of comparative studies in hearing research from the periphery to central nervous system with a focus on outstanding questions such as mechanisms for sound capture, peripheral and central processing of directional/spatial information, and non-canonical auditory processing, including efferent and hormonal effects.
Collapse
Affiliation(s)
- Grace Capshaw
- Department of Psychological and Brain Sciences, Johns Hopkins University, Baltimore, MD 21218, USA.
| | - Andrew D Brown
- Department of Speech and Hearing Sciences, University of Washington, Seattle, WA 98105, USA
| | - José L Peña
- Dominick P. Purpura Department of Neuroscience, Albert Einstein College of Medicine, Bronx, NY 10461, USA
| | - Catherine E Carr
- Department of Biology, University of Maryland, College Park, MD 20742, USA
| | | | - Daniel J Tollin
- Department of Physiology and Biophysics, University of Colorado Anschutz Medical Campus, Aurora, CO 80045, USA; Department of Otolaryngology, University of Colorado Anschutz Medical Campus, Aurora, CO 80045, USA
| | - Molly C Womack
- Department of Biology, Utah State University, Logan, UT 84322, USA.
| | - Elizabeth A McCullagh
- Department of Integrative Biology, Oklahoma State University, Stillwater, OK 74078, USA.
| |
Collapse
|
6
|
Hiraga T, Yamada Y, Kobayashi R. Theoretical investigation of active listening behavior based on the echolocation of CF-FM bats. PLoS Comput Biol 2022; 18:e1009784. [PMID: 36206507 PMCID: PMC9581360 DOI: 10.1371/journal.pcbi.1009784] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/23/2021] [Revised: 10/19/2022] [Accepted: 09/26/2022] [Indexed: 11/18/2022] Open
Abstract
Bats perceive the three-dimensional environment by emitting ultrasound pulses from their nose or mouth and receiving echoes through both ears. To determine the position of a target object, it is necessary to know the distance and direction of the target. Certain bat species that use a combined signal of long constant frequency and short frequency modulated ultrasounds synchronize their pinnae movement with pulse emission, and this behavior has been regarded as helpful for localizing the elevation angle of a reflective sound source. However, the significance of bats' ear motions remains unclear. In this study, we construct a model of an active listening system including the motion of the ears, and conduct mathematical investigations to clarify the importance of ear motion in direction detection of the reflective sound source. In the simulations, direction detection under rigid ear movements with interaural level differences was mathematically investigated by assuming that bats accomplish direction detection using the amplitude modulation in the echoes caused by ear movements. In particular, the ear motion conditions required for direction detection are theoretically investigated through exhaustive simulations of the pseudo-motion of the ears, rather than simulations of the actual ear motions of bats. The theory suggests that only certain ear motions, namely three-axis rotation, allow for accurate and robust direction detection. Our theoretical analysis also strongly supports the behavior whereby bats move their pinnae in the antiphase mode. In addition, we suggest that simple shaped hearing directionality and well-selected uncomplicated ear motions are sufficient to achieve precise and robust direction detection. Our findings and mathematical approach have the potential to be used in the design of active sensing systems in various engineering fields.
Collapse
Affiliation(s)
- Takahiro Hiraga
- Department of Mathematical and Life Sciences, Hiroshima University, Department of Sciences, Higashi-Hiroshima, Japan
| | - Yasufumi Yamada
- Program of Mathematical and Life Sciences, Hiroshima University, Department of Sciences, Higashi-Hiroshima, Japan
- * E-mail:
| | - Ryo Kobayashi
- Program of Mathematical and Life Sciences, Hiroshima University, Department of Sciences, Higashi-Hiroshima, Japan
| |
Collapse
|
7
|
Yang Q, Liu N, Yin J, Tian H, Yang Y, Ren TL. Understanding the Origin of Tensile Response in a Graphene Textile Strain Sensor with Negative Differential Resistance. ACS NANO 2022; 16:14230-14238. [PMID: 36094408 DOI: 10.1021/acsnano.2c04348] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
The flexible strain sensors based on the textile substrate have natural flexibility, high sensitivity, and wide-range tensile response. However, the textile's complex and anisotropic substructure leads to a negative differential resistance (NDR) response, lacking a deeper understanding of the mechanism. Therefore, we examined a graphene textile strain sensor with a conspicuous NDR tensile response, providing a requisite research platform for mechanism investigation. The pioneering measurement of single fiber bundles confirmed the existence of the NDR effect on the subgeometry scale. Based on the in situ characterization of tensile morphology and measurement, we conducted quantitative behavior analyses to reveal the origin of tensile electrical responses in the full range comprehensively. The results showed that the dominant factor in generating the NDR effect is the relative displacement of fibers within the textile bundles. Based on the neural spiking-like tensile response, we further demonstrated the application potential of the textile strain sensor in threshold detection and near-sensor signal processing. The proposed NDR behavior model would provide a reference for the design and application of wearable intelligent textiles.
Collapse
Affiliation(s)
- Qisheng Yang
- School of Integrated Circuits & Beijing National Research on Information Science and Technology (BNRist), Tsinghua University, Beijing 100084, China
| | - Ning Liu
- School of Integrated Circuits & Beijing National Research on Information Science and Technology (BNRist), Tsinghua University, Beijing 100084, China
| | - Jiaju Yin
- School of Integrated Circuits & Beijing National Research on Information Science and Technology (BNRist), Tsinghua University, Beijing 100084, China
- School of Aerospace Engineering, Tsinghua University, Beijing 100084, China
| | - He Tian
- School of Integrated Circuits & Beijing National Research on Information Science and Technology (BNRist), Tsinghua University, Beijing 100084, China
| | - Yi Yang
- School of Integrated Circuits & Beijing National Research on Information Science and Technology (BNRist), Tsinghua University, Beijing 100084, China
| | - Tian-Ling Ren
- School of Integrated Circuits & Beijing National Research on Information Science and Technology (BNRist), Tsinghua University, Beijing 100084, China
| |
Collapse
|
8
|
Yue Y, Tang Y, Wang Q, Xiao W, Liu J, Wang J, Chen M, Wu G, Su B. Active Perception in Non-Visual Recognition Environments by Stretchable Tentacle Sensor Arrays. ACS APPLIED MATERIALS & INTERFACES 2022; 14:26913-26922. [PMID: 35666640 DOI: 10.1021/acsami.2c04717] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Smoke fog or other light-interference environments have intrinsic obstruction for visual recognition techniques to explore objects and surroundings. Alternatively, tactile perceptions, rather than visual observations, are commonly used by burrowing or deep-sea animals to communicate with environments. Bio-inspired by this natural wisdom, here, we demonstrate stretchable tentacle sensor arrays, which can recognize surrounding objects located in non-visual conditions such as smoke fog or dark environment. Each tentacle sensor is composed of two functional parts: a retractable tentacle with a magnetic top and an elastomer bottom containing copper coils. Different from traditionally passive tactile sensors, these tentacle sensors can actively stretch under the control of a syringe pump, yielding different electrical signals when in contact with the objects. Analyzing collected sensing signals of those tactile sensor arrays by the feature analysis model, complex morphological information of irregular objects in the smoke fog can be recognized. Our study reveals a fundamental connection between stretchable tactile sensors and feature analysis and demonstrates its practical potential for active perception in a non-visual recognition environment.
Collapse
Affiliation(s)
- Yamei Yue
- State Key Laboratory of Material Processing and Die & Mould Technology, School of Materials Science and Engineering, Huazhong University of Science and Technology, Wuhan 430074, Hubei, P. R. China
- Shenzhen Huazhong University of Science and Technology Research Institute, Shenzhen 518057, Guangdong, P. R. China
| | - Yuan Tang
- School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074, Hubei, P. R. China
| | - Qi Wang
- State Key Laboratory of Advanced Electromagnetic Engineering and Technology, School of Electrical and Electronic Engineering, Huazhong University of Science and Technology, Wuhan 430074, Hubei, P. R. China
| | - Wenjing Xiao
- School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074, Hubei, P. R. China
| | - Jia Liu
- School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074, Hubei, P. R. China
| | - Jiaxi Wang
- School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074, Hubei, P. R. China
| | - Min Chen
- School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074, Hubei, P. R. China
- Sport and Health Initiative, Optical Valley Laboratory, Wuhan 430074, Hubei, P. R. China
| | - Gaoxiang Wu
- School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074, Hubei, P. R. China
| | - Bin Su
- State Key Laboratory of Material Processing and Die & Mould Technology, School of Materials Science and Engineering, Huazhong University of Science and Technology, Wuhan 430074, Hubei, P. R. China
- Shenzhen Huazhong University of Science and Technology Research Institute, Shenzhen 518057, Guangdong, P. R. China
| |
Collapse
|