Integration of deep learning and soft robotics for a biomimetic approach to nonlinear sensing

Hardware-accelerated integrated optoelectronic platform towards real-time high-resolution hyperspectral video understanding

Recent advancements in artificial intelligence have significantly expanded capabilities in processing language and images. However, the challenge of comprehensively understanding video content still needs to be solved. The main problem is the requirement to process real-time multidimensional video information at data rates exceeding 1 Tb/s, a demand that current hardware technologies cannot meet. This work introduces a hardware-accelerated integrated optoelectronic platform specifically designed for the real-time analysis of multidimensional video. By leveraging optical information processing within artificial intelligence hardware and combining it with advanced machine vision networks, the platform achieves data processing speeds of 1.2 Tb/s. This capability supports the analysis of hundreds of frequency bands with megapixel spatial resolution at video frame rates, significantly outperforming existing technologies in speed by three to four orders of magnitude. The platform demonstrates effectiveness for AI-driven tasks, such as video semantic segmentation and object understanding, across indoor and aerial scenarios. By overcoming the current data processing speed limitations, the platform shows promise in real-time AI video understanding, with potential implications for enhancing human-machine interactions and advancing cognitive processing technologies.

Self-Perceptional Soft Robotics by a Dielectric Elastomer

Soft robotics has been a rapidly growing field in recent decades due to its advantages of softness, deformability, and adaptability to various environments. However, the separation of perception and actuation in soft robot research hinders its progress toward compactness and flexibility. To address this limitation, we propose the use of a dielectric elastomer actuator (DEA), which exhibits both an actuation capability and perception stability. Specifically, we developed a DEA array to localize the 3D spatial position of objects. Subsequently, we integrate the actuation and sensing properties of DEA into soft robots to achieve self-perception. We have developed a system that integrates actuation and sensing and have proposed two modes to achieve this integration. Furthermore, we demonstrated the feasibility of this system for soft robots. When the robots detect an obstacle or an approaching object, they can swiftly respond by avoiding or escaping the obstacle. By eliminating the need for separate perception and motion considerations, self-perceptional soft robots can achieve an enhanced response performance and enable applications in a more compact and flexible field.

Bioinspiration from bats and new paradigms for autonomy in natural environments

Achieving autonomous operation in complex natural environment remains an unsolved challenge. Conventional engineering approaches to this problem have focused on collecting large amounts of sensory data that are used to create detailed digital models of the environment. However, this only postpones solving the challenge of identifying the relevant sensory information and linking it to action control to the domain of the digital world model. Furthermore, it imposes high demands in terms of computing power and introduces large processing latencies that hamper autonomous real-time performance. Certain species of bats that are able to navigate and hunt their prey in dense vegetation could be a biological model system for an alternative approach to addressing the fundamental issues associated with autonomy in complex natural environments. Bats navigating in dense vegetation rely on clutter echoes, i.e. signals that consist of unresolved contributions from many scatters. Yet, the animals are able to extract the relevant information from these input signals with brains that are often less than 1 g in mass. Pilot results indicate that information relevant to location identification and passageway finding can be directly obtained from clutter echoes, opening up the possibility that the bats' skill can be replicated in man-made autonomous systems.

A Metamaterial Computational Multi-Sensor of Grip-Strength Properties with Point-of-Care Human-Computer Interaction

Grip strength is a biomarker of frailty and an evaluation indicator of brain health, cardiovascular morbidity, and psychological health. Yet, the development of a reliable, interactive, and point-of-care device for comprehensive multi-sensing of hand grip status is challenging. Here, a relation between soft buckling metamaterial deformations and built piezoelectric voltage signals is uncovered to achieve multiple sensing of maximal grip force, grip speed, grip impulse, and endurance indicators. A metamaterial computational sensor design is established by hyperelastic model that governs the mechanical characterization, machine learning models for computational sensing, and graphical user interface to provide visual cues. A exemplify grip measurement for left and right hands of seven elderly campus workers is conducted. By taking indicators of grip status as input parameters, human-computer interactive games are incorporated into the computational sensor to improve the user compliance with measurement protocols. Two elderly female schizophrenic patients are participated in the real-time interactive point-of-care grip assessment and training for potentially sarcopenia screening. The attractive features of this advanced intelligent metamaterial computational sensing system are crucial to establish a point-of-care biomechanical platform and advancing the human-computer interactive healthcare, ultimately contributing to a global health ecosystem.

The continued importance of comparative auditory research to modern scientific discovery

A rich history of comparative research in the auditory field has afforded a synthetic view of sound information processing by ears and brains. Some organisms have proven to be powerful models for human hearing due to fundamental similarities (e.g., well-matched hearing ranges), while others feature intriguing differences (e.g., atympanic ears) that invite further study. Work across diverse "non-traditional" organisms, from small mammals to avians to amphibians and beyond, continues to propel auditory science forward, netting a variety of biomedical and technological advances along the way. In this brief review, limited primarily to tetrapod vertebrates, we discuss the continued importance of comparative studies in hearing research from the periphery to central nervous system with a focus on outstanding questions such as mechanisms for sound capture, peripheral and central processing of directional/spatial information, and non-canonical auditory processing, including efferent and hormonal effects.

Theoretical investigation of active listening behavior based on the echolocation of CF-FM bats

Bats perceive the three-dimensional environment by emitting ultrasound pulses from their nose or mouth and receiving echoes through both ears. To determine the position of a target object, it is necessary to know the distance and direction of the target. Certain bat species that use a combined signal of long constant frequency and short frequency modulated ultrasounds synchronize their pinnae movement with pulse emission, and this behavior has been regarded as helpful for localizing the elevation angle of a reflective sound source. However, the significance of bats' ear motions remains unclear. In this study, we construct a model of an active listening system including the motion of the ears, and conduct mathematical investigations to clarify the importance of ear motion in direction detection of the reflective sound source. In the simulations, direction detection under rigid ear movements with interaural level differences was mathematically investigated by assuming that bats accomplish direction detection using the amplitude modulation in the echoes caused by ear movements. In particular, the ear motion conditions required for direction detection are theoretically investigated through exhaustive simulations of the pseudo-motion of the ears, rather than simulations of the actual ear motions of bats. The theory suggests that only certain ear motions, namely three-axis rotation, allow for accurate and robust direction detection. Our theoretical analysis also strongly supports the behavior whereby bats move their pinnae in the antiphase mode. In addition, we suggest that simple shaped hearing directionality and well-selected uncomplicated ear motions are sufficient to achieve precise and robust direction detection. Our findings and mathematical approach have the potential to be used in the design of active sensing systems in various engineering fields.

Understanding the Origin of Tensile Response in a Graphene Textile Strain Sensor with Negative Differential Resistance

The flexible strain sensors based on the textile substrate have natural flexibility, high sensitivity, and wide-range tensile response. However, the textile's complex and anisotropic substructure leads to a negative differential resistance (NDR) response, lacking a deeper understanding of the mechanism. Therefore, we examined a graphene textile strain sensor with a conspicuous NDR tensile response, providing a requisite research platform for mechanism investigation. The pioneering measurement of single fiber bundles confirmed the existence of the NDR effect on the subgeometry scale. Based on the in situ characterization of tensile morphology and measurement, we conducted quantitative behavior analyses to reveal the origin of tensile electrical responses in the full range comprehensively. The results showed that the dominant factor in generating the NDR effect is the relative displacement of fibers within the textile bundles. Based on the neural spiking-like tensile response, we further demonstrated the application potential of the textile strain sensor in threshold detection and near-sensor signal processing. The proposed NDR behavior model would provide a reference for the design and application of wearable intelligent textiles.

Active Perception in Non-Visual Recognition Environments by Stretchable Tentacle Sensor Arrays

Smoke fog or other light-interference environments have intrinsic obstruction for visual recognition techniques to explore objects and surroundings. Alternatively, tactile perceptions, rather than visual observations, are commonly used by burrowing or deep-sea animals to communicate with environments. Bio-inspired by this natural wisdom, here, we demonstrate stretchable tentacle sensor arrays, which can recognize surrounding objects located in non-visual conditions such as smoke fog or dark environment. Each tentacle sensor is composed of two functional parts: a retractable tentacle with a magnetic top and an elastomer bottom containing copper coils. Different from traditionally passive tactile sensors, these tentacle sensors can actively stretch under the control of a syringe pump, yielding different electrical signals when in contact with the objects. Analyzing collected sensing signals of those tactile sensor arrays by the feature analysis model, complex morphological information of irregular objects in the smoke fog can be recognized. Our study reveals a fundamental connection between stretchable tactile sensors and feature analysis and demonstrates its practical potential for active perception in a non-visual recognition environment.