Biologically inspired artificial eyes and photonics

Anti-distortion bioinspired camera with an inhomogeneous photo-pixel array

The bioinspired camera, comprising a single lens and a curved image sensor-a photodiode array on a curved surface-, was born of flexible electronics. Its economical build lends itself well to space-constrained machine vision applications. The curved sensor, much akin to the retina, helps image focusing, but the curvature also creates a problem of image distortion, which can undermine machine vision tasks such as object recognition. Here we report an anti-distortion single-lens camera, where 4096 silicon photodiodes arrayed on a curved surface in a nonuniform pattern assimilated to the distorting optics are the key to anti-distortion engineering. That is, the photo-pixel distribution pattern itself is warped in the same manner as images are warped, which correctively reverses distortion. Acquired images feature no appreciable distortion across a 120° horizontal view, as confirmed by their neural-network recognition accuracies. This distortion correction via photo-pixel array reconfiguration is a form of in-sensor computing.

Stereoscopic artificial compound eyes for spatiotemporal perception in three-dimensional space

Arthropods' eyes are effective biological vision systems for object tracking and wide field of view because of their structural uniqueness; however, unlike mammalian eyes, they can hardly acquire the depth information of a static object because of their monocular cues. Therefore, most arthropods rely on motion parallax to track the object in three-dimensional (3D) space. Uniquely, the praying mantis (Mantodea) uses both compound structured eyes and a form of stereopsis and is capable of achieving object recognition in 3D space. Here, by mimicking the vision system of the praying mantis using stereoscopically coupled artificial compound eyes, we demonstrated spatiotemporal object sensing and tracking in 3D space with a wide field of view. Furthermore, to achieve a fast response with minimal latency, data storage/transportation, and power consumption, we processed the visual information at the edge of the system using a synaptic device and a federated split learning algorithm. The designed and fabricated stereoscopic artificial compound eye provides energy-efficient and accurate spatiotemporal object sensing and optical flow tracking. It exhibits a root mean square error of 0.3 centimeter, consuming only approximately 4 millijoules for sensing and tracking. These results are more than 400 times lower than conventional complementary metal-oxide semiconductor-based imaging systems. Our biomimetic imager shows the potential of integrating nature's unique design using hardware and software codesigned technology toward capabilities of edge computing and sensing.

Reconfigurable Microlens Array Enables Tunable Imaging Based on Shape Memory Polymers

Microlens arrays (MLAs) with a tunable imaging ability are core components of advanced micro-optical systems. Nevertheless, tunable MLAs generally suffer from high power consumption, an undeformable rigid body, large and complex systems, or limited focal length tunability. The combination of reconfigurable smart materials with MLAs may lead to distinct advantages including programmable deformation, remote manipulation, and multimodal tunability. However, unlike photopolymers that permit flexible structuring, the fabrication of tunable MLAs and compound eyes (CEs) based on transparent smart materials is still rare. In this work, we report reconfigurable MLAs that enable tunable imaging based on shape memory polymers (SMPs). The smart MLAs with closely packed 200 × 200 microlenses (40.0 μm in size) are fabricated via a combined technology that involves wet etching-assisted femtosecond laser direct writing of MLA templates on quartz, soft lithography for MLA duplication using SMPs, and the mechanical heat setting for programmable reconfiguration. By stretching or squeezing the shape memory MLAs at the transition temperature (80 °C), the size, profiles, and spatial distributions of the microlenses can be programmed. When the MLA is stretched from 0 to 120% (area ratio), the focal length is increased from 116 to 283 μm. As a proof of concept, reconfigurable MLAs and a 3D CE with a tunable field of view (FOV, 160-0°) have been demonstrated in which the thermally triggered shape memory deformation has been employed for tunable imaging. The reconfigurable MLAs and CEs with a tunable focal length and adjustable FOV may hold great promise for developing smart micro-optical systems.

Hierarchies in Visual Pathway: Functions and Inspired Artificial Vision

The development of artificial intelligence has posed a challenge to machine vision based on conventional complementary metal-oxide semiconductor (CMOS) circuits owing to its high latency and inefficient power consumption originating from the data shuffling between memory and computation units. Gaining more insights into the function of every part of the visual pathway for visual perception can bring the capabilities of machine vision in terms of robustness and generality. Hardware acceleration of more energy-efficient and biorealistic artificial vision highly necessitates neuromorphic devices and circuits that are able to mimic the function of each part of the visual pathway. In this paper, we review the structure and function of the entire class of visual neurons from the retina to the primate visual cortex within reach (Chapter 2) are reviewed. Based on the extraction of biological principles, the recent hardware-implemented visual neurons located in different parts of the visual pathway are discussed in detail in Chapters 3 and 4. Furthermore, valuable applications of inspired artificial vision in different scenarios (Chapter 5) are provided. The functional description of the visual pathway and its inspired neuromorphic devices/circuits are expected to provide valuable insights for the design of next-generation artificial visual perception systems.

Nanomaterial-Based Artificial Vision Systems: From Bioinspired Electronic Eyes to In-Sensor Processing Devices

High-performance robotic vision empowers mobile and humanoid robots to detect and identify their surrounding objects efficiently, which enables them to cooperate with humans and assist human activities. For error-free execution of these robots' tasks, efficient imaging and data processing capabilities are essential, even under diverse and complex environments. However, conventional technologies fall short of meeting the high-standard requirements of robotic vision under such circumstances. Here, we discuss recent progress in artificial vision systems with high-performance imaging and data processing capabilities enabled by distinctive electrical, optical, and mechanical characteristics of nanomaterials surpassing the limitations of traditional silicon technologies. In particular, we focus on nanomaterial-based electronic eyes and in-sensor processing devices inspired by biological eyes and animal visual recognition systems, respectively. We provide perspectives on key nanomaterials, device components, and their functionalities, as well as explain the remaining challenges and future prospects of the artificial vision systems.

Development of a bio-inspired optical system that mimics accommodation and lighting regulation like the human eye

Bio-inspired optical systems have recently been developed using polarizers and liquid or rigid lenses. In this work, we propose a bio-inspired opto-mechatronic system that imitates the accommodation and regulation of light intensity as the human eye does. The system uses a polymeric lens as a cornea, an adjustable diaphragm as an iris, a tunable solid elastic lens as a crystalline lens, and a commercial sensor as a retina. We also present the development of the electronic control system to accommodate and regulate the amount of light that enters the system, for which two stepper motors, an Arduino control system, and light and movement sensors are used. The characterization of the system is presented together with the results obtained, where it can be seen that the system works in an acceptable range as the human eye does.

Bio-inspired optical structures for enhancing luminescence

Luminescence is an essential signal for many plants, insects, and marine organisms to attract the opposite sex, avoid predators, and so on. Most luminescent living organisms have ingenious optical structures which can help them get high luminescent performances. These remarkable and efficient structures have been formed by natural selection from long-time evolution. Researchers keenly observed the enhanced luminescence phenomena and studied how these phenomena happen in order to learn the characteristics of bio-photonics. In this review, we summarize the optical structures for enhancing luminescence and their applications. The structures are classified according to their different functions. We focus on how researchers use these biological inspirations to enhance different luminescence processes, such as chemiluminescence (CL), photoluminescence (PL), and electroluminescence (EL). It lays a foundation for further research on the applications of luminescence enhancement. Furthermore, we give examples of luminescence enhancement by bio-inspired structures in information encryption, biochemical detection, and light sources. These examples show that it is possible to use bio-inspired optical structures to solve complex problems in optical applications. Our work will provide guidance for research on biomimetic optics, micro- and nano-optical structures, and enhanced luminescence.

Double-Glued Multi-Focal Bionic Compound Eye Camera

Compound eye cameras are a vital component of bionics. Compound eye lenses are currently used in light field cameras, monitoring imaging, medical endoscopes, and other fields. However, the resolution of the compound eye lens is still low at the moment, which has an impact on the application scene. Photolithography and negative pressure molding were used to create a double-glued multi-focal bionic compound eye camera in this study. The compound eye camera has 83 microlenses, with ommatidium diameters ranging from 400 μm to 660 μm, and a 92.3 degree field-of-view angle. The double-gluing structure significantly improves the optical performance of the compound eye lens, and the spatial resolution of the ommatidium is 57.00 lp mm-1. Additionally, the measurement of speed is investigated. This double-glue compound eye camera has numerous potential applications in the military, machine vision, and other fields.

Hierarchical Artificial Compound Eyes with Wide Field-of-View and Antireflection Properties Prepared by Nanotip-Focused Electrohydrodynamic Jet Printing

Artificial compound eyes (ACEs) endowed with durable superhydrophobicity, wide field-of-view (FOV), and antireflection properties are extremely appealing in advanced micro-optical systems. However, the simple and high-efficiency fabrication of ACEs with these functions is still a major challenge. Herein, inspired by moth eyes, ACEs with hierarchical macro/micro/nano structures were fabricated using the combination of nanotip-focused electrohydrodynamic jet (NFEJ) printing and air-assisted deformation processes. The NFEJ printing enables the direct and maskless fabrication of hierarchical micro/nanolens arrays (M/NLAs) without intermediate steps. The introduction of M/NLAs on the eye surface significantly improves the water hydrophobic performance with a water contact angle of 161.1° and contact angle hysteresis (CAH) of 4.2° and generally decreases the reflectance by 51% in the wavelength range of 350-1600 nm in comparison to the macroeye without any structures. The contact angle remains almost unchanged, and the CAH slightly increases from 4.2° to 8.7° after water jet impact for 20 min, indicating a durable superhydrophobicity. Moreover, the results confirm that the durable superhydrophobic ACEs with antireflection properties exhibit excellent imaging quality and a large FOV of up to 160° without distortion.

Biomimetic apposition compound eye fabricated using microfluidic-assisted 3D printing

After half a billion years of evolution, arthropods have developed sophisticated compound eyes with extraordinary visual capabilities that have inspired the development of artificial compound eyes. However, the limited 2D nature of most traditional fabrication techniques makes it challenging to directly replicate these natural systems. Here, we present a biomimetic apposition compound eye fabricated using a microfluidic-assisted 3D-printing technique. Each microlens is connected to the bottom planar surface of the eye via intracorporal, zero-crosstalk refractive-index-matched waveguides to mimic the rhabdoms of a natural eye. Full-colour wide-angle panoramic views and position tracking of a point source are realized by placing the fabricated eye directly on top of a commercial imaging sensor. As a biomimetic analogue to naturally occurring compound eyes, the eye's full-colour 3D to 2D mapping capability has the potential to enable a wide variety of applications from improving endoscopic imaging to enhancing machine vision for facilitating human-robot interactions.

Retina-like Imaging and Its Applications: A Brief Review

The properties of the human eye retina, including space-variant resolution and gaze characters, provide many advantages for numerous applications that simultaneously require a large field of view, high resolution, and real-time performance. Therefore, retina-like mechanisms and sensors have received considerable attention in recent years. This paper provides a review of state-of-the-art retina-like imaging techniques and applications. First, we introduce the principle and implementing methods, including software and hardware, and describe the comparisons between them. Then, we present typical applications combined with retina-like imaging, including three-dimensional acquisition and reconstruction, target tracking, deep learning, and ghost imaging. Finally, the challenges and outlook are discussed to further study for practical use. The results are beneficial for better understanding retina-like imaging.

Stretchable, Rehealable, Recyclable, and Reconfigurable Integrated Strain Sensor for Joint Motion and Respiration Monitoring

Cutting-edge technologies of stretchable, skin-mountable, and wearable electronics have attracted tremendous attention recently due to their very wide applications and promising performances. One direction of particular interest is to investigate novel properties in stretchable electronics by exploring multifunctional materials. Here, we report an integrated strain sensing system that is highly stretchable, rehealable, fully recyclable, and reconfigurable. This system consists of dynamic covalent thermoset polyimine as the moldable substrate and encapsulation, eutectic liquid metal alloy as the strain sensing unit and interconnects, and off-the-shelf chip components for measuring and magnifying functions. The device can be attached on different parts of the human body for accurately monitoring joint motion and respiration. Such a strain sensing system provides a reliable, economical, and ecofriendly solution to wearable technologies, with wide applications in health care, prosthetics, robotics, and biomedical devices.

Materials, Actuators, and Sensors for Soft Bioinspired Robots

Biological systems can perform complex tasks with high compliance levels. This makes them a great source of inspiration for soft robotics. Indeed, the union of these fields has brought about bioinspired soft robotics, with hundreds of publications on novel research each year. This review aims to survey fundamental advances in bioinspired soft actuators and sensors with a focus on the progress between 2017 and 2020, providing a primer for the materials used in their design.

Soft Tunable Lenses Based on Zipping Electroactive Polymer Actuators

Compact and entirely soft optics with tunable and adaptive properties drive the development of life-like soft robotic systems. Yet, existing approaches are either slow, require rigid components, or use high operating voltages of several kilovolts. Here, soft focus-tunable lenses are introduced, which operate at practical voltages, cover a high range of adjustable focal lengths, and feature response times in the milliseconds range. The nature-inspired design comprises a liquid-filled elastomeric lens membrane, which is inflated by zipping electroactive polymers to tune the focal length. An analytic description of the tunable lens supports optimized designs and accurate prediction of the lens characteristics. Focal length changes between 22 and 550 mm (numerical aperture 0.14-0.005) within 260 ms, equal in performance to human eyes, are demonstrated for a lens with 3 mm aperture radius, while applying voltages below 500 V. The presented model, design rules, and fabrication methods address central challenges of soft electrostatic actuators and optical systems, and pave the way toward autonomous bio-inspired robots and machines.

Tolerancing and characterization of curved image sensor systems

Curved image sensors, not having to correct the field curvature, are considered a relevant solution for improving the vast majority of optical systems. They offer the possibility of designing compact aberration-free optical systems. In this work, we explain the advantage of the curved sensor system using the aberration theory. A complete procedure was developed to produce functional curved sensors and functional prototypes were carried out. This paper focuses on the tolerancing process of curved sensors and its inclusion in optical design. A compact objective prototype designed and produced demonstrates the advantage of curvature and the impact of tolerancing.