1
|
Bai B, Yang X, Gan T, Li J, Mengu D, Jarrahi M, Ozcan A. Pyramid diffractive optical networks for unidirectional image magnification and demagnification. LIGHT, SCIENCE & APPLICATIONS 2024; 13:178. [PMID: 39085224 PMCID: PMC11291656 DOI: 10.1038/s41377-024-01543-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/23/2024] [Revised: 07/18/2024] [Accepted: 07/19/2024] [Indexed: 08/02/2024]
Abstract
Diffractive deep neural networks (D2NNs) are composed of successive transmissive layers optimized using supervised deep learning to all-optically implement various computational tasks between an input and output field-of-view. Here, we present a pyramid-structured diffractive optical network design (which we term P-D2NN), optimized specifically for unidirectional image magnification and demagnification. In this design, the diffractive layers are pyramidally scaled in alignment with the direction of the image magnification or demagnification. This P-D2NN design creates high-fidelity magnified or demagnified images in only one direction, while inhibiting the image formation in the opposite direction-achieving the desired unidirectional imaging operation using a much smaller number of diffractive degrees of freedom within the optical processor volume. Furthermore, the P-D2NN design maintains its unidirectional image magnification/demagnification functionality across a large band of illumination wavelengths despite being trained with a single wavelength. We also designed a wavelength-multiplexed P-D2NN, where a unidirectional magnifier and a unidirectional demagnifier operate simultaneously in opposite directions, at two distinct illumination wavelengths. Furthermore, we demonstrate that by cascading multiple unidirectional P-D2NN modules, we can achieve higher magnification factors. The efficacy of the P-D2NN architecture was also validated experimentally using terahertz illumination, successfully matching our numerical simulations. P-D2NN offers a physics-inspired strategy for designing task-specific visual processors.
Collapse
Affiliation(s)
- Bijie Bai
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, USA
- Bioengineering Department, University of California, Los Angeles, CA, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, USA
| | - Xilin Yang
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, USA
- Bioengineering Department, University of California, Los Angeles, CA, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, USA
| | - Tianyi Gan
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, USA
| | - Jingxi Li
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, USA
- Bioengineering Department, University of California, Los Angeles, CA, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, USA
| | - Deniz Mengu
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, USA
- Bioengineering Department, University of California, Los Angeles, CA, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, USA
| | - Mona Jarrahi
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, USA
| | - Aydogan Ozcan
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, USA.
- Bioengineering Department, University of California, Los Angeles, CA, USA.
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, USA.
| |
Collapse
|
2
|
Huang Z, Shi W, Wu S, Wang Y, Yang S, Chen H. Pre-sensor computing with compact multilayer optical neural network. SCIENCE ADVANCES 2024; 10:eado8516. [PMID: 39058775 PMCID: PMC11277373 DOI: 10.1126/sciadv.ado8516] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/23/2024] [Accepted: 06/25/2024] [Indexed: 07/28/2024]
Abstract
Moving computation units closer to sensors is becoming a promising approach to addressing bottlenecks in computing speed, power consumption, and data storage. Pre-sensor computing with optical neural networks (ONNs) allows extensive processing. However, the lack of nonlinear activation and dependence on laser input limits the computational capacity, practicality, and scalability. A compact and passive multilayer ONN (MONN) is proposed, which has two convolution layers and an inserted nonlinear layer, performing pre-sensor computations with designed passive masks and a quantum dot film for incoherent light. MONN has an optical length as short as 5 millimeters, two orders of magnitude smaller than state-of-the-art lens-based ONNs. MONN outperforms linear single-layer ONN across various vision tasks, off-loading up to 95% of computationally expensive operations into optics from electronics. Motivated by MONN, a paradigm is emerging for mobile vision, fulfilling the demands for practicality, miniaturization, and low power consumption.
Collapse
Affiliation(s)
- Zheng Huang
- Department of Electronic Engineering, Tsinghua University, Beijing 100084, China
- Beijing National Research Center for Information Science and Technology, Beijing 100084, China
| | - Wanxin Shi
- Department of Electronic Engineering, Tsinghua University, Beijing 100084, China
- Beijing National Research Center for Information Science and Technology, Beijing 100084, China
| | - Shukai Wu
- Department of Electronic Engineering, Tsinghua University, Beijing 100084, China
- Beijing National Research Center for Information Science and Technology, Beijing 100084, China
| | - Yaode Wang
- Department of Electronic Engineering, Tsinghua University, Beijing 100084, China
- Beijing National Research Center for Information Science and Technology, Beijing 100084, China
| | - Sigang Yang
- Department of Electronic Engineering, Tsinghua University, Beijing 100084, China
- Beijing National Research Center for Information Science and Technology, Beijing 100084, China
| | - Hongwei Chen
- Department of Electronic Engineering, Tsinghua University, Beijing 100084, China
- Beijing National Research Center for Information Science and Technology, Beijing 100084, China
| |
Collapse
|
3
|
Li Y, Li J, Ozcan A. Nonlinear encoding in diffractive information processing using linear optical materials. LIGHT, SCIENCE & APPLICATIONS 2024; 13:173. [PMID: 39043641 PMCID: PMC11266679 DOI: 10.1038/s41377-024-01529-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/02/2024] [Revised: 07/11/2024] [Accepted: 07/12/2024] [Indexed: 07/25/2024]
Abstract
Nonlinear encoding of optical information can be achieved using various forms of data representation. Here, we analyze the performances of different nonlinear information encoding strategies that can be employed in diffractive optical processors based on linear materials and shed light on their utility and performance gaps compared to the state-of-the-art digital deep neural networks. For a comprehensive evaluation, we used different datasets to compare the statistical inference performance of simpler-to-implement nonlinear encoding strategies that involve, e.g., phase encoding, against data repetition-based nonlinear encoding strategies. We show that data repetition within a diffractive volume (e.g., through an optical cavity or cascaded introduction of the input data) causes the loss of the universal linear transformation capability of a diffractive optical processor. Therefore, data repetition-based diffractive blocks cannot provide optical analogs to fully connected or convolutional layers commonly employed in digital neural networks. However, they can still be effectively trained for specific inference tasks and achieve enhanced accuracy, benefiting from the nonlinear encoding of the input information. Our results also reveal that phase encoding of input information without data repetition provides a simpler nonlinear encoding strategy with comparable statistical inference accuracy to data repetition-based diffractive processors. Our analyses and conclusions would be of broad interest to explore the push-pull relationship between linear material-based diffractive optical systems and nonlinear encoding strategies in visual information processors.
Collapse
Affiliation(s)
- Yuhang Li
- Electrical & Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Jingxi Li
- Electrical & Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Aydogan Ozcan
- Electrical & Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA.
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA.
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA.
| |
Collapse
|
4
|
Lin X, Fu Y, Zhang K, Zhang X, Feng S, Hu X. Polarization and wavelength routers based on diffractive neural network. FRONTIERS OF OPTOELECTRONICS 2024; 17:22. [PMID: 39009949 PMCID: PMC11250754 DOI: 10.1007/s12200-024-00126-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/29/2024] [Accepted: 06/18/2024] [Indexed: 07/17/2024]
Abstract
In the field of information processing, all-optical routers are significant for achieving high-speed, high-capacity signal processing and transmission. In this study, we developed three types of structurally simple and flexible routers using the deep diffractive neural network (D2NN), capable of routing incident light based on wavelength and polarization. First, we implemented a polarization router for routing two orthogonally polarized light beams. The second type is the wavelength router that can route light with wavelengths of 1550, 1300, and 1100 nm, demonstrating outstanding performance with insertion loss as low as 0.013 dB and an extinction ratio of up to 18.96 dB, while also maintaining excellent polarization preservation. The final router is the polarization-wavelength composite router, capable of routing six types of input light formed by pairwise combinations of three wavelengths (1550, 1300, and 1100 nm) and two orthogonal linearly polarized lights, thereby enhancing the information processing capability of the device. These devices feature compact structures, maintaining high contrast while exhibiting low loss and passive characteristics, making them suitable for integration into future optical components. This study introduces new avenues and methodologies to enhance performance and broaden the applications of future optical information processing systems.
Collapse
Affiliation(s)
- Xiaohong Lin
- School of Physics and Optoelectronic Engineering, Beijing University of Technology, Beijing, 100124, China
| | - Yulan Fu
- School of Physics and Optoelectronic Engineering, Beijing University of Technology, Beijing, 100124, China.
| | - Kuo Zhang
- School of Science, Minzu University of China, Beijing, 100081, China
| | - Xinping Zhang
- School of Physics and Optoelectronic Engineering, Beijing University of Technology, Beijing, 100124, China
| | - Shuai Feng
- School of Science, Minzu University of China, Beijing, 100081, China
| | - Xiaoyong Hu
- State Key Laboratory for Mesoscopic Physics and Department of Physics, Collaborative Innovation Center of Quantum Matter, Beijing Academy of Quantum Information Sciences, Nano-Optoelectronics Frontier Center of Ministry of Education, Peking University, Beijing, 100871, China.
- Collaborative Innovation Center of Extreme Optics, Shanxi University, Taiyuan, 030006, China.
- Peking University Yangtze Delta Institute of Optoelectronics, Nantong, 226010, China.
| |
Collapse
|
5
|
Yan T, Zhou T, Guo Y, Zhao Y, Shao G, Wu J, Huang R, Dai Q, Fang L. Nanowatt all-optical 3D perception for mobile robotics. SCIENCE ADVANCES 2024; 10:eadn2031. [PMID: 38968351 PMCID: PMC11225784 DOI: 10.1126/sciadv.adn2031] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/28/2023] [Accepted: 06/03/2024] [Indexed: 07/07/2024]
Abstract
Three-dimensional (3D) perception is vital to drive mobile robotics' progress toward intelligence. However, state-of-the-art 3D perception solutions require complicated postprocessing or point-by-point scanning, suffering computational burden, latency of tens of milliseconds, and additional power consumption. Here, we propose a parallel all-optical computational chipset 3D perception architecture (Aop3D) with nanowatt power and light speed. The 3D perception is executed during the light propagation over the passive chipset, and the captured light intensity distribution provides a direct reflection of the depth map, eliminating the need for extensive postprocessing. The prototype system of Aop3D is tested in various scenarios and deployed to a mobile robot, demonstrating unprecedented performance in distance detection and obstacle avoidance. Moreover, Aop3D works at a frame rate of 600 hertz and a power consumption of 33.3 nanowatts per meta-pixel experimentally. Our work is promising toward next-generation direct 3D perception techniques with light speed and high energy efficiency.
Collapse
Affiliation(s)
- Tao Yan
- Department of Automation, Tsinghua University, Beijing 100084, China
| | - Tiankuang Zhou
- Department of Electronic Engineering, Tsinghua University, Beijing 100084, China
- Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China
| | - Yanchen Guo
- Department of Automation, Tsinghua University, Beijing 100084, China
- Shenzhen International Graduate School, Tsinghua University, Shenzhen 518071, China
| | - Yun Zhao
- Department of Automation, Tsinghua University, Beijing 100084, China
- Shenzhen International Graduate School, Tsinghua University, Shenzhen 518071, China
| | - Guocheng Shao
- Department of Automation, Tsinghua University, Beijing 100084, China
- Shenzhen International Graduate School, Tsinghua University, Shenzhen 518071, China
| | - Jiamin Wu
- Department of Automation, Tsinghua University, Beijing 100084, China
- Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing 100084, China
| | - Ruqi Huang
- Shenzhen International Graduate School, Tsinghua University, Shenzhen 518071, China
| | - Qionghai Dai
- Department of Automation, Tsinghua University, Beijing 100084, China
- Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing 100084, China
| | - Lu Fang
- Department of Electronic Engineering, Tsinghua University, Beijing 100084, China
- Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing 100084, China
| |
Collapse
|
6
|
Shen CY, Li J, Gan T, Li Y, Jarrahi M, Ozcan A. All-optical phase conjugation using diffractive wavefront processing. Nat Commun 2024; 15:4989. [PMID: 38862510 PMCID: PMC11166986 DOI: 10.1038/s41467-024-49304-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/30/2023] [Accepted: 05/30/2024] [Indexed: 06/13/2024] Open
Abstract
Optical phase conjugation (OPC) is a nonlinear technique used for counteracting wavefront distortions, with applications ranging from imaging to beam focusing. Here, we present a diffractive wavefront processor to approximate all-optical phase conjugation. Leveraging deep learning, a set of diffractive layers was optimized to all-optically process an arbitrary phase-aberrated input field, producing an output field with a phase distribution that is the conjugate of the input wave. We experimentally validated this wavefront processor by 3D-fabricating diffractive layers and performing OPC on phase distortions never seen during training. Employing terahertz radiation, our diffractive processor successfully performed OPC through a shallow volume that axially spans tens of wavelengths. We also created a diffractive phase-conjugate mirror by combining deep learning-optimized diffractive layers with a standard mirror. Given its compact, passive and multi-wavelength nature, this diffractive wavefront processor can be used for various applications, e.g., turbidity suppression and aberration correction across different spectral bands.
Collapse
Affiliation(s)
- Che-Yung Shen
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, USA
- Bioengineering Department, University of California, Los Angeles, CA, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, USA
| | - Jingxi Li
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, USA
- Bioengineering Department, University of California, Los Angeles, CA, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, USA
| | - Tianyi Gan
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, USA
| | - Yuhang Li
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, USA
- Bioengineering Department, University of California, Los Angeles, CA, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, USA
| | - Mona Jarrahi
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, USA
| | - Aydogan Ozcan
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, USA.
- Bioengineering Department, University of California, Los Angeles, CA, USA.
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, USA.
| |
Collapse
|
7
|
Li J, Li Y, Gan T, Shen CY, Jarrahi M, Ozcan A. All-optical complex field imaging using diffractive processors. LIGHT, SCIENCE & APPLICATIONS 2024; 13:120. [PMID: 38802376 PMCID: PMC11130282 DOI: 10.1038/s41377-024-01482-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/29/2024] [Revised: 05/11/2024] [Accepted: 05/13/2024] [Indexed: 05/29/2024]
Abstract
Complex field imaging, which captures both the amplitude and phase information of input optical fields or objects, can offer rich structural insights into samples, such as their absorption and refractive index distributions. However, conventional image sensors are intensity-based and inherently lack the capability to directly measure the phase distribution of a field. This limitation can be overcome using interferometric or holographic methods, often supplemented by iterative phase retrieval algorithms, leading to a considerable increase in hardware complexity and computational demand. Here, we present a complex field imager design that enables snapshot imaging of both the amplitude and quantitative phase information of input fields using an intensity-based sensor array without any digital processing. Our design utilizes successive deep learning-optimized diffractive surfaces that are structured to collectively modulate the input complex field, forming two independent imaging channels that perform amplitude-to-amplitude and phase-to-intensity transformations between the input and output planes within a compact optical design, axially spanning ~100 wavelengths. The intensity distributions of the output fields at these two channels on the sensor plane directly correspond to the amplitude and quantitative phase profiles of the input complex field, eliminating the need for any digital image reconstruction algorithms. We experimentally validated the efficacy of our complex field diffractive imager designs through 3D-printed prototypes operating at the terahertz spectrum, with the output amplitude and phase channel images closely aligning with our numerical simulations. We envision that this complex field imager will have various applications in security, biomedical imaging, sensing and material science, among others.
Collapse
Affiliation(s)
- Jingxi Li
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Yuhang Li
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Tianyi Gan
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Che-Yung Shen
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Mona Jarrahi
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Aydogan Ozcan
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA.
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA.
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA.
| |
Collapse
|
8
|
Wang X, Redding B, Karl N, Long C, Zhu Z, Skowronek J, Pang S, Brady D, Sarma R. Integrated photonic encoder for low power and high-speed image processing. Nat Commun 2024; 15:4510. [PMID: 38802333 PMCID: PMC11130346 DOI: 10.1038/s41467-024-48099-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2023] [Accepted: 04/17/2024] [Indexed: 05/29/2024] Open
Abstract
Modern lens designs are capable of resolving greater than 10 gigapixels, while advances in camera frame-rate and hyperspectral imaging have made data acquisition rates of Terapixel/second a real possibility. The main bottlenecks preventing such high data-rate systems are power consumption and data storage. In this work, we show that analog photonic encoders could address this challenge, enabling high-speed image compression using orders-of-magnitude lower power than digital electronics. Our approach relies on a silicon-photonics front-end to compress raw image data, foregoing energy-intensive image conditioning and reducing data storage requirements. The compression scheme uses a passive disordered photonic structure to perform kernel-type random projections of the raw image data with minimal power consumption and low latency. A back-end neural network can then reconstruct the original images with structural similarity exceeding 90%. This scheme has the potential to process data streams exceeding Terapixel/second using less than 100 fJ/pixel, providing a path to ultra-high-resolution data and image acquisition systems.
Collapse
Affiliation(s)
- Xiao Wang
- Wyant College of Optical Sciences, University of Arizona, Tucson, Arizona, USA
| | | | - Nicholas Karl
- Sandia National Laboratories, Albuquerque, New Mexico, USA
| | | | - Zheyuan Zhu
- CREOL, The College of Optics and Photonics, University of Central Floria, Orlando, Florida, USA
| | - James Skowronek
- Wyant College of Optical Sciences, University of Arizona, Tucson, Arizona, USA
| | - Shuo Pang
- CREOL, The College of Optics and Photonics, University of Central Floria, Orlando, Florida, USA
| | - David Brady
- Wyant College of Optical Sciences, University of Arizona, Tucson, Arizona, USA.
| | - Raktim Sarma
- Sandia National Laboratories, Albuquerque, New Mexico, USA.
- Center for Integrated Nanotechnologies, Sandia National Laboratories, Albuquerque, New Mexico, USA.
| |
Collapse
|
9
|
Xu Z, Zhou T, Ma M, Deng C, Dai Q, Fang L. Large-scale photonic chiplet Taichi empowers 160-TOPS/W artificial general intelligence. Science 2024; 384:202-209. [PMID: 38603505 DOI: 10.1126/science.adl1203] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/29/2023] [Accepted: 03/11/2024] [Indexed: 04/13/2024]
Abstract
The pursuit of artificial general intelligence (AGI) continuously demands higher computing performance. Despite the superior processing speed and efficiency of integrated photonic circuits, their capacity and scalability are restricted by unavoidable errors, such that only simple tasks and shallow models are realized. To support modern AGIs, we designed Taichi-large-scale photonic chiplets based on an integrated diffractive-interference hybrid design and a general distributed computing architecture that has millions-of-neurons capability with 160-tera-operations per second per watt (TOPS/W) energy efficiency. Taichi experimentally achieved on-chip 1000-category-level classification (testing at 91.89% accuracy in the 1623-category Omniglot dataset) and high-fidelity artificial intelligence-generated content with up to two orders of magnitude of improvement in efficiency. Taichi paves the way for large-scale photonic computing and advanced tasks, further exploiting the flexibility and potential of photonics for modern AGI.
Collapse
Affiliation(s)
- Zhihao Xu
- Sigma Laboratory, Department of Electronic Engineering, Tsinghua University, Beijing, China
- Beijing National Research Center for Information Science and Technology (BNRist), Beijing, China
- Tsinghua Shenzhen International Graduate School, Shenzhen, China
| | - Tiankuang Zhou
- Sigma Laboratory, Department of Electronic Engineering, Tsinghua University, Beijing, China
- Beijing National Research Center for Information Science and Technology (BNRist), Beijing, China
- Institute for Brain and Cognitive Science, Tsinghua University (THUIBCS), Beijing, China
| | - Muzhou Ma
- Sigma Laboratory, Department of Electronic Engineering, Tsinghua University, Beijing, China
| | - ChenChen Deng
- Beijing National Research Center for Information Science and Technology (BNRist), Beijing, China
| | - Qionghai Dai
- Beijing National Research Center for Information Science and Technology (BNRist), Beijing, China
- Institute for Brain and Cognitive Science, Tsinghua University (THUIBCS), Beijing, China
- Department of Automation, Tsinghua University, Beijing, China
| | - Lu Fang
- Sigma Laboratory, Department of Electronic Engineering, Tsinghua University, Beijing, China
- Beijing National Research Center for Information Science and Technology (BNRist), Beijing, China
- Institute for Brain and Cognitive Science, Tsinghua University (THUIBCS), Beijing, China
| |
Collapse
|
10
|
Cheng Y, Zhang J, Zhou T, Wang Y, Xu Z, Yuan X, Fang L. Photonic neuromorphic architecture for tens-of-task lifelong learning. LIGHT, SCIENCE & APPLICATIONS 2024; 13:56. [PMID: 38403652 PMCID: PMC10894876 DOI: 10.1038/s41377-024-01395-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/05/2023] [Revised: 01/08/2024] [Accepted: 01/24/2024] [Indexed: 02/27/2024]
Abstract
Scalable, high-capacity, and low-power computing architecture is the primary assurance for increasingly manifold and large-scale machine learning tasks. Traditional electronic artificial agents by conventional power-hungry processors have faced the issues of energy and scaling walls, hindering them from the sustainable performance improvement and iterative multi-task learning. Referring to another modality of light, photonic computing has been progressively applied in high-efficient neuromorphic systems. Here, we innovate a reconfigurable lifelong-learning optical neural network (L2ONN), for highly-integrated tens-of-task machine intelligence with elaborated algorithm-hardware co-design. Benefiting from the inherent sparsity and parallelism in massive photonic connections, L2ONN learns each single task by adaptively activating sparse photonic neuron connections in the coherent light field, while incrementally acquiring expertise on various tasks by gradually enlarging the activation. The multi-task optical features are parallelly processed by multi-spectrum representations allocated with different wavelengths. Extensive evaluations on free-space and on-chip architectures confirm that for the first time, L2ONN avoided the catastrophic forgetting issue of photonic computing, owning versatile skills on challenging tens-of-tasks (vision classification, voice recognition, medical diagnosis, etc.) with a single model. Particularly, L2ONN achieves more than an order of magnitude higher efficiency than the representative electronic artificial neural networks, and 14× larger capacity than existing optical neural networks while maintaining competitive performance on each individual task. The proposed photonic neuromorphic architecture points out a new form of lifelong learning scheme, permitting terminal/edge AI systems with light-speed efficiency and unprecedented scalability.
Collapse
Affiliation(s)
- Yuan Cheng
- Sigma Laboratory, Department of Electronic Engineering, Tsinghua University, Beijing, 100084, China
- Beijing National Research Center for Information Science and Technology (BNRist), Beijing, 100084, China
| | - Jianing Zhang
- Sigma Laboratory, Department of Electronic Engineering, Tsinghua University, Beijing, 100084, China
- Beijing National Research Center for Information Science and Technology (BNRist), Beijing, 100084, China
| | - Tiankuang Zhou
- Sigma Laboratory, Department of Electronic Engineering, Tsinghua University, Beijing, 100084, China
- Beijing National Research Center for Information Science and Technology (BNRist), Beijing, 100084, China
| | - Yuyan Wang
- Beijing National Research Center for Information Science and Technology (BNRist), Beijing, 100084, China
| | - Zhihao Xu
- Sigma Laboratory, Department of Electronic Engineering, Tsinghua University, Beijing, 100084, China
| | - Xiaoyun Yuan
- Sigma Laboratory, Department of Electronic Engineering, Tsinghua University, Beijing, 100084, China
- Institute for Brain and Cognitive Science, Tsinghua University (THUIBCS), Beijing, 100084, China
| | - Lu Fang
- Sigma Laboratory, Department of Electronic Engineering, Tsinghua University, Beijing, 100084, China.
- Beijing National Research Center for Information Science and Technology (BNRist), Beijing, 100084, China.
- Institute for Brain and Cognitive Science, Tsinghua University (THUIBCS), Beijing, 100084, China.
| |
Collapse
|
11
|
Hu J, Mengu D, Tzarouchis DC, Edwards B, Engheta N, Ozcan A. Diffractive optical computing in free space. Nat Commun 2024; 15:1525. [PMID: 38378715 PMCID: PMC10879514 DOI: 10.1038/s41467-024-45982-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/23/2022] [Accepted: 02/09/2024] [Indexed: 02/22/2024] Open
Abstract
Structured optical materials create new computing paradigms using photons, with transformative impact on various fields, including machine learning, computer vision, imaging, telecommunications, and sensing. This Perspective sheds light on the potential of free-space optical systems based on engineered surfaces for advancing optical computing. Manipulating light in unprecedented ways, emerging structured surfaces enable all-optical implementation of various mathematical functions and machine learning tasks. Diffractive networks, in particular, bring deep-learning principles into the design and operation of free-space optical systems to create new functionalities. Metasurfaces consisting of deeply subwavelength units are achieving exotic optical responses that provide independent control over different properties of light and can bring major advances in computational throughput and data-transfer bandwidth of free-space optical processors. Unlike integrated photonics-based optoelectronic systems that demand preprocessed inputs, free-space optical processors have direct access to all the optical degrees of freedom that carry information about an input scene/object without needing digital recovery or preprocessing of information. To realize the full potential of free-space optical computing architectures, diffractive surfaces and metasurfaces need to advance symbiotically and co-evolve in their designs, 3D fabrication/integration, cascadability, and computing accuracy to serve the needs of next-generation machine vision, computational imaging, mathematical computing, and telecommunication technologies.
Collapse
Affiliation(s)
- Jingtian Hu
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Deniz Mengu
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Dimitrios C Tzarouchis
- Electrical and Systems Engineering, University of Pennsylvania, Philadelphia, PA, 19104, USA
- Meta Materials Inc., Athens, 15123, Greece
| | - Brian Edwards
- Electrical and Systems Engineering, University of Pennsylvania, Philadelphia, PA, 19104, USA
| | - Nader Engheta
- Electrical and Systems Engineering, University of Pennsylvania, Philadelphia, PA, 19104, USA
| | - Aydogan Ozcan
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA.
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA.
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA.
| |
Collapse
|
12
|
Işıl Ç, Gan T, Ardic FO, Mentesoglu K, Digani J, Karaca H, Chen H, Li J, Mengu D, Jarrahi M, Akşit K, Ozcan A. All-optical image denoising using a diffractive visual processor. LIGHT, SCIENCE & APPLICATIONS 2024; 13:43. [PMID: 38310118 PMCID: PMC10838318 DOI: 10.1038/s41377-024-01385-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Subscribe] [Scholar Register] [Received: 09/17/2023] [Revised: 01/14/2024] [Accepted: 01/15/2024] [Indexed: 02/05/2024]
Abstract
Image denoising, one of the essential inverse problems, targets to remove noise/artifacts from input images. In general, digital image denoising algorithms, executed on computers, present latency due to several iterations implemented in, e.g., graphics processing units (GPUs). While deep learning-enabled methods can operate non-iteratively, they also introduce latency and impose a significant computational burden, leading to increased power consumption. Here, we introduce an analog diffractive image denoiser to all-optically and non-iteratively clean various forms of noise and artifacts from input images - implemented at the speed of light propagation within a thin diffractive visual processor that axially spans <250 × λ, where λ is the wavelength of light. This all-optical image denoiser comprises passive transmissive layers optimized using deep learning to physically scatter the optical modes that represent various noise features, causing them to miss the output image Field-of-View (FoV) while retaining the object features of interest. Our results show that these diffractive denoisers can efficiently remove salt and pepper noise and image rendering-related spatial artifacts from input phase or intensity images while achieving an output power efficiency of ~30-40%. We experimentally demonstrated the effectiveness of this analog denoiser architecture using a 3D-printed diffractive visual processor operating at the terahertz spectrum. Owing to their speed, power-efficiency, and minimal computational overhead, all-optical diffractive denoisers can be transformative for various image display and projection systems, including, e.g., holographic displays.
Collapse
Affiliation(s)
- Çağatay Işıl
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Tianyi Gan
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Fazil Onuralp Ardic
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
| | - Koray Mentesoglu
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
| | - Jagrit Digani
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
| | - Huseyin Karaca
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
| | - Hanlong Chen
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Jingxi Li
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Deniz Mengu
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Mona Jarrahi
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Kaan Akşit
- University College London, Department of Computer Science, London, United Kingdom
| | - Aydogan Ozcan
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA.
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA.
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA.
| |
Collapse
|
13
|
Astratov VN, Sahel YB, Eldar YC, Huang L, Ozcan A, Zheludev N, Zhao J, Burns Z, Liu Z, Narimanov E, Goswami N, Popescu G, Pfitzner E, Kukura P, Hsiao YT, Hsieh CL, Abbey B, Diaspro A, LeGratiet A, Bianchini P, Shaked NT, Simon B, Verrier N, Debailleul M, Haeberlé O, Wang S, Liu M, Bai Y, Cheng JX, Kariman BS, Fujita K, Sinvani M, Zalevsky Z, Li X, Huang GJ, Chu SW, Tzang O, Hershkovitz D, Cheshnovsky O, Huttunen MJ, Stanciu SG, Smolyaninova VN, Smolyaninov II, Leonhardt U, Sahebdivan S, Wang Z, Luk’yanchuk B, Wu L, Maslov AV, Jin B, Simovski CR, Perrin S, Montgomery P, Lecler S. Roadmap on Label-Free Super-Resolution Imaging. LASER & PHOTONICS REVIEWS 2023; 17:2200029. [PMID: 38883699 PMCID: PMC11178318 DOI: 10.1002/lpor.202200029] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/17/2022] [Indexed: 06/18/2024]
Abstract
Label-free super-resolution (LFSR) imaging relies on light-scattering processes in nanoscale objects without a need for fluorescent (FL) staining required in super-resolved FL microscopy. The objectives of this Roadmap are to present a comprehensive vision of the developments, the state-of-the-art in this field, and to discuss the resolution boundaries and hurdles which need to be overcome to break the classical diffraction limit of the LFSR imaging. The scope of this Roadmap spans from the advanced interference detection techniques, where the diffraction-limited lateral resolution is combined with unsurpassed axial and temporal resolution, to techniques with true lateral super-resolution capability which are based on understanding resolution as an information science problem, on using novel structured illumination, near-field scanning, and nonlinear optics approaches, and on designing superlenses based on nanoplasmonics, metamaterials, transformation optics, and microsphere-assisted approaches. To this end, this Roadmap brings under the same umbrella researchers from the physics and biomedical optics communities in which such studies have often been developing separately. The ultimate intent of this paper is to create a vision for the current and future developments of LFSR imaging based on its physical mechanisms and to create a great opening for the series of articles in this field.
Collapse
Affiliation(s)
- Vasily N. Astratov
- Department of Physics and Optical Science, University of North Carolina at Charlotte, Charlotte, North Carolina 28223-0001, USA
| | - Yair Ben Sahel
- Department of Computer Science and Applied Mathematics, Weizmann Institute of Science, Rehovot 7610001, Israel
| | - Yonina C. Eldar
- Department of Computer Science and Applied Mathematics, Weizmann Institute of Science, Rehovot 7610001, Israel
| | - Luzhe Huang
- Electrical and Computer Engineering Department, University of California, Los Angeles, California 90095, USA
- Bioengineering Department, University of California, Los Angeles, California 90095, USA
- California Nano Systems Institute (CNSI), University of California, Los Angeles, California 90095, USA
| | - Aydogan Ozcan
- Electrical and Computer Engineering Department, University of California, Los Angeles, California 90095, USA
- Bioengineering Department, University of California, Los Angeles, California 90095, USA
- California Nano Systems Institute (CNSI), University of California, Los Angeles, California 90095, USA
- David Geffen School of Medicine, University of California, Los Angeles, California 90095, USA
| | - Nikolay Zheludev
- Optoelectronics Research Centre, University of Southampton, Southampton, SO17 1BJ, UK
- Centre for Disruptive Photonic Technologies, The Photonics Institute, School of Physical and Mathematical Sciences, Nanyang Technological University, 637371, Singapore
| | - Junxiang Zhao
- Department of Electrical and Computer Engineering, University of California, San Diego, 9500 Gilman Drive, La Jolla, California 92093, USA
| | - Zachary Burns
- Department of Electrical and Computer Engineering, University of California, San Diego, 9500 Gilman Drive, La Jolla, California 92093, USA
| | - Zhaowei Liu
- Department of Electrical and Computer Engineering, University of California, San Diego, 9500 Gilman Drive, La Jolla, California 92093, USA
- Material Science and Engineering, University of California, San Diego, 9500 Gilman Drive, La Jolla, California 92093, USA
| | - Evgenii Narimanov
- School of Electrical Engineering, and Birck Nanotechnology Center, Purdue University, West Lafayette, Indiana 47907, USA
| | - Neha Goswami
- Quantitative Light Imaging Laboratory, Beckman Institute of Advanced Science and Technology, University of Illinois at Urbana-Champaign, Illinois 61801, USA
| | - Gabriel Popescu
- Quantitative Light Imaging Laboratory, Beckman Institute of Advanced Science and Technology, University of Illinois at Urbana-Champaign, Illinois 61801, USA
| | - Emanuel Pfitzner
- Department of Chemistry, University of Oxford, Oxford OX1 3QZ, United Kingdom
| | - Philipp Kukura
- Department of Chemistry, University of Oxford, Oxford OX1 3QZ, United Kingdom
| | - Yi-Teng Hsiao
- Institute of Atomic and Molecular Sciences (IAMS), Academia Sinica 1, Roosevelt Rd. Sec. 4, Taipei 10617 Taiwan
| | - Chia-Lung Hsieh
- Institute of Atomic and Molecular Sciences (IAMS), Academia Sinica 1, Roosevelt Rd. Sec. 4, Taipei 10617 Taiwan
| | - Brian Abbey
- Australian Research Council Centre of Excellence for Advanced Molecular Imaging, La Trobe University, Melbourne, Victoria, Australia
- Department of Chemistry and Physics, La Trobe Institute for Molecular Science (LIMS), La Trobe University, Melbourne, Victoria, Australia
| | - Alberto Diaspro
- Optical Nanoscopy and NIC@IIT, CHT, Istituto Italiano di Tecnologia, Via Enrico Melen 83B, 16152 Genoa, Italy
- DIFILAB, Department of Physics, University of Genoa, Via Dodecaneso 33, 16146 Genoa, Italy
| | - Aymeric LeGratiet
- Optical Nanoscopy and NIC@IIT, CHT, Istituto Italiano di Tecnologia, Via Enrico Melen 83B, 16152 Genoa, Italy
- Université de Rennes, CNRS, Institut FOTON - UMR 6082, F-22305 Lannion, France
| | - Paolo Bianchini
- Optical Nanoscopy and NIC@IIT, CHT, Istituto Italiano di Tecnologia, Via Enrico Melen 83B, 16152 Genoa, Italy
- DIFILAB, Department of Physics, University of Genoa, Via Dodecaneso 33, 16146 Genoa, Italy
| | - Natan T. Shaked
- Tel Aviv University, Faculty of Engineering, Department of Biomedical Engineering, Tel Aviv 6997801, Israel
| | - Bertrand Simon
- LP2N, Institut d’Optique Graduate School, CNRS UMR 5298, Université de Bordeaux, Talence France
| | - Nicolas Verrier
- IRIMAS UR UHA 7499, Université de Haute-Alsace, Mulhouse, France
| | | | - Olivier Haeberlé
- IRIMAS UR UHA 7499, Université de Haute-Alsace, Mulhouse, France
| | - Sheng Wang
- School of Physics and Technology, Wuhan University, China
- Wuhan Institute of Quantum Technology, China
| | - Mengkun Liu
- Department of Physics and Astronomy, Stony Brook University, USA
- National Synchrotron Light Source II, Brookhaven National Laboratory, USA
| | - Yeran Bai
- Boston University Photonics Center, Boston, MA 02215, USA
| | - Ji-Xin Cheng
- Boston University Photonics Center, Boston, MA 02215, USA
| | - Behjat S. Kariman
- Optical Nanoscopy and NIC@IIT, CHT, Istituto Italiano di Tecnologia, Via Enrico Melen 83B, 16152 Genoa, Italy
- DIFILAB, Department of Physics, University of Genoa, Via Dodecaneso 33, 16146 Genoa, Italy
| | - Katsumasa Fujita
- Department of Applied Physics and the Advanced Photonics and Biosensing Open Innovation Laboratory (AIST); and the Transdimensional Life Imaging Division, Institute for Open and Transdisciplinary Research Initiatives, Osaka University, Osaka, Japan
| | - Moshe Sinvani
- Faculty of Engineering and the Nano-Technology Center, Bar-Ilan University, Ramat Gan, 52900 Israel
| | - Zeev Zalevsky
- Faculty of Engineering and the Nano-Technology Center, Bar-Ilan University, Ramat Gan, 52900 Israel
| | - Xiangping Li
- Guangdong Provincial Key Laboratory of Optical Fiber Sensing and Communications, Institute of Photonics Technology, Jinan University, Guangzhou 510632, China
| | - Guan-Jie Huang
- Department of Physics and Molecular Imaging Center, National Taiwan University, Taipei 10617, Taiwan
- Brain Research Center, National Tsing Hua University, Hsinchu 30013, Taiwan
| | - Shi-Wei Chu
- Department of Physics and Molecular Imaging Center, National Taiwan University, Taipei 10617, Taiwan
- Brain Research Center, National Tsing Hua University, Hsinchu 30013, Taiwan
| | - Omer Tzang
- School of Chemistry, The Sackler faculty of Exact Sciences, and the Center for Light matter Interactions, and the Tel Aviv University Center for Nanoscience and Nanotechnology, Tel Aviv 69978, Israel
| | - Dror Hershkovitz
- School of Chemistry, The Sackler faculty of Exact Sciences, and the Center for Light matter Interactions, and the Tel Aviv University Center for Nanoscience and Nanotechnology, Tel Aviv 69978, Israel
| | - Ori Cheshnovsky
- School of Chemistry, The Sackler faculty of Exact Sciences, and the Center for Light matter Interactions, and the Tel Aviv University Center for Nanoscience and Nanotechnology, Tel Aviv 69978, Israel
| | - Mikko J. Huttunen
- Laboratory of Photonics, Physics Unit, Tampere University, FI-33014, Tampere, Finland
| | - Stefan G. Stanciu
- Center for Microscopy – Microanalysis and Information Processing, Politehnica University of Bucharest, 313 Splaiul Independentei, 060042, Bucharest, Romania
| | - Vera N. Smolyaninova
- Department of Physics Astronomy and Geosciences, Towson University, 8000 York Rd., Towson, MD 21252, USA
| | - Igor I. Smolyaninov
- Department of Electrical and Computer Engineering, University of Maryland, College Park, MD 20742, USA
| | - Ulf Leonhardt
- Weizmann Institute of Science, Rehovot 7610001, Israel
| | - Sahar Sahebdivan
- EMTensor GmbH, TechGate, Donau-City-Strasse 1, 1220 Wien, Austria
| | - Zengbo Wang
- School of Computer Science and Electronic Engineering, Bangor University, Bangor, LL57 1UT, United Kingdom
| | - Boris Luk’yanchuk
- Faculty of Physics, Lomonosov Moscow State University, Moscow 119991, Russia
| | - Limin Wu
- Department of Materials Science and State Key Laboratory of Molecular Engineering of Polymers, Fudan University, Shanghai 200433, China
| | - Alexey V. Maslov
- Department of Radiophysics, University of Nizhny Novgorod, Nizhny Novgorod, 603022, Russia
| | - Boya Jin
- Department of Physics and Optical Science, University of North Carolina at Charlotte, Charlotte, North Carolina 28223-0001, USA
| | - Constantin R. Simovski
- Department of Electronics and Nano-Engineering, Aalto University, FI-00076, Espoo, Finland
- Faculty of Physics and Engineering, ITMO University, 199034, St-Petersburg, Russia
| | - Stephane Perrin
- ICube Research Institute, University of Strasbourg - CNRS - INSA de Strasbourg, 300 Bd. Sébastien Brant, 67412 Illkirch, France
| | - Paul Montgomery
- ICube Research Institute, University of Strasbourg - CNRS - INSA de Strasbourg, 300 Bd. Sébastien Brant, 67412 Illkirch, France
| | - Sylvain Lecler
- ICube Research Institute, University of Strasbourg - CNRS - INSA de Strasbourg, 300 Bd. Sébastien Brant, 67412 Illkirch, France
| |
Collapse
|
14
|
Li Y, Li J, Zhao Y, Gan T, Hu J, Jarrahi M, Ozcan A. Universal Polarization Transformations: Spatial Programming of Polarization Scattering Matrices Using a Deep Learning-Designed Diffractive Polarization Transformer. ADVANCED MATERIALS (DEERFIELD BEACH, FLA.) 2023; 35:e2303395. [PMID: 37633311 DOI: 10.1002/adma.202303395] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/12/2023] [Revised: 08/09/2023] [Indexed: 08/28/2023]
Abstract
Controlled synthesis of optical fields having nonuniform polarization distributions presents a challenging task. Here, a universal polarization transformer is demonstrated that can synthesize a large set of arbitrarily-selected, complex-valued polarization scattering matrices between the polarization states at different positions within its input and output field-of-views (FOVs). This framework comprises 2D arrays of linear polarizers positioned between isotropic diffractive layers, each containing tens of thousands of diffractive features with optimizable transmission coefficients. After its deep learning-based training, this diffractive polarization transformer can successfully implement Ni No = 10 000 different spatially-encoded polarization scattering matrices with negligible error, where Ni and No represent the number of pixels in the input and output FOVs, respectively. This universal polarization transformation framework is experimentally validated in the terahertz spectrum by fabricating wire-grid polarizers and integrating them with 3D-printed diffractive layers to form a physical polarization transformer. Through this set-up, an all-optical polarization permutation operation of spatially-varying polarization fields is demonstrated, and distinct spatially-encoded polarization scattering matrices are simultaneously implemented between the input and output FOVs of a compact diffractive processor. This framework opens up new avenues for developing novel devices for universal polarization control and may find applications in, e.g., remote sensing, medical imaging, security, material inspection, and machine vision.
Collapse
Affiliation(s)
- Yuhang Li
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Jingxi Li
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Yifan Zhao
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Tianyi Gan
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Jingtian Hu
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Mona Jarrahi
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Aydogan Ozcan
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| |
Collapse
|
15
|
Yuan X, Wang Y, Xu Z, Zhou T, Fang L. Training large-scale optoelectronic neural networks with dual-neuron optical-artificial learning. Nat Commun 2023; 14:7110. [PMID: 37925451 PMCID: PMC10625607 DOI: 10.1038/s41467-023-42984-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2023] [Accepted: 10/25/2023] [Indexed: 11/06/2023] Open
Abstract
Optoelectronic neural networks (ONN) are a promising avenue in AI computing due to their potential for parallelization, power efficiency, and speed. Diffractive neural networks, which process information by propagating encoded light through trained optical elements, have garnered interest. However, training large-scale diffractive networks faces challenges due to the computational and memory costs of optical diffraction modeling. Here, we present DANTE, a dual-neuron optical-artificial learning architecture. Optical neurons model the optical diffraction, while artificial neurons approximate the intensive optical-diffraction computations with lightweight functions. DANTE also improves convergence by employing iterative global artificial-learning steps and local optical-learning steps. In simulation experiments, DANTE successfully trains large-scale ONNs with 150 million neurons on ImageNet, previously unattainable, and accelerates training speeds significantly on the CIFAR-10 benchmark compared to single-neuron learning. In physical experiments, we develop a two-layer ONN system based on DANTE, which can effectively extract features to improve the classification of natural images.
Collapse
Affiliation(s)
- Xiaoyun Yuan
- Department of Electronic Engineering, Tsinghua University, Beijing, China
- Beijing National Research Center for Information Science and Technology (BNRist), Beijing, China
- Institute for Brain and Cognitive Science, Tsinghua University (THUIBCS), Beijing, China
| | - Yong Wang
- Department of Electronic Engineering, Tsinghua University, Beijing, China
| | - Zhihao Xu
- Department of Electronic Engineering, Tsinghua University, Beijing, China
- Tsinghua Shenzhen International Graduate School, Shenzhen, China
| | - Tiankuang Zhou
- Department of Electronic Engineering, Tsinghua University, Beijing, China
- Beijing National Research Center for Information Science and Technology (BNRist), Beijing, China
- Institute for Brain and Cognitive Science, Tsinghua University (THUIBCS), Beijing, China
| | - Lu Fang
- Department of Electronic Engineering, Tsinghua University, Beijing, China.
- Beijing National Research Center for Information Science and Technology (BNRist), Beijing, China.
- Institute for Brain and Cognitive Science, Tsinghua University (THUIBCS), Beijing, China.
| |
Collapse
|
16
|
Park J, Bai B, Ryu D, Liu T, Lee C, Luo Y, Lee MJ, Huang L, Shin J, Zhang Y, Ryu D, Li Y, Kim G, Min HS, Ozcan A, Park Y. Artificial intelligence-enabled quantitative phase imaging methods for life sciences. Nat Methods 2023; 20:1645-1660. [PMID: 37872244 DOI: 10.1038/s41592-023-02041-4] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/25/2023] [Accepted: 09/11/2023] [Indexed: 10/25/2023]
Abstract
Quantitative phase imaging, integrated with artificial intelligence, allows for the rapid and label-free investigation of the physiology and pathology of biological systems. This review presents the principles of various two-dimensional and three-dimensional label-free phase imaging techniques that exploit refractive index as an intrinsic optical imaging contrast. In particular, we discuss artificial intelligence-based analysis methodologies for biomedical studies including image enhancement, segmentation of cellular or subcellular structures, classification of types of biological samples and image translation to furnish subcellular and histochemical information from label-free phase images. We also discuss the advantages and challenges of artificial intelligence-enabled quantitative phase imaging analyses, summarize recent notable applications in the life sciences, and cover the potential of this field for basic and industrial research in the life sciences.
Collapse
Affiliation(s)
- Juyeon Park
- Department of Physics, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
- KAIST Institute for Health Science and Technology, KAIST, Daejeon, Republic of Korea
| | - Bijie Bai
- Electrical and Computer Engineering Department, University of California, Los Angeles, Los Angeles, CA, USA
- Bioengineering Department, University of California, Los Angeles, Los Angeles, CA, USA
| | - DongHun Ryu
- Department of Physics, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
- KAIST Institute for Health Science and Technology, KAIST, Daejeon, Republic of Korea
- Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Tairan Liu
- Electrical and Computer Engineering Department, University of California, Los Angeles, Los Angeles, CA, USA
| | - Chungha Lee
- Department of Physics, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
- KAIST Institute for Health Science and Technology, KAIST, Daejeon, Republic of Korea
| | - Yi Luo
- Electrical and Computer Engineering Department, University of California, Los Angeles, Los Angeles, CA, USA
| | - Mahn Jae Lee
- KAIST Institute for Health Science and Technology, KAIST, Daejeon, Republic of Korea
- Graduate School of Medical Science and Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
| | - Luzhe Huang
- Electrical and Computer Engineering Department, University of California, Los Angeles, Los Angeles, CA, USA
| | - Jeongwon Shin
- KAIST Institute for Health Science and Technology, KAIST, Daejeon, Republic of Korea
- Department of Biological Sciences, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
| | - Yijie Zhang
- Electrical and Computer Engineering Department, University of California, Los Angeles, Los Angeles, CA, USA
| | | | - Yuzhu Li
- Electrical and Computer Engineering Department, University of California, Los Angeles, Los Angeles, CA, USA
| | - Geon Kim
- Department of Physics, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
- KAIST Institute for Health Science and Technology, KAIST, Daejeon, Republic of Korea
| | | | - Aydogan Ozcan
- Electrical and Computer Engineering Department, University of California, Los Angeles, Los Angeles, CA, USA.
- Bioengineering Department, University of California, Los Angeles, Los Angeles, CA, USA.
| | - YongKeun Park
- Department of Physics, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea.
- KAIST Institute for Health Science and Technology, KAIST, Daejeon, Republic of Korea.
- Tomocube, Daejeon, Republic of Korea.
| |
Collapse
|
17
|
Chen Y, Nazhamaiti M, Xu H, Meng Y, Zhou T, Li G, Fan J, Wei Q, Wu J, Qiao F, Fang L, Dai Q. All-analog photoelectronic chip for high-speed vision tasks. Nature 2023; 623:48-57. [PMID: 37880362 PMCID: PMC10620079 DOI: 10.1038/s41586-023-06558-8] [Citation(s) in RCA: 9] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2023] [Accepted: 08/21/2023] [Indexed: 10/27/2023]
Abstract
Photonic computing enables faster and more energy-efficient processing of vision data1-5. However, experimental superiority of deployable systems remains a challenge because of complicated optical nonlinearities, considerable power consumption of analog-to-digital converters (ADCs) for downstream digital processing and vulnerability to noises and system errors1,6-8. Here we propose an all-analog chip combining electronic and light computing (ACCEL). It has a systemic energy efficiency of 74.8 peta-operations per second per watt and a computing speed of 4.6 peta-operations per second (more than 99% implemented by optics), corresponding to more than three and one order of magnitude higher than state-of-the-art computing processors, respectively. After applying diffractive optical computing as an optical encoder for feature extraction, the light-induced photocurrents are directly used for further calculation in an integrated analog computing chip without the requirement of analog-to-digital converters, leading to a low computing latency of 72 ns for each frame. With joint optimizations of optoelectronic computing and adaptive training, ACCEL achieves competitive classification accuracies of 85.5%, 82.0% and 92.6%, respectively, for Fashion-MNIST, 3-class ImageNet classification and time-lapse video recognition task experimentally, while showing superior system robustness in low-light conditions (0.14 fJ μm-2 each frame). ACCEL can be used across a broad range of applications such as wearable devices, autonomous driving and industrial inspections.
Collapse
Affiliation(s)
- Yitong Chen
- Department of Automation, Tsinghua University, Beijing, China
| | | | - Han Xu
- Department of Electronic Engineering, Tsinghua University, Beijing, China
| | - Yao Meng
- Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, China
| | - Tiankuang Zhou
- Department of Automation, Tsinghua University, Beijing, China
- Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, China
- Shenzhen International Graduate School, Tsinghua University, Shenzhen, China
| | - Guangpu Li
- Department of Automation, Tsinghua University, Beijing, China
- Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, China
| | - Jingtao Fan
- Department of Automation, Tsinghua University, Beijing, China
| | - Qi Wei
- Department of Precision Instruments, Tsinghua University, Beijing, China
| | - Jiamin Wu
- Department of Automation, Tsinghua University, Beijing, China.
- Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, China.
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China.
| | - Fei Qiao
- Department of Electronic Engineering, Tsinghua University, Beijing, China.
| | - Lu Fang
- Department of Electronic Engineering, Tsinghua University, Beijing, China.
- Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, China.
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China.
| | - Qionghai Dai
- Department of Automation, Tsinghua University, Beijing, China.
- Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, China.
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China.
| |
Collapse
|
18
|
Li J, Li X, Yardimci NT, Hu J, Li Y, Chen J, Hung YC, Jarrahi M, Ozcan A. Rapid sensing of hidden objects and defects using a single-pixel diffractive terahertz sensor. Nat Commun 2023; 14:6791. [PMID: 37880258 PMCID: PMC10600253 DOI: 10.1038/s41467-023-42554-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/05/2023] [Accepted: 10/13/2023] [Indexed: 10/27/2023] Open
Abstract
Terahertz waves offer advantages for nondestructive detection of hidden objects/defects in materials, as they can penetrate most optically-opaque materials. However, existing terahertz inspection systems face throughput and accuracy restrictions due to their limited imaging speed and resolution. Furthermore, machine-vision-based systems using large-pixel-count imaging encounter bottlenecks due to their data storage, transmission and processing requirements. Here, we report a diffractive sensor that rapidly detects hidden defects/objects within a 3D sample using a single-pixel terahertz detector, eliminating sample scanning or image formation/processing. Leveraging deep-learning-optimized diffractive layers, this diffractive sensor can all-optically probe the 3D structural information of samples by outputting a spectrum, directly indicating the presence/absence of hidden structures or defects. We experimentally validated this framework using a single-pixel terahertz time-domain spectroscopy set-up and 3D-printed diffractive layers, successfully detecting unknown hidden defects inside silicon samples. This technique is valuable for applications including security screening, biomedical sensing and industrial quality control.
Collapse
Affiliation(s)
- Jingxi Li
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Xurong Li
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Nezih T Yardimci
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Jingtian Hu
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Yuhang Li
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Junjie Chen
- Physics & Astronomy Department, University of California, Los Angeles, CA, 90095, USA
| | - Yi-Chun Hung
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
| | - Mona Jarrahi
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Aydogan Ozcan
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA.
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA.
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA.
| |
Collapse
|
19
|
Feng J, Chen H, Yang D, Hao J, Lin J, Jin P. Multi-wavelength diffractive neural network with the weighting method. OPTICS EXPRESS 2023; 31:33113-33122. [PMID: 37859098 DOI: 10.1364/oe.499840] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/05/2023] [Accepted: 09/04/2023] [Indexed: 10/21/2023]
Abstract
Recently, the diffractive deep neural network (D2NN) has demonstrated the advantages to achieve large-scale computational tasks in terms of high speed, low power consumption, parallelism, and scalability. A typical D2NN with cascaded diffractive elements is designed for monochromatic illumination. Here, we propose a framework to achieve the multi-wavelength D2NN (MW-D2NN) based on the method of weight coefficients. In training, each wavelength is assigned a specific weighting and their output planes construct the wavelength weighting loss function. The trained MW-D2NN can implement the classification of images of handwritten digits at multi-wavelength incident beams. The designed 3-layers MW-D2NN achieves a simulation classification accuracy of 83.3%. We designed a 1-layer MW-D2NN. The simulation and experiment classification accuracy are 71.4% and 67.5% at RGB wavelengths. Furthermore, the proposed MW-D2NN can be extended to intelligent machine vision systems for multi-wavelength and incoherent illumination.
Collapse
|
20
|
Li X, Li J, Li Y, Ozcan A, Jarrahi M. High-throughput terahertz imaging: progress and challenges. LIGHT, SCIENCE & APPLICATIONS 2023; 12:233. [PMID: 37714865 PMCID: PMC10504281 DOI: 10.1038/s41377-023-01278-0] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/28/2023] [Revised: 08/04/2023] [Accepted: 08/28/2023] [Indexed: 09/17/2023]
Abstract
Many exciting terahertz imaging applications, such as non-destructive evaluation, biomedical diagnosis, and security screening, have been historically limited in practical usage due to the raster-scanning requirement of imaging systems, which impose very low imaging speeds. However, recent advancements in terahertz imaging systems have greatly increased the imaging throughput and brought the promising potential of terahertz radiation from research laboratories closer to real-world applications. Here, we review the development of terahertz imaging technologies from both hardware and computational imaging perspectives. We introduce and compare different types of hardware enabling frequency-domain and time-domain imaging using various thermal, photon, and field image sensor arrays. We discuss how different imaging hardware and computational imaging algorithms provide opportunities for capturing time-of-flight, spectroscopic, phase, and intensity image data at high throughputs. Furthermore, the new prospects and challenges for the development of future high-throughput terahertz imaging systems are briefly introduced.
Collapse
Affiliation(s)
- Xurong Li
- Department of Electrical & Computer Engineering, University of California Los Angeles (UCLA), Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California Los Angeles (UCLA), Los Angeles, CA, 90095, USA
| | - Jingxi Li
- Department of Electrical & Computer Engineering, University of California Los Angeles (UCLA), Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California Los Angeles (UCLA), Los Angeles, CA, 90095, USA
- Department of Bioengineering, University of California Los Angeles (UCLA), Los Angeles, CA, 90095, USA
| | - Yuhang Li
- Department of Electrical & Computer Engineering, University of California Los Angeles (UCLA), Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California Los Angeles (UCLA), Los Angeles, CA, 90095, USA
- Department of Bioengineering, University of California Los Angeles (UCLA), Los Angeles, CA, 90095, USA
| | - Aydogan Ozcan
- Department of Electrical & Computer Engineering, University of California Los Angeles (UCLA), Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California Los Angeles (UCLA), Los Angeles, CA, 90095, USA
- Department of Bioengineering, University of California Los Angeles (UCLA), Los Angeles, CA, 90095, USA
| | - Mona Jarrahi
- Department of Electrical & Computer Engineering, University of California Los Angeles (UCLA), Los Angeles, CA, 90095, USA.
- California NanoSystems Institute (CNSI), University of California Los Angeles (UCLA), Los Angeles, CA, 90095, USA.
| |
Collapse
|
21
|
Rahman MSS, Yang X, Li J, Bai B, Ozcan A. Universal linear intensity transformations using spatially incoherent diffractive processors. LIGHT, SCIENCE & APPLICATIONS 2023; 12:195. [PMID: 37582771 PMCID: PMC10427714 DOI: 10.1038/s41377-023-01234-y] [Citation(s) in RCA: 8] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/22/2023] [Revised: 07/14/2023] [Accepted: 07/15/2023] [Indexed: 08/17/2023]
Abstract
Under spatially coherent light, a diffractive optical network composed of structured surfaces can be designed to perform any arbitrary complex-valued linear transformation between its input and output fields-of-view (FOVs) if the total number (N) of optimizable phase-only diffractive features is ≥~2NiNo, where Ni and No refer to the number of useful pixels at the input and the output FOVs, respectively. Here we report the design of a spatially incoherent diffractive optical processor that can approximate any arbitrary linear transformation in time-averaged intensity between its input and output FOVs. Under spatially incoherent monochromatic light, the spatially varying intensity point spread function (H) of a diffractive network, corresponding to a given, arbitrarily-selected linear intensity transformation, can be written as H(m, n; m', n') = |h(m, n; m', n')|2, where h is the spatially coherent point spread function of the same diffractive network, and (m, n) and (m', n') define the coordinates of the output and input FOVs, respectively. Using numerical simulations and deep learning, supervised through examples of input-output profiles, we demonstrate that a spatially incoherent diffractive network can be trained to all-optically perform any arbitrary linear intensity transformation between its input and output if N ≥ ~2NiNo. We also report the design of spatially incoherent diffractive networks for linear processing of intensity information at multiple illumination wavelengths, operating simultaneously. Finally, we numerically demonstrate a diffractive network design that performs all-optical classification of handwritten digits under spatially incoherent illumination, achieving a test accuracy of >95%. Spatially incoherent diffractive networks will be broadly useful for designing all-optical visual processors that can work under natural light.
Collapse
Affiliation(s)
- Md Sadman Sakib Rahman
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Xilin Yang
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Jingxi Li
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Bijie Bai
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Aydogan Ozcan
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA.
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA.
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA.
| |
Collapse
|
22
|
Li J, Gan T, Zhao Y, Bai B, Shen CY, Sun S, Jarrahi M, Ozcan A. Unidirectional imaging using deep learning-designed materials. SCIENCE ADVANCES 2023; 9:eadg1505. [PMID: 37115928 DOI: 10.1126/sciadv.adg1505] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/05/2022] [Accepted: 03/27/2023] [Indexed: 06/19/2023]
Abstract
A unidirectional imager would only permit image formation along one direction, from an input field-of-view (FOV) A to an output FOV B, and in the reverse path, B → A, the image formation would be blocked. We report the first demonstration of unidirectional imagers, presenting polarization-insensitive and broadband unidirectional imaging based on successive diffractive layers that are linear and isotropic. After their deep learning-based training, the resulting diffractive layers are fabricated to form a unidirectional imager. Although trained using monochromatic illumination, the diffractive unidirectional imager maintains its functionality over a large spectral band and works under broadband illumination. We experimentally validated this unidirectional imager using terahertz radiation, well matching our numerical results. We also created a wavelength-selective unidirectional imager, where two unidirectional imaging operations, in reverse directions, are multiplexed through different illumination wavelengths. Diffractive unidirectional imaging using structured materials will have numerous applications in, e.g., security, defense, telecommunications, and privacy protection.
Collapse
Affiliation(s)
- Jingxi Li
- Electrical and Computer Engineering Department, University of California, Los Angeles, Los Angeles, CA 90095, USA
- Bioengineering Department, University of California, Los Angeles, Los Angeles, CA 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, Los Angeles, CA 90095, USA
| | - Tianyi Gan
- Electrical and Computer Engineering Department, University of California, Los Angeles, Los Angeles, CA 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, Los Angeles, CA 90095, USA
| | - Yifan Zhao
- Electrical and Computer Engineering Department, University of California, Los Angeles, Los Angeles, CA 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, Los Angeles, CA 90095, USA
| | - Bijie Bai
- Electrical and Computer Engineering Department, University of California, Los Angeles, Los Angeles, CA 90095, USA
- Bioengineering Department, University of California, Los Angeles, Los Angeles, CA 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, Los Angeles, CA 90095, USA
| | - Che-Yung Shen
- Electrical and Computer Engineering Department, University of California, Los Angeles, Los Angeles, CA 90095, USA
- Bioengineering Department, University of California, Los Angeles, Los Angeles, CA 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, Los Angeles, CA 90095, USA
| | - Songyu Sun
- Electrical and Computer Engineering Department, University of California, Los Angeles, Los Angeles, CA 90095, USA
| | - Mona Jarrahi
- Electrical and Computer Engineering Department, University of California, Los Angeles, Los Angeles, CA 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, Los Angeles, CA 90095, USA
| | - Aydogan Ozcan
- Electrical and Computer Engineering Department, University of California, Los Angeles, Los Angeles, CA 90095, USA
- Bioengineering Department, University of California, Los Angeles, Los Angeles, CA 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, Los Angeles, CA 90095, USA
| |
Collapse
|
23
|
Mengu D, Tabassum A, Jarrahi M, Ozcan A. Snapshot multispectral imaging using a diffractive optical network. LIGHT, SCIENCE & APPLICATIONS 2023; 12:86. [PMID: 37024463 PMCID: PMC10079962 DOI: 10.1038/s41377-023-01135-0] [Citation(s) in RCA: 10] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/30/2022] [Revised: 03/14/2023] [Accepted: 03/17/2023] [Indexed: 06/19/2023]
Abstract
Multispectral imaging has been used for numerous applications in e.g., environmental monitoring, aerospace, defense, and biomedicine. Here, we present a diffractive optical network-based multispectral imaging system trained using deep learning to create a virtual spectral filter array at the output image field-of-view. This diffractive multispectral imager performs spatially-coherent imaging over a large spectrum, and at the same time, routes a pre-determined set of spectral channels onto an array of pixels at the output plane, converting a monochrome focal-plane array or image sensor into a multispectral imaging device without any spectral filters or image recovery algorithms. Furthermore, the spectral responsivity of this diffractive multispectral imager is not sensitive to input polarization states. Through numerical simulations, we present different diffractive network designs that achieve snapshot multispectral imaging with 4, 9 and 16 unique spectral bands within the visible spectrum, based on passive spatially-structured diffractive surfaces, with a compact design that axially spans ~72λm, where λm is the mean wavelength of the spectral band of interest. Moreover, we experimentally demonstrate a diffractive multispectral imager based on a 3D-printed diffractive network that creates at its output image plane a spatially repeating virtual spectral filter array with 2 × 2 = 4 unique bands at terahertz spectrum. Due to their compact form factor and computation-free, power-efficient and polarization-insensitive forward operation, diffractive multispectral imagers can be transformative for various imaging and sensing applications and be used at different parts of the electromagnetic spectrum where high-density and wide-area multispectral pixel arrays are not widely available.
Collapse
Affiliation(s)
- Deniz Mengu
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute, University of California, Los Angeles, CA, 90095, USA
| | - Anika Tabassum
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute, University of California, Los Angeles, CA, 90095, USA
| | - Mona Jarrahi
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute, University of California, Los Angeles, CA, 90095, USA
| | - Aydogan Ozcan
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA.
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA.
- California NanoSystems Institute, University of California, Los Angeles, CA, 90095, USA.
| |
Collapse
|
24
|
Bai B, Li Y, Luo Y, Li X, Çetintaş E, Jarrahi M, Ozcan A. All-optical image classification through unknown random diffusers using a single-pixel diffractive network. LIGHT, SCIENCE & APPLICATIONS 2023; 12:69. [PMID: 36894546 PMCID: PMC9998891 DOI: 10.1038/s41377-023-01116-3] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/20/2022] [Revised: 02/22/2023] [Accepted: 02/22/2023] [Indexed: 06/01/2023]
Abstract
Classification of an object behind a random and unknown scattering medium sets a challenging task for computational imaging and machine vision fields. Recent deep learning-based approaches demonstrated the classification of objects using diffuser-distorted patterns collected by an image sensor. These methods demand relatively large-scale computing using deep neural networks running on digital computers. Here, we present an all-optical processor to directly classify unknown objects through unknown, random phase diffusers using broadband illumination detected with a single pixel. A set of transmissive diffractive layers, optimized using deep learning, forms a physical network that all-optically maps the spatial information of an input object behind a random diffuser into the power spectrum of the output light detected through a single pixel at the output plane of the diffractive network. We numerically demonstrated the accuracy of this framework using broadband radiation to classify unknown handwritten digits through random new diffusers, never used during the training phase, and achieved a blind testing accuracy of 87.74 ± 1.12%. We also experimentally validated our single-pixel broadband diffractive network by classifying handwritten digits "0" and "1" through a random diffuser using terahertz waves and a 3D-printed diffractive network. This single-pixel all-optical object classification system through random diffusers is based on passive diffractive layers that process broadband input light and can operate at any part of the electromagnetic spectrum by simply scaling the diffractive features proportional to the wavelength range of interest. These results have various potential applications in, e.g., biomedical imaging, security, robotics, and autonomous driving.
Collapse
Affiliation(s)
- Bijie Bai
- Electrical and Computer Engineering Department, University of California, Los Angeles, California, 90095, USA
- Bioengineering Department, University of California, Los Angeles, California, 90095, USA
- California Nano Systems Institute (CNSI), University of California, Los Angeles, California, 90095, USA
| | - Yuhang Li
- Electrical and Computer Engineering Department, University of California, Los Angeles, California, 90095, USA
- Bioengineering Department, University of California, Los Angeles, California, 90095, USA
- California Nano Systems Institute (CNSI), University of California, Los Angeles, California, 90095, USA
| | - Yi Luo
- Electrical and Computer Engineering Department, University of California, Los Angeles, California, 90095, USA
- Bioengineering Department, University of California, Los Angeles, California, 90095, USA
- California Nano Systems Institute (CNSI), University of California, Los Angeles, California, 90095, USA
| | - Xurong Li
- Electrical and Computer Engineering Department, University of California, Los Angeles, California, 90095, USA
- California Nano Systems Institute (CNSI), University of California, Los Angeles, California, 90095, USA
| | - Ege Çetintaş
- Electrical and Computer Engineering Department, University of California, Los Angeles, California, 90095, USA
- Bioengineering Department, University of California, Los Angeles, California, 90095, USA
- California Nano Systems Institute (CNSI), University of California, Los Angeles, California, 90095, USA
| | - Mona Jarrahi
- Electrical and Computer Engineering Department, University of California, Los Angeles, California, 90095, USA
- California Nano Systems Institute (CNSI), University of California, Los Angeles, California, 90095, USA
| | - Aydogan Ozcan
- Electrical and Computer Engineering Department, University of California, Los Angeles, California, 90095, USA.
- Bioengineering Department, University of California, Los Angeles, California, 90095, USA.
- California Nano Systems Institute (CNSI), University of California, Los Angeles, California, 90095, USA.
| |
Collapse
|
25
|
Hazan A, Ratzker B, Zhang D, Katiyi A, Sokol M, Gogotsi Y, Karabchevsky A. MXene-Nanoflakes-Enabled All-Optical Nonlinear Activation Function for On-Chip Photonic Deep Neural Networks. ADVANCED MATERIALS (DEERFIELD BEACH, FLA.) 2023; 35:e2210216. [PMID: 36641139 DOI: 10.1002/adma.202210216] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/04/2022] [Revised: 12/01/2022] [Indexed: 06/17/2023]
Abstract
2D metal carbides and nitrides (MXene) are promising material platforms for on-chip neural networks owing to their nonlinear saturable absorption effect. The localized surface plasmon resonances in metallic MXene nanoflakes may play an important role in enhancing the electromagnetic absorption; however, their contribution is not determined due to the lack of a precise understanding of its localized surface plasmon behavior. Here, a saturable absorber made of MXene thin film and a silicon waveguide with MXene flakes overlayer are developed to perform neuromorphic tasks. The proposed configurations are reconfigurable and can therefore be adjusted for various applications without the need to modify the physical structure of the proposed MXene-based activator configurations via tuning the wavelength of operation. The capability and feasibility of the obtained results of machine-learning applications are confirmed via handwritten digit classification task, with near 99% accuracy. These findings can guide the design of advanced ultrathin saturable absorption materials on a chip for a broad range of applications.
Collapse
Affiliation(s)
- Adir Hazan
- School of Electrical and Computer Engineering, Electro-Optics and Photonics Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, 8410501, Israel
| | - Barak Ratzker
- Department of Materials Science and Engineering, Tel Aviv University, Ramat Aviv, 6997801, Israel
| | - Danzhen Zhang
- A. J. Drexel Nanomaterials Institute and Department of Materials Science and Engineering, Drexel University, Philadelphia, PA, 19104, USA
| | - Aviad Katiyi
- School of Electrical and Computer Engineering, Electro-Optics and Photonics Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, 8410501, Israel
| | - Maxim Sokol
- Department of Materials Science and Engineering, Tel Aviv University, Ramat Aviv, 6997801, Israel
| | - Yury Gogotsi
- A. J. Drexel Nanomaterials Institute and Department of Materials Science and Engineering, Drexel University, Philadelphia, PA, 19104, USA
| | - Alina Karabchevsky
- School of Electrical and Computer Engineering, Electro-Optics and Photonics Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, 8410501, Israel
| |
Collapse
|
26
|
Cecconi V, Kumar V, Pasquazi A, Totero Gongora JS, Peccianti M. Nonlinear field-control of terahertz waves in random media for spatiotemporal focusing. OPEN RESEARCH EUROPE 2023; 2:32. [PMID: 37645307 PMCID: PMC10445851 DOI: 10.12688/openreseurope.14508.2] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 01/30/2023] [Indexed: 09/15/2023]
Abstract
Controlling the transmission of broadband optical pulses in scattering media is a critical open challenge in photonics. To date, wavefront shaping techniques at optical frequencies have been successfully applied to control the spatial properties of multiple-scattered light. However, a fundamental restriction in achieving an equivalent degree of control over the temporal properties of a broadband pulse is the limited availability of experimental techniques to detect the coherent properties (i.e., the spectral amplitude and absolute phase) of the transmitted field. Terahertz experimental frameworks, on the contrary, enable measuring the field dynamics of broadband pulses at ultrafast (sub-cycle) time scales directly. In this work, we provide a theoretical/numerical demonstration that, within this context, complex scattering can be used to achieve spatio-temporal control of instantaneous fields and manipulate the temporal properties of single-cycle pulses by solely acting on spatial degrees of freedom of the illuminating field. As direct application scenarios, we demonstrate spatio-temporal focusing, chirp compensation, and control of the carrier-envelope-phase (CEP) of a CP-stable, transform-limited THz pulse.
Collapse
Affiliation(s)
- Vittorio Cecconi
- Emergent Photonics (EPic) Lab, Department of Physics and Astronomy, University of Sussex, Brighton, BN19QH, UK
- Emergent Photonics Research Centre and Dept. of Physics, Loughborough University, Loughborough, LE11 3TU, UK
| | - Vivek Kumar
- Emergent Photonics (EPic) Lab, Department of Physics and Astronomy, University of Sussex, Brighton, BN19QH, UK
| | - Alessia Pasquazi
- Emergent Photonics (EPic) Lab, Department of Physics and Astronomy, University of Sussex, Brighton, BN19QH, UK
- Emergent Photonics Research Centre and Dept. of Physics, Loughborough University, Loughborough, LE11 3TU, UK
| | - Juan Sebastian Totero Gongora
- Emergent Photonics (EPic) Lab, Department of Physics and Astronomy, University of Sussex, Brighton, BN19QH, UK
- Emergent Photonics Research Centre and Dept. of Physics, Loughborough University, Loughborough, LE11 3TU, UK
| | - Marco Peccianti
- Emergent Photonics (EPic) Lab, Department of Physics and Astronomy, University of Sussex, Brighton, BN19QH, UK
- Emergent Photonics Research Centre and Dept. of Physics, Loughborough University, Loughborough, LE11 3TU, UK
| |
Collapse
|
27
|
Cecconi V, Kumar V, Pasquazi A, Totero Gongora JS, Peccianti M. Nonlinear field-control of terahertz waves in random media for spatiotemporal focusing. OPEN RESEARCH EUROPE 2023; 2:32. [PMID: 37645307 PMCID: PMC10445851 DOI: 10.12688/openreseurope.14508.3] [Citation(s) in RCA: 8] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 01/30/2023] [Indexed: 08/31/2023]
Abstract
Controlling the transmission of broadband optical pulses in scattering media is a critical open challenge in photonics. To date, wavefront shaping techniques at optical frequencies have been successfully applied to control the spatial properties of multiple-scattered light. However, a fundamental restriction in achieving an equivalent degree of control over the temporal properties of a broadband pulse is the limited availability of experimental techniques to detect the coherent properties (i.e., the spectral amplitude and absolute phase) of the transmitted field. Terahertz experimental frameworks, on the contrary, enable measuring the field dynamics of broadband pulses at ultrafast (sub-cycle) time scales directly. In this work, we provide a theoretical/numerical demonstration that, within this context, complex scattering can be used to achieve spatio-temporal control of instantaneous fields and manipulate the temporal properties of single-cycle pulses by solely acting on spatial degrees of freedom of the illuminating field. As direct application scenarios, we demonstrate spatio-temporal focusing, chirp compensation, and control of the carrier-envelope-phase (CEP) of a CP-stable, transform-limited THz pulse.
Collapse
Affiliation(s)
- Vittorio Cecconi
- Emergent Photonics (EPic) Lab, Department of Physics and Astronomy, University of Sussex, Brighton, BN19QH, UK
- Emergent Photonics Research Centre and Dept. of Physics, Loughborough University, Loughborough, LE11 3TU, UK
| | - Vivek Kumar
- Emergent Photonics (EPic) Lab, Department of Physics and Astronomy, University of Sussex, Brighton, BN19QH, UK
| | - Alessia Pasquazi
- Emergent Photonics (EPic) Lab, Department of Physics and Astronomy, University of Sussex, Brighton, BN19QH, UK
- Emergent Photonics Research Centre and Dept. of Physics, Loughborough University, Loughborough, LE11 3TU, UK
| | - Juan Sebastian Totero Gongora
- Emergent Photonics (EPic) Lab, Department of Physics and Astronomy, University of Sussex, Brighton, BN19QH, UK
- Emergent Photonics Research Centre and Dept. of Physics, Loughborough University, Loughborough, LE11 3TU, UK
| | - Marco Peccianti
- Emergent Photonics (EPic) Lab, Department of Physics and Astronomy, University of Sussex, Brighton, BN19QH, UK
- Emergent Photonics Research Centre and Dept. of Physics, Loughborough University, Loughborough, LE11 3TU, UK
| |
Collapse
|
28
|
Grigorev GV, Lebedev AV, Wang X, Qian X, Maksimov GV, Lin L. Advances in Microfluidics for Single Red Blood Cell Analysis. BIOSENSORS 2023; 13:117. [PMID: 36671952 PMCID: PMC9856164 DOI: 10.3390/bios13010117] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/31/2022] [Revised: 12/04/2022] [Accepted: 12/23/2022] [Indexed: 05/24/2023]
Abstract
The utilizations of microfluidic chips for single RBC (red blood cell) studies have attracted great interests in recent years to filter, trap, analyze, and release single erythrocytes for various applications. Researchers in this field have highlighted the vast potential in developing micro devices for industrial and academia usages, including lab-on-a-chip and organ-on-a-chip systems. This article critically reviews the current state-of-the-art and recent advances of microfluidics for single RBC analyses, including integrated sensors and microfluidic platforms for microscopic/tomographic/spectroscopic single RBC analyses, trapping arrays (including bifurcating channels), dielectrophoretic and agglutination/aggregation studies, as well as clinical implications covering cancer, sepsis, prenatal, and Sickle Cell diseases. Microfluidics based RBC microarrays, sorting/counting and trapping techniques (including acoustic, dielectrophoretic, hydrodynamic, magnetic, and optical techniques) are also reviewed. Lastly, organs on chips, multi-organ chips, and drug discovery involving single RBC are described. The limitations and drawbacks of each technology are addressed and future prospects are discussed.
Collapse
Affiliation(s)
- Georgii V. Grigorev
- Data Science and Information Technology Research Center, Tsinghua Berkeley Shenzhen Institute, Tsinghua University, Shenzhen 518055, China
- Mechanical Engineering Department, University of California in Berkeley, Berkeley, CA 94720, USA
- School of Information Technology, Cherepovets State University, 162600 Cherepovets, Russia
| | - Alexander V. Lebedev
- Machine Building Department, Bauman Moscow State University, 105005 Moscow, Russia
| | - Xiaohao Wang
- Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China
| | - Xiang Qian
- Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China
| | - George V. Maksimov
- Faculty of Biology, Lomonosov Moscow State University, 119991 Moscow, Russia
- Physical metallurgy Department, Federal State Autonomous Educational Institution of Higher Education National Research Technological University “MISiS”, 119049 Moscow, Russia
| | - Liwei Lin
- Mechanical Engineering Department, University of California in Berkeley, Berkeley, CA 94720, USA
| |
Collapse
|
29
|
Lai W, Lei G, Meng Q, Shi D, Cui W, Ma P, Wang Y, Han K. Single-pixel imaging using discrete Zernike moments. OPTICS EXPRESS 2022; 30:47761-47775. [PMID: 36558696 DOI: 10.1364/oe.473912] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/30/2022] [Accepted: 11/28/2022] [Indexed: 06/17/2023]
Abstract
A novel single-pixel imaging (SPI) technique based on discrete orthogonal Zernike moments is proposed. In this technique, the target object is illuminated by two sets of Zernike basis patterns which satisfy the Zernike polynomials. The Zernike moments of object image are obtained by measuring the reflected light intensities through a single-pixel detector. And the object image is reconstructed by summing the product of Zernike polynomials and detected intensities iteratively. By theoretical and experimental demonstrations, an image with high quality is retrieved under compressive sampling. Moreover, the Zernike illuminating patterns are used for object classification due to the rotation invariant of Zernike moments. By measuring the amplitudes of a few specific Zernike moments through the SPI system, the rotated images with different angles and the same content are classified into the same class on experiment. This classification technique has the advantages of high efficiency and high accuracy due to the high modulation speed and high sensitivity of SPI system.
Collapse
|
30
|
Işıl Ç, Mengu D, Zhao Y, Tabassum A, Li J, Luo Y, Jarrahi M, Ozcan A. Super-resolution image display using diffractive decoders. SCIENCE ADVANCES 2022; 8:eadd3433. [PMID: 36459555 PMCID: PMC10936058 DOI: 10.1126/sciadv.add3433] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/06/2022] [Accepted: 10/18/2022] [Indexed: 06/17/2023]
Abstract
High-resolution image projection over a large field of view (FOV) is hindered by the restricted space-bandwidth product (SBP) of wavefront modulators. We report a deep learning-enabled diffractive display based on a jointly trained pair of an electronic encoder and a diffractive decoder to synthesize/project super-resolved images using low-resolution wavefront modulators. The digital encoder rapidly preprocesses the high-resolution images so that their spatial information is encoded into low-resolution patterns, projected via a low SBP wavefront modulator. The diffractive decoder processes these low-resolution patterns using transmissive layers structured using deep learning to all-optically synthesize/project super-resolved images at its output FOV. This diffractive image display can achieve a super-resolution factor of ~4, increasing the SBP by ~16-fold. We experimentally validate its success using 3D-printed diffractive decoders that operate at the terahertz spectrum. This diffractive image decoder can be scaled to operate at visible wavelengths and used to design large SBP displays that are compact, low power, and computationally efficient.
Collapse
Affiliation(s)
- Çağatay Işıl
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA 90095, USA
| | - Deniz Mengu
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA 90095, USA
| | - Yifan Zhao
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA 90095, USA
| | - Anika Tabassum
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA 90095, USA
| | - Jingxi Li
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA 90095, USA
| | - Yi Luo
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA 90095, USA
| | - Mona Jarrahi
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA 90095, USA
| | - Aydogan Ozcan
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA 90095, USA
| |
Collapse
|
31
|
Zhou L, Shi J, Zhang X. The mr-MDA: An Invariant to Shifting, Scaling, and Rotating Variance for 3D Object Recognition Using Diffractive Deep Neural Network. SENSORS (BASEL, SWITZERLAND) 2022; 22:7754. [PMID: 36298105 PMCID: PMC9610052 DOI: 10.3390/s22207754] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/05/2022] [Revised: 10/03/2022] [Accepted: 10/10/2022] [Indexed: 06/16/2023]
Abstract
The diffractive deep neural network (D2NN) can efficiently accomplish 2D object recognition based on rapid optical manipulation. Moreover, the multiple-view D2NN array (MDA) possesses the obvious advantage of being able to effectively achieve 3D object classification. At present, 3D target recognition should be performed in a high-speed and dynamic way. It should be invariant to the typical shifting, scaling, and rotating variance of targets in relatively complicated circumstances, which remains a shortcoming of optical neural network architectures. In order to efficiently recognize 3D targets based on the developed D2NN, a more robust MDA (mr-MDA) is proposed in this paper. Through utilizing a new training strategy to tackle several random disturbances introduced into the optical neural network system, a trained mr-MDA model constructed by us was numerically verified, demonstrating that the training strategy is able to dynamically recognize 3D objects in a relatively stable way.
Collapse
Affiliation(s)
- Liang Zhou
- National Key Laboratory of Science & Technology on Multispectral Information Processing, Huazhong University of Science & Technology, Wuhan 430074, China
- School of Artificial Intelligence & Automation, Huazhong University of Science & Technology, Wuhan 430074, China
| | - Jiashuo Shi
- National Key Laboratory of Science & Technology on Multispectral Information Processing, Huazhong University of Science & Technology, Wuhan 430074, China
- School of Artificial Intelligence & Automation, Huazhong University of Science & Technology, Wuhan 430074, China
| | - Xinyu Zhang
- National Key Laboratory of Science & Technology on Multispectral Information Processing, Huazhong University of Science & Technology, Wuhan 430074, China
- School of Artificial Intelligence & Automation, Huazhong University of Science & Technology, Wuhan 430074, China
| |
Collapse
|
32
|
Zarei S, Khavasi A. Realization of optical logic gates using on-chip diffractive optical neural networks. Sci Rep 2022; 12:15747. [PMID: 36130987 PMCID: PMC9492711 DOI: 10.1038/s41598-022-19973-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/19/2022] [Accepted: 09/07/2022] [Indexed: 11/09/2022] Open
Abstract
Optical computing is highly desired as a potential strategy for circumventing the performance limitations of semiconductor-based electronic devices and circuits. Optical logic gates are considered as fundamental building blocks for optical computation and they enable logic functions to be performed extremely quickly without the generation of heat and crosstalk. Here, we discuss the design of a multi-functional optical logic gate based on an on-chip diffractive optical neural network that can perform AND, NOT and OR logic operations at the wavelength of 1.55 µm. The wavelength-independent operation of the multi-functional logic gate at seven wavelengths (over a bandwidth of 60 nm) is also studied which paves the way for wavelength division multiplexed parallel computation. This simple, highly-integrable, low-loss, energy-efficient and broadband optical logic gate provides a path for the development of high-speed on-chip nanophotonic processors for future optical computing applications.
Collapse
Affiliation(s)
- Sanaz Zarei
- Department of Electrical Engineering, Sharif University of Technology, Tehran, Iran.
| | - Amin Khavasi
- Department of Electrical Engineering, Sharif University of Technology, Tehran, Iran
| |
Collapse
|
33
|
Sun X, Yang Y, Jia H, Yang J. Physics-aware training for the physical machine learning model building. Innovation (N Y) 2022; 3:100287. [PMID: 35898945 PMCID: PMC9310120 DOI: 10.1016/j.xinn.2022.100287] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/26/2022] [Accepted: 07/07/2022] [Indexed: 11/29/2022] Open
Affiliation(s)
- Xuecong Sun
- Key Laboratory of Noise and Vibration Research, Institute of Acoustics, Chinese Academy of Sciences, Beijing 100190, China.,University of Chinese Academy of Sciences, Beijing 100049, China
| | - Yuzhen Yang
- Key Laboratory of Noise and Vibration Research, Institute of Acoustics, Chinese Academy of Sciences, Beijing 100190, China
| | - Han Jia
- University of Chinese Academy of Sciences, Beijing 100049, China.,State Key Laboratory of Acoustics, Institute of Acoustics, Chinese Academy of Sciences, Beijing 100190, China
| | - Jun Yang
- Key Laboratory of Noise and Vibration Research, Institute of Acoustics, Chinese Academy of Sciences, Beijing 100190, China.,University of Chinese Academy of Sciences, Beijing 100049, China
| |
Collapse
|
34
|
Xu Z, Yuan X, Zhou T, Fang L. A multichannel optical computing architecture for advanced machine vision. LIGHT, SCIENCE & APPLICATIONS 2022; 11:255. [PMID: 35977940 PMCID: PMC9385649 DOI: 10.1038/s41377-022-00945-y] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/05/2022] [Revised: 07/14/2022] [Accepted: 07/21/2022] [Indexed: 06/03/2023]
Abstract
Endowed with the superior computing speed and energy efficiency, optical neural networks (ONNs) have attracted ever-growing attention in recent years. Existing optical computing architectures are mainly single-channel due to the lack of advanced optical connection and interaction operators, solving simple tasks such as hand-written digit classification, saliency detection, etc. The limited computing capacity and scalability of single-channel ONNs restrict the optical implementation of advanced machine vision. Herein, we develop Monet: a multichannel optical neural network architecture for a universal multiple-input multiple-channel optical computing based on a novel projection-interference-prediction framework where the inter- and intra- channel connections are mapped to optical interference and diffraction. In our Monet, optical interference patterns are generated by projecting and interfering the multichannel inputs in a shared domain. These patterns encoding the correspondences together with feature embeddings are iteratively produced through the projection-interference process to predict the final output optically. For the first time, Monet validates that multichannel processing properties can be optically implemented with high-efficiency, enabling real-world intelligent multichannel-processing tasks solved via optical computing, including 3D/motion detections. Extensive experiments on different scenarios demonstrate the effectiveness of Monet in handling advanced machine vision tasks with comparative accuracy as the electronic counterparts yet achieving a ten-fold improvement in computing efficiency. For intelligent computing, the trends of dealing with real-world advanced tasks are irreversible. Breaking the capacity and scalability limitations of single-channel ONN and further exploring the multichannel processing potential of wave optics, we anticipate that the proposed technique will accelerate the development of more powerful optical AI as critical support for modern advanced machine vision.
Collapse
Affiliation(s)
- Zhihao Xu
- Sigma Laboratory, Department of Electronic Engineering, Tsinghua University, Beijing, China
- Beijing National Research Center for Information Science and Technology (BNRist), Beijing, China
- Tsinghua Shenzhen International Graduate School, Shenzhen, China
| | - Xiaoyun Yuan
- Sigma Laboratory, Department of Electronic Engineering, Tsinghua University, Beijing, China
- Institute for Brain and Cognitive Science, Tsinghua University (THUIBCS), Beijing, China
| | - Tiankuang Zhou
- Sigma Laboratory, Department of Electronic Engineering, Tsinghua University, Beijing, China
- Tsinghua Shenzhen International Graduate School, Shenzhen, China
| | - Lu Fang
- Sigma Laboratory, Department of Electronic Engineering, Tsinghua University, Beijing, China.
- Beijing National Research Center for Information Science and Technology (BNRist), Beijing, China.
- Institute for Brain and Cognitive Science, Tsinghua University (THUIBCS), Beijing, China.
| |
Collapse
|
35
|
Wu L, Zhang Z. Only-train-electrical-to-optical-conversion (OTEOC): simple diffractive neural networks with optical readout. OPTICS EXPRESS 2022; 30:28024-28037. [PMID: 36236959 DOI: 10.1364/oe.462370] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/02/2022] [Accepted: 07/01/2022] [Indexed: 06/16/2023]
Abstract
Machine learning hardware based on optical diffraction is emerging as a new computing platform with high throughput and low latency. The current all-optical diffractive deep neural networks often suffer from complex optical configuration, lack of efficient optical nonlinear activation, and critical alignment between optical layers for system integration. The opto-electronic diffractive neural networks can partially address these issues by shifting some computation load, e.g., nonlinear activation and adaptive training, to the electronic domain. However, these hybrid networks require extra optical-to-electrical conversion that inevitably slows the overall process down. Here, we propose a simple opto-electronic diffractive neural network with just one optical layer enabled by a standard phase-only spatial light modulator. The proposed system can classify images by optical readout and does not need to collect the light distribution for subsequent electronic computation. The nonlinear function is intrinsically integrated in the essential encoding process from the electronic input to the modulated wavefront of light. Thanks to its simplicity, the system can reach high classification accuracy without calibration and can be reconfigured by updating the weights without changing or moving any physical component. We believe this technology brings diffractive neural networks a step closer to building realistic optics-based neurocomputers.
Collapse
|
36
|
Zuo C, Chen Q. Exploiting optical degrees of freedom for information multiplexing in diffractive neural networks. LIGHT, SCIENCE & APPLICATIONS 2022; 11:208. [PMID: 35794086 PMCID: PMC9259600 DOI: 10.1038/s41377-022-00903-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Exploiting internal degrees of freedom of light, such as polarization, provides efficient ways to scale the capacity of optical diffractive computing, which may ultimately lead to high-throughput, multifunctional all-optical diffractive processors that can execute a diverse range of tasks in parallel.
Collapse
Affiliation(s)
- Chao Zuo
- Smart Computational Imaging Laboratory (SCILab), School of Electronic and Optical Engineering, Nanjing University of Science and Technology, 210094, Nanjing, Jiangsu Province, China.
- Smart Computational Imaging Research Institute (SCIRI) of Nanjing University of Science and Technology, 210019, Nanjing, Jiangsu Province, China.
- Jiangsu Key Laboratory of Spectral Imaging & Intelligent Sense, 210094, Nanjing, Jiangsu Province, China.
| | - Qian Chen
- Jiangsu Key Laboratory of Spectral Imaging & Intelligent Sense, 210094, Nanjing, Jiangsu Province, China
| |
Collapse
|
37
|
Abstract
Recent years have witnessed promising artificial intelligence (AI) applications in many disciplines, including optics, engineering, medicine, economics, and education. In particular, the synergy of AI and meta-optics has greatly benefited both fields. Meta-optics are advanced flat optics with novel functions and light-manipulation abilities. The optical properties can be engineered with a unique design to meet various optical demands. This review offers comprehensive coverage of meta-optics and artificial intelligence in synergy. After providing an overview of AI and meta-optics, we categorize and discuss the recent developments integrated by these two topics, namely AI for meta-optics and meta-optics for AI. The former describes how to apply AI to the research of meta-optics for design, simulation, optical information analysis, and application. The latter reports the development of the optical Al system and computation via meta-optics. This review will also provide an in-depth discussion of the challenges of this interdisciplinary field and indicate future directions. We expect that this review will inspire researchers in these fields and benefit the next generation of intelligent optical device design.
Collapse
Affiliation(s)
- Mu Ku Chen
- Department of Electrical Engineering, City University of Hong Kong, Kowloon, Hong Kong 999077.,Centre for Biosystems, Neuroscience, and Nanotechnology, City University of Hong Kong, Kowloon, Hong Kong 999077.,The State Key Laboratory of Terahertz and Millimeter Waves, City University of Hong Kong, Kowloon, Hong Kong 999077
| | - Xiaoyuan Liu
- Department of Electrical Engineering, City University of Hong Kong, Kowloon, Hong Kong 999077
| | - Yanni Sun
- Department of Electrical Engineering, City University of Hong Kong, Kowloon, Hong Kong 999077
| | - Din Ping Tsai
- Department of Electrical Engineering, City University of Hong Kong, Kowloon, Hong Kong 999077.,Centre for Biosystems, Neuroscience, and Nanotechnology, City University of Hong Kong, Kowloon, Hong Kong 999077.,The State Key Laboratory of Terahertz and Millimeter Waves, City University of Hong Kong, Kowloon, Hong Kong 999077
| |
Collapse
|
38
|
Yan T, Yang R, Zheng Z, Lin X, Xiong H, Dai Q. All-optical graph representation learning using integrated diffractive photonic computing units. SCIENCE ADVANCES 2022; 8:eabn7630. [PMID: 35704580 PMCID: PMC9200271 DOI: 10.1126/sciadv.abn7630] [Citation(s) in RCA: 13] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/17/2021] [Accepted: 04/29/2022] [Indexed: 05/28/2023]
Abstract
Photonic neural networks perform brain-inspired computations using photons instead of electrons to achieve substantially improved computing performance. However, existing architectures can only handle data with regular structures but fail to generalize to graph-structured data beyond Euclidean space. Here, we propose the diffractive graph neural network (DGNN), an all-optical graph representation learning architecture based on the diffractive photonic computing units (DPUs) and on-chip optical devices to address this limitation. Specifically, the graph node attributes are encoded into strip optical waveguides, transformed by DPUs, and aggregated by optical couplers to extract their feature representations. DGNN captures complex dependencies among node neighborhoods during the light-speed optical message passing over graph structures. We demonstrate the applications of DGNN for node and graph-level classification tasks with benchmark databases and achieve superior performance. Our work opens up a new direction for designing application-specific integrated photonic circuits for high-efficiency processing large-scale graph data structures using deep learning.
Collapse
Affiliation(s)
- Tao Yan
- Department of Automation, Tsinghua University, Beijing 100084, China
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing 100084, China
| | - Rui Yang
- Department of Automation, Tsinghua University, Beijing 100084, China
- Department of Electronic Engineering, Tsinghua University, Beijing 100084, China
- Department of Electronic Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Ziyang Zheng
- Department of Automation, Tsinghua University, Beijing 100084, China
- Department of Electronic Engineering, Tsinghua University, Beijing 100084, China
- Department of Electronic Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Xing Lin
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing 100084, China
- Department of Electronic Engineering, Tsinghua University, Beijing 100084, China
- Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China
- Beijing Laboratory of Brain and Cognitive Intelligence, Beijing Municipal Education Commission, Beijing 100084, China
| | - Hongkai Xiong
- Department of Electronic Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Qionghai Dai
- Department of Automation, Tsinghua University, Beijing 100084, China
- Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing 100084, China
- Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China
- Beijing Laboratory of Brain and Cognitive Intelligence, Beijing Municipal Education Commission, Beijing 100084, China
| |
Collapse
|
39
|
Luo X, Hu Y, Ou X, Li X, Lai J, Liu N, Cheng X, Pan A, Duan H. Metasurface-enabled on-chip multiplexed diffractive neural networks in the visible. LIGHT, SCIENCE & APPLICATIONS 2022; 11:158. [PMID: 35624107 PMCID: PMC9142536 DOI: 10.1038/s41377-022-00844-2] [Citation(s) in RCA: 27] [Impact Index Per Article: 13.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/19/2021] [Revised: 05/06/2022] [Accepted: 05/10/2022] [Indexed: 05/16/2023]
Abstract
Replacing electrons with photons is a compelling route toward high-speed, massively parallel, and low-power artificial intelligence computing. Recently, diffractive networks composed of phase surfaces were trained to perform machine learning tasks through linear optical transformations. However, the existing architectures often comprise bulky components and, most critically, they cannot mimic the human brain for multitasking. Here, we demonstrate a multi-skilled diffractive neural network based on a metasurface device, which can perform on-chip multi-channel sensing and multitasking in the visible. The polarization multiplexing scheme of the subwavelength nanostructures is applied to construct a multi-channel classifier framework for simultaneous recognition of digital and fashionable items. The areal density of the artificial neurons can reach up to 6.25 × 106 mm-2 multiplied by the number of channels. The metasurface is integrated with the mature complementary metal-oxide semiconductor imaging sensor, providing a chip-scale architecture to process information directly at physical layers for energy-efficient and ultra-fast image processing in machine vision, autonomous driving, and precision medicine.
Collapse
Affiliation(s)
- Xuhao Luo
- National Research Center for High-Efficiency Grinding, College of Mechanical and Vehicle Engineering, Hunan University, Changsha, 410082, China
- Institute of Precision Optical Engineering, School of Physics Science and Engineering, Tongji University, Shanghai, 200092, China
| | - Yueqiang Hu
- National Research Center for High-Efficiency Grinding, College of Mechanical and Vehicle Engineering, Hunan University, Changsha, 410082, China.
- Advanced Manufacturing Laboratory of Micro-Nano Optical Devices, Shenzhen Research Institute, Hunan University, Shenzhen, 518000, China.
| | - Xiangnian Ou
- National Research Center for High-Efficiency Grinding, College of Mechanical and Vehicle Engineering, Hunan University, Changsha, 410082, China
| | - Xin Li
- National Research Center for High-Efficiency Grinding, College of Mechanical and Vehicle Engineering, Hunan University, Changsha, 410082, China
| | - Jiajie Lai
- National Research Center for High-Efficiency Grinding, College of Mechanical and Vehicle Engineering, Hunan University, Changsha, 410082, China
| | - Na Liu
- 2nd Physics Institute, University of Stuttgart, Pfaffenwaldring 57, 70569, Stuttgart, Germany
- Max Planck Institute for Solid State Research, Heisenbergstrasse 1, 70569, Stuttgart, Germany
| | - Xinbin Cheng
- Institute of Precision Optical Engineering, School of Physics Science and Engineering, Tongji University, Shanghai, 200092, China
| | - Anlian Pan
- National Research Center for High-Efficiency Grinding, College of Mechanical and Vehicle Engineering, Hunan University, Changsha, 410082, China
| | - Huigao Duan
- National Research Center for High-Efficiency Grinding, College of Mechanical and Vehicle Engineering, Hunan University, Changsha, 410082, China.
- Greater Bay Area Institute for Innovation, Hunan University, Guangzhou, 511300, China.
| |
Collapse
|
40
|
Li J, Hung YC, Kulce O, Mengu D, Ozcan A. Polarization multiplexed diffractive computing: all-optical implementation of a group of linear transformations through a polarization-encoded diffractive network. LIGHT, SCIENCE & APPLICATIONS 2022; 11:153. [PMID: 35614046 PMCID: PMC9133014 DOI: 10.1038/s41377-022-00849-x] [Citation(s) in RCA: 18] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/14/2022] [Revised: 05/10/2022] [Accepted: 05/11/2022] [Indexed: 05/15/2023]
Abstract
Research on optical computing has recently attracted significant attention due to the transformative advances in machine learning. Among different approaches, diffractive optical networks composed of spatially-engineered transmissive surfaces have been demonstrated for all-optical statistical inference and performing arbitrary linear transformations using passive, free-space optical layers. Here, we introduce a polarization-multiplexed diffractive processor to all-optically perform multiple, arbitrarily-selected linear transformations through a single diffractive network trained using deep learning. In this framework, an array of pre-selected linear polarizers is positioned between trainable transmissive diffractive materials that are isotropic, and different target linear transformations (complex-valued) are uniquely assigned to different combinations of input/output polarization states. The transmission layers of this polarization-multiplexed diffractive network are trained and optimized via deep learning and error-backpropagation by using thousands of examples of the input/output fields corresponding to each one of the complex-valued linear transformations assigned to different input/output polarization combinations. Our results and analysis reveal that a single diffractive network can successfully approximate and all-optically implement a group of arbitrarily-selected target transformations with a negligible error when the number of trainable diffractive features/neurons (N) approaches [Formula: see text], where Ni and No represent the number of pixels at the input and output fields-of-view, respectively, and Np refers to the number of unique linear transformations assigned to different input/output polarization combinations. This polarization-multiplexed all-optical diffractive processor can find various applications in optical computing and polarization-based machine vision tasks.
Collapse
Affiliation(s)
- Jingxi Li
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Yi-Chun Hung
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
| | - Onur Kulce
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Deniz Mengu
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Aydogan Ozcan
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA.
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA.
- California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA.
| |
Collapse
|
41
|
Classification and reconstruction of spatially overlapping phase images using diffractive optical networks. Sci Rep 2022; 12:8446. [PMID: 35589729 PMCID: PMC9120207 DOI: 10.1038/s41598-022-12020-y] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/10/2021] [Accepted: 05/03/2022] [Indexed: 01/09/2023] Open
Abstract
Diffractive optical networks unify wave optics and deep learning to all-optically compute a given machine learning or computational imaging task as the light propagates from the input to the output plane. Here, we report the design of diffractive optical networks for the classification and reconstruction of spatially overlapping, phase-encoded objects. When two different phase-only objects spatially overlap, the individual object functions are perturbed since their phase patterns are summed up. The retrieval of the underlying phase images from solely the overlapping phase distribution presents a challenging problem, the solution of which is generally not unique. We show that through a task-specific training process, passive diffractive optical networks composed of successive transmissive layers can all-optically and simultaneously classify two different randomly-selected, spatially overlapping phase images at the input. After trained with ~ 550 million unique combinations of phase-encoded handwritten digits from the MNIST dataset, our blind testing results reveal that the diffractive optical network achieves an accuracy of > 85.8% for all-optical classification of two overlapping phase images of new handwritten digits. In addition to all-optical classification of overlapping phase objects, we also demonstrate the reconstruction of these phase images based on a shallow electronic neural network that uses the highly compressed output of the diffractive optical network as its input (with e.g., ~ 20-65 times less number of pixels) to rapidly reconstruct both of the phase images, despite their spatial overlap and related phase ambiguity. The presented phase image classification and reconstruction framework might find applications in e.g., computational imaging, microscopy and quantitative phase imaging fields.
Collapse
|
42
|
Shi W, Huang Z, Huang H, Hu C, Chen M, Yang S, Chen H. LOEN: Lensless opto-electronic neural network empowered machine vision. LIGHT, SCIENCE & APPLICATIONS 2022; 11:121. [PMID: 35508469 PMCID: PMC9068799 DOI: 10.1038/s41377-022-00809-5] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/30/2021] [Revised: 04/15/2022] [Accepted: 04/20/2022] [Indexed: 06/14/2023]
Abstract
Machine vision faces bottlenecks in computing power consumption and large amounts of data. Although opto-electronic hybrid neural networks can provide assistance, they usually have complex structures and are highly dependent on a coherent light source; therefore, they are not suitable for natural lighting environment applications. In this paper, we propose a novel lensless opto-electronic neural network architecture for machine vision applications. The architecture optimizes a passive optical mask by means of a task-oriented neural network design, performs the optical convolution calculation operation using the lensless architecture, and reduces the device size and amount of calculation required. We demonstrate the performance of handwritten digit classification tasks with a multiple-kernel mask in which accuracies of as much as 97.21% were achieved. Furthermore, we optimize a large-kernel mask to perform optical encryption for privacy-protecting face recognition, thereby obtaining the same recognition accuracy performance as no-encryption methods. Compared with the random MLS pattern, the recognition accuracy is improved by more than 6%.
Collapse
Affiliation(s)
- Wanxin Shi
- Beijing National Research Center for Information Science and Technology (BNRist), Department of Electronic Engineering, Tsinghua University, Beijing, 100084, China
| | - Zheng Huang
- Beijing National Research Center for Information Science and Technology (BNRist), Department of Electronic Engineering, Tsinghua University, Beijing, 100084, China
| | - Honghao Huang
- Beijing National Research Center for Information Science and Technology (BNRist), Department of Electronic Engineering, Tsinghua University, Beijing, 100084, China
| | - Chengyang Hu
- Beijing National Research Center for Information Science and Technology (BNRist), Department of Electronic Engineering, Tsinghua University, Beijing, 100084, China
| | - Minghua Chen
- Beijing National Research Center for Information Science and Technology (BNRist), Department of Electronic Engineering, Tsinghua University, Beijing, 100084, China
| | - Sigang Yang
- Beijing National Research Center for Information Science and Technology (BNRist), Department of Electronic Engineering, Tsinghua University, Beijing, 100084, China
| | - Hongwei Chen
- Beijing National Research Center for Information Science and Technology (BNRist), Department of Electronic Engineering, Tsinghua University, Beijing, 100084, China.
| |
Collapse
|
43
|
Luo Y, Mengu D, Ozcan A. Cascadable all-optical NAND gates using diffractive networks. Sci Rep 2022; 12:7121. [PMID: 35505083 PMCID: PMC9065113 DOI: 10.1038/s41598-022-11331-4] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/20/2022] [Accepted: 04/11/2022] [Indexed: 01/24/2023] Open
Abstract
Owing to its potential advantages such as scalability, low latency and power efficiency, optical computing has seen rapid advances over the last decades. Here, we present the design and analysis of cascadable all-optical NAND gates using diffractive neural networks. We encoded the logical values at the input and output planes of a diffractive NAND gate using the relative optical power of two spatially-separated apertures. Based on this architecture, we numerically optimized the design of a diffractive neural network composed of 4 passive layers to all-optically perform NAND operation using diffraction of light, and cascaded these diffractive NAND gates to perform complex logical functions by successively feeding the output of one diffractive NAND gate into another. We numerically demonstrated the cascadability of our diffractive NAND gates by using identical diffractive designs to all-optically perform AND and OR operations, which can be formulated as [Formula: see text] and [Formula: see text], respectively. We also designed an all-optical half-adder that takes two logical values as input and returns their sum and the carry using cascaded diffractive NAND gates. Cascadable all-optical NAND gates composed of spatially-engineered passive diffractive layers can serve optical computing platforms.
Collapse
Affiliation(s)
- Yi Luo
- Electrical and Computer Engineering Department, University of California, 420 Westwood Plaza, Engr. IV 68-119, UCLA, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California Nano Systems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Deniz Mengu
- Electrical and Computer Engineering Department, University of California, 420 Westwood Plaza, Engr. IV 68-119, UCLA, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California Nano Systems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA
| | - Aydogan Ozcan
- Electrical and Computer Engineering Department, University of California, 420 Westwood Plaza, Engr. IV 68-119, UCLA, Los Angeles, CA, 90095, USA.
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA.
- California Nano Systems Institute (CNSI), University of California, Los Angeles, CA, 90095, USA.
| |
Collapse
|
44
|
Wang Z, Chang L, Wang F, Li T, Gu T. Integrated photonic metasystem for image classifications at telecommunication wavelength. Nat Commun 2022; 13:2131. [PMID: 35440131 PMCID: PMC9018697 DOI: 10.1038/s41467-022-29856-7] [Citation(s) in RCA: 12] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/30/2021] [Accepted: 04/01/2022] [Indexed: 11/15/2022] Open
Abstract
Miniaturized image classifiers are potential for revolutionizing their applications in optical communication, autonomous vehicles, and healthcare. With subwavelength structure enabled directional diffraction and dispersion engineering, the light propagation through multi-layer metasurfaces achieves wavelength-selective image recognitions on a silicon photonic platform at telecommunication wavelength. The metasystems implement high-throughput vector-by-matrix multiplications, enabled by near 103 nanoscale phase shifters as weight elements within 0.135 mm2 footprints. The diffraction manifested computing capability incorporates the fabrication and measurement related phase fluctuations, and thus the pre-trained metasystem can handle uncertainties in inputs without post-tuning. Here we demonstrate three functional metasystems: a 15-pixel spatial pattern classifier that reaches near 90% accuracy with femtosecond inputs, a multi-channel wavelength demultiplexer, and a hyperspectral image classifier. The diffractive metasystem provides an alternative machine learning architecture for photonic integrated circuits, with densely integrated phase shifters, spatially multiplexed throughput, and data processing capabilities. Metasystem architectures are attractive alternatives to waveguide-based integrated photonic processors due to the subwavelength structures. Here, the authors report a 1D passive silicon photonic metasystem with near 90% spatial pattern classification accuracy at telecommunication wavelength.
Collapse
Affiliation(s)
- Zi Wang
- Department of Electrical and Computer Engineering, University of Delaware, Newark, DE, 19711, USA
| | - Lorry Chang
- Department of Electrical and Computer Engineering, University of Delaware, Newark, DE, 19711, USA
| | - Feifan Wang
- Department of Electrical and Computer Engineering, University of Delaware, Newark, DE, 19711, USA
| | - Tiantian Li
- Department of Electrical and Computer Engineering, University of Delaware, Newark, DE, 19711, USA
| | - Tingyi Gu
- Department of Electrical and Computer Engineering, University of Delaware, Newark, DE, 19711, USA.
| |
Collapse
|
45
|
Zheng S, Xu S, Fan D. Orthogonality of diffractive deep neural network. OPTICS LETTERS 2022; 47:1798-1801. [PMID: 35363738 DOI: 10.1364/ol.449899] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/29/2021] [Accepted: 02/28/2022] [Indexed: 06/14/2023]
Abstract
Some rules of the diffractive deep neural network (D2NN) are discovered. They reveal that the inner product of any two optical fields in D2NN is invariant and the D2NN acts as a unitary transformation for optical fields. If the output intensities of the two inputs are separated spatially, the input fields must be orthogonal. These rules imply that the D2NN is not only suitable for the classification of general objects but also more suitable for applications aimed at optical orthogonal modes. Our simulation shows the D2NN performs well in applications like mode conversion, mode multiplexing/demultiplexing, and optical mode recognition.
Collapse
|
46
|
Roadmap on Digital Holography-Based Quantitative Phase Imaging. J Imaging 2021; 7:jimaging7120252. [PMID: 34940719 PMCID: PMC8703719 DOI: 10.3390/jimaging7120252] [Citation(s) in RCA: 16] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/30/2021] [Revised: 11/11/2021] [Accepted: 11/15/2021] [Indexed: 12/02/2022] Open
Abstract
Quantitative Phase Imaging (QPI) provides unique means for the imaging of biological or technical microstructures, merging beneficial features identified with microscopy, interferometry, holography, and numerical computations. This roadmap article reviews several digital holography-based QPI approaches developed by prominent research groups. It also briefly discusses the present and future perspectives of 2D and 3D QPI research based on digital holographic microscopy, holographic tomography, and their applications.
Collapse
|
47
|
Lv X, Yang Z, Wang Y, Zhou K, Lin J, Jin P. Channeled imaging spectropolarimeter reconstruction by neural networks. OPTICS EXPRESS 2021; 29:35556-35569. [PMID: 34808986 DOI: 10.1364/oe.441850] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/30/2021] [Accepted: 10/08/2021] [Indexed: 06/13/2023]
Abstract
Snapshot channeled imaging spectropolarimetry (SCISP), which can achieve spectral and polarization imaging without scanning (a single exposure), is a promising optical technique. As Fourier transform is used to reconstruct information, SCISP has its inherent limitations such as channel crosstalk, resolution and accuracy drop, the complex phase calibration, et al. To overcome these drawbacks, a nonlinear technique based on neural networks (NNs) is introduced to replace the role of Fourier reconstruction. Herein, abundant spectral and polarization datasets were built through specially designed generators. The established NNs can effectively learn the forward conversion procedure through minimizing a loss function, subsequently enabling a stable output containing spectral, polarization, and spatial information. The utility and reliability of the proposed technique is confirmed by experiments, which are proved to maintain high spectral and polarization accuracy.
Collapse
|
48
|
Sun T. Light People: Professor Aydogan Ozcan. LIGHT, SCIENCE & APPLICATIONS 2021; 10:208. [PMID: 34611128 PMCID: PMC8491441 DOI: 10.1038/s41377-021-00643-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
In 2016, the news that Google's artificial intelligence (AI) robot AlphaGo, based on the principle of deep learning, won the victory over lee Sedol, the former world Go champion and the famous 9th Dan competitor of Korea, caused a sensation in both fields of AI and Go, which brought epoch-making significance to the development of deep learning. Deep learning is a complex machine learning algorithm that uses multiple layers of artificial neural networks to automatically analyze signals or data. At present, deep learning has penetrated into our daily life, such as the applications of face recognition and speech recognition. Scientists have also made many remarkable achievements based on deep learning. Professor Aydogan Ozcan from the University of California, Los Angeles (UCLA) led his team to research deep learning algorithms, which provided new ideas for the exploring of optical computational imaging and sensing technology, and introduced image generation and reconstruction methods which brought major technological innovations to the development of related fields. Optical designs and devices are moving from being physically driven to being data-driven. We are much honored to have Aydogan Ozcan, Fellow of the National Academy of Inventors and Chancellor's Professor of UCLA, to unscramble his latest scientific research results and foresight for the future development of related fields, and to share his journey of pursuing Optics, his indissoluble relationship with Light: Science & Applications (LSA), and his experience in talent cultivation.
Collapse
Affiliation(s)
- Tingting Sun
- Light Publishing Group, Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, 3888 Dong Nan Hu Road, Changchun, 130033, China.
| |
Collapse
|
49
|
Kulce O, Mengu D, Rivenson Y, Ozcan A. All-optical synthesis of an arbitrary linear transformation using diffractive surfaces. LIGHT, SCIENCE & APPLICATIONS 2021; 10:196. [PMID: 34561415 PMCID: PMC8463717 DOI: 10.1038/s41377-021-00623-5] [Citation(s) in RCA: 25] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/20/2021] [Revised: 08/20/2021] [Accepted: 08/23/2021] [Indexed: 05/08/2023]
Abstract
Spatially-engineered diffractive surfaces have emerged as a powerful framework to control light-matter interactions for statistical inference and the design of task-specific optical components. Here, we report the design of diffractive surfaces to all-optically perform arbitrary complex-valued linear transformations between an input (Ni) and output (No), where Ni and No represent the number of pixels at the input and output fields-of-view (FOVs), respectively. First, we consider a single diffractive surface and use a matrix pseudoinverse-based method to determine the complex-valued transmission coefficients of the diffractive features/neurons to all-optically perform a desired/target linear transformation. In addition to this data-free design approach, we also consider a deep learning-based design method to optimize the transmission coefficients of diffractive surfaces by using examples of input/output fields corresponding to the target transformation. We compared the all-optical transformation errors and diffraction efficiencies achieved using data-free designs as well as data-driven (deep learning-based) diffractive designs to all-optically perform (i) arbitrarily-chosen complex-valued transformations including unitary, nonunitary, and noninvertible transforms, (ii) 2D discrete Fourier transformation, (iii) arbitrary 2D permutation operations, and (iv) high-pass filtered coherent imaging. Our analyses reveal that if the total number (N) of spatially-engineered diffractive features/neurons is ≥Ni × No, both design methods succeed in all-optical implementation of the target transformation, achieving negligible error. However, compared to data-free designs, deep learning-based diffractive designs are found to achieve significantly larger diffraction efficiencies for a given N and their all-optical transformations are more accurate for N < Ni × No. These conclusions are generally applicable to various optical processors that employ spatially-engineered diffractive surfaces.
Collapse
Affiliation(s)
- Onur Kulce
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute, University of California, Los Angeles, CA, 90095, USA
| | - Deniz Mengu
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute, University of California, Los Angeles, CA, 90095, USA
| | - Yair Rivenson
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA
- California NanoSystems Institute, University of California, Los Angeles, CA, 90095, USA
| | - Aydogan Ozcan
- Electrical and Computer Engineering Department, University of California, Los Angeles, CA, 90095, USA.
- Bioengineering Department, University of California, Los Angeles, CA, 90095, USA.
- California NanoSystems Institute, University of California, Los Angeles, CA, 90095, USA.
| |
Collapse
|