1
|
Cheng J, Huang C, Zhang J, Wu B, Zhang W, Liu X, Zhang J, Tang Y, Zhou H, Zhang Q, Gu M, Dong J, Zhang X. Multimodal deep learning using on-chip diffractive optics with in situ training capability. Nat Commun 2024; 15:6189. [PMID: 39043669 PMCID: PMC11266606 DOI: 10.1038/s41467-024-50677-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2024] [Accepted: 07/18/2024] [Indexed: 07/25/2024] Open
Abstract
Multimodal deep learning plays a pivotal role in supporting the processing and learning of diverse data types within the realm of artificial intelligence generated content (AIGC). However, most photonic neuromorphic processors for deep learning can only handle a single data modality (either vision or audio) due to the lack of abundant parameter training in optical domain. Here, we propose and demonstrate a trainable diffractive optical neural network (TDONN) chip based on on-chip diffractive optics with massive tunable elements to address these constraints. The TDONN chip includes one input layer, five hidden layers, and one output layer, and only one forward propagation is required to obtain the inference results without frequent optical-electrical conversion. The customized stochastic gradient descent algorithm and the drop-out mechanism are developed for photonic neurons to realize in situ training and fast convergence in the optical domain. The TDONN chip achieves a potential throughput of 217.6 tera-operations per second (TOPS) with high computing density (447.7 TOPS/mm2), high system-level energy efficiency (7.28 TOPS/W), and low optical latency (30.2 ps). The TDONN chip has successfully implemented four-class classification in different modalities (vision, audio, and touch) and achieve 85.7% accuracy on multimodal test sets. Our work opens up a new avenue for multimodal deep learning with integrated photonic processors, providing a potential solution for low-power AI large models using photonic technology.
Collapse
Affiliation(s)
- Junwei Cheng
- Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, 430074, China
| | - Chaoran Huang
- Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong, 999077, China
| | - Jialong Zhang
- Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, 430074, China
| | - Bo Wu
- Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, 430074, China
| | - Wenkai Zhang
- Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, 430074, China
| | - Xinyu Liu
- Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, 430074, China
| | - Jiahui Zhang
- Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, 430074, China
| | - Yiyi Tang
- Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, 430074, China
| | - Hailong Zhou
- Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, 430074, China
| | - Qiming Zhang
- Institute of Photonic Chips, University of Shanghai for Science and Technology, Shanghai, 200093, China
| | - Min Gu
- Institute of Photonic Chips, University of Shanghai for Science and Technology, Shanghai, 200093, China
| | - Jianji Dong
- Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, 430074, China.
- Optics Valley Laboratory, Wuhan, 430074, China.
| | - Xinliang Zhang
- Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, 430074, China
- Optics Valley Laboratory, Wuhan, 430074, China
| |
Collapse
|
2
|
Kutluyarov RV, Zakoyan AG, Voronkov GS, Grakhova EP, Butt MA. Neuromorphic Photonics Circuits: Contemporary Review. NANOMATERIALS (BASEL, SWITZERLAND) 2023; 13:3139. [PMID: 38133036 PMCID: PMC10745993 DOI: 10.3390/nano13243139] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/13/2023] [Revised: 12/08/2023] [Accepted: 12/10/2023] [Indexed: 12/23/2023]
Abstract
Neuromorphic photonics is a cutting-edge fusion of neuroscience-inspired computing and photonics technology to overcome the constraints of conventional computing architectures. Its significance lies in the potential to transform information processing by mimicking the parallelism and efficiency of the human brain. Using optics and photonics principles, neuromorphic devices can execute intricate computations swiftly and with impressive energy efficiency. This innovation holds promise for advancing artificial intelligence and machine learning while addressing the limitations of traditional silicon-based computing. Neuromorphic photonics could herald a new era of computing that is more potent and draws inspiration from cognitive processes, leading to advancements in robotics, pattern recognition, and advanced data processing. This paper reviews the recent developments in neuromorphic photonic integrated circuits, applications, and current challenges.
Collapse
Affiliation(s)
- Ruslan V. Kutluyarov
- School of Photonics Engineering and Research Advances (SPhERA), Ufa University of Science and Technology, 32, Z. Validi St., 450076 Ufa, Russia
| | - Aida G. Zakoyan
- School of Photonics Engineering and Research Advances (SPhERA), Ufa University of Science and Technology, 32, Z. Validi St., 450076 Ufa, Russia
| | - Grigory S. Voronkov
- School of Photonics Engineering and Research Advances (SPhERA), Ufa University of Science and Technology, 32, Z. Validi St., 450076 Ufa, Russia
| | - Elizaveta P. Grakhova
- School of Photonics Engineering and Research Advances (SPhERA), Ufa University of Science and Technology, 32, Z. Validi St., 450076 Ufa, Russia
| | | |
Collapse
|
3
|
Huang Y, Yue H, Ma W, Zhang Y, Xiao Y, Tang Y, Tang H, Chu T. Parallel photonic acceleration processor for matrix-matrix multiplication. OPTICS LETTERS 2023; 48:3231-3234. [PMID: 37319069 DOI: 10.1364/ol.488464] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/22/2023] [Accepted: 05/16/2023] [Indexed: 06/17/2023]
Abstract
We propose and experimentally demonstrate a highly parallel photonic acceleration processor based on a wavelength division multiplexing (WDM) system and a non-coherent Mach-Zehnder interferometer (MZI) array for matrix-matrix multiplication. The dimensional expansion is achieved by WDM devices, which play a crucial role in realizing matrix-matrix multiplication together with the broadband characteristics of an MZI. We implemented a 2 × 2 arbitrary nonnegative valued matrix using a reconfigurable 8 × 8 MZI array structure. Through experimentation, we verified that this structure could achieve 90.5% inference accuracy in a classification task for the Modified National Institute of Standards and Technology (MNIST) handwritten dataset. This provides a new effective solution for large-scale integrated optical computing systems based on convolution acceleration processors.
Collapse
|
4
|
Zhang J, Yan Q, Liu H, Tang Y, Zhou T, Jiang T. Coherent optical neuron control based on reinforcement learning. OPTICS LETTERS 2023; 48:1084-1087. [PMID: 36791016 DOI: 10.1364/ol.484435] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/26/2022] [Accepted: 01/18/2023] [Indexed: 06/18/2023]
Abstract
Optical neural networks take optical neurons as the cornerstone to achieve complex functions. The coherent optical neuron has become one of the mainstream implementations because it can effectively perform natural and even complex number calculations. However, its state variability and requirement for reliability and effectiveness render traditional control methods no longer applicable. In this Letter, deep reinforcement coherent optical neuron control (DRCON) is proposed, and its effectiveness is experimentally demonstrated. Compared with the standard stochastic gradient descent, the average convergence rate of DRCON is 33% faster, while the effective number of bits increases from less than 2 bits to 5.5 bits. DRCON is a promising first step for large-scale optical neural network control.
Collapse
|
5
|
Gao D, Zhou Z. Silicon-based optoelectronics: progress towards large scale optoelectronic integration and applications. FRONTIERS OF OPTOELECTRONICS 2022; 15:27. [PMID: 36637555 PMCID: PMC9756240 DOI: 10.1007/s12200-022-00030-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 03/07/2022] [Indexed: 06/17/2023]
Affiliation(s)
- Dingshan Gao
- Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, 430074, China.
| | - Zhiping Zhou
- State Key Laboratory of Advanced Optical Communication Systems and Networks, School of Electronics, Peking University, Beijing, 100871, China.
| |
Collapse
|