All-optical graph representation learning using integrated diffractive photonic computing units

Optical neural networks: progress and challenges

Artificial intelligence has prevailed in all trades and professions due to the assistance of big data resources, advanced algorithms, and high-performance electronic hardware. However, conventional computing hardware is inefficient at implementing complex tasks, in large part because the memory and processor in its computing architecture are separated, performing insufficiently in computing speed and energy consumption. In recent years, optical neural networks (ONNs) have made a range of research progress in optical computing due to advantages such as sub-nanosecond latency, low heat dissipation, and high parallelism. ONNs are in prospect to provide support regarding computing speed and energy consumption for the further development of artificial intelligence with a novel computing paradigm. Herein, we first introduce the design method and principle of ONNs based on various optical elements. Then, we successively review the non-integrated ONNs consisting of volume optical components and the integrated ONNs composed of on-chip components. Finally, we summarize and discuss the computational density, nonlinearity, scalability, and practical applications of ONNs, and comment on the challenges and perspectives of the ONNs in the future development trends.

Partial coherence enhances parallelized photonic computing

Advancements in optical coherence control1-5 have unlocked many cutting-edge applications, including long-haul communication, light detection and ranging (LiDAR) and optical coherence tomography6-8. Prevailing wisdom suggests that using more coherent light sources leads to enhanced system performance and device functionalities9-11. Our study introduces a photonic convolutional processing system that takes advantage of partially coherent light to boost computing parallelism without substantially sacrificing accuracy, potentially enabling larger-size photonic tensor cores. The reduction of the degree of coherence optimizes bandwidth use in the photonic convolutional processing system. This breakthrough challenges the traditional belief that coherence is essential or even advantageous in integrated photonic accelerators, thereby enabling the use of light sources with less rigorous feedback control and thermal-management requirements for high-throughput photonic computing. Here we demonstrate such a system in two photonic platforms for computing applications: a photonic tensor core using phase-change-material photonic memories that delivers parallel convolution operations to classify the gaits of ten patients with Parkinson's disease with 92.2% accuracy (92.7% theoretically) and a silicon photonic tensor core with embedded electro-absorption modulators (EAMs) to facilitate 0.108 tera operations per second (TOPS) convolutional processing for classifying the Modified National Institute of Standards and Technology (MNIST) handwritten digits dataset with 92.4% accuracy (95.0% theoretically).

Polarization and wavelength routers based on diffractive neural network

In the field of information processing, all-optical routers are significant for achieving high-speed, high-capacity signal processing and transmission. In this study, we developed three types of structurally simple and flexible routers using the deep diffractive neural network (D2NN), capable of routing incident light based on wavelength and polarization. First, we implemented a polarization router for routing two orthogonally polarized light beams. The second type is the wavelength router that can route light with wavelengths of 1550, 1300, and 1100 nm, demonstrating outstanding performance with insertion loss as low as 0.013 dB and an extinction ratio of up to 18.96 dB, while also maintaining excellent polarization preservation. The final router is the polarization-wavelength composite router, capable of routing six types of input light formed by pairwise combinations of three wavelengths (1550, 1300, and 1100 nm) and two orthogonal linearly polarized lights, thereby enhancing the information processing capability of the device. These devices feature compact structures, maintaining high contrast while exhibiting low loss and passive characteristics, making them suitable for integration into future optical components. This study introduces new avenues and methodologies to enhance performance and broaden the applications of future optical information processing systems.

Nanowatt all-optical 3D perception for mobile robotics

Three-dimensional (3D) perception is vital to drive mobile robotics' progress toward intelligence. However, state-of-the-art 3D perception solutions require complicated postprocessing or point-by-point scanning, suffering computational burden, latency of tens of milliseconds, and additional power consumption. Here, we propose a parallel all-optical computational chipset 3D perception architecture (Aop3D) with nanowatt power and light speed. The 3D perception is executed during the light propagation over the passive chipset, and the captured light intensity distribution provides a direct reflection of the depth map, eliminating the need for extensive postprocessing. The prototype system of Aop3D is tested in various scenarios and deployed to a mobile robot, demonstrating unprecedented performance in distance detection and obstacle avoidance. Moreover, Aop3D works at a frame rate of 600 hertz and a power consumption of 33.3 nanowatts per meta-pixel experimentally. Our work is promising toward next-generation direct 3D perception techniques with light speed and high energy efficiency.

Large-scale photonic computing with nonlinear disordered media

Neural networks find widespread use in scientific and technological applications, yet their implementations in conventional computers have encountered bottlenecks due to ever-expanding computational needs. Photonic computing is a promising neuromorphic platform with potential advantages of massive parallelism, ultralow latency and reduced energy consumption but mostly for computing linear operations. Here we demonstrate a large-scale, high-performance nonlinear photonic neural system based on a disordered polycrystalline slab composed of lithium niobate nanocrystals. Mediated by random quasi-phase-matching and multiple scattering, linear and nonlinear optical speckle features are generated as the interplay between the simultaneous linear random scattering and the second-harmonic generation, defining a complex neural network in which the second-order nonlinearity acts as internal nonlinear activation functions. Benchmarked against linear random projection, such nonlinear mapping embedded with rich physical computational operations shows improved performance across a large collection of machine learning tasks in image classification, regression and graph classification. Demonstrating up to 27,648 input and 3,500 nonlinear output nodes, the combination of optical nonlinearity and random scattering serves as a scalable computing engine for diverse applications.

Demixing microwave signals using system-on-chip photonic processor

The integrated photonic processor, co-packaged with electronic peripherals, is proposed for blind source separation of microwave signals, which separates signal-of-interest from dynamic interference with real-time adaptability.

Optical authentication scheme based on all-optical neural network

Diffractive deep neural network is architectural designs based on the principles of neural networks, which consists of multiple diffraction layers and has the remarkable ability to perform machine learning tasks at the speed of light. In this paper, a novel optical authentication system was presented that utilizes the diffractive deep neural network principle. By carefully manipulating a light beam with both a public key and a private key, we are able to generate a unique and secure image representation at a precise distance. The generated image can undergo authentication by being processed through the proposed authentication system. Leveraging the utilization of invisible terahertz light, the certification system possesses inherent characteristics of concealment and enhanced security. Additionally, the entire certification process operates solely through the manipulation of the light beam, eliminating the need for electronic calculations. As a result, the system offers rapid certification speed. The proposed optical authentication scheme is further validated through computer simulations, which showcase its robust security and high precision. This method holds immense potential for diverse applications in optical neural network authentication, warranting a broad scope of future prospects.

Photonic neuromorphic architecture for tens-of-task lifelong learning

Scalable, high-capacity, and low-power computing architecture is the primary assurance for increasingly manifold and large-scale machine learning tasks. Traditional electronic artificial agents by conventional power-hungry processors have faced the issues of energy and scaling walls, hindering them from the sustainable performance improvement and iterative multi-task learning. Referring to another modality of light, photonic computing has been progressively applied in high-efficient neuromorphic systems. Here, we innovate a reconfigurable lifelong-learning optical neural network (L2ONN), for highly-integrated tens-of-task machine intelligence with elaborated algorithm-hardware co-design. Benefiting from the inherent sparsity and parallelism in massive photonic connections, L2ONN learns each single task by adaptively activating sparse photonic neuron connections in the coherent light field, while incrementally acquiring expertise on various tasks by gradually enlarging the activation. The multi-task optical features are parallelly processed by multi-spectrum representations allocated with different wavelengths. Extensive evaluations on free-space and on-chip architectures confirm that for the first time, L2ONN avoided the catastrophic forgetting issue of photonic computing, owning versatile skills on challenging tens-of-tasks (vision classification, voice recognition, medical diagnosis, etc.) with a single model. Particularly, L2ONN achieves more than an order of magnitude higher efficiency than the representative electronic artificial neural networks, and 14× larger capacity than existing optical neural networks while maintaining competitive performance on each individual task. The proposed photonic neuromorphic architecture points out a new form of lifelong learning scheme, permitting terminal/edge AI systems with light-speed efficiency and unprecedented scalability.

Correlated optical convolutional neural network with "quantum speedup"

Compared with electrical neural networks, optical neural networks (ONNs) have the potentials to break the limit of the bandwidth and reduce the consumption of energy, and therefore draw much attention in recent years. By far, several types of ONNs have been implemented. However, the current ONNs cannot realize the acceleration as powerful as that indicated by the models like quantum neural networks. How to construct and realize an ONN with the quantum speedup is a huge challenge. Here, we propose theoretically and demonstrate experimentally a new type of optical convolutional neural network by introducing the optical correlation. It is called the correlated optical convolutional neural network (COCNN). We show that the COCNN can exhibit "quantum speedup" in the training process. The character is verified from the two aspects. One is the direct illustration of the faster convergence by comparing the loss function curves of the COCNN with that of the traditional convolutional neural network (CNN). Such a result is compatible with the training performance of the recently proposed quantum convolutional neural network (QCNN). The other is the demonstration of the COCNN's capability to perform the QCNN phase recognition circuit, validating the connection between the COCNN and the QCNN. Furthermore, we take the COCNN analog to the 3-qubit QCNN phase recognition circuit as an example and perform an experiment to show the soundness and the feasibility of it. The results perfectly match the theoretical calculations. Our proposal opens up a new avenue for realizing the ONNs with the quantum speedup, which will benefit the information processing in the era of big data.

Training large-scale optoelectronic neural networks with dual-neuron optical-artificial learning

Optoelectronic neural networks (ONN) are a promising avenue in AI computing due to their potential for parallelization, power efficiency, and speed. Diffractive neural networks, which process information by propagating encoded light through trained optical elements, have garnered interest. However, training large-scale diffractive networks faces challenges due to the computational and memory costs of optical diffraction modeling. Here, we present DANTE, a dual-neuron optical-artificial learning architecture. Optical neurons model the optical diffraction, while artificial neurons approximate the intensive optical-diffraction computations with lightweight functions. DANTE also improves convergence by employing iterative global artificial-learning steps and local optical-learning steps. In simulation experiments, DANTE successfully trains large-scale ONNs with 150 million neurons on ImageNet, previously unattainable, and accelerates training speeds significantly on the CIFAR-10 benchmark compared to single-neuron learning. In physical experiments, we develop a two-layer ONN system based on DANTE, which can effectively extract features to improve the classification of natural images.

Graphene/silicon heterojunction for reconfigurable phase-relevant activation function in coherent optical neural networks

Optical neural networks (ONNs) herald a new era in information and communication technologies and have implemented various intelligent applications. In an ONN, the activation function (AF) is a crucial component determining the network performances and on-chip AF devices are still in development. Here, we first demonstrate on-chip reconfigurable AF devices with phase activation fulfilled by dual-functional graphene/silicon (Gra/Si) heterojunctions. With optical modulation and detection in one device, time delays are shorter, energy consumption is lower, reconfigurability is higher and the device footprint is smaller than other on-chip AF strategies. The experimental modulation voltage (power) of our Gra/Si heterojunction achieves as low as 1 V (0.5 mW), superior to many pure silicon counterparts. In the photodetection aspect, a high responsivity of over 200 mA/W is realized. Special nonlinear functions generated are fed into a complex-valued ONN to challenge handwritten letters and image recognition tasks, showing improved accuracy and potential of high-efficient, all-component-integration on-chip ONN. Our results offer new insights for on-chip ONN devices and pave the way to high-performance integrated optoelectronic computing circuits.

High-throughput terahertz imaging: progress and challenges

Many exciting terahertz imaging applications, such as non-destructive evaluation, biomedical diagnosis, and security screening, have been historically limited in practical usage due to the raster-scanning requirement of imaging systems, which impose very low imaging speeds. However, recent advancements in terahertz imaging systems have greatly increased the imaging throughput and brought the promising potential of terahertz radiation from research laboratories closer to real-world applications. Here, we review the development of terahertz imaging technologies from both hardware and computational imaging perspectives. We introduce and compare different types of hardware enabling frequency-domain and time-domain imaging using various thermal, photon, and field image sensor arrays. We discuss how different imaging hardware and computational imaging algorithms provide opportunities for capturing time-of-flight, spectroscopic, phase, and intensity image data at high throughputs. Furthermore, the new prospects and challenges for the development of future high-throughput terahertz imaging systems are briefly introduced.

C-DONN: compact diffractive optical neural network with deep learning regression

A new method to improve the integration level of an on-chip diffractive optical neural network (DONN) is proposed based on a standard silicon-on-insulator (SOI) platform. The metaline, which represents a hidden layer in the integrated on-chip DONN, is composed of subwavelength silica slots, providing a large computation capacity. However, the physical propagation process of light in the subwavelength metalinses generally requires an approximate characterization using slot groups and extra length between adjacent layers, which limits further improvements of the integration of on-chip DONN. In this work, a deep mapping regression model (DMRM) is proposed to characterize the process of light propagation in the metalines. This method improves the integration level of on-chip DONN to over 60,000 and elimnates the need for approximate conditions. Based on this theory, a compact-DONN (C-DONN) is exploited and benchmarked on the Iris plants dataset to verify the performance, yielding a testing accuracy of 93.3%. This method provides a potential solution for future large-scale on-chip integration.

Planar Fourier optics for slab waveguides, surface plasmon polaritons, and 2D materials

Recent experimental work has demonstrated the potential of combining the merits of diffractive and on-chip photonic information processing devices in a single chip by making use of planar (or slab) waveguides. Here, arguments are developed to show that diffraction formulas familiar from 3D Fourier optics can be adapted to 2D under certain mild conditions on the operating speeds of the devices in question. In addition to serving those working in on-chip photonics, this Letter provides analytical tools for the study of surface plasmon polaritons, surface waves, and the optical, acoustic, and crystallographic properties of 2D materials.

Optical neural ordinary differential equations

Increasing the layer number of on-chip photonic neural networks (PNNs) is essential to improve its model performance. However, the successive cascading of network hidden layers results in larger integrated photonic chip areas. To address this issue, we propose the optical neural ordinary differential equations (ON-ODEs) architecture that parameterizes the continuous dynamics of hidden layers with optical ODE solvers. The ON-ODE comprises the PNNs followed by the photonic integrator and optical feedback loop, which can be configured to represent residual neural networks (ResNets) and implement the function of recurrent neural networks with effectively reduced chip area occupancy. For the interference-based optoelectronic nonlinear hidden layer, the numerical experiments demonstrate that the single hidden layer ON-ODE can achieve approximately the same accuracy as the two-layer optical ResNets in image classification tasks. In addition, the ON-ODE improves the model classification accuracy for the diffraction-based all-optical linear hidden layer. The time-dependent dynamics property of ON-ODE is further applied for trajectory prediction with high accuracy.

Optical computing powers graph neural networks

Graph-based neural networks have promising perspectives but are limited by electronic bottlenecks. Our work explores the advantages of optical neural networks in the graph domain. We propose an optical graph neural network (OGNN) based on inverse-designed optical processing units (OPUs) to classify graphs with optics. The OPUs, combined with two types of optical components, can perform multiply-accumulate, matrix-vector multiplication, and matrix-matrix multiplication operations. The proposed OGNN can classify typical non-Euclidean MiniGCDataset graphs and successfully predict 1000 test graphs with 100% accuracy. The OPU-formed optical-electrical graph attention network is also scalable to handle more complex graph data, such as the Cora dataset, with 89.0% accuracy.

Chip-Based High-Dimensional Optical Neural Network

Parallel multi-thread processing in advanced intelligent processors is the core to realize high-speed and high-capacity signal processing systems. Optical neural network (ONN) has the native advantages of high parallelization, large bandwidth, and low power consumption to meet the demand of big data. Here, we demonstrate the dual-layer ONN with Mach-Zehnder interferometer (MZI) network and nonlinear layer, while the nonlinear activation function is achieved by optical-electronic signal conversion. Two frequency components from the microcomb source carrying digit datasets are simultaneously imposed and intelligently recognized through the ONN. We successfully achieve the digit classification of different frequency components by demultiplexing the output signal and testing power distribution. Efficient parallelization feasibility with wavelength division multiplexing is demonstrated in our high-dimensional ONN. This work provides a high-performance architecture for future parallel high-capacity optical analog computing.

Nonlinear germanium-silicon photodiode for activation and monitoring in photonic neuromorphic networks

Silicon photonics is promising for artificial neural networks computing owing to its superior interconnect bandwidth, low energy consumption and scalable fabrication. However, the lack of silicon-integrated and monitorable optical neurons limits its revolution in large-scale artificial neural networks. Here, we highlight nonlinear germanium-silicon photodiodes to construct on-chip optical neurons and a self-monitored all-optical neural network. With specifically engineered optical-to-optical and optical-to-electrical responses, the proposed neuron merges the all-optical activation and non-intrusive monitoring functions in a compact footprint of 4.3 × 8 μm2. Experimentally, a scalable three-layer photonic neural network enables in situ training and learning in object classification and semantic segmentation tasks. The performance of this neuron implemented in a deep-scale neural network is further confirmed via handwriting recognition, achieving a high accuracy of 97.3%. We believe this work will enable future large-scale photonic intelligent processors with more functionalities but simplified architecture.