Integrated photonic metasystem for image classifications at telecommunication wavelength

Diffractive optical computing in free space

Structured optical materials create new computing paradigms using photons, with transformative impact on various fields, including machine learning, computer vision, imaging, telecommunications, and sensing. This Perspective sheds light on the potential of free-space optical systems based on engineered surfaces for advancing optical computing. Manipulating light in unprecedented ways, emerging structured surfaces enable all-optical implementation of various mathematical functions and machine learning tasks. Diffractive networks, in particular, bring deep-learning principles into the design and operation of free-space optical systems to create new functionalities. Metasurfaces consisting of deeply subwavelength units are achieving exotic optical responses that provide independent control over different properties of light and can bring major advances in computational throughput and data-transfer bandwidth of free-space optical processors. Unlike integrated photonics-based optoelectronic systems that demand preprocessed inputs, free-space optical processors have direct access to all the optical degrees of freedom that carry information about an input scene/object without needing digital recovery or preprocessing of information. To realize the full potential of free-space optical computing architectures, diffractive surfaces and metasurfaces need to advance symbiotically and co-evolve in their designs, 3D fabrication/integration, cascadability, and computing accuracy to serve the needs of next-generation machine vision, computational imaging, mathematical computing, and telecommunication technologies.

Analog spatiotemporal feature extraction for cognitive radio-frequency sensing with integrated photonics

Analog feature extraction (AFE) is an appealing strategy for low-latency and efficient cognitive sensing systems since key features are much sparser than the Nyquist-sampled data. However, applying AFE to broadband radio-frequency (RF) scenarios is challenging due to the bandwidth and programmability bottlenecks of analog electronic circuitry. Here, we introduce a photonics-based scheme that extracts spatiotemporal features from broadband RF signals in the analog domain. The feature extractor structure inspired by convolutional neural networks is implemented on integrated photonic circuits to process RF signals from multiple antennas, extracting valid features from both temporal and spatial dimensions. Because of the tunability of the photonic devices, the photonic spatiotemporal feature extractor is trainable, which enhances the validity of the extracted features. Moreover, a digital-analog-hybrid transfer learning method is proposed for the effective and low-cost training of the photonic feature extractor. To validate our scheme, we demonstrate a radar target recognition task with a 4-GHz instantaneous bandwidth. Experimental results indicate that the photonic analog feature extractor tackles broadband RF signals and reduces the sampling rate of analog-to-digital converters to 1/4 of the Nyquist sampling while maintaining a high target recognition accuracy of 97.5%. Our scheme offers a promising path for exploiting the AFE strategy in the realm of cognitive RF sensing, with the potential to contribute to the efficient signal processing involved in applications such as autonomous driving, robotics, and smart factories.

Correlated optical convolutional neural network with "quantum speedup"

Compared with electrical neural networks, optical neural networks (ONNs) have the potentials to break the limit of the bandwidth and reduce the consumption of energy, and therefore draw much attention in recent years. By far, several types of ONNs have been implemented. However, the current ONNs cannot realize the acceleration as powerful as that indicated by the models like quantum neural networks. How to construct and realize an ONN with the quantum speedup is a huge challenge. Here, we propose theoretically and demonstrate experimentally a new type of optical convolutional neural network by introducing the optical correlation. It is called the correlated optical convolutional neural network (COCNN). We show that the COCNN can exhibit "quantum speedup" in the training process. The character is verified from the two aspects. One is the direct illustration of the faster convergence by comparing the loss function curves of the COCNN with that of the traditional convolutional neural network (CNN). Such a result is compatible with the training performance of the recently proposed quantum convolutional neural network (QCNN). The other is the demonstration of the COCNN's capability to perform the QCNN phase recognition circuit, validating the connection between the COCNN and the QCNN. Furthermore, we take the COCNN analog to the 3-qubit QCNN phase recognition circuit as an example and perform an experiment to show the soundness and the feasibility of it. The results perfectly match the theoretical calculations. Our proposal opens up a new avenue for realizing the ONNs with the quantum speedup, which will benefit the information processing in the era of big data.

Simulating an Integrated Photonic Image Classifier for Diffractive Neural Networks

The slowdown of Moore's law and the existence of the "von Neumann bottleneck" has led to electronic-based computing systems under von Neumann's architecture being unable to meet the fast-growing demand for artificial intelligence computing. However, all-optical diffractive neural networks provide a possible solution to this challenge. They can outperform conventional silicon-based electronic neural networks due to the significantly higher speed of the propagation of optical signals (≈108 m.s-1) compared to electrical signals (≈105 m.s-1), their parallelism in nature, and their low power consumption. The integrated diffractive deep neural network (ID2NN) uses an on-chip fully passive photonic approach to achieve the functionality of neural networks (matrix-vector operations) and can be fabricated via the CMOS process, which is technologically more amenable to implementing an artificial intelligence processor. In this paper, we present a detailed design framework for the integrated diffractive deep neural network and corresponding silicon-on-insulator integration implementation through Python-based simulations. The performance of our proposed ID2NN was evaluated by solving image classification problems using the MNIST dataset.

Recent advances in metasurface design and quantum optics applications with machine learning, physics-informed neural networks, and topology optimization methods

As a two-dimensional planar material with low depth profile, a metasurface can generate non-classical phase distributions for the transmitted and reflected electromagnetic waves at its interface. Thus, it offers more flexibility to control the wave front. A traditional metasurface design process mainly adopts the forward prediction algorithm, such as Finite Difference Time Domain, combined with manual parameter optimization. However, such methods are time-consuming, and it is difficult to keep the practical meta-atom spectrum being consistent with the ideal one. In addition, since the periodic boundary condition is used in the meta-atom design process, while the aperiodic condition is used in the array simulation, the coupling between neighboring meta-atoms leads to inevitable inaccuracy. In this review, representative intelligent methods for metasurface design are introduced and discussed, including machine learning, physics-information neural network, and topology optimization method. We elaborate on the principle of each approach, analyze their advantages and limitations, and discuss their potential applications. We also summarize recent advances in enabled metasurfaces for quantum optics applications. In short, this paper highlights a promising direction for intelligent metasurface designs and applications for future quantum optics research and serves as an up-to-date reference for researchers in the metasurface and metamaterial fields.

C-DONN: compact diffractive optical neural network with deep learning regression

A new method to improve the integration level of an on-chip diffractive optical neural network (DONN) is proposed based on a standard silicon-on-insulator (SOI) platform. The metaline, which represents a hidden layer in the integrated on-chip DONN, is composed of subwavelength silica slots, providing a large computation capacity. However, the physical propagation process of light in the subwavelength metalinses generally requires an approximate characterization using slot groups and extra length between adjacent layers, which limits further improvements of the integration of on-chip DONN. In this work, a deep mapping regression model (DMRM) is proposed to characterize the process of light propagation in the metalines. This method improves the integration level of on-chip DONN to over 60,000 and elimnates the need for approximate conditions. Based on this theory, a compact-DONN (C-DONN) is exploited and benchmarked on the Iris plants dataset to verify the performance, yielding a testing accuracy of 93.3%. This method provides a potential solution for future large-scale on-chip integration.

Folded Digital Meta-Lenses for on-Chip Spectrometer

In-plane diffractive optical networks based on meta-surfaces are promising for on-chip application. The design constraints of regular antenna unit place ultimate limits on the functionalities of the meta-systems. This fundamental limitation has been reflected by the large footprints of cascaded meta-surfaces. Here, we propose a digital meta-lens with a large degree of design freedom, enabling significantly improved beam focusing, collimation, and deflection capabilities. A highly dispersive and compact diffractive optical system is constructed for spectrometer via five layers of meta-lenses in a folded configuration. The device only occupies a 100 μm × 100 μm chip area on a silicon photonic platform. Sparse and continuous spectra reconstruction is achieved over a 35 nm bandwidth. Fine spectral lines separated by 0.14 nm are resolved. In addition to such a compact and high-resolution on-chip spectrometer, it is also expected to be promising for imaging, optical computing, and other applications due to the great versatility of the digital lens design.

Photonic unsupervised learning variational autoencoder for high-throughput and low-latency image transmission

Following the explosive growth of global data, there is an ever-increasing demand for high-throughput processing in image transmission systems. However, existing methods mainly rely on electronic circuits, which severely limits the transmission throughput. Here, we propose an end-to-end all-optical variational autoencoder, named photonic encoder-decoder (PED), which maps the physical system of image transmission into an optical generative neural network. By modeling the transmission noises as the variation in optical latent space, the PED establishes a large-scale high-throughput unsupervised optical computing framework that integrates main computations in image transmission, including compression, encryption, and error correction to the optical domain. It reduces the system latency of computation by more than four orders of magnitude compared with the state-of-the-art devices and transmission error ratio by 57% than on-off keying. Our work points to the direction for a wide range of artificial intelligence-based physical system designs and next-generation communications.

Effects of interlayer reflection and interpixel interaction in diffractive optical neural networks

Multilayer diffractive optical neural networks (DONNs) can perform machine learning (ML) tasks at the speed of light with low energy consumption. Decreasing the number of diffractive layers can reduce inevitable material and diffraction losses to improve system performance, and incorporating compact devices can reduce the system footprint. However, current analytical DONN models cannot accurately describe such physical systems. Here we show the ever-ignored effects of interlayer reflection and interpixel interaction on the deployment performance of DONNs through full-wave electromagnetic simulations and terahertz (THz) experiments. We demonstrate that the drop of handwritten digit classification accuracy due to reflection is negligible with conventional low-index THz polymer materials, while it can be substantial with high-index materials. We further show that one- and few-layer DONN systems can achieve high classification accuracy, but there is a trade-off between accuracy and model-system matching rate because of the fast-varying spatial distribution of optical responses in diffractive masks. Deep DONNs can break down such a trade-off because of reduced mask spatial complexity. Our results suggest that new accurate and trainable DONN models are needed to advance the development and deployment of compact DONN systems for sophisticated ML tasks.

Realization of optical logic gates using on-chip diffractive optical neural networks

Optical computing is highly desired as a potential strategy for circumventing the performance limitations of semiconductor-based electronic devices and circuits. Optical logic gates are considered as fundamental building blocks for optical computation and they enable logic functions to be performed extremely quickly without the generation of heat and crosstalk. Here, we discuss the design of a multi-functional optical logic gate based on an on-chip diffractive optical neural network that can perform AND, NOT and OR logic operations at the wavelength of 1.55 µm. The wavelength-independent operation of the multi-functional logic gate at seven wavelengths (over a bandwidth of 60 nm) is also studied which paves the way for wavelength division multiplexed parallel computation. This simple, highly-integrable, low-loss, energy-efficient and broadband optical logic gate provides a path for the development of high-speed on-chip nanophotonic processors for future optical computing applications.

Metasurface-enabled on-chip multiplexed diffractive neural networks in the visible

Replacing electrons with photons is a compelling route toward high-speed, massively parallel, and low-power artificial intelligence computing. Recently, diffractive networks composed of phase surfaces were trained to perform machine learning tasks through linear optical transformations. However, the existing architectures often comprise bulky components and, most critically, they cannot mimic the human brain for multitasking. Here, we demonstrate a multi-skilled diffractive neural network based on a metasurface device, which can perform on-chip multi-channel sensing and multitasking in the visible. The polarization multiplexing scheme of the subwavelength nanostructures is applied to construct a multi-channel classifier framework for simultaneous recognition of digital and fashionable items. The areal density of the artificial neurons can reach up to 6.25 × 10⁶ mm^-2 multiplied by the number of channels. The metasurface is integrated with the mature complementary metal-oxide semiconductor imaging sensor, providing a chip-scale architecture to process information directly at physical layers for energy-efficient and ultra-fast image processing in machine vision, autonomous driving, and precision medicine.