Inference in artificial intelligence with deep optics and photonics

All-Optical Data Processing with Photon-Avalanching Nanocrystalline Photonic Synapse

Data processing and storage in electronic devices are typically performed as a sequence of elementary binary operations. Alternative approaches, such as neuromorphic or reservoir computing, are rapidly gaining interest where data processing is relatively slow, but can be performed in a more comprehensive way or massively in parallel, like in neuronal circuits. Here, time-domain all-optical information processing capabilities of photon-avalanching (PA) nanoparticles at room temperature are discovered. Demonstrated functionality resembles properties found in neuronal synapses, such as: paired-pulse facilitation and short-term internal memory, in situ plasticity, multiple inputs processing, and all-or-nothing threshold response. The PA-memory-like behavior shows capability of machine-learning-algorithm-free feature extraction and further recognition of 2D patterns with simple 2 input artificial neural network. Additionally, high nonlinearity of luminescence intensity in response to photoexcitation mimics and enhances spike-timing-dependent plasticity that is coherent in nature with the way a sound source is localized in animal neuronal circuits. Not only are yet unexplored fundamental properties of photon-avalanche luminescence kinetics studied, but this approach, combined with recent achievements in photonics, light confinement and guiding, promises all-optical data processing, control, adaptive responsivity, and storage on photonic chips.

Multi-wavelength diffractive neural network with the weighting method

Recently, the diffractive deep neural network (D2NN) has demonstrated the advantages to achieve large-scale computational tasks in terms of high speed, low power consumption, parallelism, and scalability. A typical D2NN with cascaded diffractive elements is designed for monochromatic illumination. Here, we propose a framework to achieve the multi-wavelength D2NN (MW-D2NN) based on the method of weight coefficients. In training, each wavelength is assigned a specific weighting and their output planes construct the wavelength weighting loss function. The trained MW-D2NN can implement the classification of images of handwritten digits at multi-wavelength incident beams. The designed 3-layers MW-D2NN achieves a simulation classification accuracy of 83.3%. We designed a 1-layer MW-D2NN. The simulation and experiment classification accuracy are 71.4% and 67.5% at RGB wavelengths. Furthermore, the proposed MW-D2NN can be extended to intelligent machine vision systems for multi-wavelength and incoherent illumination.

Referenceless characterization of complex media using physics-informed neural networks

In this work, we present a method to characterize the transmission matrices of complex scattering media using a physics-informed, multi-plane neural network (MPNN) without the requirement of a known optical reference field. We use this method to accurately measure the transmission matrix of a commercial multi-mode fiber without the problems of output-phase ambiguity and dark spots, leading to up to 58% improvement in focusing efficiency compared with phase-stepping holography. We demonstrate how our method is significantly more noise-robust than phase-stepping holography and show how it can be generalized to characterize a cascade of transmission matrices, allowing one to control the propagation of light between independent scattering media. This work presents an essential tool for accurate light control through complex media, with applications ranging from classical optical networks, biomedical imaging, to quantum information processing.

High-throughput terahertz imaging: progress and challenges

Many exciting terahertz imaging applications, such as non-destructive evaluation, biomedical diagnosis, and security screening, have been historically limited in practical usage due to the raster-scanning requirement of imaging systems, which impose very low imaging speeds. However, recent advancements in terahertz imaging systems have greatly increased the imaging throughput and brought the promising potential of terahertz radiation from research laboratories closer to real-world applications. Here, we review the development of terahertz imaging technologies from both hardware and computational imaging perspectives. We introduce and compare different types of hardware enabling frequency-domain and time-domain imaging using various thermal, photon, and field image sensor arrays. We discuss how different imaging hardware and computational imaging algorithms provide opportunities for capturing time-of-flight, spectroscopic, phase, and intensity image data at high throughputs. Furthermore, the new prospects and challenges for the development of future high-throughput terahertz imaging systems are briefly introduced.

Single-pixel image reconstruction using coherent nonlinear optics

We propose and experimentally demonstrate a novel, to the best of our knowledge, hybrid optoelectronic system that utilizes mode-selective frequency upconversion, single-pixel detection, and a deep neural network to achieve the reliable reconstruction of two-dimensional (2D) images from a noise-contaminated database of handwritten digits. Our system is designed to maximize the multi-scale structural similarity index measure (MS-SSIM) and minimize the mean absolute error (MAE) during the training process. Through extensive evaluation, we have observed that the reconstructed images exhibit high-quality results, with a peak signal-to-noise ratio (PSNR) reaching approximately 20 dB and a structural similarity index measure (SSIM) of around 0.85. These impressive metrics demonstrate the effectiveness and fidelity of our image reconstruction technique. The versatility of our approach allows its application in various fields, including Lidar, compressive imaging, volumetric reconstruction, and so on.

Experimental realization of convolution processing in photonic synthetic frequency dimensions

Convolution is an essential operation in signal and image processing and consumes most of the computing power in convolutional neural networks. Photonic convolution has the promise of addressing computational bottlenecks and outperforming electronic implementations. Performing photonic convolution in the synthetic frequency dimension, which harnesses the dynamics of light in the spectral degrees of freedom for photons, can lead to highly compact devices. Here, we experimentally realize convolution operations in the synthetic frequency dimension. Using a modulated ring resonator, we synthesize arbitrary convolution kernels using a predetermined modulation waveform with high accuracy. We demonstrate the convolution computation between input frequency combs and synthesized kernels. We also introduce the idea of an additive offset to broaden the kinds of kernels that can be implemented experimentally when the modulation strength is limited. Our work demonstrate the use of synthetic frequency dimension to efficiently encode data and implement computation tasks, leading to a compact and scalable photonic computation architecture.

Tunable nonlinear optical mapping in a multiple-scattering cavity

Nonlinear disordered systems are not only a model system for fundamental studies but also in high demand for practical applications. However, optical nonlinearity based on intrinsic material response is weak in random scattering systems. Here, we propose and experimentally realize a highly nonlinear mapping between the scattering potential and the emerging light of a reconfigurable multiple-scattering cavity. A quantitative analysis of the degree of nonlinearity reveals its dependence on the number of scattering events. The effective order of nonlinear mapping can be tuned over a wide range at low optical lower. The strong nonlinear mapping enhances output intensity fluctuations and long-range correlations. The flexibility, robustness, and energy efficiency of our approach provides a versatile platform for exploring such nonlinear mappings for various applications.

Optofluidic memory and self-induced nonlinear optical phase change for reservoir computing in silicon photonics

Nanophotonics allows to employ light-matter interaction to induce nonlinear optical effects and realize non-conventional memory and computation capabilities, however to date, light-liquid interaction was not considered as a potential mechanism to achieve computation on a nanoscale. Here, we experimentally demonstrate self-induced phase change effect which relies on the coupling between geometric changes of thin liquid film to optical properties of photonic waveguide modes, and then employ it for neuromorphic computing. In our optofluidic silicon photonics system we utilize thermocapillary-based deformation of thin liquid film capable to induce nonlinear effect which is more than one order of magnitude higher compared to the more traditional heat-based thermo-optical effect, and allowing operation as a nonlinear actuator and memory element, both residing at the same compact spatial region. The resulting dynamics allows to implement Reservoir Computing at spatial region which is approximately five orders of magnitude smaller compared to state-of-the-art experimental liquid-based systems.

Biasing the quantum vacuum to control macroscopic probability distributions

Quantum field theory suggests that electromagnetic fields naturally fluctuate, and these fluctuations can be harnessed as a source of perfect randomness. Many potential applications of randomness rely on controllable probability distributions. We show that vacuum-level bias fields injected into multistable optical systems enable a controllable source of quantum randomness, and we demonstrated this concept in an optical parametric oscillator (OPO). By injecting bias pulses with less than one photon on average, we controlled the probabilities of the two possible OPO output states. The potential of our approach for sensing sub-photon-level fields was demonstrated by reconstructing the temporal shape of fields below the single-photon level. Our results provide a platform to study quantum dynamics in nonlinear driven-dissipative systems and point toward applications in probabilistic computing and weak field sensing.

Large depth-of-field ultra-compact microscope by progressive optimization and deep learning

The optical microscope is customarily an instrument of substantial size and expense but limited performance. Here we report an integrated microscope that achieves optical performance beyond a commercial microscope with a 5×, NA 0.1 objective but only at 0.15 cm³ and 0.5 g, whose size is five orders of magnitude smaller than that of a conventional microscope. To achieve this, a progressive optimization pipeline is proposed which systematically optimizes both aspherical lenses and diffractive optical elements with over 30 times memory reduction compared to the end-to-end optimization. By designing a simulation-supervision deep neural network for spatially varying deconvolution during optical design, we accomplish over 10 times improvement in the depth-of-field compared to traditional microscopes with great generalization in a wide variety of samples. To show the unique advantages, the integrated microscope is equipped in a cell phone without any accessories for the application of portable diagnostics. We believe our method provides a new framework for the design of miniaturized high-performance imaging systems by integrating aspherical optics, computational optics, and deep learning.

A Comprehensive Review of Bio-Inspired Optimization Algorithms Including Applications in Microelectronics and Nanophotonics

The application of artificial intelligence in everyday life is becoming all-pervasive and unavoidable. Within that vast field, a special place belongs to biomimetic/bio-inspired algorithms for multiparameter optimization, which find their use in a large number of areas. Novel methods and advances are being published at an accelerated pace. Because of that, in spite of the fact that there are a lot of surveys and reviews in the field, they quickly become dated. Thus, it is of importance to keep pace with the current developments. In this review, we first consider a possible classification of bio-inspired multiparameter optimization methods because papers dedicated to that area are relatively scarce and often contradictory. We proceed by describing in some detail some more prominent approaches, as well as those most recently published. Finally, we consider the use of biomimetic algorithms in two related wide fields, namely microelectronics (including circuit design optimization) and nanophotonics (including inverse design of structures such as photonic crystals, nanoplasmonic configurations and metamaterials). We attempted to keep this broad survey self-contained so it can be of use not only to scholars in the related fields, but also to all those interested in the latest developments in this attractive area.

Optical micro-phase-shift dropvolume in a diffractive deep neural network

To provide a desirable number of parallel subnetworks as required to reach a robust inference in an active modulation diffractive deep neural network, a random micro-phase-shift dropvolume that involves five-layer statistically independent dropconnect arrays is monolithically embedded into the unitary backpropagation, which does not require any mathematical derivations with respect to the multilayer arbitrary phase-only modulation masks, even maintaining the nonlinear nested characteristic of neural networks, and generating an opportunity to realize a structured-phase encoding within the dropvolume. Further, a drop-block strategy is introduced into the structured-phase patterns designed to flexibly configure a credible macro-micro phase dropvolume allowing for convergence. Concretely, macro-phase dropconnects concerning fringe griddles that encapsulate sparse micro-phase are implemented. We numerically validate that macro-micro phase encoding is a good plan to the types of encoding within a dropvolume.

Ultrafast dynamic machine vision with spatiotemporal photonic computing

Ultrafast dynamic machine vision in the optical domain can provide unprecedented perspectives for high-performance computing. However, owing to the limited degrees of freedom, existing photonic computing approaches rely on the memory's slow read/write operations to implement dynamic processing. Here, we propose a spatiotemporal photonic computing architecture to match the highly parallel spatial computing with high-speed temporal computing and achieve a three-dimensional spatiotemporal plane. A unified training framework is devised to optimize the physical system and the network model. The photonic processing speed of the benchmark video dataset is increased by 40-fold on a space-multiplexed system with 35-fold fewer parameters. A wavelength-multiplexed system realizes all-optical nonlinear computing of dynamic light field with a frame time of 3.57 nanoseconds. The proposed architecture paves the way for ultrafast advanced machine vision free from the limits of memory wall and will find applications in unmanned systems, autonomous driving, ultrafast science, etc.

Microring-based programmable coherent optical neural networks

Coherent programmable integrated photonics circuits have shown great potential as specialized hardware accelerators for deep learning tasks, which usually involve the use of linear matrix multiplication and nonlinear activation components. We design, simulate and train an optical neural network fully based on microring resonators, which shows advantages in terms of device footprint and energy efficiency. We use tunable coupled double ring structures as the interferometer components for the linear multiplication layers and modulated microring resonators as the reconfigurable nonlinear activation components. We then develop optimization algorithms to train the direct tuning parameters such as applied voltages based on the transfer matrix method and using automatic differentiation for all optical components.

Artificial Intelligence and Advanced Materials

Artificial intelligence (AI) is gaining strength, and materials science can both contribute to and profit from it. In a simultaneous progress race, new materials, systems, and processes can be devised and optimized thanks to machine learning (ML) techniques, and such progress can be turned into innovative computing platforms. Future materials scientists will profit from understanding how ML can boost the conception of advanced materials. This review covers aspects of computation from the fundamentals to directions taken and repercussions produced by computation to account for the origins, procedures, and applications of AI. ML and its methods are reviewed to provide basic knowledge of its implementation and its potential. The materials and systems used to implement AI with electric charges are finding serious competition from other information-carrying and processing agents. The impact these techniques have on the inception of new advanced materials is so deep that a new paradigm is developing where implicit knowledge is being mined to conceive materials and systems for functions instead of finding applications to found materials. How far this trend can be carried is hard to fathom, as exemplified by the power to discover unheard of materials or physical laws buried in data.

Neuromorphic Photonics Based on Phase Change Materials

Neuromorphic photonics devices based on phase change materials (PCMs) and silicon photonics technology have emerged as promising solutions for addressing the limitations of traditional spiking neural networks in terms of scalability, response delay, and energy consumption. In this review, we provide a comprehensive analysis of various PCMs used in neuromorphic devices, comparing their optical properties and discussing their applications. We explore materials such as GST (Ge₂Sb₂Te₅), GeTe-Sb₂Te₃, GSST (Ge₂Sb₂Se₄Te₁), Sb₂S₃/Sb₂Se₃, Sc_0.2Sb₂Te₃ (SST), and In₂Se₃, highlighting their advantages and challenges in terms of erasure power consumption, response rate, material lifetime, and on-chip insertion loss. By investigating the integration of different PCMs with silicon-based optoelectronics, this review aims to identify potential breakthroughs in computational performance and scalability of photonic spiking neural networks. Further research and development are essential to optimize these materials and overcome their limitations, paving the way for more efficient and high-performance photonic neuromorphic devices in artificial intelligence and high-performance computing applications.

Noise Cancellation Effects in Integrated Photonics with Wilkinson Power Dividers

Wilkinson power dividers (WPDs) are a popular element in RF and microwave technologies known for providing isolation capabilities. However, the benefits that WPDs could offer to integrated photonic systems are far less studied. Here, we investigate the thermal emission from and the noise performance of silicon-on-insulator (SOI) WPDs. We find that WPDs exhibit a noiseless port, with important implications for receiving systems and absorption-based quantum state transformations. At the same time, the thermal signals exiting noisy ports exhibit nontrivial correlations, opening the possibility for noise cancellation. We analyze passive and active networks containing WPDs showing how such nontrivial correlations can prevent the amplification of the thermal noise introduced by WPDs while benefiting from their isolation capabilities. Using this insight, we propose a modified ring-resonator amplifier that improves by N times the SNR in comparison with conventional traveling wave and ring-resonator amplifiers, with N being the number of inputs/outputs of the WPD. We believe that our results represent an important step forward in the implementation of SOI-WPDs and their integration in complex photonic networks, particularly for mid-IR and quantum photonics applications.

Fast reconstruction of programmable integrated interferometers

Programmable linear optical interferometers are important for classical and quantum information technologies, as well as for building hardware-accelerated artificial neural networks. Recent results showed the possibility of constructing optical interferometers that could implement arbitrary transformations of input fields even in the case of high manufacturing errors. The building of detailed models of such devices drastically increases the efficiency of their practical use. The integral design of interferometers complicates its reconstruction since the internal elements are hard to address. This problem can be approached by using optimization algorithms [Opt. Express29, 38429 (2021)10.1364/OE.432481]. In this paper, we present what we believe to be a novel efficient algorithm based on linear algebra only, which does not use computationally expensive optimization procedures. We show that this approach makes it possible to perform fast and accurate characterization of high-dimensional programmable integrated interferometers. Moreover, the method provides access to the physical characteristics of individual interferometer layers.

Dammann gratings-based truly parallel optical matrix multiplication accelerator

Matrix multiplication (MM) is a fundamental operation in various scientific and engineering computations, as well as in artificial intelligence algorithms. Efficient implementation of MM is crucial for speeding up numerous applications. Photonics presents an opportunity for efficient acceleration of dense matrix computation, owing to its intrinsic advantages, such as huge parallelism, low latency, and low power consumption. However, most optical matrix computing architectures have been limited to realizing single-channel vector-matrix multiplication or using complex configurations to expand the number of channels, which does not fully exploit the parallelism of optics. In this study, we propose a novel, to the best of our knowledge, scheme for the implementation of large-scale two-dimensional optical MM with truly massive parallelism based on a specially designed Dammann grating. We demonstrate a sequence of MMs of 50 pairs of randomly generated 4 × 8 and 8 × 4 matrices in our proof-of-principle experiment. The results indicate that the mean relative error is approximately 0.048, thereby demonstrating optical robustness and high accuracy.

CHARLES: A C++ fixed-point library for Photonic-Aware Neural Networks

In this paper we present CHARLES (C++ pHotonic Aware neuRaL nEtworkS), a C++ library aimed at providing a flexible tool to simulate the behavior of Photonic-Aware Neural Network (PANN). PANNs are neural network architectures aware of the constraints due to the underlying photonic hardware, mostly in terms of low equivalent precision of the computations. For this reason, CHARLES exploits fixed-point computations for inference, while it supports both floating-point and fixed-point numerical formats for training. In this way, we can compare the effects due to the quantization in the inference phase when the training phase is performed on a classical floating-point model and on a model exploiting high-precision fixed-point numbers. To validate CHARLES and identify the most suited numerical format for PANN training, we report the simulation results obtained considering three datasets: Iris, MNIST, and Fashion-MNIST. Fixed-training is shown to outperform floating-training when executing inference on bitwidths suitable for photonic implementation. Indeed, performing the training phase in the floating-point domain and then quantizing to lower bitwidths results in a very high accuracy loss. Instead, when fixed-point numbers are exploited in the training phase, the accuracy loss due to quantization to lower bitwidths is significantly reduced. In particular, we show that for Iris dataset, fixed-training achieves a performance similar to floating-training. Fixed-training allows to obtain an accuracy of 90.4% and 68.1% with the MNIST and Fashion-MNIST datasets using only 6 bits, while the floating-training reaches an accuracy of just 25.4% and 50.0% when exploiting the same bitwidths.

Learning photons go backward

Efficient learning algorithms are implemented in a silicon photonic neural network chip.

Real-time adaptive optical self-interference cancellation for in-band full-duplex transmission using SARSA(λ) reinforcement learning

Self-interference (SI) due to signal leakage from a local transmitter is an issue in an in-band full-duplex (IBFD) transmission system, which would cause severe distortions to a receiving signal of interest (SOI). By superimposing a local reference signal with the same amplitude and opposite phase, the SI signal can be fully canceled. However, as the manipulation of the reference signal is usually operated manually, it is difficult to ensure a high speed and high accurate cancellation. To overcome this problem, a real-time adaptive optical SI cancellation (RTA-OSIC) scheme using a SARSA(λ) reinforcement learning (RL) algorithm is proposed and experimentally demonstrated. The proposed RTA-OSIC scheme can automatically adjust the amplitude and phase of a reference signal by adjusting a variable optical attenuator (VOA) and a variable optical delay line (VODL) achieved through an adaptive feedback signal, which is generated by evaluating the quality of the received SOI. To verify the feasibility of the proposed scheme, a 5 GHz 16QAM OFDM IBFD transmission experiment is demonstrated. By using the proposed RTA-OSIC scheme, for an SOI at three different bandwidths of 200, 400, and 800 MHz, the signal can be adaptively and correctly recovered within 8 time periods (TPs), which is the required time of a single adaptive control step. The cancellation depth for the SOI with a bandwidth of 800 MHz is 20.18 dB. The short- and long-term stability of the proposed RTA-OSIC scheme is also evaluated. The experimental results indicate that the proposed approach could be a promising solution for real-time adaptive SI cancellation in future IBFD transmission systems.

Simultaneously Sorting Overlapping Quantum States of Light

The efficient manipulation, sorting, and measurement of optical modes and single-photon states is fundamental to classical and quantum science. Here, we realize simultaneous and efficient sorting of nonorthogonal, overlapping states of light, encoded in the transverse spatial degree of freedom. We use a specifically designed multiplane light converter to sort states encoded in dimensions ranging from d=3 to d=7. Through the use of an auxiliary output mode, the multiplane light converter simultaneously performs the unitary operation required for unambiguous discrimination and the basis change for the outcomes to be spatially separated. Our results lay the groundwork for optimal image identification and classification via optical networks, with potential applications ranging from self-driving cars to quantum communication systems.

Snapshot multispectral imaging using a diffractive optical network

Multispectral imaging has been used for numerous applications in e.g., environmental monitoring, aerospace, defense, and biomedicine. Here, we present a diffractive optical network-based multispectral imaging system trained using deep learning to create a virtual spectral filter array at the output image field-of-view. This diffractive multispectral imager performs spatially-coherent imaging over a large spectrum, and at the same time, routes a pre-determined set of spectral channels onto an array of pixels at the output plane, converting a monochrome focal-plane array or image sensor into a multispectral imaging device without any spectral filters or image recovery algorithms. Furthermore, the spectral responsivity of this diffractive multispectral imager is not sensitive to input polarization states. Through numerical simulations, we present different diffractive network designs that achieve snapshot multispectral imaging with 4, 9 and 16 unique spectral bands within the visible spectrum, based on passive spatially-structured diffractive surfaces, with a compact design that axially spans ~72λ_m, where λ_m is the mean wavelength of the spectral band of interest. Moreover, we experimentally demonstrate a diffractive multispectral imager based on a 3D-printed diffractive network that creates at its output image plane a spatially repeating virtual spectral filter array with 2 × 2 = 4 unique bands at terahertz spectrum. Due to their compact form factor and computation-free, power-efficient and polarization-insensitive forward operation, diffractive multispectral imagers can be transformative for various imaging and sensing applications and be used at different parts of the electromagnetic spectrum where high-density and wide-area multispectral pixel arrays are not widely available.

Imaging-based intelligent spectrometer on a plasmonic rainbow chip

Compact, lightweight, and on-chip spectrometers are required to develop portable and handheld sensing and analysis applications. However, the performance of these miniaturized systems is usually much lower than their benchtop laboratory counterparts due to oversimplified optical architectures. Here, we develop a compact plasmonic "rainbow" chip for rapid, accurate dual-functional spectroscopic sensing that can surpass conventional portable spectrometers under selected conditions. The nanostructure consists of one-dimensional or two-dimensional graded metallic gratings. By using a single image obtained by an ordinary camera, this compact system can accurately and precisely determine the spectroscopic and polarimetric information of the illumination spectrum. Assisted by suitably trained deep learning algorithms, we demonstrate the characterization of optical rotatory dispersion of glucose solutions at two-peak and three-peak narrowband illumination across the visible spectrum using just a single image. This system holds the potential for integration with smartphones and lab-on-a-chip systems to develop applications for in situ analysis.

Single-shot polarimetry of vector beams by supervised learning

States of light encoding multiple polarizations - vector beams - offer unique capabilities in metrology and communication. However, their practical application is limited by the lack of methods for measuring many polarizations in a scalable and compact way. Here we demonstrate polarimetry of vector beams in a single shot without any polarization optics. We map the beam polarization content into a spatial intensity distribution through light scattering and exploit supervised learning for single-shot measurements of multiple polarizations. We characterize structured light encoding up to nine polarizations with accuracy beyond 95% on each Stokes parameter. The method also allows us to classify beams with an unknown number of polarization modes, a functionality missing in conventional techniques. Our findings enable a fast and compact polarimeter for polarization-structured light, a general tool that may radically impact optical devices for sensing, imaging, and computing.

High-contrast optical bistability using a subwavelength epsilon-near-zero material

Optical bistability opens up a promising avenue toward various optical nonlinear functions analogous to their electrical counterparts, such as switches, logic gates, and memory. Free-space bistable devices have unique advantages in large-scale integration. However, most proposed free-space schemes for optical bistability have limitations in one or more aspects of low contrast ratio, compromised compatibility, slow switching speed, and bulk size. Epsilon-near-zero (ENZ) materials have recently shown an ultrafast and giant optical nonlinearity within a subwavelength scale, potentially overcoming these obstacles. Using large-mobility indium-doped cadmium oxide (CdO) as the ENZ material, we numerically demonstrate two efficient schemes for high-contrast optical bistability within a deep subwavelength size based on the ENZ mode and the Berreman mode. The ENZ wavelength can be optically tuned with a typical time scale of sub-picoseconds, giving rise to a switchable bistability between the near-zero state and the high-reflection state. Our work contributes to the advances on compact and ultrafast all-optical signal processing.

High-efficiency FBG array sensor interrogation system via a neural network working with sparse data

FBG array sensors have been widely used in the multi-point monitoring of large structures due to their excellent optical multiplexing capability. This paper proposes a cost-effective demodulation system for FBG array sensors based on a Neural Network (NN). The stress variations applied to the FBG array sensor are encoded by the array waveguide grating (AWG) as transmitted intensities under different channels and fed to an end-to-end NN model, which receives them and simultaneously establishes a complex nonlinear relationship between the transmitted intensity and the actual wavelength to achieve absolute interrogation of the peak wavelength. In addition, a low-cost data augmentation strategy is introduced to break the data size bottleneck common in data-driven methods so that the NN can still achieve superior performance with small-scale data. In summary, the demodulation system provides an efficient and reliable solution for multi-point monitoring of large structures based on FBG array sensors.

Optical-electronic hybrid Fourier convolutional neural network based on super-pixel complex-valued modulation

An optical-electronic hybrid convolutional neural network (CNN) system is proposed and investigated for its parallel processing capability and system design robustness. It is regarded as a practical way to implement real-time optical computing. In this paper, we propose a complex-valued modulation method based on an amplitude-only liquid-crystal-on-silicon spatial light modulator and a fixed four-level diffractive optical element. A comparison of computational results of convolutions between different modulation methods in the Fourier plane shows the feasibility of the proposed complex-valued modulation method. A hybrid CNN model with one convolutional layer of multiple channels is proposed and trained electrically for different classification tasks. Our simulation results show that this model has a classification accuracy of 97.55% for MNIST, 88.81% for Fashion MNIST, and 56.16% for Cifar10, which outperforms models using only amplitude or phase modulation and is comparable to the ideal complex-valued modulation method.

Learned digital lens enabled single optics achromatic imaging

High-quality imaging with reduced optical complexity has been extensively investigated owing to its promising future in academic and industrial research. However, the practical performance of most imaging systems has encountered a bottleneck posed by optics rather than electronics. Here, we propose a digital lens (DL) to compensate for the chromatic aberration induced by physical optical elements, while the residual wavelength-independent degradation is tackled through a self-designed neural network. By transforming physical aberration correction to an algorithm-based computational imaging task, the proposed DL enables our framework to reduce optical complexity and achieve achromatic imaging in the analog domain. Real experiments have been conducted with an off-the-shelf single lens and recovered images show up to 14.62 dB higher peak signal-to-noise ratio (PSNR) than the original chromatic input. Furthermore, we run a comprehensive ablation study to highlight the contribution of embedding the proposed DL, which shows a 4.83 dB PSNR improvement compared with the methods without DL. Technically, the proposed method can be an alternative for future applications that require both simple optics and high-fidelity visualization.

Evolving scattering networks for engineering disorder

Network science provides a powerful tool for unraveling the complexities of social, technological and biological systems. Constructing networks using wave phenomena is also of great interest in devising advanced hardware for machine learning, as shown in optical neural networks. Although most wave-based networks have employed static network models, the impact of evolving models in network science provides strong motivation to apply dynamical network modeling to wave physics. Here the concept of evolving scattering networks for scattering phenomena is developed. The network is defined by links, node degrees and their evolution processes modeling multi-particle interferences, which directly determine scattering from disordered materials. I demonstrate the concept by examining network-based material classification, microstructure screening and preferential attachment in evolutions, which are applied to stealthy hyperuniformity. The results enable independent control of scattering from different length scales, revealing superdense material phases in short-range order. The proposed concept provides a bridge between wave physics and network science to resolve multiscale material complexities and open-system material design.

Multiple-scattering simulator-trained neural network for intensity diffraction tomography

Recovering 3D phase features of complex biological samples traditionally sacrifices computational efficiency and processing time for physical model accuracy and reconstruction quality. Here, we overcome this challenge using an approximant-guided deep learning framework in a high-speed intensity diffraction tomography system. Applying a physics model simulator-based learning strategy trained entirely on natural image datasets, we show our network can robustly reconstruct complex 3D biological samples. To achieve highly efficient training and prediction, we implement a lightweight 2D network structure that utilizes a multi-channel input for encoding the axial information. We demonstrate this framework on experimental measurements of weakly scattering epithelial buccal cells and strongly scattering C. elegans worms. We benchmark the network's performance against a state-of-the-art multiple-scattering model-based iterative reconstruction algorithm. We highlight the network's robustness by reconstructing dynamic samples from a living worm video. We further emphasize the network's generalization capabilities by recovering algae samples imaged from different experimental setups. To assess the prediction quality, we develop a quantitative evaluation metric to show that our predictions are consistent with both multiple-scattering physics and experimental measurements.

Why optics needs thickness

This study shows why and when optical systems need thickness as well as width or area. Wave diffraction explains the fundamental need for area or diameter of a lens or aperture to achieve some resolution or number of pixels in microscopes and cameras. This work demonstrates that if we know what the optics is to do, even before design, we can also deduce the minimum required thickness. This limit comes from diffraction combined with a concept called overlapping nonlocality C that can be deduced rigorously from just the mathematical description of what the device is to do. C expresses how much the input regions for different output regions overlap. This limit applies broadly to optics, from cameras to metasurfaces, and to wave systems generally.

Quantitative phase contrast imaging with a nonlocal angle-selective metasurface

Phase contrast microscopy has played a central role in the development of modern biology, geology, and nanotechnology. It can visualize the structure of translucent objects that remains hidden in regular optical microscopes. The optical layout of a phase contrast microscope is based on a 4 f image processing setup and has essentially remained unchanged since its invention by Zernike in the early 1930s. Here, we propose a conceptually new approach to phase contrast imaging that harnesses the non-local optical response of a guided-mode-resonator metasurface. We highlight its benefits and demonstrate the imaging of various phase objects, including biological cells, polymeric nanostructures, and transparent metasurfaces. Our results showcase that the addition of this non-local metasurface to a conventional microscope enables quantitative phase contrast imaging with a 0.02π phase accuracy. At a high level, this work adds to the growing body of research aimed at the use of metasurfaces for analog optical computing.

Adaptive Memory and In Materia Reinforcement Learning Enabled by Flexoelectric-like Response from Ultrathin HfO2

Reinforcement learning (RL) is a mathematical framework of neural learning by trial and error that revolutionized the field of artificial intelligence. However, until now, RL has been implemented in algorithms with the compatibly of traditional complementary metal-oxide-semiconductor-based von Neumann digital platforms, which thus limits performance in terms of latency, fault tolerance, and robustness. Here, we demonstrate that nanocolumnar (∼12 nm) HfO₂ structures can be used as building blocks to conduct the RL within the material by combining its stress-adjustable charge transport and memory functions. Specifically, HfO₂ nanostructures grown by the sputtering method exhibit self-assembled vertical nanocolumnar structures that generate voltage depending on the impact of stress under self-biased conditions. The observed results are attributed to the flexoelectric-like response of HfO₂. Further, multilevel current (>10^-3 A) modulation with touch and controlled suspension (∼10^-12 A) with a short electric pulse (100 ms) were demonstrated, yielding a proof-of-concept memory with an on/off ratio greater than 10⁹. Utilizing multipattern dynamic memory and tactile sensing, RL was implemented to successfully solve a maze game using an array of 6 × 4. This work could pave the way to do RL within materials for a variety of applications such as memory storage, neuromorphic sensors, smart robots, and human-machine interaction systems.

Direct retrieval of Zernike-based pupil functions using integrated diffractive deep neural networks

Retrieving the pupil phase of a beam path is a central problem for optical systems across scales, from telescopes, where the phase information allows for aberration correction, to the imaging of near-transparent biological samples in phase contrast microscopy. Current phase retrieval schemes rely on complex digital algorithms that process data acquired from precise wavefront sensors, reconstructing the optical phase information at great expense of computational resources. Here, we present a compact optical-electronic module based on multi-layered diffractive neural networks printed on imaging sensors, capable of directly retrieving Zernike-based pupil phase distributions from an incident point spread function. We demonstrate this concept numerically and experimentally, showing the direct pupil phase retrieval of superpositions of the first 14 Zernike polynomials. The integrability of the diffractive elements with CMOS sensors shows the potential for the direct extraction of the pupil phase information from a detector module without additional digital post-processing.

Programmable low-power consumption all-optical nonlinear activation functions using a micro-ring resonator with phase-change materials

A programmable hardware implementation of all-optical nonlinear activation functions for different scenarios and applications in all-optical neural networks is essential. We demonstrate a programmable, low-loss all-optical activation function device based on a silicon micro-ring resonator loaded with phase change materials. Four different nonlinear activation functions of Relu, ELU, Softplus and radial basis functions are implemented for incident signal light of the same wavelength. The maximum power consumption required to switch between the four different nonlinear activation functions in calculation is only 1.748 nJ. The simulation of classification of hand-written digit images also shows that they can perform well as alternative nonlinear activation functions. The device we design can serve as nonlinear units in photonic neural networks, while its nonlinear transfer function can be flexibly programmed to optimize the performance of different neuromorphic tasks.

Learned end-to-end high-resolution lensless fiber imaging towards real-time cancer diagnosis

Recent advances in label-free histology promise a new era for real-time diagnosis in neurosurgery. Deep learning using autofluorescence is promising for tumor classification without histochemical staining process. The high image resolution and minimally invasive diagnostics with negligible tissue damage is of great importance. The state of the art is raster scanning endoscopes, but the distal lens optics limits the size. Lensless fiber bundle endoscopy offers both small diameters of a few 100 microns and the suitability as single-use probes, which is beneficial in sterilization. The problem is the inherent honeycomb artifacts of coherent fiber bundles (CFB). For the first time, we demonstrate an end-to-end lensless fiber imaging with exploiting the near-field. The framework includes resolution enhancement and classification networks that use single-shot CFB images to provide both high-resolution imaging and tumor diagnosis. The well-trained resolution enhancement network not only recovers high-resolution features beyond the physical limitations of CFB, but also helps improving tumor recognition rate. Especially for glioblastoma, the resolution enhancement network helps increasing the classification accuracy from 90.8 to 95.6%. The novel technique enables histological real-time imaging with lensless fiber endoscopy and is promising for a quick and minimally invasive intraoperative treatment and cancer diagnosis in neurosurgery.

Translation-invariant optical neural network for image classification

The classification performance of all-optical Convolutional Neural Networks (CNNs) is greatly influenced by components' misalignment and translation of input images in the practical applications. In this paper, we propose a free-space all-optical CNN (named Trans-ONN) which accurately classifies translated images in the horizontal, vertical, or diagonal directions. Trans-ONN takes advantages of an optical motion pooling layer which provides the translation invariance property by implementing different optical masks in the Fourier plane for classifying translated test images. Moreover, to enhance the translation invariance property, global average pooling (GAP) is utilized in the Trans-ONN structure, rather than fully connected layers. The comparative studies confirm that taking advantage of vertical and horizontal masks along GAP operation provide the best translation invariance property, compared to the alternative network models, for classifying horizontally and vertically shifted test images up to 50 pixel shifts of Kaggle Cats and Dogs, CIFAR-10, and MNIST datasets, respectively. Also, adopting the diagonal mask along GAP operation achieves the best classification accuracy for classifying translated test images in the diagonal direction for large number of pixel shifts (i.e. more than 30 pixel shifts). It is worth mentioning that the proposed translation invariant networks are capable of classifying the translated test images not included in the training procedure.

Key frames assisted hybrid encoding for high-quality compressive video sensing

Snapshot compressive imaging (SCI) encodes high-speed scene video into a snapshot measurement and then computationally makes reconstructions, allowing for efficient high-dimensional data acquisition. Numerous algorithms, ranging from regularization-based optimization and deep learning, are being investigated to improve reconstruction quality, but they are still limited by the ill-posed and information-deficient nature of the standard SCI paradigm. To overcome these drawbacks, we propose a new key frames assisted hybrid encoding paradigm for compressive video sensing, termed KH-CVS, that alternatively captures short-exposure key frames without coding and long-exposure encoded compressive frames to jointly reconstruct high-quality video. With the use of optical flow and spatial warping, a deep convolutional neural network framework is constructed to integrate the benefits of these two types of frames. Extensive experiments on both simulations and real data from the prototype we developed verify the superiority of the proposed method.

Basis function approach for diffractive pattern generation with Dammann vortex metasurfaces

In mathematics, general functions can be decomposed into a linear combination of basis functions. This principle can be used for creating an infinite number of distinct geometric patterns based on a finite number of basis patterns. Here, we propose a Dammann vortex metasurface (DVM) for optically generating an array of diverse, diffraction-multiplexed vortex patterns, based on three custom-defined basis patterns. The proposed DVM, with its capability of quantitatively correlating phase and intensity distribution in different diffraction orders, opens up doors for various applications including orbital angular momentum encryptions and quantum entanglement.

Nano-electromechanical spatial light modulator enabled by asymmetric resonant dielectric metasurfaces

Spatial light modulators (SLMs) play essential roles in various free-space optical technologies, offering spatio-temporal control of amplitude, phase, or polarization of light. Beyond conventional SLMs based on liquid crystals or microelectromechanical systems, active metasurfaces are considered as promising SLM platforms because they could simultaneously provide high-speed and small pixel size. However, the active metasurfaces reported so far have achieved either limited phase modulation or low efficiency. Here, we propose nano-electromechanically tunable asymmetric dielectric metasurfaces as a platform for reflective SLMs. Exploiting the strong asymmetric radiation of perturbed high-order Mie resonances, the metasurfaces experimentally achieve a phase-shift close to 290^∘, over 50% reflectivity, and a wavelength-scale pixel size. Electrical control of diffraction patterns is also achieved by displacing the Mie resonators using nano-electro-mechanical forces. This work paves the ways for future exploration of the asymmetric metasurfaces and for their application to the next-generation SLMs.

This work experimentally demonstrates nano-electromechanically tunable asymmetric dielectric metasurfaces. The metasurfaces enable large phase tuning, high reflection, a wavelength-scale pixel size, and electrical control of diffraction patterns.

Study of Intelligent Wireless Network Management in the Context of Artificial Intelligence for the Improvement of Chinese Language Mandarin Test Training Programmes

One of the most prominent ways of communication between people is through language, which plays a significant role in expressing thoughts. Different ways of expressing a language can be through speech, writing, signing, or gesture. Each country has their own traditional language and they are getting upgraded with technology in a complex environment. The country of China is termed as Chinese and they follow a dialect named Putonghua. Besides this Putonghua, the people follow different dialects, but this Putonghua is considered the official dialect. Also, the Putonghua Proficiency Test is to test the fluency of Chinese-speaking skills. In the traditional system, the test is conducted by the authorities manually. This process will be difficult when multiple people appear for the test, and in some circumstances, complex situations will arise. Hence, technological advancement can be leveraged to simplify the processes. In this research, Chinese language learning and the Putonghua Test were performed with the implementation of the Deep Learning (DL) model. This process involves the design of a DL model for mobile phones and training the model according to the application. Later, the concept is implemented through intelligent wireless network communication for learning and testing of the language. LIDA is implemented in this research work to train the system with DL. The main functionality of LIDA is template matching, which is required for testing the proficiency of the Chinese language by the candidate in PPT. When the new LIDA model is compared to the existing Vector Spaced Model (VSM), it is found that the LIDA achieves 98.67% of the test accuracy.

Two-layer integrated photonic architectures with multiport photodetectors for high-fidelity and energy-efficient matrix multiplications

Photonic integrated circuits (PICs) are emerging as a promising tool for accelerating matrix multiplications in deep learning. Previous PIC architectures, primarily focusing on the matrix-vector multiplication (MVM), have large hardware errors that increase with the device scale. In this work, we propose a novel PIC architecture for MVM, which features an intrinsically small hardware error that does not increase with the device scale. Moreover, we further develop this concept and propose a PIC architecture for the general matrix-matrix multiplication (GEMM), which allows the GEMM to be directly performed on a photonic chip with a high energy efficiency unattainable by parallel or sequential MVMs. This work provides a promising approach to realize a high fidelity and high energy efficiency optical computing platform.

Mathematical operations and equation solving with reconfigurable metadevices

Performing analog computations with metastructures is an emerging wave-based paradigm for solving mathematical problems. For such devices, one major challenge is their reconfigurability, especially without the need for a priori mathematical computations or computationally-intensive optimization. Their equation-solving capabilities are applied only to matrices with special spectral (eigenvalue) distribution. Here we report the theory and design of wave-based metastructures using tunable elements capable of solving integral/differential equations in a fully-reconfigurable fashion. We consider two architectures: the Miller architecture, which requires the singular-value decomposition, and an alternative intuitive direct-complex-matrix (DCM) architecture introduced here, which does not require a priori mathematical decomposition. As examples, we demonstrate, using system-level simulation tools, the solutions of integral and differential equations. We then expand the matrix inverting capabilities of both architectures toward evaluating the generalized Moore-Penrose matrix inversion. Therefore, we provide evidence that metadevices can implement generalized matrix inversions and act as the basis for the gradient descent method for solutions to a wide variety of problems. Finally, a general upper bound of the solution convergence time reveals the rich potential that such metadevices can offer for stationary iterative schemes.

Using electrical resistance asymmetries to infer the geometric shapes of foundry patterned nanophotonic structures

While silicon photonics has leveraged the nanofabrication tools and techniques from the microelectronics industry, it has also inherited the metrological methods from the same. Photonics fabrication is inherently different from microelectronics in its intrinsic sensitivity to 3D shape and geometry, especially in a high-index contrast platform like silicon-on-insulator. In this work, we show that electrical resistance measurements can in principle be used to infer the geometry of such nanophotonic structures and reconstruct the micro-loading curves of foundry etch processes. We implement our ideas to infer 3D geometries from a standard silicon photonics foundry and discuss some of the potential sources of error that need to be calibrated out. By using electrical measurements, pre-designed structures can be rapidly tested at wafer-scale, without the added complexity of optical alignment and spectral measurement and analysis, providing both a route towards predictive optical device performance and a means to control the geometry variation.

Motion detection and direction recognition in a photonic spiking neural network consisting of VCSELs-SA

Motion detection and direction recognition are two important fundamental visual functions among the many cognitive functions performed by the human visual system. The retina and visual cortex are indispensable for composing the visual nervous system. The retina is responsible for transmitting electrical signals converted from light signals to the visual cortex of the brain. We propose a photonic spiking neural network (SNN) based on vertical-cavity surface-emitting lasers with an embedding saturable absorber (VCSELs-SA) with temporal integration effects, and demonstrate that the motion detection and direction recognition tasks can be solved by mimicking the visual nervous system. Simulation results reveal that the proposed photonic SNN with a modified supervised algorithm combining the tempotron and the STDP rule can correctly detect the motion and recognize the direction angles, and is robust to time jitter and the current difference between VCSEL-SAs. The proposed approach adopts a low-power photonic neuromorphic system for real-time information processing, which provides theoretical support for the large-scale application of hardware photonic SNN in the future.

Flexible method for generating needle-shaped beams and its application in optical coherence tomography

Needle-shaped beams (NBs) featuring a long depth-of-focus (DOF) can drastically improve the resolution of microscopy systems. However, thus far, the implementation of a specific NB has been onerous due to the lack of a common, flexible generation method. Here we develop a spatially multiplexed phase pattern that creates many axially closely spaced foci as a universal platform for customizing various NBs, allowing flexible manipulations of beam length and diameter, uniform axial intensity, and sub-diffraction-limit beams. NBs designed via this method successfully extended the DOF of our optical coherence tomography (OCT) system. It revealed clear individual epidermal cells of the entire human epidermis, fine structures of human dermal-epidermal junction in a large depth range, and a high-resolution dynamic heartbeat of alive Drosophila larvae.

Light-driven single-cell rotational adhesion frequency assay

The interaction between cell surface receptors and extracellular ligands is highly related to many physiological processes in living systems. Many techniques have been developed to measure the ligand-receptor binding kinetics at the single-cell level. However, few techniques can measure the physiologically relevant shear binding affinity over a single cell in the clinical environment. Here, we develop a new optical technique, termed single-cell rotational adhesion frequency assay (scRAFA), that mimics in vivo cell adhesion to achieve label-free determination of both homogeneous and heterogeneous binding kinetics of targeted cells at the subcellular level. Moreover, the scRAFA is also applicable to analyze the binding affinities on a single cell in native human biofluids. With its superior performance and general applicability, scRAFA is expected to find applications in study of the spatial organization of cell surface receptors and diagnosis of infectious diseases.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1186/s43593-022-00020-4.